how to implement following requirement:
some operators can add attachments, but cannot remove them from contacts.
I'm not sure I understood your second comment correctly:
As long as the user has the right to update the contac record, they should have these two (very different) buttons.
Raise a bug-report, if the second button is missing when they can update the contact record and add another attachment.