HI,

If you do like this(the above code), the existing groups assigned to the menu wont get removed, instead the newly given group also will get added with the existing groups assigned to the menu.

If you activate the developer mode and navigate to Settings -> Technical -> User Interface -> Menu items, open your menu and check groups tab, now you can see there will be old group and newly added group. 

So what you have to do is that first of all you have to remove the existing groups from the menu and then assign the new group, for that you can do it like this.

<record id="contacts.menu_contacts" model="ir.ui.menu">
    <field name="groups_id" eval="[(5,0),(4, ref('hr.group_hr_manager'))]"/>
</record>

Thanks