With the help of this post :

https://www.odoo.com/fr_FR/forum/aide-1/question/problem-with-security-and-ir-attachment-104606

We have now a public user in res.user. This user has only the 'Public' group and the 'employee' group.

Now, the public user can download an attachement only with a url like this :

http://ourodoo.com/web/binary/saveas?model=ir.attachment&field=datas&filename_field=name&id=56

But now that we know that it works well, we can't let the public user with the 'employe' group. So we will make a new module to adjust this. May be we will create a new group and probably we will have to override the Python web controller to meet the needs of our customer.