Google can't fetch Odoo robots.txt (Failed: Robots.txt unreachable)

Question

Hello,

Google can't for some reason fetch generated robots.txt, although it should be able to do so. I can access robots.txt using curl with following output:

curl http://www.mydomain.com/robots.txt [1]
User-agent: *
Disallow: /web/login
Allow: *
User-Agent: Googlebot
Disallow: /web/login

Google is complaining:
Failed: Robots.txt unreachable
Any idea what is wrong?
Also, because of that Google can't access sitemap.xml.
Another problem is about sitemap.xml. I contains URL's with http, not http

Lumír Jasiok · Accepted Answer

I have found a problem and fixed it. In our case problem was, that we had issue with Nginx proxy settings. When we were accessing the our domain webpages using curl (command line) or Safari, everything seems to be working. But when we tried access website using Firefox, we received an SSL error:
SSL_ERROR_RX_UNEXPECTED_NEW_SESSION_TICKET
We had to move from all sites handled by the Nginx line
ssl_session_tickets off;
to the /etc/nginx/nginx.conf, section http {} and restart the Nginx.
More info

Relaterede indlæg	Besvarelser	Visninger
Urls are not showing in HTTPs version website sitemap robots.txt	1 jan. 23	6848
Empty Quotation from public users - Odoo 10 index sitemap robots.txt odoo10	1 jun. 17	8764
Sitemap.xml created incorrectly with reverse proxy Løst google url website_builder sitemap	2 jul. 15	13364
Pages that aren’t indexed can’t be served on Google Løst robots.txt	1 feb. 25	2720
url_root not https so the sitemap and robots don't have links with https website sitemap robots.txt odoo12 v14	2 dec. 24	9720

Dette spørgsmål er blevet anmeldt

Kan du lide at diskutere? Læs ikke bare med, deltag aktivt i debatten!