Skip to Content
This question has been flagged
googlesitemaprobots.txt
1 Respondre
6023 Vistes
Avatar
Lumír Jasiok

Hello,


Google can't for some reason fetch generated robots.txt, although it should be able to do so. I can access robots.txt using curl with following output:

curl http://www.mydomain.com/robots.txt

User-agent: *
Disallow: /web/login
Allow: *

User-Agent: Googlebot
Disallow: /web/login


Google is complaining:


Failed: Robots.txt unreachable

Any idea what is wrong?

Also, because of that Google can't access sitemap.xml. 

Another problem is about sitemap.xml. I contains URL's with http, not https prefix. They are valid, as we have http->https redirection rule, but I would prefer to have it correctly in sitemap in the first place. Any help with that?


Many thanks in advance.


Lumir

Avatar
Descartar
Avatar
Lumír Jasiok
Autor Best Answer

I have found a problem and fixed it. In our case problem was, that we had issue with Nginx proxy settings. When we were accessing the our domain webpages using curl (command line) or Safari, everything seems to be working. But when we tried access website using Firefox, we received an SSL error:

SSL_ERROR_RX_UNEXPECTED_NEW_SESSION_TICKET

We had to move from all sites handled by the Nginx line 

ssl_session_tickets off;

to the /etc/nginx/nginx.conf, section http {} and restart the Nginx.

More info here https://serverfault.com/questions/1021041/browsers-reported-ssl-error-when-one-of-the-server-blocks-in-nginx-configur

This was preventing Google accessing URL's of the domain. Now it's fixed.


Avatar
Descartar
Related Posts Respostes Vistes Activitat
Urls are not showing in HTTPs version
website sitemap robots.txt
Avatar
Avatar
1
de gen. 23
6805
Empty Quotation from public users - Odoo 10
index sitemap robots.txt odoo10
Avatar
Avatar
1
de juny 17
8746
Sitemap.xml created incorrectly with reverse proxy Solved
google url website_builder sitemap
Avatar
Avatar
2
de jul. 15
13260
Pages that aren’t indexed can’t be served on Google Solved
robots.txt
Avatar
Avatar
1
de febr. 25
2589
url_root not https so the sitemap and robots don't have links with https
website sitemap robots.txt odoo12 v14
Avatar
Avatar
Avatar
2
de des. 24
9575