We just launched an awesome new robots.txt testing tool in Webmaster Tools. I recommend you try it out, even if you're sure that your robots.txt file is fine. Some of these issues can be subtle and easy to miss. While you're at it, also double-check how the important pages of your site render with Googlebot, and if you're accidentally blocking any JS or CSS files from crawling.
114 plus ones
Shared publicly•View activity
View 13 previous comments
- that can happen if we have trouble reaching your server. We'll usually retry as soon as we can, so if the initial attempt fails, and the retry works, we may list that as separate events there. Technically, it's not awesome, practically at least the retry worked, so nothing got lost :).Jul 22, 2014
- Thank you John, all is well, as the retry always works. I love Fetch and Render by the way. :)Jul 22, 2014
- Why many time i get message in webmaster that your robots.txt is not accessible?
I have to use fetch as Google every time.
Is there any way to fix robots.txt .Aug 5, 2014
- the "robots.txt unreachable" message means that we couldn't reach your server at all. Usually that's a sign of connectivity problems on your side (maybe your hoster is limiting the number of users on your site). It doesn't mean that there's anything wrong with your robots.txt file, it's just that we couldn't check.Aug 6, 2014
- Google dont pay tax non of them doAug 10, 2014
- Is there a step-by-step guide for using the robots.txt testing tool. I need to test my robots.txt file and could not find it anywhere on the internet.Mar 3, 2016