We just launched an update to Webmaster Tools that helps webmasters debug how pages could be rendered by Googlebot.

Main takeaway from me: watch what you block with your robots.txt
Webmaster level: all. The Fetch as Google feature in Webmaster Tools provides webmasters with the results of Googlebot attempting to fetch their pages. The server headers and HTML shown are useful to diagnose technical problems and hacking side-effects, but sometimes make double-checking the ...
95
48
Imri Sagive's profile photomed amine aboud's profile photoRajeev Gaur's profile photoChuck Croll's profile photo
28 comments
 
Hello,
Did you change quota of 500 URL submissions per week to 500 URLs per month, or it's just an error?
 
Excellent. Can't wait to start using this.
 
Das war eine gute Idee - vielleicht sollte man die Sachen die nicht geladen werden können aber unkritisch sind mit grün markieren wie die Google Analytics Tracking Codes etc.
Translate
 
John,

I am not blocking anything on one of my sites, but Fetch and Render says Google Fonts API is blocked by robots.txt.  What is going on there?

There are no reported crawl errors on this site.
 
Yeh, I agree with +Barry Schwartz here. It's all well and good saying don't block anything that contributes to the visuals of your website but we use Google API's fonts and maps - all of which are blocked at Google's end so not really a lot we can do about that and Google contradict there own advice.
 
Yes, that's why we said "meaningfully contributes to your site's visible content, or to its layout" -- some of these elements do have some effect, but they tend not to significantly / meaningfully change the content. For example, the fonts just change the font that's used - the content is there just the same. Similarly, things like Analytics code, counters, etc all don't change the content at all, so that's fine too. If these are the only kinds of blocked resources you're seeing, you're doing a great job & deserve a coffee, tea, or cookie :-)
 
The bigger issues we see are when sites use JavaScript to pull in content, or when they use CSS to significantly change their layout, and because these things are blocked, we can't count any of that for the site. Similarly, sometimes AJAX/JSON request URLs are blocked, and prevent us from fetching the content that browsers would see. We'd love to rank your pages for that content, if only we could crawl it!  
 
Thanks for clarification +John Mueller - the weird thing is though, after fetch and render, the page Google showed looks nothing like our actual website - yet it was only the Google resources that it showed as being blocked.

Looks like some more digging into this will be required :)
 
I'm happy to take a look if you send me the URL. Maybe things are timing out? Maybe you're accidentally cloaking to Googlebot (eg using JavaScript to check the user-agent and serve slightly different content)? It's always good for us to have examples, if you're able to share the URL.
 
+John Mueller Is or will the JS rendered content appear in the google cache, or when 'viewing as text only' version? 
 
+John Mueller I mostly build sites using Wordpress and by default Wordpress blocks access to wp-includes. Should I be concerned enough to create my own robots.txt and give Google access to this folder?
 
I'm not convinced that needs answering really +Mark Nolan - Font that doesn't fit on the page due to the typography style isn't a form of hiding content from search engines or visitors and I'm pretty sure Googlebot is clever enough to realise that +Ralph Corderoy 
 
+Geoff Jackson, Googlebot considers what's below the fold, and Analytics tracks display sizes, so it seems quite possible that a right-column div pushed off the side due to a Google self-blocked font might have its content value weakened; making a user scroll sideways is certainly a black mark I'd consider if writing Googlebot.

But we're just guessing. It would be nice if Google said either it doesn't matter and it isn't something we think ever should, or yes, that's not great, we might try and unblock ourselves from our own fonts so stay tuned.
 
The font size would generally just result in text wrapping, and even if it moved it somewhat to the side, that wouldn't be an issue for us. 
 
love the render preview. It would be great if the preview window were bigger though so we can see how small elements on the page render more clearly
 
+John Mueller  What about background elements and themes?  I see many system.theme.css parts blocked by robots.txt....  the rendering of my homepage looks awful without colors in places if it was a user using Googlebot's eyes.
 
+Karen Haley I'd generally recommend not disallowing crawling of those either, especially if - as you mentioned - they contribute significantly to what's shown.
 
Thanks +John Mueller!  I'd love any other advice you could give as to why our site (http://goo.gl/jL8YeL) dropped horrendously in the SERPS since last fall.  It's really nice to hear from you in person!   
 
+Karen Haley, have you poked around its results on webpagetest.org, e.g. http://www.webpagetest.org/result/140724_2W_ZD6/ is one I ran just now. Could do better on time to first byte, and looking at the images it seems some photo-like ones are PNG rather than JPEG, bloating their size: http://www.webpagetest.org/pageimages.php?test=140724_2W_ZD6&run=1&cached=0  Also, that long vertical line of green in the waterfall shows all the many small (because they're minimised) CSS files being loaded, of the first 56 things loaded by the browser all but five seem to be CSS. Combining them into one or two CSS files and then minimising would result in more throughput.

These are just some points I see from an initial look; Google value site speed more and more, especially due to mobile's accessing sites so perhaps as they've valued it more the site has fared less well?
 
It looks like you still have a reconsideration request pending, +Karen Haley - so I'd wait to see how that goes. If you're not sure how to proceed from there (well, depending on how it goes :)), I'd check back in the forums (feel free to send me the link to your thread, if you want).
 
+Ralph Corderoy - thank you very much.  All the images I can change (without our webmaster/hosting company) have been optimized to 99% qual JPEGS as our CMS doesn't support a WebP upload.  +John Mueller - here is our thread in the forums, thank you! (http://goo.gl/okgxNF)
Add a comment...