Public
Jan 11, 2013
HELP! MY SITE HAS 939 CRAWL ERRORS!!1
I see this kind of question several times a week; you’re not alone - many websites have crawl errors.
1) 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
2) In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
3) What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
4) You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it.
http://support.google.com/webmasters/bin/answer.py?answer=2467403
5) We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages.
http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
6) There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors.
http://support.google.com/webmasters/bin/answer.py?answer=181708
7) Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.
I see this kind of question several times a week; you’re not alone - many websites have crawl errors.
1) 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
2) In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
3) What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
4) You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it.
http://support.google.com/webmasters/bin/answer.py?answer=2467403
5) We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages.
http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
6) There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors.
http://support.google.com/webmasters/bin/answer.py?answer=181708
7) Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.
View 81 previous comments
So is there no way to stop Googlebot spamming servers with invalid requests it's been guessing by scraping JavaScript? I guess passing on the cost of that to us is your preferred method of dealing with suboptimal code your end?Aug 24, 2015
Thanks for such a great post +John Mueller :) It seems i am late to the comment 'party' but i am picking up a lot more soft 404's on a customer site.
https://hubblehq.com/office-space-london/manor-park
This page is an example. This was originally returning no search results and we used
<meta ng-if="meta" name="robots" content="noindex, follow" class="ng-scope">
However we have now added a 'here are some others you might like' feature and the rate of soft 404's has increased. Does this have to do with the fact that this is still "noindex"?
Thanks. :)48w
Hi +John Mueller I am getting a lot of 'soft 404' errors for pages that exist on our website and are rendering fine. I am getting about 4 or 5 a day and I believe Google is delisting these healthy pages. When I click on a URL that GWMT is saying that is a soft 404 it ALWAYS renders fine and does not 404 or fail to provide a proper page. Here is an example of the latest one http://www.divertimenti.co.uk/rosie-birkett-smoked-trout-griddled-lemon.html - I am scratching my head on how to fix this when I worry this might be a bug at google's end? (this has been going on 6 months now and I cannot find how to fix and have been through all the normal channels.)33w
Lots of404's errors appear after web admin has killed spammers/hackers who inject a lot of spammy pages and links to you site. It would be truly unfair if google punish websites that fight with spammers .30w
+Rekin Finansów that's a perfect use of 404 -- great to see sites clean up! (and remember to make sure to fix the security issues too :)).30w
Custom 404 page of my website started appearing in Google. What could be the reason? and same is appearing as soft 404 page in GSC.
How to get rid of it and stop showing custom 404 page in search engines. Should custom 404 page has robots(noindex) meta tag added on it?14w
