Your site is being crawled at a rate calculated as optimal by Google.

Hi guys, I’m working with a client whose pages crawled by day in Search Console has dropped to 1/10th its previous rate and flatlined for the past 2 months. Looking at the crawl rate in site settings, the usual option to select a crawl rate (or let Google decide) is replaced with the message above.

We’ve improved server performance, capacity and unnecessary rate limiting and have seen the number of 5xx server errors is dropping. We’re also working on improving page speed to reduce the time spent downloading a page. The crawl rate is slowly increasing - a few thousand more pages a day - but it very much feels like this would happen anyway as Google works out the ‘optimal’ rate, and still well short of its former glory.

Any suggestions on how we can get the crawl rate back up? It’s becoming problematic because we’re finding that new content isn’t getting indexed.

I did find a page that lets you report a problem with how Googlebot crawls your site (https://www.google.com/webmasters/tools/googlebot-report) but it seems to be intended for limiting crawls.

Many thanks,
Martin

Last week marked the end of 6 amazing years for me at +DEJAN. It's been a pleasure working with such an awesome team, they'll certainly be missed.

Looking forward to a break and some travel before starting my next adventure in London!

Counting down to the Orion test flight. http://www.ustream.tv/nasahdtv

Ouch.

Uh oh. Flag on display at the Australia store in Cologne.

So this pretty much sums up my day. Productive, yet unproductive.

http://www.dailymotion.com/video/x2gp98t

Want to revert to the old avatar menu in Chrome?

1. Visit chrome://flags page
2. Find ‘Enable new avatar menu’ flag and select ‘Disabled‘ from drop down and restart the browser.

Q: How long does it take to funk across the Sydney Harbour Bridge?

A: 20 minutes. And then some.

Photo day at the +Dejan SEO office. This pretty much sums it up for me.