Stream

Martin Reed

General Tech Talk  - 
 
Your site is being crawled at a rate calculated as optimal by Google.

Hi guys, I’m working with a client whose pages crawled by day in Search Console has dropped to 1/10th its previous rate and flatlined for the past 2 months. Looking at the crawl rate in site settings, the usual option to select a crawl rate (or let Google decide) is replaced with the message above.

We’ve improved server performance, capacity and unnecessary rate limiting and have seen the number of 5xx server errors is dropping. We’re also working on improving page speed to reduce the time spent downloading a page. The crawl rate is slowly increasing - a few thousand more pages a day - but it very much feels like this would happen anyway as Google works out the ‘optimal’ rate, and still well short of its former glory.

Any suggestions on how we can get the crawl rate back up? It’s becoming problematic because we’re finding that new content isn’t getting indexed.

I did find a page that lets you report a problem with how Googlebot crawls your site (https://www.google.com/webmasters/tools/googlebot-report) but it seems to be intended for limiting crawls.

Many thanks,
Martin
1
John Vantine's profile photoMartin Reed's profile photo
4 comments
 
Thanks John, sure thing, I'll report back once something exciting happens 
Add a comment...

Katherine Watier Ong

Google & Bing Tools  - 
 
I'm looking for an outside perspective. I have a client who has seen a drop in impressions and CTR for their branded keywords in Google Search Console but NOT in rankings, and a corresponding drop in homepage organic traffic (as that's the page that receives most of the branded queries).

I'm a bit stumped as to what might have gone one. From a technical perspective, there doesn't seem to be any change, the homepage design has stayed the same, etc.

They have a few associated login type pages that are now 404ing which probably ranked for branded terms, but there doesn't seem to be enough of those to warrant the drop in traffic, and I can't wrap my brain around how rankings would stay the same but impressions and CTR would drop.

Any other things that I could check?

1
Tony “Tiggerito” McCreath's profile photoDave Elliott's profile photo
12 comments
 
i'd check for seasonality as suggested but i'd also chuck the brand into Google Trends to see if the company has become less popular recently.

I'd also look at the SERPs and see if one of the competitors has increased their ad-words spend and is bidding on your branded terms.
Add a comment...

Hyderali Shaikh

General Tech Talk  - 
 
Is Google no longer accepting sitemap in html format?

I read the google guidelines also in which they didn't mention about html files.

Due to some security reason, we've blocked .xml & .txt in robots. So, I made sitemap file in .html but it has been three days the status in search console is still showing 'pending'.

Currently our robots.txt file is as below:-

Sitemap: http://[URL].com/sitemap.xml

User-agent: Googlebot
Allow: /sitemap.xml

User-agent: *
Disallow: /*.txt$
Disallow: /*.xml$

I still don't know how many URLs has been indexed in Google & how many are still pending.

Any solution on the above query?

Thanks.


1
Randy Milanovic's profile photoJaaved Khatree's profile photo
2 comments
 
As far as I know, you can't submit an HTML sitemap the same way you would an XML sitemap.

+1 for what Randy suggested above: Fetch and Submit the relevant pages that make up your HTML sitemap and see how that goes.

Depending on the size of your site and your internal linking structure, you could get away without using an XML sitemap.

Let us know how it goes :)
Add a comment...

Nikhil Raj

Google & Bing Tools  - 
 
Q on Search Console: The index status number in search console hasn't changed for last 4 weeks. But the sitemap index count is growing. Anyone see similar pattern in search console? 
1
Add a comment...

Hyderali Shaikh

General Tech Talk  - 
 
Hi,

One of my client offers content in more than one language i.e. Hindi, Portuguese, Turkish, Gujarati etc. The main site is in .com.

The sub-directory is like below

http://www.xyz.com/hi (for hindi)
http://www.xyz.com/pt (for portugues)
http://www.xyz.com/tr (for turkey)

After checking the view source, I found the below tag below /head

<a title="xyz" href="http://www.xyz.com/hi/" rel="alternate" hreflang="hi"></a>

<a title="xyz" href="http://www.xyz.com/pt/" rel="alternate" hreflang="pt"></a>

I'm not much familiar with multilingual seo. So, I wanted to know if the above code is perfectly implemented by the developer? Or I've to feed him something else?

Also, I read somewhere that you have to create multilingual sitemap of each subdirectory & submit those in Google. Is this true?

P.S. In search console we've added India as a international targeting. Should I select unlisted?
1
Hyderali Shaikh's profile photoFederico Sasso's profile photo
5 comments
 
Hi +Hyderali Shaikh setting geo-targeting to "Unlisted" in GSC would have a positive effect for non-Hindi queries on non-Indian SERPs.
Regarding the possible impact on today positioning in Indian SERP, I don't really know; I'd at least restrict geo-targeting to India for /hi/ in order not to alter today setting for Hindi contents.
Add a comment...

Chris Koszo

General Tech Talk  - 
 
Hello guys and girls, I can't figure this out.. I'm looking at a website for a company that ranks well (hundreds of #1-2 rankings, has knowledge graph, featured snippets etc.) but for its actual product (only sells one product) the rankings are nonexistant (200+ position in Google).. This has been the case for them for 5+ years (before Penguin). Since they're the largest company in their industry I think searchers expect  to see them on page 1 but don't. What gives?
 
To me it seems like a simple case of Penguin where they did something to overoptimize for their main keyword and got hit, however, they've had this ranking issue years before Penguin even came out, and BEFORE they even hired any SEOs. Never in the company’s history did they rank for their main keyword in Google. In Bing their historic position varies from #1-15. They’re in a competitive niche but I think at least they’d deserve to rank on page 3 or something in Google? They have 1000 employees and links from Wikipedia and hundreds of other publications due to PR efforts.
 
 One theory going around is that since their domain is a prime 3 letter .com domain, and the previous company they purchased it from was so strong, that Google is still associating it with that/doesn’t trust them enough to rank the new site for their money keywords? It doesn’t make sense though because they’re in the top 3 for hundreds of mid to long tail queries. Their content is excellent and satisfies searcher intent. Also they don't and never had a manual penalty etc. either. One of their recent SEO companies acquired questionable links in 2015, but those were disavowed already by them.
 
Should they just sit out and wait for Penguin to actually run and see what that brings? I really can’t think of anything else. Feel free to PM me I will share the company name.
 
+John Mueller, this is the same question I mentioned to you on Wednesday Hangouts, but I think this provides more detail. Let’s see if +Gary Illyes  is on this board too :D  Thank you!!
1
Chris Koszo's profile photo
13 comments
 
FYI, here's rankings for a variation of that keyword. So weird! https://i.imgur.com/aMmR5afh.jpg +Terry Van Horne 
Add a comment...

Hyderali Shaikh

General Tech Talk  - 
 
Hi,

Today, I got a message in our search console account that there has been increased in authorization permission errors i.e. 403 error. But I checked the list, most of the URLs are login & logout page.

Like this -> [URL]?data%5Bregflag%5D=0&data%5Bmd5res%5D=TestName&login_uid=TestName&login_password=Password&submit=Submit

We haven't added /login to robots.txt. Should I do that?

How to fix this issue?

Thanks in advance.
1
Hyderali Shaikh's profile photoCollin Davis's profile photo
9 comments
 
Unless it is a clickable link, Google crawlers wouldn't be able to execute it. You need not add anything in that case.
Add a comment...

Dave Ashworth

General Tech Talk  - 
 
rel prev/next and canonical tag combination

When you have a series of paginated pages, and they are being reported as dupe content, have always gone with rel prev/next to conslidate them

As I understand it, canonical tags are used in a different way to inform search engines which of many similar pages is the primary page

So, why is mixing rel prev next and canonical tags common practice? More to the point, why do Google say it's ok to mix the two?

https://webmasters.googleblog.com/2011/09/pagination-with-relnext-and-relprev.html

Reason being, you have page 1, 2, 3 & 4 - I would use rel prev next to say these are a sequence and consolidate them.

But if each page then has a canonical tag specifying itself, is that not then saying to treat each page independently of the others in the sequence? i.e. a conflicting signal?
1
Dave Ashworth's profile photoKathy Alice Brown's profile photo
5 comments
 
With sites that have a lot of filters there was a reason to have both. With the filters that we deemed as not significant we would canonicalize the URL to a more authoritative URL and then use rel prev/next to aggregate the sequence to the first page. We were not using a View All page.
Add a comment...

Daniel Laidler

SEO Tools & SAAS Providers  - 
 
Hey All,

TL;DR - Can fixed HTTP links (a SAAS restriction) within site template have a negative effect on a site that is running site-wide HTTPS with 301 pattern match redirects?

All comments appreciated!

- +John Mueller +Jim Stewart 
4 votes  -  votes visible to Public
Yes
75%
No
25%
2
Terry Van Horne's profile photoTony “Tiggerito” McCreath's profile photo
10 comments
 
You could add some JavaScript to update all internal links on a page to https. Googlebot will pick up on that.
Add a comment...

Mark Taylor

General Tech Talk  - 
 
Question - If a business has collected a reasonable number of customer reviews using a third party partner (Feefo, TrustPilot etc) but has not previously added them to their website, will Google see this as a problem if a product page goes from zero to 10+ reviews between the page being indexed?
1
Emile Pukander's profile photoChris Koszo's profile photo
8 comments
 
Trust Pilot is tricky because they let you use their API to populate your reviews on your site, but it's only for users because they're blocking the JS files to bots on their side. Makes sense for them since they want to get that traffic and not have duplicate content issues.

Anyone have experience in this and how to get around it? For one of my sites I have hundreds of Trust Pilot reviews, so I'm just thinking of manually pulling in a sample and writing my own JS on the site to pull from Trust Pilot's API to keep my aggregate score honest and up to date.

Another follow up question though (sorry): Is it ok to mark up a page on a site with an aggregate review schema without having any of the actual reviews on the company/site itself that's being reviewed?
Add a comment...

Katherine Watier Ong

Technical Audits  - 
 
Is transcraping a penalty?

My gut says yes, but I wanted to check with others. I have a client whose business model is to take news in Arabic, translate it to English and publish it without any additional commentary.

That's scraping, correct?
1
Katherine Watier Ong's profile photoMaurizio Ceravolo's profile photo
5 comments
 
Well, it's hard to be penalized with human translation
Add a comment...

About this community

Technical SEO is a community to discuss the technical issues around building sites aimed at performing really well - for Google and users. We look at new technology and methodologies, help each other solve technical problems, and create experiments. We moderate! No self-promotion, no link dropping. We do allow links if you're creating a relevant discussion & adding value. If you have questions, ask a moderator. If you have basic SEO or marketing questions, we recommend checking out Google's Webmaster Central or one of the many other G+ communities. BELOW ARE LINKS TO FRIENDS OF THE COMMUNITY

Katherine Watier Ong

SEO Tools & SAAS Providers  - 
 
Anyone using URL Profiler for a Panda audit on a large site?

I wanted to give it a whirl, but the site I'm looking at has 500K+ pages and I think URL profiler might run forever and I'll be spending a pretty penny on proxies.

Any suggestions?

I'm also using Deepcrawl...
1
1
Gerry White's profile photoManuel Lemos's profile photo
2 comments
 
That is interesting. Personally I have been using a tool that I developed to tell me what is the level of low quality pages that Google is not sending traffic to you.

Since I have a user generated content site. it has helped me automating marking as NOINDEX irrelevant pages and reduce the chances the site be so affected by Panda.

I have been considering providing this tool to other SEO or Web site managers.

Is this something you would be interested to be able to use?
Add a comment...

Joanna Laznicka

General Tech Talk  - 
 
I am presuming I am doing something wrong and would love some guidance. -- I understand that google can come up with their own Meta Title and Meta Description but feel in my gut something else is happening and it is something I am doing wrong. Time to time I search all my sites in google by site:MYSITE to just see what is happening and one of my site in search almost every meta title has the site name at the end of it, even thought I have not requested this to be in the back end. See screenshot of my search results and you will see VC List at the end of each title. I really don't want the verbage VC List at the end of the title for a few different reasons such as it throws off the keywords I am targeting, as well not every page of my site/publication is about a list of VC's but advice on other topics such as crowdfunding or valuation or the many other topics startups deal with. I can't imagine my meta titles are so bad in Googles eyes that they have to take my blog title and then throw VC List at the end of each one so where am I going wrong? I would really appreciate your guidance before I start messing with markup and breaking more things. - I also see that in Bing the meta titles are correct and don't have a forced VC List at the end of them - So why is this happening in Google

1
1
Joanna Laznicka's profile photoMichael Rhodes's profile photo
11 comments
 
Well what happened?
Add a comment...

Chris Koszo

Mobile SEO  - 
 
Hello all, anyone know of a Wordpress plugin for AMP that already supports pages and not just Posts? +Joost de Valk I'm sure you guys have something coming down the pipe :D
1
Mark Taylor's profile photo
 
This one says it supports pages but I haven't tested it .. https://wordpress.org/plugins/amp-supremacy/
Add a comment...

Dave Ashworth

General Tech Talk  - 
 
Not sure if this is a bug or not, but the Index Status within Google Search Console doesn't look to be updating - had the same figures across every website for 3 weeks in a row now.

Anyone else seeing similar?
3
Dave Ashworth's profile photo
12 comments
Add a comment...

Mark Taylor

General Tech Talk  - 
 
A friend of mine runs a local football Club website and wants to add a section to offer football related products, which will be fed from affiliate programs.

They have a good site that is technically strong from a SEO perspective and I've warned him of the impact of doing this but I understand why they want to do this.

Whilst it's good practice to block this section in robots.txt we all know this is a guide to the SEs rather than a command.

What steps should they take to prevent the whole site's SEO devaluing, losing natural search traffic or worse, being penalised?

Is there any reason to chose a sub-domain over a sub-directory?
1
Dave Elliott's profile photoJoanna Laznicka's profile photo
8 comments
 
Another option would be to promote affiliate links in their social media (of course follow the FTC guidelines of disclosing) - that way they are not missing with their SEO of their site - I find promoting targeted quality products via social media a much better return then adding them to my sites.

Add a comment...

Chris Bolton

General Tech Talk  - 
 
So my client had their Google My Business page hijacked by an offshoot of their company. The offshoot changed the name and the address of the Google business page, but the knowledge graph info is still showing for my clients brand name.

I've created a new Google My Business page with the correct info, but it's not showing yet.

Is there anything you guys would recommend to get the new Google Business page to usurp the one with incorrect info?
1
Chris Koszo's profile photoJenny Stradling's profile photo
6 comments
 
Wha?? Someone had success with their Twitter? I've uncovered hoards of issues and spam... We did research weeks (maybe months) ago on drug rehab center SPAM on Google Maps. I noticed some weird trends with our addiction
treatment clients and so we started looking closer. Turns out when Google decided to remove "maps" from Google+ and created Google My Business (GMB) they reverted the old Google maps back to maps that had since been claimed, cleaned up and optimized. When they reverted the maps back to maps, they removed all claim. Rehab marketing "spammers" went ahead and claimed these maps for themselves. There is so much OUTRIGHT spam happening, it's insane. I guess I have to write a blog post. Sigh 
Add a comment...

Hyderali Shaikh

SEO Tools & SAAS Providers  - 
 
Is there a tool or way to find all the old URLs which has now been redirected to new URLs?

An ex-employee who 301 redirected around 1000 URLs but didn't keep the list of OLD URLs.

Thanks. 
1
Kyle Sutton's profile photoDave Elliott's profile photo
10 comments
 
screaming frog should find them all. Also majestic's historic page data is very useful for finding old urls.
Add a comment...

David Wilding

Technical Audits  - 
 
Google Locale-aware crawling - country list & practical experience

Hi guys,

I am working with a international news publisher website that is considering utilising language based folders, such as /en/, which change the content within each language 'website' dependent on the users IP. So a English speaker from Nigeria would get custom navigation and content lists on the homepage (for example) specific to Nigeria. A English speaker from the UK would get their own UK version. All this occurs on the same URL, /en/.

I am very familiar with hreflang and how to implement this successfully, i.e multiple URLs for location/language and hreflang references.

Now, Locale-aware crawling seemingly resolves my issue by proving geo-distributed crawling, meaning G will see the various IP dependent content. Google of course though still recommends using hreflang even though this functionality seemingly exists.

Ref:
https://support.google.com/webmasters/answer/6144055
https://plus.google.com/u/0/+PierreFar/posts/FTjsSsvsXy7

There is not a lot of information out there regards this.

My questions are,

Does anyone have a definitive list of countries that geo-distrubted crawling emanates from?

Does anyone here have practical experience of successfully using locale-aware crawling?

Anyone have any tips of how to best 'trigger' Google sending various local-aware bots to a website?

Thanks for any info you may have!
1
Rick Bucich's profile photo
 
Sorry, message got flagged as spam.

The latest greatest info from Google indicates that they highly recommend that you do not build a site against location based crawling yet. They may have the ability to do so in a limited manner but it isn't ready for public consumption. The best alternative for now is hreflang.

P.S. I'm getting this feedback from a recent Twitter conversation between John Mueller and Dan Petrovic where John simply says "just don't" or thereabouts. I quickly looked but didn't see the tweet to reference. 
Add a comment...

Nikhil Raj

Google & Bing Tools  - 
 
I was tracking the sitemap index numbers for all the files I have submitted using Search Console. Noticed that in some weeks when I compare it with the last week, the index numbers goes down for the sitemap files, in the range of a few thousands. Any idea why this might be the case? Why would a page in the index get removed? There is no change in robots.txt, meta robots tag, canonical link.
1
Gerry White's profile photo
 
Google is pretty clever, the algo behind it is constantly improving and adapting. To check everything follow these steps


Install SEO tools for Excel

Open the XML sitemap in Excel (literally copy and paste the URL into the open dialoge box )

- Check the canonical = the url
- Check the headers
- Check the title tags for duplication
- Check the meta robots

No issues? It maybe that when google crawled your site you had some pages returning an unexpected response, it should tell you this in crawl errors.

Finally it may simply be that this site has "too much" content for Google to index, if your PageRank is low and your number of pages is high that can cause this, but it is pretty unlikely.

Add a comment...