Googlebot joined on the trip to SMX Milano, and decided to go visit Majestic-bot. Hopefully Googlebot'll post some selfies from the trip! :-)
50 plus ones
Shared publicly•View activity
View 4 previous comments
- Hi Rusty. Are you saying that when we crawl a page, Majestic should identify the page we came from? Does any other bot do this? Is there a good business case that can be made for this?Nov 15, 2014
- GroßartigNov 15, 2014
Yes, if you could send a referer header to indicate where the link came from, it would be very useful. The main reason would be finding where the bot gets its mangled / broken urls that it tries to crawl.
I understand that the bot doesn't actually 'follow' links, but rather records the link for following later. And so a link may have several referers. In that case just including one of the referers is better than none at all.
I just had a quick look through my logs and
the only bots I saw sending referer headers were the archive.org_bot, QihooBot, and msnbot-media /1.0b. The majority don't send referer headers, however I think it would be useful if they did.
Actually, I just checked again and those bots only send referer headers from the same domain they are crawling. Presumably to get around any anti hotlinking rules. TinEye-bot is the only bot I could see actually sending referer headers pointing to an outside source.Nov 15, 2014
- see belowNov 17, 2014
- Right - that won't work on several fronts. The largest being that it would create referral spam all across the internet which would upset webmasters much more than the bot as it is now. Happy to continue this conversation on a support ticket (ask for Dixon) but would rather not hijack John's thread... especially because he was kind enough to show a budding love affair picture between two inanimate objects.Nov 17, 2014
- No probs, I have opened a ticket.Nov 17, 2014