#102 of the things to keep in mind when working on a big website: If you have a giant robots.txt file, remember that Googlebot will only read the first 500kB ( http://code.google.com/web/controlcrawlindex/docs/robots_txt.html ). If your robots.txt is longer, it can result in a line being truncated in an unwanted way. The simple solution is to limit your robots.txt files to a reasonable size :-).
Shared publiclyView activity