Google spidering the invisible.

by Jimboot on September 8, 2010

It wasn’t that long ago that many SEOs were complaining it would take Google weeks or months to find their content. As we demonstrated previously, the Google caffeine algorithm is fast! It is finding a lot more content a lot faster. However over the last couple of weeks we’ve seen Google find pages that have not been linked from anywhere and are listed in our robots.txt file. If pages are listed in your robots.txt file as disallowed then a well behaved bot should stay away from them. However this doesn’t seem to be the case with the new caffeine. Certainly Google will list a link to a page in robots.txt if it finds it on nother page but it will not actually download the page. It would seem that is not the case with caffeine. The other disturbing thing is that how does Google find these pages if there is no link to them anywhere?

Also we picked up another 60,000 or so backlinks and I have a look at a viewers site.

Leave a Comment

Previous post:

Next post: