Matt Dispels More Rumors

We love Matt Cutts. Or at least, I love Matt Cutts. Why? Because he’s always on hand to dispel rumors and calm the fears of the paranoid — and when you’re dealing with the Internet, the rumors and the lot of invented conspiracies add up.

If you spent any time trolling the forums last week, it’s likely you caught some of the buzz regarding the AdSense bot and the rumor that it was crawling sites and giving AdSense users an unfair advantage in the search engines. Lots of people got themselves all worked up over it.

So, does participating in AdSense get your pages indexed faster and give you a boost in the search engines? Matt says no, of course not. That would be wrong. Why/ how are the AdSense bots crawling the Web, you ask? According to Matt, its all part of Google’s plan for ‘smarter crawling’ and ‘better indexing’.

One of Bigdaddy’s new features is the ‘crawl caching proxy’ update which cache’s all of Google’s bots (GoogleBot, AdSense bot, Blog search bot, News bot, etc.) crawls in one place in order to save user bandwidth. If one bot crawls your sites and then another comes poking around a few hours later, it will know that your site has already been visited and will pass on by. Matt provided a series of flow charts on his blog to illustrate his point, so if you don’t find my text description adequate you may want to check that out. Basically, it’s part of Google’s effort for bandwidth reduction. Smart. Very smart.

It should also be noted that the robots.txt rules still apply to each crawl. Matt explains:

“If service X was allowed to fetch a page, but a robots.txt file prevents service Y from fetching the page, service Y wouldn’t get the page from the caching proxy.”

Good to know. And in case you were wondering, the crawl caching proxy is not the same as the cached page users see if they click on the "Cached" link on the search engine results page. Those cached pages are only updated when a new page is added to the index.

So, participating in AdSense gets you no additional crawling in Google’s index. You don’t get indexed faster and you don’t get a boost in the ranking. Can we stop talking about this now?

Matt was also on hand to discuss Lee Odden’s question regarding Google and Toolbar data. Lee asked if Google uses toolbar data for its rankings. Matt says he will never answer an outright ‘no’ to the endless stream of ‘Does Google Deem [X] Important?’ questions. He says it is wise not ‘to rule something out absolutely’. We agree. Kind of like what we said about Meta tags. Don’t not do it. If you’re going to do something, you may as well do it well.

Lisa Barone is a writer, content marketer & VP of strategy at Overit Media. She's also a very active Twitterer, much to the dismay of the rest of the world.

See Lisa's author page for links to connect on social media.