Microsoft Boycott

Martin Remy takes issue with my call for a boycott of the Microsoft search crawler.

On the practical level, he questions the effectiveness of such an effort:

Maybe I'm missing some subtle intended effect here, but on the surface this seems a bit like withholding my listing from the phone directory out of spite for Qwest. Anyone who blocks the MS crawler will shut themselves out of the new search service. ...a few thousand bloggers missing from Microsoft's new search engine, while they may become the subject of some interesting internal memos in Redmond, won't make a dent in Goliath's armor.

On the philosophical level, he argues that we should treat Microsoft like any other competitor, when it comes to web search:

Google demonstrated first-hand that quality can win in this space, and I don't see any reason to deny Microsoft an opportunity to spend some money on its own attempt, and pay some developers' mortgages. And it draws Microsoft out from the desktop and onto the Web, arguably a more level playing field. I never thought I'd be saying something like that out loud, but there it is.

I disagree with both points.

First, the question of effectiveness. There's no comparison to taking yourself out of the phone book - the Microsoft crawl is the precursor to a brand new search engine, not an established service people already use. No one will switch to a new phone book if it's missing lots of names. Waiting until MSN is fully entrenched will squander the best opportunity we have for making sure the search engine market stays diverse.

A crawler boycott by bloggers will be extremely effective, even if only a few thousand people participate. That's because the hard part of creating a good search engine isn't finding or storing huge amounts of content - it's making accurate determinations of relevance.

To determine relevance, you can either hire a building full of people to look at websites and classify them (the Yahoo model), or analyze patterns in hyperlinks that other people create (the Google model). We've seen which approach scales best.

But the hyperlink model depends on a distributed network of volunteers who create links to interesting content, and allow search engine sites to harvest that information. In effect, we're all working for Google (and Technorati, and others) for free. We allow them to crawl our sites and harvest our links because our relationship is symbiotic. We give them link information, they give us useful search tools.

Because weblogs are (by definition) frequently updated, topical, and rich in hyperlinks, they are a gold mine for search engines. Weblogs serve as an amplifier for interesting and timely content.

If a search engine can't crawl the blogosphere, its relevance judgements and its ability to identify breaking news will suffer. That degradation in link analysis will hurt it far more than a few thousand pages of missing content.

As for the second question - whether a boycott is justified - all you have to do is look at Microsoft's history. Microsoft isn't entering the search space on an equal footing with anyone. They have a monopoly on desktops, and they will use that monopoly to promote MSN search, just like they used it to promote Internet Explorer. The browser wars showed us that their search engine won't have to be better than Google, or even as good as Google. As long as it's good enough for most users, it will take over, because of the overriding convenience of having web search seamlessly integrated into the operating system, office software, and email client.

Microsoft would be stupid not to pursue this strategy. They ended up with a slap on the wrist after years of anti-trust litigation stemming from the browser wars. What disincentive do they have that will keep them from repeating the exercise? Another few million in legal fees, at worst.

As Martin points out, Google won on quality. Right now, another search engine could come along and win on quality again (perhaps even Martin's own effort). But as soon as Microsoft reaches a critical mass, the game is over. No one will be able to compete with a search service embedded in Windows. Microsoft will make sure of it.

Luckily, webloggers have an opportunity to keep Microsoft from spoiling the party. By blocking their crawler in your robots.txt file, you'll be refusing to work for free to help Microsoft extend its monopoly to yet another market. Every weblog that participates in the boycott will be helping offset the unfair advantage of Windows integration.

All it takes is this:

User-agent: MSNBOT
Disallow: /