« Cormac TwomeyMoustiers »

Microsoft Boycott

Martin Remy takes issue with my call for a boycott of the Microsoft search crawler.

On the practical level, he questions the effectiveness of such an effort:

Maybe I'm missing some subtle intended effect here, but on the surface this seems a bit like withholding my listing from the phone directory out of spite for Qwest. Anyone who blocks the MS crawler will shut themselves out of the new search service. ...a few thousand bloggers missing from Microsoft's new search engine, while they may become the subject of some interesting internal memos in Redmond, won't make a dent in Goliath's armor.

On the philosophical level, he argues that we should treat Microsoft like any other competitor, when it comes to web search:

Google demonstrated first-hand that quality can win in this space, and I don't see any reason to deny Microsoft an opportunity to spend some money on its own attempt, and pay some developers' mortgages. And it draws Microsoft out from the desktop and onto the Web, arguably a more level playing field. I never thought I'd be saying something like that out loud, but there it is.

I disagree with both points.

First, the question of effectiveness. There's no comparison to taking yourself out of the phone book - the Microsoft crawl is the precursor to a brand new search engine, not an established service people already use. No one will switch to a new phone book if it's missing lots of names. Waiting until MSN is fully entrenched will squander the best opportunity we have for making sure the search engine market stays diverse.

A crawler boycott by bloggers will be extremely effective, even if only a few thousand people participate. That's because the hard part of creating a good search engine isn't finding or storing huge amounts of content - it's making accurate determinations of relevance.

To determine relevance, you can either hire a building full of people to look at websites and classify them (the Yahoo model), or analyze patterns in hyperlinks that other people create (the Google model). We've seen which approach scales best.

But the hyperlink model depends on a distributed network of volunteers who create links to interesting content, and allow search engine sites to harvest that information. In effect, we're all working for Google (and Technorati, and others) for free. We allow them to crawl our sites and harvest our links because our relationship is symbiotic. We give them link information, they give us useful search tools.

Because weblogs are (by definition) frequently updated, topical, and rich in hyperlinks, they are a gold mine for search engines. Weblogs serve as an amplifier for interesting and timely content.

If a search engine can't crawl the blogosphere, its relevance judgements and its ability to identify breaking news will suffer. That degradation in link analysis will hurt it far more than a few thousand pages of missing content.

As for the second question - whether a boycott is justified - all you have to do is look at Microsoft's history. Microsoft isn't entering the search space on an equal footing with anyone. They have a monopoly on desktops, and they will use that monopoly to promote MSN search, just like they used it to promote Internet Explorer. The browser wars showed us that their search engine won't have to be better than Google, or even as good as Google. As long as it's good enough for most users, it will take over, because of the overriding convenience of having web search seamlessly integrated into the operating system, office software, and email client.

Microsoft would be stupid not to pursue this strategy. They ended up with a slap on the wrist after years of anti-trust litigation stemming from the browser wars. What disincentive do they have that will keep them from repeating the exercise? Another few million in legal fees, at worst.

As Martin points out, Google won on quality. Right now, another search engine could come along and win on quality again (perhaps even Martin's own effort). But as soon as Microsoft reaches a critical mass, the game is over. No one will be able to compete with a search service embedded in Windows. Microsoft will make sure of it.

Luckily, webloggers have an opportunity to keep Microsoft from spoiling the party. By blocking their crawler in your robots.txt file, you'll be refusing to work for free to help Microsoft extend its monopoly to yet another market. Every weblog that participates in the boycott will be helping offset the unfair advantage of Windows integration.

All it takes is this:

User-agent: MSNBOT
Disallow: /

« Cormac TwomeyMoustiers »

Greatest Hits

The Alameda-Weehawken Burrito Tunnel
The story of America's most awesome infrastructure project.

Argentina on Two Steaks A Day
Eating the happiest cows in the world

Scott and Scurvy
Why did 19th century explorers forget the simple cure for scurvy?

No Evidence of Disease
A cancer story with an unfortunate complication.

Controlled Tango Into Terrain
Trying to learn how to dance in Argentina

Dabblers and Blowhards
Calling out Paul Graham for a silly essay about painting

Attacked By Thugs
Warsaw police hijinks

Dating Without Kundera
Practical alternatives to the Slavic Dave Matthews

A Rocket To Nowhere
A Space Shuttle rant

Best Practices For Time Travelers
The story of John Titor, visitor from the future

100 Years Of Turbulence
The Wright Brothers and the harmful effects of patent law

Every Damn Thing

2020 Mar Apr Jun Aug Sep Oct
2019 May Jun Jul Aug Dec
2018 Oct Nov Dec
2017 Feb Sep
2016 May Oct
2015 May Jul Nov
2014 Jul Aug
2013 Feb Dec
2012 Feb Sep Nov Dec
2011 Aug
2010 Mar May Jun Jul
2009 Jan Feb Mar Apr May Jun Jul Aug Sep
2008 Jan Apr May Aug Nov
2007 Jan Mar Apr May Jul Dec
2006 Feb Mar Apr May Jun Jul Aug Sep Oct Nov
2005 Jan Feb Mar Apr Jul Aug Sep Oct Nov Dec
2004 Jan Feb Mar Apr May Jun Jul Aug Oct Nov Dec
2003 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2002 May Jun Jul Aug Sep Oct Nov Dec

Your Host

Maciej Cegłowski


Please ask permission before reprinting full-text posts or I will crush you.