« Don't Trust GoogleWe Don't Know What We're Doing »

Popdex Treachery

Anil and others have already pointed out that Popdex demonstrates the sincerest form of flattery, both in design and in name. But there's yet another reason to be annoyed by the site.

For the past few weeks, I've seen the same IP address come up in my log stats with great frequency, sometimes six or ten times a day. The address resolved to a DSL connection in St. Louis, and I figured it must be a loyal—if somewhat obsessive—reader. After all, any self-respecting bot would ask for a robots.txt file, or at the very least identify itself in the User Agent field. That's just common courtesy, after all—the guidelines have been around since 1993.

Well, it turns out Popdex spiders in stealth mode, never asking for a robots.txt file, and pretending to be a regular browser (although once in a while, it does slip and reveal its Perl nature. Even then, it's only because the programmer was too lazy to change the default user agent setting on the Perl LWP module)., I have unmasked you!

« Don't Trust GoogleWe Don't Know What We're Doing »

Greatest Hits

The Alameda-Weehawken Burrito Tunnel
The story of America's most awesome infrastructure project.

Argentina on Two Steaks A Day
Eating the happiest cows in the world

Scott and Scurvy
Why did 19th century explorers forget the simple cure for scurvy?

No Evidence of Disease
A cancer story with an unfortunate complication.

Controlled Tango Into Terrain
Trying to learn how to dance in Argentina

Dabblers and Blowhards
Calling out Paul Graham for a silly essay about painting

Attacked By Thugs
Warsaw police hijinks

Dating Without Kundera
Practical alternatives to the Slavic Dave Matthews

A Rocket To Nowhere
A Space Shuttle rant

Best Practices For Time Travelers
The story of John Titor, visitor from the future

100 Years Of Turbulence
The Wright Brothers and the harmful effects of patent law

Every Damn Thing

2020 Mar Apr Jun Aug Sep Oct
2019 May Jun Jul Aug Dec
2018 Oct Nov Dec
2017 Feb Sep
2016 May Oct
2015 May Jul Nov
2014 Jul Aug
2013 Feb Dec
2012 Feb Sep Nov Dec
2011 Aug
2010 Mar May Jun Jul
2009 Jan Feb Mar Apr May Jun Jul Aug Sep
2008 Jan Apr May Aug Nov
2007 Jan Mar Apr May Jul Dec
2006 Feb Mar Apr May Jun Jul Aug Sep Oct Nov
2005 Jan Feb Mar Apr Jul Aug Sep Oct Nov Dec
2004 Jan Feb Mar Apr May Jun Jul Aug Oct Nov Dec
2003 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2002 May Jun Jul Aug Sep Oct Nov Dec

Your Host

Maciej Cegłowski


Please ask permission before reprinting full-text posts or I will crush you.