« Don't Trust Google | We Don't Know What We're Doing » |
Anil and others have already pointed out that Popdex demonstrates the sincerest form of flattery, both in design and in name. But there's yet another reason to be annoyed by the site.
For the past few weeks, I've seen the same IP address come up in my log stats with great frequency, sometimes six or ten times a day. The address resolved to a DSL connection in St. Louis, and I figured it must be a loyal—if somewhat obsessive—reader. After all, any self-respecting bot would ask for a robots.txt file, or at the very least identify itself in the User Agent field. That's just common courtesy, after all—the guidelines have been around since 1993.
Well, it turns out Popdex spiders in stealth mode, never asking for a robots.txt file, and pretending to be a regular browser (although once in a while, it does slip and reveal its Perl nature. Even then, it's only because the programmer was too lazy to change the default user agent setting on the Perl LWP module). 65.69.49.203, I have unmasked you!
« Don't Trust Google | We Don't Know What We're Doing » |
brevity is for the weak
Greatest Hits
The Alameda-Weehawken Burrito TunnelThe story of America's most awesome infrastructure project.
Argentina on Two Steaks A Day
Eating the happiest cows in the world
Scott and Scurvy
Why did 19th century explorers forget the simple cure for scurvy?
No Evidence of Disease
A cancer story with an unfortunate complication.
Controlled Tango Into Terrain
Trying to learn how to dance in Argentina
Dabblers and Blowhards
Calling out Paul Graham for a silly essay about painting
Attacked By Thugs
Warsaw police hijinks
Dating Without Kundera
Practical alternatives to the Slavic Dave Matthews
A Rocket To Nowhere
A Space Shuttle rant
Best Practices For Time Travelers
The story of John Titor, visitor from the future
100 Years Of Turbulence
The Wright Brothers and the harmful effects of patent law
Every Damn Thing
Your Host
Maciej Cegłowski
maciej @ ceglowski.com
Threat
Please ask permission before reprinting full-text posts or I will crush you.