DEBRIS.COMgood for a laugh, or possibly an aneurysm

Continuing Web 2.0 coverage…

I’m fascinated by Technorati, the original blog search engine, for a couple of reasons:

• The technical challenge they’ve taken on is astounding — they’re facing a Google-class scaling problem. (See statistics below.)
• and perhaps due to the first point, the performance of the system has, in my experience, been poor. Searches take 30-60 seconds. I’ve never used the site because I don’t have the patience for it.
• Even so, the recent public abandonment/shaming of Technorati by brand-name bloggers strikes me as somewhat unfair.
• New entries into Technorati’s blog-search market, e.g. Google’s blogsearch site, introduce a huge challenge to Technorati’s long-term success, especially if it’s true that Technorati is not executing well.

Technorati’s archive (a retrospective search architecture, btw) is tiered into 3 layers based on recency. They’ve made the assumption that the most recent hits for a search are more interesting than older hits, and of course in many cases that’s true. But the data i/o requirements, of simultaneously pushing new data into the front of each archive while expiring it out the other side (and from there into the next-most-recent archive) would seem to be unmanageable.

I learned this week that the write to read ratio within technorati’s database is 5:1. That presents a horrible scaling problem, because writes are slow and can’t be easily parallelized. Compare to Tribe.net’s write to read ratio: 1:99. Tribe’s is probably an extreme example (in fact I suspect that figure needs to be qualified) but in any case if your write rate is bigger than your read rate, you’ll be hurting under load. E.g., searches will take 30-60 seconds.

Part of Technorati’s problem is volume. Check out these “state of the blogosphere” stats from Dave Sifry:

• There are now 18.9 million blogs
• 70,000 new blogs appear every day, about one per second
• 55% are still active 3 months later
• 13% of blogs are updated weekly or more

It’s a big pile of data to read, store, and index.