Old news

Will notes that you can get really old news when doing a web search on a hot new topic (‘vice presidential debate’), and he’s right about why.

First of all, of course, the new stuff may simply not be in the search engine’s index yet. In addition to that, though, most web search engines pay attention to inlinks and link text, and those take time to accumulate. This makes for an inherent bias towards stuff that’s been around long enough to get linked to. (Sure, there are things you can do about it, but they’re all second-order.)

Before you tell me how all this is because we just don’t grok the intertwingliness of the future which is now, notice that this problem is not really about search engines being out of date, it’s about the web being out of date. Even if your engine learns about all new documents and links instantaneously, the newest document is not yet linked.It’s not even specific to the web — paying attention to later meta-information about anything means a bias toward things that are old enough to have some. A Thing is created (say, a weblog post), and then meta-Things begin to accumulate on top of it (say, trackbacks). If the meta-Things are important to your retrieval, then you’ve just made it much harder to find the newly-born Thing.

Pings and real-time notification will speed this up enormously — when I’m alerted about my friend’s blog post, I can link/trackback/comment/react, and start the meta-comment ball rolling right away — but this just compresses the timeframe. The newest things still are too fresh for pointers, and you won’t find them easily if the pointers drive your retrieval. Sure, you get notified about what your buddies just said, but search is supposed to help you find stuff that you _don’t_ already know about.

This is one very large, very noisy cocktail party. Somewhere, right now, someone who is _not_ your friend (yet) is softly saying something that is exactly what you’re interested in, but you can’t hear it … because of all that yammering!