Biggest News That No One Cared

Nothing in the blogosphere is really considered “little known” these days especially given the # of trackbacks on Versign’s blog . . . but comparatively, the purchase of by Verisgn got very little attention given the implications. . . (maybe cause people were confusing it with deal or that the web 2.0 conference was more interesting) . . . (here are some other posts on the topic . . . Ochman, Kottke, Paid Content, and Winer’s own account, and lastly John Furrier)

The purchase is a watershed moment because the way current search engines are architected is wholly inadequate given the increasingly real time nature of the web. Limited by the # of crawler they can sent out to the web, (computing cost + just bogging down the web in general), search engine indexes are usually out of date the moment they are refreshed. The next time any webpage might be re-indexed could be hours if not days. The web has changed significantly in the last 3 years as blogs became one of the major sources of “pages” and information on the web. Because people instead of corporations are now creators and distributions channels for content and information, the web is “changing” and “updating” itself much faster than it was when corporations controlled the creation and distribution of information.

Good analogy is eBay. Because pricing information updates in real time for auctions, a traditional “search engine” implementation is useless to index the tens of million plus items that are live on the site at any time. Instead, the index must be updated to the second in order to give searchers on eBay relevant information regarding products.

The web is becoming more and more like eBay . . . timeliness is as important as static relevancy.

Yahoo received a lot of crap for the paid inclusion service when first launched, but there was a token of a good idea beyond making money that made a lot of sense. I.E., instead of “pulling” for content, Yahoo wanted web sites themselves to “push” content to the search engine. This not only saves Yahoo’s crawlers from crawling non-updated sites, but also help Yahoo keep informed of the latest changes on the web. Not surprisingly, all the major search engines are doing this today. . . ie asking major contributor of content/information to push or ping them with new content as they are updated. (although its not highly publicized for PR reasons)

As the # of search engines proliferate, it will no longer make sense for ONE website to push content to MULTIPLE search engines. . . it is both inefficient and expansive. The centralized ping and crawl RSS model makes a lot more sense. Only when there are changes to the website does the website ping the centralized server ONCE, which in turn lets ALL the search engines (or RSS readers) know that its time to send a crawler/reader out to read the RSS feed of the website to gather new content.

This is the future of search engines if they ever want to become real-time repositories of “human knowledge.” Blog Search is not only a nice-to-have vertical search engine for Yahoo and Google, it’s a mandatory upgrade if they ever want to remain competitive and true to their mission.

Today, Versign in one swoop has become the gatekeeper of a major source of internet content . . . while it will certainly keep the service “open” . . . in the near future a premium service will allow paid customers to receive priority “service” and the ability to skip the queue . . . in effect those who pay will have newer and more recent indexes . . . as more of the web subscribe to the RSS model of search inclusion . . . Versign will be well poised to profit from this acquisition. . . If I was Google or Yahoo (Yahoo does not have blogger and Google does not have ), I would have ponied up the low single digit millions for . . . in 5 years weblogs’s assets will be worth more than Flikr . . . mark my word. .