Stories by Malcolm Burgess

FAST search brings Web into view

Is thoroughness a Nordic trait? In a peculiar parallel to their Icelandic cousins’ contribution to the Human Genome project, the Norwegians now have their own claim to fame with — a search engine which attempts to index — and keep up with — the entire content of the Web.
And as with DNA, there’s a lot of junk and duplicate information floating around. Most search engines saunter their way through 16% at most of documents on the Web. Add to this the fact that about 40% of information contained therein is duplicate and the massive fuzzy inefficiency of it all becomes clear.
The Web contains about 800 million documents, but according to the company behind the effort, Norway-based FAST (Fast Search and Transfer), none of the big Web search engines have so far been able to make more than 160 million documents searchable. In May 1999, FAST announced a catalogue the size of 80 million documents. On August 2, 1999 more than 200 million documents were made searchable, with hopes of growing with the Web to one billion documents and beyond.
In any review of a speed machine you have to ask how it handles corners. It’s all very well that you can get so many million hits, but can it pull over in time to drop off the kids or pick up hitchhikers? FAST’s parallel server approach delivers exceptional search speed — a typical query races through all 200 million documents in less than a second — but its massive size allows it to place the most relevant items in the first two pages of results — reducing the number of "gems" that are missed. Drawing on scalablitity theory, FAST aims for its search engine to grow "organically" with the Web. — Malcolm Burgess
All the Web All the Time