• by David Berkowitz , June 21, 2005

Ever search for something online and come up short? A number of recent announcements target that sense of loss.

The parties coming to the rescue include Yahoo!, the Federal Aviation Administration, and the National Security Agency. The last two aren't the typical agencies referred to when it comes to search engine marketing; it's a sign that the industry is just starting to get interesting. Everyone has a vested interest in search.

Let's start with the search engine. Yahoo! launched its Search Subscriptions beta site, allowing searches for content from Consumer Reports, FT.com, Forrester Research, IEEE publications, the New England Journal of Medicine, TheStreet.com, and the Wall Street Journal. Searchers can read a short intro to the content, but after that, they need to pay for access.

There's a pressing question surrounding the launch: Why would people want to search for content they can't readily view?

The publishers are the biggest winners in the deal. The real value for them will come if Search Subscriptions is integrated into Yahoo! News. As for the publishers now involved, only two, TheStreet.com and WSJ.com, have any content with overlapping coverage areas. A physician won't go to Search Subscriptions to search the New England Journal of Medicine; he or she will have that site bookmarked already. And an average person seeking health information will find little of interest on that site; free content at sites such as WebMD and iVillage generally suffices.

If the integration with Yahoo! News occurs, as PaidContent.org suggests could happen, then the publishers will be able to reach a massive swath of Internet users, exposing them to content that people may well find of value if it's relevant enough to their search. The challenge for Yahoo! will be balancing providing the publishers with the necessary reach while not alienating its visitors, many of whom may balk if they discover they suddenly have to pay for accessing Yahoo!'s results.

PaidContent also suggested that in time, Yahoo! could adapt the Search Subscriptions model to a revenue-share arrangement with publishers, with the search engine taking a cut of content purchased. Yahoo!'s paid search advertising model is far too lucrative to risk for the relative pocket change that would come in from revenue shares on subscriptions, so expect the engine to delve gently into this arena.

Yahoo! isn't the only party expanding the scope of search engines. MarketingVox pointed to a release from New York's University of Buffalo describing a government-sponsored projected to find new connections through search. The Federal Aviation Administration is backing a search engine developed by the school based on Unintended Information Revelation (UIR) as a way to reveal "hidden" information, such as seemingly unrelated or unexpected terms and concepts.

The Buffalo release explains, "Existing search engines process individual documents based on the number of times a key word appears in a single document... UIR is based on the construction of concept chain graphs that search for the best path connecting two concepts within a multitude of documents."

It seems like LinkedIn for search results, examining the paths connecting concepts to shed light on the concepts. For instance, a search engine can provide a wealth of information about Google and AOL. If this UIR engine were in the public domain, you might find out that Microsoft is very often connecting the two, and examining Microsoft's role could shed new light on the direction both sites are moving in.

The UIR engine prototype was designed to scour information in the 9/11 Commission Report - hence the FAA's interest. Broader applications are coming. You have to think the major search engines are working along similar themes. Logic would dictate that a single search algorithm couldn't account for all of the ways information is connected.

MarketingVox also pointed to a related New York Times piece from last week, "Enough Keyword Searches. Just Answer My Question," describing the AQUAINT project by the National Security Agency, Central Intelligence Agency, and others. The acronym stands for "advanced question and answering for intelligence," which aims to dig up answers to complicated or interrelated questions. For instance, Times writer James Fallows says a question posed to AQUAINT may be, "Did any potential terrorist just buy an airline ticket?"

It feels like the opposite of the paradigm that gave birth to the Internet, which started as a Department of Defense research project and was fomented by universities before any corporate interests took hold. Search engines have had a less linear path, with some, such as Google, emerging from university researchers. Search engine companies popularized the medium of search, and it's now heading back to the universities and government agencies, which are thinking, "What else can we do with this?"

That's one more question a search engine may be able to answer someday.