Traditional search engines collect and index data under a centralized scheme from each distributed location and replicates data across distributed server clusters in various geographical locations. Google continues to attempt to collect local data efficiently and "support geo-distributed clusters" remains an open task on Google's radar.
A properly designed distributed search engine may achieve the goal of sharing local data collected locally in an efficient way that does not require the data replication among all the geographic locations. And the data reflects the dynamic products and services information from the hidden web instead of only being from static web pages. Moreover, the data being collected must have an expiration date according to the real world so that expired information will be purged in a timely manner. This is in tern to achieve the goal of reducing the energy consumption to a reasonable level.
Existing distributed search engine solutions are lacking a systematic way to ensure data coverage due to the arbitrary number of nodes scattered all over the internet.
Using Agent Technology to implement this kind of distributed search engine is an appealing solution leading towards solving the problems that traditional search engines are facing. Agent Technology provides a complete infrastructure to set up an agents and brokers network as well as the messaging protocol for the communication between the peers. This meets the requirement of the "distributed systems" from Wikipedia we quoted in the previous blog page.
Each node only needs to collect and index the local information and share it with other nodes when remote usersrequest it. Remote data can be cached locally in order to provide quick search results. By using human intelligence, each node only needs to save the indexed link information for every web page, instead of saving all the words in every document of every website over the internet.
For those who are not familiar with Agent Technology, please refer to Tim Finin's PowerPoint slide show of "Software Agents - Knowledge Sharing".