Introduction to Web Robots
What is a Web Robot
A Web robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. Note that "recursive" here doesn't limit the definition to any specific traversal algorithm; even if a robot applies some heuristic to the selection and order of documents to visit and spaces out requests over a long space of time, it is still a robot. Normal Web browsers are not robots, because the are operated by a human, and don't automatically retrieve referenced documents (other than inline images). Web robots are sometimes referred to as Web Wanderers, Web Crawlers, Web Walkers or Spiders . These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them.
Web Robots can do in a few minutes what it may take a human several hours to do. Robots provide a valuable service to internet users; without them it would be impossible to build functional web indices and keep them up-to-date. They are an essentiality of life on the internet, and they are increasing in number everyday. But the robot's greed to try and document the entire web could be its own undoing - in the sense that no single robot will be in a position to provide a comprehensive, relevant listing of all available websites for a particular search.
Intelligent Agents or Bots
Web Robots belong to a class called "Intelligent Agents" or simply "Bots". Formally - "Bots have a life of their own. They make the mechanical equivalent of decisions and take action without intervention from humans. They are self actuating. In fact, the ability to act autonomously is the very essence of bothood." Many researchers believe that to be called ``intelligent,'' an agent must satisfy several interrelated criteria. There are five attributes which capture the essence of an intelligent agent:
There are many types of Bots such as :