Search engine spiders or crawlers or robots are actually softwares made my search engine companied to read and scan the information on a site and store them in the databases and keep them up to date. Every search engine performs four basic functions: web crawling, document indexing, query processing and ranking results.

All search engines have automated programs called search engine spiders which “crawl” thorough the web pages on the World Wide Web. But it’s not always right to have the spiders crawling across certain sections of the website. For example, the payment details of the customers for purchases made on the website are considered to be private information on which the search engine spiders should have no access to.

The search engine spider is the most important tool that controls the ratings of websites in the listings of major search engines like Google, MSN or Yahoo.

So as to ease the crawl of the spider over a particular site the site should be light and take minimum time to load. A complex site, say, one built on a flash platform or one which comprises of a number of photographs will not be spider friendly hence chances are of them not showing on the Search Engine Results Page (SERP).

If a particular site is well connected internally by linking, it is easier for the visitors to navigate through the pages. It also means that it will, likewise be easier for the search engine spiders to crawl through the lines and aid in greater visibility of the page(s).

It is a general notion that search engine spiders have crawled over the entire page in the World Wide Web. Bu certain studies prove otherwise. Recently it has been established that no search engine indexes more than 16% of all the billions of pages of online text. The spiders only download a fraction of the entire website and sample them. So it is imperative to have all the relevant information on these pages.