It is the task of search engines to bring your website to the notice of the target consumers. So it is good to know how these search engines actually work and how they provide relevant information to the target customer who is involved in the search process.

There are broadly two types of search engines. The first is by robots and is called crawlers or spiders. Search Engines use spiders to test all the websites. When you deliver your website to a search engine by completing their submission procedure, their spider checks your entire website.

A ‘spider’ is a kind of program that is run by the search engine system. Spider visits your site, read the information available on the actual site, the site's Meta tags and also follow all the the links that the site has to offer. The spider then returns all that data and knowledge back to a central depository, where the data is indexed. It visits every link that your site has to offer and index those sites as well. Some spiders only go through certain number of pages on the website, so you should not create a website with 500 pages! The frequency with which it all happens is solely determined by the moderators of the particular search engine.

A spider acts like a book where it shows the table of contents, the actual information and different links for all the websites it finds during its indexing, and it may search up to a million pages a day.

Certain examples are: Excite, Lycos, AltaVista and Google.

When you refer to a particular search engine to search information, it actually locates through the index which it has created and not the whole the Web. Different search engines all different kinds of ranking procedures because not every search engine adopts the same algorithm to search through the topics.

One of the main points that a search engine scans for is the frequency of certain keywords on a web page, but it can also detect if any kind of spamdexing is done on the website. Then the search engine analyze the way that pages link to other pages in the Web.
By checking how these pages are linked with each other, a search engine can easily determine what a page is all about and also if the keywords of the linked pages are similar to the keywords on the original page or not