For what main reason do search engines employ web crawlers, also known as spiders or bots?

किस मुख्य कारण से खोज इंजन वेब कॉलर का उपयोग करते हैं, जिन्हें मकड़ियों या बॉट के रूप में भी जाना जाता है?

A web crawler (spider or bot) is a program that keeps following links and reading pages. What it finds is stored in the search engine's index, so that your query can be answered in a fraction of a second without searching the live Web each time.

(A) Crawlers read content; they do not write it.

(C) Search engines want users to reach websites, not be blocked from them.

(D) Email attachments are handled by mail software, which has nothing to do with crawling.

Crawling finds pages, indexing stores them, ranking orders them.

A file called robots.txt tells crawlers which parts of a site they should not visit.

SEO (Search Engine Optimization) is the work of making a site easier to find in search results.

The deep web is content not indexed by search engines, such as your email inbox or bank account pages; it is not the same as the dark web.