How do search engines work? Crawling, indexing and ranking explained

How do search engines work, in one paragraph?

A search engine sends a program called a crawler to discover and download web pages, analyses and stores those pages in a huge database called the index, and when someone searches, ranks the indexed pages by how relevant and useful they are for that query. Google's In-depth guide to how Google Search works calls the three stages crawling, indexing and serving search results, and notes that not every page makes it through each one. Microsoft's How Bing delivers search results describes the same three stages for Bing.

The whole process is automated. Google describes its search engine as fully automated, and says it doesn't accept payment to crawl a site more often or rank it higher. Bing says the same of its organic "blue links". That is why SEO is about making pages easy to crawl, easy to understand and worth ranking, not about paying anyone. If you are new to the term, start with What is SEO?

What are the steps from web page to search result?

A page goes through six steps: it is discovered, crawled, rendered, analysed, stored as part of a canonical group in the index, and finally ranked and shown for a query. Here is the pipeline as Google and Bing document it.

1. 1. Discovery: the engine learns the URL exists

   Google says there is no central registry of all web pages, so it constantly looks for new and updated ones. Most are found by following links from pages it already knows; others arrive through sitemaps that site owners submit. Bing's Webmaster Guidelines list IndexNow submissions, XML sitemaps, crawlable internal links and external links from relevant websites as the discovery routes.

2. 2. Crawling: a bot fetches the page

   Google's crawler is Googlebot; Bing's is Bingbot. Google says an algorithmic process decides which sites to crawl, how often and how many pages to fetch from each, and that its crawlers try not to overload a site. Server responses steer this: Google gives HTTP 500 errors as an example of a signal to slow down. Pages blocked by robots.txt or behind a login are not crawled.

3. 3. Rendering: the page is run like a browser would

   Many sites build content with JavaScript. Google says it renders pages and runs their JavaScript using a recent version of Chrome, because some content only appears after that. Bing's guidelines warn against hiding critical content behind client-side rendering, and say content that cannot be reliably rendered may not be indexed.

4. 4. Analysis: the engine works out what the page is about

   Google analyses the text, key tags and attributes such as the <title> element and image alt text, and the images and videos. It also collects signals about the page, such as its language, the country it is local to, and its usability.

5. 5. Indexing: duplicates are grouped and a canonical is stored

   Google clusters pages with similar content and selects one as the canonical, the version that may be shown in search results; the others may be served in other contexts, such as to mobile users. The result is stored in the Google index. Google is clear that indexing is not guaranteed and that not every page it processes is indexed.

6. 6. Ranking and serving: the best matches are shown

   When someone searches, the engine looks up matching pages in its index and returns the ones it considers most relevant and highest quality. Google says relevance depends on hundreds of factors, including the searcher's location, language and device, and that the features on the results page (local results, images and so on) change with the query.

What is crawling and what stops it?

Crawling is the stage where a search engine's bot discovers a URL and downloads the page; it fails when the bot cannot find the URL, is told not to fetch it, or cannot get a working response from the server. Google lists the common reasons as problems with the server handling the site, network issues, and robots.txt rules that prevent Googlebot's access.

A few points that trip people up:

• Orphan pages. A page that nothing links to and that is not in a sitemap may never be discovered. Our guide to orphan pages shows how to find them.
• robots.txt controls crawling, not indexing. Bing's guidelines say this in as many words, and tell you to use noindex when a URL should not appear in Bing search or Copilot. A page blocked in robots.txt cannot be read, so a noindex tag on it cannot be seen either.
• Firewalls and bot protection. A CDN or security plugin that challenges bots can block crawlers even when robots.txt allows them. Check what the crawler actually received, not what your browser shows.
• Crawl waste. Bing says it allocates crawl capacity based on site health, efficiency and crawl value, and that large numbers of duplicate or low-value URLs (for example, endless parameter combinations) can delay the indexing of important pages.

For a full list of crawler names and what blocking each one changes, see our guide to allowing or blocking AI crawlers in robots.txt.

What is indexing and why are some pages not indexed?

Indexing is the stage where the search engine analyses a crawled page, decides whether it is a duplicate of another page, and stores the chosen version so it can be returned for searches; a page can be crawled and still not be indexed. Google names three common reasons: low-quality content, robots meta rules that disallow indexing, and website designs that make indexing difficult.

Two ideas explain most indexing surprises:

• Canonicalization. If the same content is available at several URLs (with and without www, with tracking parameters, a print version), Google picks one as canonical and treats the others as duplicates. You can suggest your preferred URL, but Google may choose a different one. Bing's guidelines add that duplicate URLs dilute signals and recommend 301 redirects for permanent moves.
• Not everything should be indexed. Search Console's Page indexing report help says not to expect every URL on your site to be indexed. Duplicates, intentionally blocked pages and pages without meaningful content are normal exclusions. What matters is that your important pages are in.

Being indexed is also not the same as appearing. Google's URL Inspection help says a page reported as "URL is on Google" is eligible to appear, but that this doesn't guarantee it will appear in results.

How do search engines rank pages?

Search engines rank indexed pages by estimating how relevant and useful each one is for the exact query, combining many signals about the page, the site and the searcher; Google says it uses hundreds of factors and does not publish a full list. Google mentions the searcher's location, language and device among them.

Microsoft is more specific about Bing. Its How Bing delivers search results page lists the main parameters, in general order of importance, while noting the weighting varies by search and changes over time:

Two things both engines agree on: ads are separate and labelled, and organic positions cannot be bought. A worked example of ranking in practice: a search for "dentist open Saturday" from a phone in Calgary is likely to get local results, while "how long does a filling last" gets articles. Same engine, same index, different results, because the query and the searcher differ. For what Google's "average position" figure actually measures, see Search Console average position explained.

How do AI answers choose their sources?

AI answers in search, such as Google's AI Overviews and AI Mode, Microsoft Copilot and ChatGPT search, retrieve pages through a search index and then write a response that links to the pages supporting it, so they depend on the same crawling and indexing as ordinary results. A page that fails either stage cannot be cited.

• Google. Its AI features documentation says a page must be indexed and eligible to show a snippet to appear as a supporting link, and that there are no additional requirements or special optimizations. AI Overviews and AI Mode may run several related searches at once (query fan-out) to find supporting pages.
• Bing and Copilot. Microsoft's guidelines say Bing and Copilot rely on the same crawling, indexing and ranking foundation as traditional search, and that the guidelines cover Bing search, Copilot and grounding API results.
• ChatGPT search. OpenAI's help article Searching the web with ChatGPT says ChatGPT sometimes partners with other search providers and rewrites a question into one or more targeted queries. To be eligible for inclusion, a site must allow OAI-SearchBot to crawl it and let traffic from OpenAI's published IP addresses through its host or CDN. OpenAI says placement is not guaranteed.

We will not repeat the details here. For what Google says about getting cited, the controls and how to measure it, read How to get cited in AI Overviews; for other assistants, see our GEO guide.

How do you tell which stage failed?

Match the symptom to the stage: if the search engine does not know the URL or could not fetch it, crawling failed; if it fetched the page but did not store it, indexing failed; if the page is indexed but gets no impressions, it is losing at ranking. Google Search Console's Page indexing report and URL Inspection tool tell you which, for free.

Work top to bottom: a ranking fix is wasted on a page that is not indexed. For a whole-site version of these checks, see how to do an SEO audit, and for setting up Bing, our Bing Webmaster Tools guide.

How MrKeyword helps at each stage

MrKeyword helps with crawling and indexing problems through its Site Audit, and with ranking through rank tracking built on your own Search Console and Bing Webmaster Tools data. It is a way of seeing the same pipeline across a whole site rather than one URL at a time.

• Crawling and rendering: the Site Audit follows links like a search crawler, respects robots.txt, and reports errors, redirects and blocked pages, sorted by how many pages each issue affects.
• Ranking: rank tracking reads the average positions Search Console and Bing report for your verified site and keeps a daily copy, by device and country.
• AI answers: AI visibility asks ChatGPT, Gemini, Claude and Perplexity questions you choose each week and records whether they named you and which sites they cited.

Prices: Free is $0 for one website; Starter is $10/month, Growth $19/month for 3 websites and Agency $49/month for 10. Full limits are on the pricing page.

What MrKeyword does not do

MrKeyword is not a search engine and cannot see inside Google's or Bing's index; it shows you what their own tools report and what its crawler finds on your site. Specifically:

• No view of Google's index. Whether a page is indexed, and which canonical Google chose, comes only from Search Console. MrKeyword's crawler sees your pages as a crawler would, but it is not Googlebot.
• No live rank checks. Positions are Search Console and Bing averages over a period, with their reporting delay, not a live scrape of the results page. MrKeyword does not scrape Google.
• No guarantee of indexing, ranking or AI citations. Fixing what the audit finds removes obstacles; it cannot make a search engine include or rank a page.
• No measured data for other people's sites. Competitor positions are labelled Predicted or Not available.

Questions and answers