Indexing: how Google processes and stores pages
Separate discovery, crawling, indexing, and serving. Diagnose noindex, robots.txt, canonical, rendering, duplicate, and quality-related exclusions.
Indexing is the stage where Google analyzes a crawled page’s text, images, video, metadata, and other signals, then may store information about it in the Google index. Google describes Search as three broad stages: crawling, indexing, and serving results. Passing one stage does not guarantee the next.
Discovery, crawling, indexing, and serving
| Stage | Question | Useful evidence |
|---|---|---|
| Discovery | Does Google know the URL? | Internal links, sitemap discovery, URL Inspection |
| Crawling | Can Googlebot fetch it? | Crawl logs, status code, robots.txt, rendered response |
| Indexing | Did Google select and process this canonical content? | Page Indexing report and URL Inspection |
| Serving | Does it appear for a particular query? | Search Console performance data |
A page can be crawled but not indexed, indexed but rarely served, or known by URL while crawling is blocked.
Diagnostic order
- Confirm the exact canonical URL. Normalize protocol, hostname, path, parameters, and trailing slash.
- Check discovery. Link from an indexable page and include the canonical URL in the sitemap when appropriate.
- Check the response. The intended page should return a stable
200, not a redirect loop, soft 404, or intermittent server error. - Inspect rendered content. Ensure the primary content and links appear in the HTML Google receives.
- Check directives. Look for meta robots and
X-Robots-Tag; do not block the URL in robots.txt if Google must readnoindex. - Check canonical selection. A canonical is a signal, not a command. Conflicting links, sitemaps, redirects, and page content can lead Google to select another URL.
- Evaluate distinct value. Near-duplicate or low-value pages may be excluded even when technically valid.
Search Console workflow
Use URL Inspection for an individual URL and the Page Indexing report for patterns. Compare “user-declared canonical” with “Google-selected canonical,” inspect the last crawl and rendered HTML, then group exclusions by template or directory before editing pages one by one.
Request indexing only after fixing the underlying issue. Google says recrawling can take days to weeks and that repeated requests do not make crawling faster.
Common mistakes
- Treating a
site:query as a complete index inventory. - Blocking a URL in robots.txt and adding
noindex, which prevents the crawler from seeing the directive. - Submitting every parameter or filter URL in a sitemap.
- Assuming “Crawled - currently not indexed” is a manual penalty.
- Creating more similar pages instead of consolidating intent and strengthening the canonical page.
Primary sources
- Google: how Search works
- Google: technical requirements
- Google: noindex implementation
- Google: request a recrawl
See also: Crawling, canonical URLs, soft 404, and the Indexing glossary definition.
Q&A
Why isn't my page indexed?
Check discovery, crawl access, HTTP status, rendered content, noindex, canonical selection, duplication, soft-404 signals, and whether the page provides distinct value.
Can I force Google to index a page?
No. URL Inspection can request a recrawl for a URL you manage, but a request does not guarantee indexing or serving.
Does robots.txt remove a page from Google?
No. Robots.txt controls crawling. A blocked URL can still be known and shown as a URL-only result; use noindex on a crawlable page when removal from the index is the goal.