Indexation & Crawl Control at Scale
Large programmatic sites must carefully manage indexation to protect crawl budget and index quality.
Learning Focus
After this lesson you can implement indexation rules, manage crawl budget, segment sitemaps, and suppress low-value pages at scale.
This lesson covers the seven indexation and crawl control areas (leaves 9.4.1–9.4.7): indexation rules, noindex rules, canonical rules, sitemap segmentation, crawl depth management, parameter control, and low-value page suppression.
Indexation Rules
Define rules for which programmatic pages should be indexed.
Indexation decision criteria:
| Criterion | Index | Noindex |
|---|---|---|
| Data completeness | All required fields populated | Missing fields |
| Content quality | Passes quality thresholds | Below quality thresholds |
| Search demand | Minimum search volume for the combination | Very low or no search volume |
| User value | Provides unique value | Better served by a database or search |
| Duplication | Unique content | Near-duplicate of another page |
| Engagement potential | Likely to generate engagement | Unlikely to satisfy user needs |
Indexation rule implementation:
- During page generation, each page receives an indexation flag (index/noindex).
- Noindexed pages are excluded from sitemaps.
- Noindexed pages use
noindex, followto pass link equity. - Periodically review noindexed pages for potential indexation as quality improves.
Noindex Rules
Define when noindex should be applied.
Noindex application:
| Scenario | Noindex Directive | Reasoning |
|---|---|---|
| Incomplete data | noindex, follow | Page may still be useful but not ready for index |
| Thin content | noindex, follow | Pass equity, protect index quality |
| Test pages | noindex, nofollow | No need for equity or indexation |
| User-generated pages pending review | noindex, follow | Index after human approval |
| Seasonal pages out of season | noindex, follow | Re-index when seasonally relevant |
Canonical Rules
Define canonical rules for programmatic pages.
Canonical patterns:
| Page Type | Canonical Rule |
|---|---|
| Single entity | Self-canonical |
| List/combination page | Self-canonical |
| Faceted filter combination | Canonical to parent category page |
| Parameter variation | Canonical to clean URL version |
| Paginated page 2+ | Self-canonical or canonical to page 1 (deprecated; use self) |
Canonical enforcement:
- Template must generate canonical tag for every page.
- Canonical must be absolute URL.
- Cross-domain canonical only when content is intentionally shared.
Sitemap Segmentation
Core Concept
Organize programmatic pages into segmented sitemaps.
Sitemap segmentation strategy:
| Sitemap | Contents | Update Frequency |
|---|---|---|
| High-priority pages | Pages with traffic, high quality, strong engagement | Daily or weekly |
| Standard pages | Most indexable programmatic pages | Weekly |
| Product/listings | Product or listing pages | Daily (for inventory changes) |
| New pages | Recently generated pages | Daily |
| Suppressed pages | Noindex pages (for monitoring) | Weekly |
Sitemap size management:
- Each sitemap limited to 50,000 URLs or 50 MB.
- Sitemap index for larger sets.
- Only indexable (noindex, follow) URLs in sitemaps.
Crawl Depth Management
Manage crawl depth for large-scale sites to ensure important pages are crawled.
Crawl depth optimization:
| Strategy | Implementation |
|---|---|
| Shallow linking | Important pages linked from category or hub pages (depth 2-3) |
| Sitemap prioritization | High-priority sitemaps submitted for quicker crawl |
| Orphan prevention | Ensure all important pages have at least one internal link |
| Hub pages | Create intermediate hub pages to reduce depth |
| Freshness signals | Update sitemaps and content freshness to encourage recrawl |
Parameter Control
Manage URL parameters that can create infinite crawl paths.
Parameter management strategies (from Lesson 3.7.5):
| Parameter Type | Control |
|---|---|
Sorting (?sort=) | Block in robots.txt or noindex |
Filtering (?color=) | Noindex, canonical, or JavaScript-based |
Pagination (?page=) | Keep crawlable with noindex on deep pages |
Tracking (?utm_) | Ignore (Google ignores) |
Session IDs (?sid=) | Remove from URLs |
Low-Value Page Suppression
Suppress low-value pages from the index to protect crawl budget and index quality.
Suppression methods:
| Method | When to Use |
|---|---|
| Noindex | Most common for low-quality programmatic pages |
| Robots.txt block | Only for very low-value parameter combinations (use with caution) |
| 404/410 | If the combination should not exist (not recommended — creates 404s) |
| Consolidation | Combine several low-value pages into one higher-value page |
Suppression review cycle:
- Quarterly, review performance of programmatic pages.
- Pages with zero impressions in 90 days → review for suppression.
- Pages with high impressions but low engagement → review content quality.
- Pages with good performance → keep and potentially expand.
Workflow
- Define indexation rules: pages meeting quality thresholds → index, follow. Pages with incomplete data, thin content, or below value threshold → noindex, follow. Test pages → noindex, nofollow.
- Implement canonical rules: self-canonical for primary entity and combination pages. Canonical to parent for faceted/filtered variants. Absolute URLs only.
- Segment sitemaps by priority: high-priority (traffic-generating, high quality), standard (most indexable pages), new pages (recently generated), suppressed (noindex, for monitoring only).
- Manage crawl depth: ensure all important pages are within 2-3 clicks from a crawled hub. Submit high-priority sitemaps. Prevent orphan pages.
- Control URL parameters: block sorting, filtering, session, and tracking parameters in robots.txt or with noindex. Keep only canonical, clean URLs in sitemaps.
Common Mistakes
warning
- Indexing everything and hoping for the best: Without explicit indexation rules, thin and low-quality pages dilute index quality and waste crawl budget. Default to noindex; promote to index when quality thresholds are met.
- Blocking important pages in robots.txt: Blocking filter/sort parameter URLs is correct, but accidentally blocking category or location pages via overly broad rules can remove pages from the index entirely. Test robots.txt rules carefully.
warning
- No sitemap segmentation: Throwing all 50,000+ URLs into a single sitemap makes it impossible to prioritize crawling. Segment sitemaps so high-value pages get crawled first.
- Ignoring crawl budget entirely: On large programmatic sites, Google may spend 70%+ of crawl on low-value parameter URLs. Without crawl budget management, important pages may go uncrawled or refreshed slowly.
- Using noindex,nofollow across the board: Using nofollow on noindexed pages that have inbound links wastes link equity. Use noindex,follow to preserve link flow to indexable pages.
Checklist
- Define explicit indexation rules based on quality and data completeness thresholds
- Default new pages to noindex; index only when thresholds are met
- Implement self-canonical on all primary entity and combination pages
- Segment sitemaps by priority (high, standard, new, suppressed)
- Block low-value parameter URLs (sort, filter, session, tracking) from crawling
- Ensure all important pages are linked within 2-3 clicks from a crawled hub
- Use noindex,follow on suppressed pages to preserve link equity
- Monitor crawl stats: low-value URLs should consume <20% of crawl budget
- Review indexation rules quarterly based on page performance data