Crawl Trap
A crawl trap is an infinite URL space (params, filters, calendars) that wastes crawl budget and slows indexing of important pages.
Definition
A crawl trap is when a site generates a near-infinite number of low-value URLs (parameter combinations, faceted filters, calendar pages) that crawlers keep discovering. This wastes crawl budget and reduces crawling/refresh for important pages.
Why it matters
- Wastes crawl budget and slows discovery/refresh of key pages
- Creates duplication and canonical confusion
- Common in e-commerce, faceted navigation, and programmatic sites
How to implement
- Constrain indexable parameter combinations (noindex/canonicals)
- Use robots.txt cautiously to block low-value infinite paths
- Improve internal linking so key pages aren’t buried
Related
Tools
FAQ
Common questions about this term.