Robots Directives
Robots directives include robots.txt (crawl control) and noindex directives (index control). Mixing them incorrectly can break indexing.
Definition
Robots directives control crawler behavior and indexing behavior. robots.txt primarily controls crawling, while meta robots and X-Robots-Tag primarily control indexing (noindex). Understanding the difference is essential for debugging indexing issues.
Why it matters
- Wrong directives can block crawling or indexing
- Prevents Disallow/noindex conflicts
- Core toolset for managing crawl budget at scale
How to implement
- To keep URLs out of SERPs: use noindex, not only Disallow
- To block crawling: use robots.txt Disallow carefully
- Keep sitemaps limited to canonical, indexable URLs
Related
FAQ
Common questions about this term.