Should I use X-Robots-Tag or meta robots?

For HTML pages either works. For PDFs/images and server-side rules, X-Robots-Tag is the better fit.

X-Robots-Tag

X-Robots-Tag is an HTTP response header that can apply noindex/nofollow directives to HTML and non-HTML resources (PDFs, images).

Definition

X-Robots-Tag is an HTTP header sent by the server, similar to meta robots but more flexible. It can target non-HTML assets (like PDFs) and can be applied via server/CDN rules to entire paths.

Why it matters

Control indexing for PDFs and other non-HTML assets
Apply rules at the server/CDN layer faster than editing many pages
Helps manage index scope with reverse proxies and edge platforms

How to implement

Use noindex for assets that shouldn’t appear in SERPs
Avoid conflicts: if you block crawling entirely, updates may take longer
Verify headers in the HTTP response to ensure the directive is present

Examples

http

X-Robots-Tag: noindex, nofollow

Tutorials

Technical SEO Indexability Checklist

Tools

Indexability Checker

FAQ

Common questions about this term.

Back to glossary