What’s the difference between Disallow and noindex?

Disallow (robots.txt) blocks crawling; noindex (meta robots or X-Robots-Tag) blocks indexing. For noindex to work, the crawler must be able to fetch the page to see the directive.

Does robots.txt affect rankings?

Not directly, but mistakes can prevent crawling or block CSS/JS, which can harm rendering and indexability—so it can indirectly hurt rankings.

Should I include a Sitemap directive in robots.txt?

Yes. It’s optional but recommended: `Sitemap: https://example.com/sitemap.xml` helps discovery and reduces crawl waste.

robots.txt (Developer Guide)

robots.txt is the gatekeeper of crawling. Done right, it reduces crawl waste; done wrong, it can block your entire site.

1) robots.txt is not noindex

Disallow blocks crawling
noindex blocks indexing (meta robots / X-Robots-Tag)

2) A practical template

text

User-agent: *
Disallow: /admin
Disallow: /api

Allow: /assets/

Sitemap: https://example.com/sitemap.xml

Avoid blocking important CSS/JS—Google may fail to render content correctly.

3) Fast workflow with our generator

Open robots.txt Generator and pick paths to block.
Review for accidental blocks (assets, important pages).
Publish it at `/robots.txt` on your domain.

More: Indexability checklist

FAQ

Common robots.txt mistakes and misconceptions.