Meta Robots
Meta robots controls whether a page is indexed (index/noindex) and whether links are followed (follow/nofollow).
Definition
Meta robots is an HTML <meta name="robots" content="..."> directive in the <head>. It affects indexing and link-following behavior. For example, noindex tells search engines not to show the page in SERPs, while follow allows link discovery.
Why it matters
- Keep low-value pages (internal search, test pages) out of SERPs
- Avoid index bloat and diluted quality signals
- Control index scope alongside canonicals and sitemaps
- Protect unfinished pages — use noindex on dev/staging/test pages to prevent indexing
- Manage pagination strategy — noindex page 2+ to avoid duplicate content issues
- Control link equity flow — nofollow prevents PageRank from flowing to low-trust pages
- Fine-grained crawler control — set different rules per bot (googlebot, bingbot)
How to implement
- To remove from SERPs: use noindex (don't block crawling via robots.txt or the bot can't see it)
- If you still want link discovery: use noindex,follow (as appropriate)
- Don't list noindex URLs in sitemaps and keep correct HTTP statuses
- Use noindex on faceted/filtered pages to prevent index bloat
- Use X-Robots-Tag HTTP header for non-HTML resources (PDFs, images)
- Add site-wide noindex on staging environments or use robots.txt Disallow
- Verify meta robots with Search Console URL Inspection tool
Examples
html
<!-- Default (can be omitted): allow indexing and link following -->
<meta name="robots" content="index, follow" />
<!-- Don't index but follow links (common for pagination, filters) -->
<meta name="robots" content="noindex, follow" />
<!-- Index but don't follow links (for external link pages) -->
<meta name="robots" content="index, nofollow" />
<!-- Block completely -->
<meta name="robots" content="noindex, nofollow" />
<!-- Additional directives: no caching, no snippets -->
<meta name="robots" content="noarchive, nosnippet" />
<!-- Target specific crawlers -->
<meta name="googlebot" content="noindex" />
<meta name="bingbot" content="noindex" />typescript
// Next.js App Router - layout.tsx or page.tsx
import { Metadata } from 'next';
// Set based on page type
export function generateMetadata({ params, searchParams }): Metadata {
const isFilterPage = Object.keys(searchParams).length > 0;
const isPaginatedPage = searchParams.page && Number(searchParams.page) > 1;
return {
robots: {
index: !isFilterPage && !isPaginatedPage,
follow: true,
nocache: false,
googleBot: {
index: !isFilterPage && !isPaginatedPage,
follow: true,
'max-video-preview': -1,
'max-image-preview': 'large',
'max-snippet': -1,
},
},
};
}
// Staging site-wide noindex
// next.config.js
const isProduction = process.env.NODE_ENV === 'production';
module.exports = {
async headers() {
if (!isProduction) {
return [{
source: '/:path*',
headers: [{ key: 'X-Robots-Tag', value: 'noindex, nofollow' }],
}];
}
return [];
},
};Related
FAQ
Common questions about this term.