The meta robots tag is a page-level instruction telling search engines whether to index the page and whether to follow its links. It is the precise counterpart to the site-level robots.txt file.
Placed in the head as a meta element with name="robots", it accepts values including index, noindex, follow, nofollow, noarchive, nosnippet and max-image-preview. The equivalent X-Robots-Tag HTTP header applies the same directives to non-HTML files such as PDFs, which cannot carry meta tags.
The distinction from robots.txt is critical and routinely confused. Robots.txt controls crawling; the meta robots tag controls indexing. A page blocked in robots.txt is never fetched, so its noindex is never seen — which is why combining the two to remove a page from the index reliably fails.
Use noindex to keep pages out of the index while allowing crawling, and remove any robots.txt block on those URLs until they have dropped out. Apply the X-Robots-Tag header for PDFs and other non-HTML resources. Audit for noindexed pages that are also submitted in your sitemap, which is a contradictory signal.
Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.