What Is Meta Robots Tag? SEO Guide | Nepras Technologies
Gurugram, India  ·  Serving clients in India, the US & UK +91 85339 89944 info@neprastechnologies.com Free website audit →
Nepras Technologies NEPRAS
About Sample Projects Blog Contact Get a Quote →
SEO Glossary

Meta Robots Tag

The meta robots tag is a page-level instruction telling search engines whether to index the page and whether to follow its links. It is the precise counterpart to the site-level robots.txt file.

What Is Meta Robots Tag?

Placed in the head as a meta element with name="robots", it accepts values including index, noindex, follow, nofollow, noarchive, nosnippet and max-image-preview. The equivalent X-Robots-Tag HTTP header applies the same directives to non-HTML files such as PDFs, which cannot carry meta tags.

Why Is Meta Robots Tag Important for SEO?

The distinction from robots.txt is critical and routinely confused. Robots.txt controls crawling; the meta robots tag controls indexing. A page blocked in robots.txt is never fetched, so its noindex is never seen — which is why combining the two to remove a page from the index reliably fails.

How Does Meta Robots Tag Work?

Use noindex to keep pages out of the index while allowing crawling, and remove any robots.txt block on those URLs until they have dropped out. Apply the X-Robots-Tag header for PDFs and other non-HTML resources. Audit for noindexed pages that are also submitted in your sitemap, which is a contradictory signal.

Related Terms

Frequently Asked Questions

What does Meta Robots Tag mean?

The meta robots tag is a page-level instruction telling search engines whether to index a page and whether to follow its links.

What is the difference between meta robots and robots.txt?

Robots.txt controls whether a page is crawled. The meta robots tag controls whether a crawled page is indexed.

How do I noindex a PDF?

Use the X-Robots-Tag HTTP header, since non-HTML files cannot contain a meta robots tag.

Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.