The X-Robots-Tag is an HTTP response header carrying the same indexing directives as the meta robots tag, but applicable to any file type rather than only HTML.
It accepts the same values — noindex, nofollow, noarchive, nosnippet and others — and is set in the server configuration or application response. Because it is a header rather than markup, it works for PDFs, images, videos and other non-HTML resources that cannot contain a meta tag.
Without it there is no way to noindex a PDF, which matters because PDFs are indexed and frequently outrank the HTML pages they duplicate. It is also the practical way to apply directives across many files at once through a server rule rather than editing each document.
Use it for non-HTML resources and for pattern-based rules across directories. As with the meta robots tag, the URL must remain crawlable for the header to be read. Verify it with curl or browser developer tools, since a header is invisible in page source and easily assumed rather than confirmed.
Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.