Robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may or may not request. It controls crawling, not indexing.
It follows the Robots Exclusion Protocol, using User-agent, Disallow and Allow directives, and commonly declares the sitemap location. It must be at the domain root to be found. Compliance is voluntary — well-behaved crawlers obey it, malicious ones ignore it entirely.
The most consequential misunderstanding in SEO is treating robots.txt as a way to remove pages from search. Blocking a URL prevents it being fetched, so any noindex directive on it is never read, and a blocked page can still appear in results based on external links. A misplaced Disallow: / has taken entire sites out of search.
Use robots.txt to manage crawl efficiency — blocking parameter URLs, internal search and infinite calendar paths. Use noindex to remove pages from the index, keeping them crawlable. Never block CSS or JavaScript required for rendering, and check the file immediately after any launch or migration.
Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.