CodeMaster AcademyWhat robots.txt actually controls (and what it doesn't)
A robots.txt file sits at your domain's root and tells well-behaved search engine crawlers which parts of your site they're allowed to request — but it's a voluntary convention, not an enforcement mechanism. Legitimate crawlers like Googlebot respect it; malicious scrapers and bots that ignore it entirely aren't stopped by anything written in this file. It's also not a privacy tool — disallowing a page in robots.txt doesn't remove it from search results if other sites link to it, and it doesn't hide the page from anyone who has the direct URL; it only asks crawlers not to fetch it.
The most common mistake: blocking too much by accident
A single misplaced Disallow: / blocks your entire site from being crawled, and it's a surprisingly common accidental deployment mistake — often left over from a staging environment's robots.txt that was never changed before going live. This is one of the most damaging technical SEO errors possible, since it can silently prevent an entire site from being indexed while everything else about the site looks fine to a human visitor.
What belongs in a well-formed robots.txt
- User-agent: Which crawler the following rules apply to (
*means all crawlers). - Disallow: Paths crawlers shouldn't request (e.g., admin areas, internal search results, duplicate parameter-based URLs).
- Allow: Exceptions within a disallowed path that should still be crawlable.
- Sitemap: A pointer to your sitemap.xml location, helping crawlers discover your full page list efficiently.
Using this generator
Specify which paths to allow or block and add your sitemap URL, and the tool outputs a correctly formatted robots.txt ready to place at your domain's root (e.g., yoursite.com/robots.txt). After publishing, test it with Google Search Console's URL inspection tool to confirm your important pages aren't accidentally blocked.
Common questions
Does robots.txt keep a page out of Google entirely?
Not reliably — if other sites link to a disallowed page, Google can still index the URL itself (without crawling its content). Use a noindex meta tag on the page itself for a real guarantee against indexing.
Can I have multiple Sitemap entries?
Yes — a robots.txt file can list multiple sitemap URLs if your site's sitemap is split across several files.
Related tools
Page last reviewed: July 2026