Robots.txt Generator
Four lines of plain text that can quietly remove your entire site from search results if you get one character wrong. This builds a valid file before you upload it.
- Presets for the common cases
- Sitemap and host lines
- Copy or download
- Free, no sign-up
Four lines that can remove your site from search
Disallow: / is a single stroke away from Disallow: — and the difference between those two is your entire website being crawled, or none of it.
This builds a valid file from presets and inputs, so the syntax is right before it goes anywhere near your server.
It is a request, not a restriction
The single most important thing to understand about this file, and the misunderstanding does real damage.
- Malicious bots ignore it entirely. Compliance is voluntary.
- Listing a sensitive path is effectively publishing a list of directories you would rather people did not visit — the file is public by design.
- Use authentication, or noindex where appropriate. This manages crawl behaviour for cooperative crawlers; it is not a security control.
Blocking crawling is not blocking indexing
These are different things, and conflating them produces the opposite of what people intend.
- Disallowing a URL stops compliant crawlers fetching it. It does not keep it out of results — a page can appear based on links pointing to it, even if never crawled.
- If you need a page out of the index, noindex is the correct tool.
- And here is the trap: noindex only works if the crawler can reach the page to read it. Blocking that page in robots.txt prevents the noindex from ever being seen.
User-agent blocks do not inherit
The part people get wrong most often, and the failure is invisible — nothing errors, the rule simply never applies.
- Each block applies only to the crawler it names. Rules do not carry over between blocks.
- Writing a general rule and assuming a specific crawler will follow it is a common and silent mistake.
- Robots.txt is now also how you express whether AI systems may crawl your content for training or retrieval — which means these blocks matter more than they used to.
Where the file goes
Upload it to your domain root, reachable at https://yoursite.com/robots.txt.
- Not a subfolder. Crawlers only look in that exact location.
- A robots.txt anywhere else is simply a text file nobody reads.
- Two optional lines finish it: your sitemap URL, which is the standard way to point crawlers at it, and host.
- 1Build it from presetsSyntax correct before it leaves here.
- 2Upload to the domain rootNot a subfolder.
- 3Fetch it yourselfOpen /robots.txt and read what you shipped.
Four numbers worth keeping
Why Parix.ai gives this away
A four-line file that can deindex a whole site deserves to be generated rather than typed. These tools are the same checks we run for clients before launch — which is why there is no paywall and no sign-up.
Build my robots.txtWho uses it
Developers setting up a new site, and anyone who has ever shipped a staging robots.txt to production — which is how sites disappear from search for a fortnight before somebody notices.
Pair it with
The XML Sitemap Generator builds the file this one points at, and the llms.txt Generator covers the newer question of what AI assistants should read.
Check it after you upload
Open https://yoursite.com/robots.txt in a browser and read what is actually there. It takes five seconds and catches the staging file before Google does.