Build a valid robots.txt to control how search engines and AI crawlers access your site — with one-click presets.
A robots.txt file is four lines of plain text that can quietly remove your entire site from search results if you get one character wrong. Disallow: / is a single stroke away from Disallow: — and the difference between the two is your whole website being crawled or none of it.
This generator builds a valid file from presets and inputs, so the syntax is right before you upload it.
Quick presets cover the standard scenarios in one click, and you can add another user-agent block for anything more specific.
That structure matters because user-agent blocks are the part people get wrong. Each block applies to the crawler it names, and rules do not carry over between blocks. Writing a general rule and assuming a specific crawler will follow it is a common and invisible mistake.
Two optional lines complete the file: your sitemap URL — worth including, since it is the standard way to point crawlers at your sitemap — and host.
Copy the result or download the file.
Upload to your domain root, reachable at https://yoursite.com/robots.txt. Not a subfolder — crawlers only look in that exact location, and a robots.txt anywhere else is simply a text file nobody reads.
This is the single most important thing to understand about the file, and the misunderstanding causes real damage.
Well-behaved crawlers obey robots.txt. Malicious bots ignore it entirely — and worse, your robots.txt is public, so listing a sensitive path is effectively publishing a list of directories you would rather people did not visit.
Never use robots.txt to hide sensitive pages. Use authentication, or noindex where appropriate. Robots.txt manages crawl behaviour for cooperative crawlers; it is not a security control.
The other frequent confusion. Disallowing a URL stops compliant crawlers from fetching it — it does not guarantee the URL stays out of search results. A page can appear in results based on links pointing to it, even when never crawled.
If you need a page kept out of the index, a noindex directive is the correct tool. And it only works if the crawler can reach the page to see it, which means blocking that page in robots.txt actually prevents the noindex from being read.
Robots.txt has become the main mechanism for expressing whether AI systems may crawl your content for training or retrieval.
This is a genuine business decision rather than a technical default. Blocking AI crawlers protects your content from being used without attribution. Allowing them makes your content available to be cited when someone asks an assistant a question your site answers.
Neither is obviously right. Decide deliberately rather than inheriting whatever your CMS generated.
A developer discovers the client's staging site is fully indexed — duplicate content competing with the live site, with test data visible in the results.
The immediate fix is a robots.txt on the staging domain disallowing everything, plus removing the already-indexed URLs through Search Console. For the live site, he generates a proper file: standard crawl rules, the sitemap line pointing at the real sitemap, and no accidental disallow on anything that matters.
Two files, five minutes, and a duplicate-content problem that had been running for months.
Developers launching sites. SEO specialists auditing crawl behaviour. Site owners deciding how AI crawlers should treat their content. Anyone who has ever been afraid to edit robots.txt by hand — reasonably. Pair it with the XML Sitemap Generator and the llms.txt Generator. Our web development team handles technical SEO properly.
One wrong character in robots.txt can deindex a site. Use the presets above, add your sitemap line, and upload a file you know is valid.
Everything you might be wondering about the Robots.txt Generator.
Hand-picked free tools that pair well with the Robots.txt Generator.
Practical reads on the topics this tool touches.
Join our newsletter to receive automation tips, product updates, and business insights.
