Robots.txt Generator

How to use this generator

1. Enter your website URL. This auto-fills the sitemap line — Google and Bing both use it to find every page worth crawling.
2. Tick the common blocks that apply to your site. WordPress admin and internal search are safe defaults for almost every site; cart/checkout only applies if you run an online store.
3. Add any custom paths. One per line, each starting with a slash.
4. Watch for the red warning. If your custom paths would block the whole site, the tool tells you before you ship it.
5. Copy the output and upload it to your site root as robots.txt — not inside a subfolder, and not through a CMS page editor unless your CMS explicitly supports it.

What robots.txt actually controls — and what it doesn't

This is the single most common misunderstanding about the file. Disallow stops crawling, not indexing. If a page is disallowed in robots.txt but another site links to it, Google can still show that URL in search results — usually with no title or description, because it was never allowed to read the page. To reliably keep a page out of search results, use a noindex meta tag or HTTP header on the page itself, not a robots.txt disallow.

Robots.txt also isn't a security tool. It's a publicly readable file — anyone, including someone with bad intent, can open yoursite.com/robots.txt and see exactly which folders you didn't want crawled. Never use it to "hide" anything sensitive; use proper authentication instead.

The mistake that de-indexes a site

Other frequent mistakes worth checking for:

• Blocking CSS or JS folders. Google renders pages like a browser before ranking them; if it can't fetch your stylesheets or scripts, it may judge the page as broken or poorly designed.
• Forgetting the sitemap line. It costs one line and materially helps crawlers find new and updated pages faster.
• Wrong case. Robots.txt paths are case-sensitive. /Private/ and /private/ are different paths to a crawler.
• One file per subdomain. A robots.txt at example.com/robots.txt does not cover blog.example.com — each subdomain needs its own.

Test before you rely on it

Once the file is live, check it in Google Search Console before assuming it's working as intended — crawlers can take time to pick up a changed robots.txt, and a syntax mistake that looks fine to the eye can behave differently to different crawlers. If you're not sure a path is blocked correctly, treat the file as a draft until you've confirmed it in Search Console.

The syntax rules and directive reference live in our full robots.txt guide — this page is for building the file quickly; that one is for understanding every directive in depth.