Robots.txt is the smallest file on your website and the one most capable of destroying your traffic.
Two lines in the wrong place can remove an entire site from Google. It happens most often at launch, when a staging-site block gets pushed live with the code.
The file itself is simple. What causes damage is a misunderstanding about what it does. It sits in the technical layer covered by our SEO guide, alongside technical SEO.
The distinction that matters most
This trips up experienced people. "Block it in robots.txt" is the reflex answer to "keep this out of Google", and it is the wrong one.
Worse, the two directives conflict: if you block a page in robots.txt, Google cannot crawl it, so it never sees the noindex tag on it. Using both at once means the noindex is invisible and the page can stay indexed indefinitely.
Where the file lives
- Always at the root: example.com/robots.txt. Nowhere else works.
- One file per host and protocol. The https and http versions, and each subdomain, are separate files.
- Plain text, UTF-8. Not HTML.
- If the file does not exist, everything is crawlable. That is the correct default for most sites.
The syntax
| Directive | What it does | Note |
|---|---|---|
| User-agent | Names the crawler the following rules apply to | An asterisk means all crawlers. Rules are not combined across groups — the most specific matching group wins. |
| Disallow | Blocks a path prefix | Matches the start of the path. Disallow slash blocks the entire site. |
| Allow | Carves an exception out of a Disallow | Supported by Google. The more specific rule wins, not the first one. |
| Sitemap | Points to your XML sitemap | Absolute URL. Independent of user-agent groups. |
Two wildcards are supported by Google: an asterisk matches any sequence of characters, and a dollar sign anchors the end of a URL. Both are useful for parameter and file-type patterns.
Comments start with a hash. Blank lines separate groups. Paths are case-sensitive.
The mistakes that cause real damage
- Shipping the staging block to production. Disallow slash under a wildcard user-agent, pushed live at launch. This is the single most common catastrophic SEO error, and it can go unnoticed for weeks.
- Blocking CSS and JavaScript. Google renders pages. Block the assets and it sees a broken page, which affects how the page is assessed.
- Using robots.txt to hide private content. The file is public and readable by anyone. Listing your admin paths in it is a directory of exactly where to look. Use authentication.
- Combining Disallow with noindex on the same URL. The block prevents Google from ever seeing the noindex.
- Blocking a page you have also canonicalised. Google cannot read a canonical tag it is not allowed to fetch.
- Assuming it is enforcement. Well-behaved crawlers obey it. Scrapers and bad actors ignore it entirely.
What a sensible file looks like
For most sites: almost empty.
Block your internal search results, your cart and checkout paths, and any parameter patterns that generate duplicates. Point to the sitemap. Stop there.
Long, elaborate robots.txt files are usually a symptom of a site architecture problem being papered over. If you are blocking hundreds of patterns, the URLs themselves are the thing to fix — see crawl budget.
How to test it
- Search Console's robots.txt report shows the file Google actually fetched and flags syntax errors. Start here, always.
- URL Inspection tells you whether a specific URL is blocked, which is the question you usually actually have.
- Fetch it yourself in a browser after every deployment. It takes five seconds and catches the launch-day disaster.
Where to go from here
Open your robots.txt now and read every line. If you cannot explain why a rule exists, it probably should not.
Then check the Search Console report for parse errors, and confirm that nothing important is blocked with URL Inspection.
Frequently asked questions
What is robots.txt used for?
Robots.txt is a plain text file at the root of a domain that tells search engine crawlers which URL paths they may request. It is used to keep crawlers away from low-value areas like internal search results, cart pages and parameter-generated duplicates, and to point them at your XML sitemap.
Does robots.txt stop a page appearing in Google?
No. It stops crawling, not indexing. If other sites link to a blocked URL, Google can still index it and show it in results, usually without a description. To keep a page out of the index, allow it to be crawled and serve a noindex tag instead.
Can I use robots.txt and noindex together?
You should not. If robots.txt blocks the URL, Google cannot fetch the page, so it never sees the noindex tag. The two directives cancel each other out and the page can remain indexed indefinitely. Pick one: block it, or noindex it.
Where should the robots.txt file be placed?
At the root of the host, so example.com/robots.txt. No other location works. Each protocol and each subdomain needs its own file, so the https version, the http version and any subdomains are all separate.
Is robots.txt a security measure?
No, and treating it as one is dangerous. The file is publicly readable by anyone who types the URL, so listing admin or private paths in it advertises exactly where to look. Well-behaved crawlers obey it; scrapers and malicious bots ignore it. Use real authentication for anything sensitive.
We check robots.txt on day one of every audit
It takes five minutes and it is the fastest way to find out why a site vanished.