What each robots.txt line blocks, and what it does not
A robots.txt is a few lines, and most of them do something other than what they seem to. An empty Disallow opens the whole site, a group for Googlebot cancels everything written for every other crawler, and none of it keeps a page out of search.
Written · Updated
Disallow: with nothing after it allows everything; Disallow: / blocks everything for the crawlers its group names. A crawler obeys only the group that names it most specifically and ignores the group for *, so rules written for every crawler do not reach one that has a group of its own. Within a group the longest matching rule wins, and Allow wins a tie. And robots.txt stops crawling, not indexing: a blocked page can still appear in search by its address.
Line by line
| Line | What it does |
|---|---|
Disallow: | Nothing. An empty rule allows every path; it is the way to say a group may crawl the whole site. |
Disallow: / | Blocks every path for the group, usually a staging site’s file copied to production. |
Disallow: /private | Blocks every path that begins /private: /private/, but also /private-events and /privately. End it with a slash to mean the folder only. |
Disallow: /*.pdf$ | * stands for any run of characters and $ for the end of the address, so this blocks addresses ending in .pdf. /report.pdf?v=2 does not end in .pdf and is not blocked. |
Allow: /public/*.pdf | Opens the PDFs under /public/ again: it is longer than /*.pdf$, so it wins for those addresses. |
Noindex: /tag | Nothing. Google stopped reading Noindex in robots.txt in 2019, and it was never part of the standard. |
The matching rules, the wildcard and the end marker are those of RFC 9309, the standard Google, Bing and the other large crawlers follow.
The group that cancels the others
User-agent: *
Disallow: /private
User-agent: Googlebot
Disallow: /tmp
This reads as “keep everyone out of /private, and Googlebot out of /tmp as well”. It means something else. Googlebot has a group of its own, so it reads that group alone: it stays out of /tmp and may crawl /private. Every other crawler reads the group for *. A crawler with no exact group falls back to the one whose name begins its own, so Googlebot-Image reads the Googlebot group here too.
The fix is to repeat in each named group every line it should still obey. WELLKNOWN says when a file does this, naming the crawler and the paths it opens to it.
What robots.txt cannot do
- Keep a page out of search. A blocked page can still be listed by its address when other pages link to it, without a description. To keep it out, let it be crawled and give it
<meta name="robots" content="noindex">; a crawler that is not allowed to fetch the page never sees that tag. - Hide anything. The file is public, and a list of disallowed paths is a list of where to look. It is a request well-behaved crawlers honour, not access control.
- Stop a crawler that ignores it. That takes the server: a firewall rule, a login, or refusing the crawler’s address.
Testing an address against a robots.txt
- Paste the file into WELLKNOWN. It lists each group and its rules, an empty Disallow read as allowing everything, and the problems first: a
Disallow: /for every crawler, a rule before anyUser-agent, a named group that cancels the rules for*. - Put an address under it, a path or a full URL, and the crawler’s name. It says whether that crawler may fetch it, by which group and which rule, such as “Blocked: Googlebot may not fetch /private/plan.pdf, by the group for *, because
disallow: /privateis the longest rule that matches”. - Change the file until the answers are the ones you meant, then publish it.
Questions people ask about what robots.txt lines block
What does an empty Disallow mean in robots.txt?
That nothing is disallowed: Disallow: with no path allows every page for the crawlers its group names. Disallow: / is the line that blocks everything.
Does Googlebot follow the rules under User-agent: *?
Only if there is no group for Googlebot. A crawler obeys the group that names it most specifically and ignores the group for *, so a Googlebot group must repeat any rule from * that Googlebot should still follow.
Which rule wins when Allow and Disallow both match?
The longer one, measured by the length of its path, and Allow if they are the same length. That is RFC 9309’s rule, which Google and Bing follow.
Does robots.txt stop a page appearing in Google?
No. It stops crawling, and a blocked page can still be listed by its address when other pages link to it. Use a noindex meta tag on a page crawlers are allowed to fetch.
Does Disallow: /private also block /private-events?
Yes. A rule matches every address that begins with it. Write Disallow: /private/ to mean that folder alone.
Related tools
- Was this file changed?Hashes, signatures, a PDF's saved versions, a photo's edit history and the marks a cut leaves: what each shows about a file, and where each one stops.
- Is an email really from who it says?Ignore the From line: it is text the sender typed, and nothing in delivery checks it.
- How to check a file has not been changedTake the SHA-256 of the file you have and compare it, character for character, with the value the publisher gave you.
- Was this PDF changed after it was signed?A signature covers a stated range of bytes, not the whole file, so anything appended afterwards leaves it verifying perfectly.
- What a PDF keeps after you delete a pageA PDF can be saved by appending, so the deleted page, the removed comment and the pre-redaction text are all still in the file and the reader just hides them.
- How to tell if a photo has been editedEverybody says check whether Software says Photoshop, which is the weakest signal in the file.
What your files give away
A one-page checklist of what a photo, a document or a PDF says about you, free now. Then one letter a month, with one new tip.