Settings

Disallow: with nothing after it allows everything; Disallow: / blocks everything for the crawlers its group names. A crawler obeys only the group that names it most specifically and ignores the group for *, so rules written for every crawler do not reach one that has a group of its own. Within a group the longest matching rule wins, and Allow wins a tie. And robots.txt stops crawling, not indexing: a blocked page can still appear in search by its address.

Line by line

LineWhat it does
Disallow:Nothing. An empty rule allows every path; it is the way to say a group may crawl the whole site.
Disallow: /Blocks every path for the group, usually a staging site’s file copied to production.
Disallow: /privateBlocks every path that begins /private: /private/, but also /private-events and /privately. End it with a slash to mean the folder only.
Disallow: /*.pdf$* stands for any run of characters and $ for the end of the address, so this blocks addresses ending in .pdf. /report.pdf?v=2 does not end in .pdf and is not blocked.
Allow: /public/*.pdfOpens the PDFs under /public/ again: it is longer than /*.pdf$, so it wins for those addresses.
Noindex: /tagNothing. Google stopped reading Noindex in robots.txt in 2019, and it was never part of the standard.

The matching rules, the wildcard and the end marker are those of RFC 9309, the standard Google, Bing and the other large crawlers follow.

The group that cancels the others

User-agent: *
Disallow: /private

User-agent: Googlebot
Disallow: /tmp

This reads as “keep everyone out of /private, and Googlebot out of /tmp as well”. It means something else. Googlebot has a group of its own, so it reads that group alone: it stays out of /tmp and may crawl /private. Every other crawler reads the group for *. A crawler with no exact group falls back to the one whose name begins its own, so Googlebot-Image reads the Googlebot group here too.

The fix is to repeat in each named group every line it should still obey. WELLKNOWN says when a file does this, naming the crawler and the paths it opens to it.

What robots.txt cannot do

  • Keep a page out of search. A blocked page can still be listed by its address when other pages link to it, without a description. To keep it out, let it be crawled and give it <meta name="robots" content="noindex">; a crawler that is not allowed to fetch the page never sees that tag.
  • Hide anything. The file is public, and a list of disallowed paths is a list of where to look. It is a request well-behaved crawlers honour, not access control.
  • Stop a crawler that ignores it. That takes the server: a firewall rule, a login, or refusing the crawler’s address.

Testing an address against a robots.txt

  1. Paste the file into WELLKNOWN. It lists each group and its rules, an empty Disallow read as allowing everything, and the problems first: a Disallow: / for every crawler, a rule before any User-agent, a named group that cancels the rules for *.
  2. Put an address under it, a path or a full URL, and the crawler’s name. It says whether that crawler may fetch it, by which group and which rule, such as “Blocked: Googlebot may not fetch /private/plan.pdf, by the group for *, because disallow: /private is the longest rule that matches”.
  3. Change the file until the answers are the ones you meant, then publish it.

Questions people ask about what robots.txt lines block

What does an empty Disallow mean in robots.txt?

That nothing is disallowed: Disallow: with no path allows every page for the crawlers its group names. Disallow: / is the line that blocks everything.

Does Googlebot follow the rules under User-agent: *?

Only if there is no group for Googlebot. A crawler obeys the group that names it most specifically and ignores the group for *, so a Googlebot group must repeat any rule from * that Googlebot should still follow.

Which rule wins when Allow and Disallow both match?

The longer one, measured by the length of its path, and Allow if they are the same length. That is RFC 9309’s rule, which Google and Bing follow.

Does robots.txt stop a page appearing in Google?

No. It stops crawling, and a blocked page can still be listed by its address when other pages link to it. Use a noindex meta tag on a page crawlers are allowed to fetch.

Does Disallow: /private also block /private-events?

Yes. A rule matches every address that begins with it. Write Disallow: /private/ to mean that folder alone.

Related tools

More on this

All 12 guides under “Was this changed, and is it genuine?”.

What your files give away

A one-page checklist of what a photo, a document or a PDF says about you, free now. Then one letter a month, with one new tip.

No spam, unsubscribe at any time, nothing is shared.