Robots.txt Tester
Paste a robots.txt file, an address and a crawler’s name to see whether it’s allowed, which rule decides and why. The site itself is never visited.
Paste the file’s text, up to 5 MB. This page never fetches it from a site: open the file in your browser and copy it.
Its product token, such as Googlebot. Leave it empty for a crawler without a group of its own.
Paste a robots.txt file above.
| Line | Group | Rule | Length | Result |
|---|
Everything happens on this page: no site is visited, nothing you paste is sent anywhere, and nothing is saved.
See what a robots.txt file really allows
A robots.txt file is short, but small details change what it does: which group a crawler follows, which of two matching rules wins, what a * or a $ covers. This page reads the file the way RFC 9309, the Robots Exclusion Protocol, says a crawler must, and shows its working, so you can find out why a page is blocked before a search engine does.
How to use it
- Paste the robots.txt text, or press Fill in an example to try it.
- Enter the address to check, in full or as a path such as /shop/cart.
- Enter the crawler’s name, such as Googlebot, or leave it empty for any crawler without a group of its own.
- Read the verdict, the rule that decided it and the group it came from. The table below lists every rule and whether it matched.
The rules it follows
- Groups: a crawler follows the groups that name it, combined; if none does, the * group; if there’s neither, nothing is blocked.
- Longest match: of the rules that match, the longest path decides, and Allow wins a tie.
- Wildcards: * matches any run of characters, and $ at the end means the address must end there.
- Encoding: accented letters and escapes such as %62 are compared in one standard form on both sides, so /café and /caf%C3%A9 match.
- Always allowed: /robots.txt itself, whatever the file says.
Reading the table
Each row is one rule from the file, with its line number, the crawlers its group names and its length as compared. “Matches, decides” marks the rule that settled the verdict. “Another crawler’s group” means the rule doesn’t apply to this crawler at all, however well its path fits the address.
Good to know
- It doesn’t visit anything. No address you type is fetched or checked online, so it can’t tell you whether a page exists, only what the file says about it.
- One file covers one host. The file on www.example.com says nothing about shop.example.com.
- Skipped lines are listed, such as misspelled keys, Crawl-delay and rules before any User-agent line, so you can fix them.
- Writing a new file instead of checking one? The Robots.txt Generator builds one from a form. This page reads a file you already have.
- Unsure how an address is encoded? The URL Encoder shows its percent-escapes.