Modern Niche World MNW

Robots.txt Generator

Choose which crawlers may visit which parts of your site, and get a robots.txt file to upload. Every name and path is checked before you copy it.

Start from

A starting point replaces the groups below. Sitemap lines stay as they are.

Names for language-model training (optional)

Each company documents its own name, so check its page before you rely on one: names change and new ones appear. Remember that robots.txt is a request, not a lock. Crawlers that follow it stay out; nothing forces the others to.

Full addresses. They’re written into the file as they are and never visited.

The file is built on this page from what you enter. No address is visited or checked online, and nothing is sent or saved.

Tell crawlers where they may go

A robots.txt file sits at the top of a website and asks crawlers, the programs search engines and others use to read pages, to stay out of certain addresses. Keeping them away from carts, internal search results or endless filtered lists saves your server work and keeps their attention on pages that matter. This page writes the file as RFC 9309, the Robots Exclusion Protocol, describes it.

How to use it

  1. Pick a starting point: allow everything, block everything, or block one folder.
  2. In each group, list the crawlers it’s for, one per line, then its rules: Disallow for paths to stay out of, Allow for exceptions.
  3. Add your sitemap’s full address if you have one.
  4. Fix anything listed in red, then copy the file or download it and upload it as robots.txt at the top of your site.

How crawlers read it

  • Groups: a crawler follows the group that names it. If none does, it follows the * group, and only that one.
  • Longest rule wins: when an Allow and a Disallow both match an address, the one with the longer path decides, and Allow wins a tie. So Disallow: /shop/ with Allow: /shop/help/ keeps the help pages open.
  • Paths start with / and are case-sensitive: /Photos/ and /photos/ are different.
  • Blank lines don’t end a group, which is why each group here needs a rule: on its own, a User-agent line joins the next group.

Language-model training

Some companies document a name you can block to keep your pages out of their language-model training. GPTBot, ClaudeBot and Meta-ExternalAgent are crawlers; Google-Extended and Applebot-Extended are only names, read by the company’s usual crawler; CCBot fills Common Crawl’s open archive, which Common Crawl says is widely used to train language models. Adding one makes an ordinary group that asks it to stay out of the whole site. Blocking one doesn’t block the company’s search crawler, which has its own name.

Good to know

  • It isn’t security. Anyone can read the file, and a crawler can ignore it.
  • Save it as plain text named robots.txt, all lowercase, in UTF-8, which is how most text editors save files; the download is already in that form.
  • Check it before you upload: the Robots.txt Tester shows whether a given address is allowed, and which rule decides.
  • No sitemap yet? The XML Sitemap Generator makes one from a list of addresses.
  • Spaces and accents in a path are written as percent-escapes; the URL Encoder converts them.

Frequently asked questions

Does a robots.txt file keep pages private?
No. It’s a request that well-behaved crawlers follow, not a lock, and anyone can read the file, so listing a folder in it tells people the folder exists. Protect private pages with a password or a login instead.
Where does the file go?
At the top of your site, so its address is https://www.example.com/robots.txt. A file in a folder is ignored, and each host needs its own: one on www.example.com doesn’t cover shop.example.com.
Will blocking a page take it out of search results?
Not always. A blocked page can still be listed if other sites link to it, just without a description, because the crawler never reads it. To keep a page out of results, let crawlers read it and give it a noindex tag.
What do * and $ do in a path?
A * stands for any run of characters, so /*.pdf matches every address with .pdf in it. A $ at the end means the address must end there: /*.pdf$ matches /guide.pdf but not /guide.pdf?page=2.

Related tools

Which rule wins

Robots.txt Tester

Paste a robots.txt file to see whether a crawler may fetch an address and which rule decides, matched as RFC 9309 says.

Open tool
Split at 50,000

XML Sitemap Generator

Turn a list of page addresses into an XML sitemap, with repeats and other hosts left out and an index past 50,000.

Open tool
%20 · +

URL Encoder and Decoder

Percent-encode or decode text for links and forms, and split a URL into its decoded parts and query parameters.

Open tool