Robots.txt Generator

What this tool does

This robots.txt generator builds the file from rules you declare — which crawlers, which paths to block, which exceptions to allow, and where your sitemap lives — and flags the mistakes that usually go unnoticed until traffic drops. Presets for WordPress, Shopify and Next.js fill in the paths those platforms typically want excluded.

The file is produced in your browser and never leaves it.

How to use it

1. Pick a mode. Allow everything is the right default for most live sites; block everything belongs on staging and nowhere else.
2. List the paths to disallow, one per line. A path without a leading slash is normalised for you, because without it many crawlers skip the rule silently.
3. Add exceptions under Allow — they take precedence over a broader Disallow.
4. Point to your sitemap with an absolute URL.
5. Copy the result to the root of your domain as robots.txt.

Recipes

Each row is what the generator emits for that intent — or, where the file has more around it, the part that changes. The order is the generator's own: user-agent lines first, then Allow, then Disallow, then Crawl-delay. Recipes that need more than one group are written out in full in the next section.

An empty Disallow: is not a mistake. It is how the format says "nothing is blocked for this group", and it is why the allow-everything file has a directive at all.

Recipes that need more than one group

The generator's form writes one group at a time, so for these you generate each group and stack them, separated by a blank line. The output is the same either way — a group ends where the next User-agent line begins.

Blocking every AI training crawler while leaving search engines alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Blocking training crawls but staying eligible for ChatGPT's search citations:

User-agent: GPTBot
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow:

Different rules per crawler, with a catch-all at the end:

User-agent: Googlebot
Disallow: /drafts/

User-agent: Bingbot
Disallow: /drafts/
Disallow: /beta/

User-agent: *
Disallow: /

A crawler reads one group — the one whose user-agent matches it most specifically — and ignores every other. That is the trap in the last example: once Googlebot has its own group, the * group no longer applies to it at all, so anything you wanted blocked for everyone has to be repeated inside the Googlebot group.

Longest match wins

Disallow: / followed by Allow: /public/ looks like a contradiction, and the reason it works is the rule Google actually applies: for any given URL, the directive with the longest matching path decides. For /public/report.html, Allow: /public/ is eight characters of match against one, so the Allow wins. For /private/report.html nothing but Disallow: / matches, and the URL stays blocked.

When two rules match with exactly the same length, Allow wins. That tie-break is what makes Disallow: /folder/ plus Allow: /folder/ a way to say "allowed", though writing it that way helps nobody.

Line order is irrelevant to Google, and the generator puts Allow first only because it reads better above the rule it carves an exception out of. It is not irrelevant everywhere: older crawlers, and a long tail of smaller bots still running on the original 1994 convention, take the first rule that matches and stop. Putting the specific Allow above the broad Disallow costs you nothing with Google and behaves correctly with them.

Wildcards, and what they are not

Two special characters are supported by Google, Bing and most modern crawlers:

• * matches any run of characters, including none.
• $ anchors the rule to the end of the URL.

That is the entire syntax. It is not a regular expression: there are no character classes, no alternation, no optional groups, and no anchor for the start of the path other than the leading slash every rule already has.

The mistake this causes is Disallow: /*.php, written by someone who means "PHP files". Without the $, the rule matches any URL containing .php anywhere after the slash — so it also blocks /index.php?id=2, /index.php/comments, and /blog/2026/php-tips. If you mean files that end in .php, the rule is Disallow: /*.php$, and if you also have query strings on those files you need both, because $ and ? cannot be true at once.

The same logic applies to /*?, which blocks every URL containing a question mark — the usual way to keep crawlers out of faceted navigation. Check it against a real URL before publishing, because a query string on a page you do want indexed is caught by it too.

Blocking is not hiding

The most expensive misunderstanding about this file is that blocking a URL removes it from search. It does not. Disallow prevents the crawler from fetching the page; it does nothing about the URL being indexed from external links. The result is the worst of both worlds — the page appears in results with no title or description, and the crawler cannot read the noindex you added, because you told it not to look.

If a page must not appear in search, let the crawler fetch it and return a noindex directive. Use robots.txt for the opposite problem: preventing crawlers from spending your crawl budget on faceted-search URLs, endless calendars and internal search results.

AI crawlers, honestly

The AI bots are the most common reason people open this file in 2026, and the honest framing is that robots.txt is a request. The major operators publish their user-agent strings and say they honour it; smaller ones are inconsistent, and a crawler that ignores the file leaves no trace in it either way. Blocking is a stated preference, not an access control.

The names worth knowing, and what blocking each one actually costs you:

• GPTBot — OpenAI's training crawler. Blocking it removes your content from training data.
• ChatGPT-User — fetches a page because a user asked ChatGPT to look at it. Blocking it means a person who pastes your URL gets nothing back.
• OAI-SearchBot — powers ChatGPT's search results and citations. Blocking this one removes you from a referral channel that is growing, which is a different decision from the other two.
• ClaudeBot and anthropic-ai — Anthropic's crawlers; the second is the older name and still worth listing.
• CCBot — Common Crawl. Not an AI company, but its archive feeds many models, so blocking it has knock-on effects you cannot see.
• Google-Extended — a token, not a crawler. It controls whether content already fetched by Googlebot is used for Gemini and grounding. Blocking it has no effect on Google Search rankings or indexing, which makes it the least costly of these to set.
• Applebot-Extended — the same idea for Apple Intelligence, separate from Applebot, which serves Siri and Spotlight.
• PerplexityBot — cites sources and sends clicks, so blocking it is a trade-off rather than a win.
• Bytespider — ByteDance. Frequently reported as aggressive; the most common entry on a block list written for load rather than principle.

There is no single right answer here. A publisher selling subscriptions and a business hoping to be recommended by an assistant want opposite files. What is always wrong is copying someone else's AI block list without checking which of those crawlers sends you traffic.

What belongs in the file, and what does not

Block what wastes crawling: parameterised URLs that produce near-identical pages, internal search results, cart and checkout paths. Leave alone what search engines need to render the page — blocking CSS or JavaScript means Google evaluates a broken version of your layout, and mobile usability suffers for it.

Never treat the file as a security boundary. It is published at a predictable address and read by anyone curious about your site, so a Disallow: /internal-reports/ is an invitation, not a lock.

After you publish it

Check the live file at /robots.txt and confirm it is served as plain text — a CMS that returns HTML for that path has effectively no robots file. Then test the rules against real URLs in Search Console before assuming they behave the way they read.

One file per host, and the subdomain trap

Rules apply to the host that serves them. example.com/robots.txt says nothing about blog.example.com, shop.example.com or the staging subdomain someone spun up last quarter — each needs its own file, and a subdomain without one is crawled freely.

This is where sites accidentally expose a staging environment to search. The production file is carefully written; the copy running on staging.example.com inherits nothing, gets crawled, and starts competing with the real site for its own content. If you run staging on a public subdomain, give it a robots.txt that blocks everything and an authentication layer in front, because the file alone is a request, not an enforcement.

Reading someone else's file

The file is public by design, which makes it a fast way to understand how a competitor thinks about their own site. The paths they block tell you which sections they consider low value, the sitemaps they declare tell you how their content is organised, and a Disallow on faceted parameters tells you they have a crawl budget problem worth learning from.