Robots.txt Validator

What this tool does

This robots.txt validator reads the file you paste line by line, the way a crawler does, and reports every line that a crawler would skip or misread: lines with no colon, misspelled directives, rules that sit above any User-agent, groups that end up with no rules, sitemaps written as relative paths, and directives that Google stopped honouring years ago.

It is also a robots.txt tester. Type a user-agent and a URL or path, and it tells you whether that crawler may fetch it, which line decided, and which group was used. That answers the question behind most searches for "is my URL blocked by robots.txt" without waiting for a search console report to refresh.

How to use it

1. Open https://your-site.com/robots.txt, select everything and paste it into the box. A draft you have not deployed yet works just as well.
2. Read the findings, which are sorted by line. Errors are lines no crawler will obey; warnings are lines that probably do not do what you think; notes are there to confirm intent.
3. Enter a crawler name such as Googlebot, Bingbot or GPTBot, then a full URL or a path. The verdict updates as you type.

How crawlers actually read the file

RFC 9309, published in 2022, finally turned the 1994 convention into a standard, and it settles most of the arguments this robots.txt syntax checker is built around.

Groups. A group starts with one or more consecutive User-agent lines and continues with the Allow and Disallow lines under it. Two User-agent lines in a row share the rules that follow. A Disallow before any User-agent belongs to nobody. Blank lines do not end a group, which is why a stray User-agent line with nothing under it can quietly swallow the rules you meant for someone else.

Choosing a group. A crawler looks for the group that names it, case-insensitively. Google adds one step: a crawler such as Googlebot-Image with no group of its own falls back to the Googlebot group before it falls back to *. Only one group applies. If several groups name the same crawler, they are merged into one.

Choosing a rule. Within that group, the rule whose path matches and is longest wins. A tie between Allow and Disallow of the same length goes to Allow. Paths are prefixes: /admin also matches /admin-tools and /administrator, so add the trailing slash when you mean the directory. The only wildcards are *, which matches any run of characters, and $, which anchors the end of the URL and only means that when it is the last character.

What is always allowed. /robots.txt itself can always be fetched, and a missing or empty file allows everything. Google reads at most the first 500 KiB, and the validator warns if you paste more than that.

Mistakes worth checking for

The costly ones are rarely syntax errors. They are a Disallow: / left over from staging, a Googlebot group added for one rule that silently exempts Google from everything under *, or a CSS and JavaScript directory blocked so Google renders a broken page. Test a few of your real templates, not only the paths you meant to block.

If you are writing the file from scratch, the robots.txt generator builds one from presets. To keep a crawlable page out of the index, use the meta robots tag generator instead of a Disallow, and list the pages you do want found with the sitemap generator.