What this meta robots tag generator produces
This meta robots tag generator turns a set of choices about indexing into two things at once: the
<meta name="robots"> element for the head of an HTML page, and the equivalent X-Robots-Tag
response header written out for Apache and for Nginx. The directives are identical in both. What
differs is where they can live, and that difference is the whole reason the header exists.
Tick the boxes, name the crawlers you want to address, and the output updates as you go. Anything
contradictory is called out underneath — a snippet limit sitting next to nosnippet, an expiry date
on a page already excluded, a token written with a capital letter that no crawler will match.
Crawling and indexing are different switches
The single most expensive misunderstanding in on-page SEO is treating robots.txt and this tag as
two ways of saying the same thing. They are not. A Disallow line stops the fetch. This tag is read
after the fetch, from inside the response.
Follow that through and the consequence is uncomfortable: a URL blocked in robots.txt can still appear in results, listed by its address alone with no title of yours, because Google learned about it from a link somewhere and was never allowed to read the page that says keep it out. So if the goal is genuine removal, the crawl has to be permitted first. Build the crawl rules with the robots.txt generator, then let the tag decide what happens to the page once it has been read. The tool has a checkbox for the blocked case for exactly this reason.
The default is index, follow — and it is implicit
No tag at all means index and follow. Both are assumed. Writing them out is not wrong, but it is also not doing anything, which is why the tool marks the empty state with a note rather than pretending it produced something useful. The tag earns its place the moment you deviate: a thin tag archive, a search results page, an internal staging copy, a thank-you page that has no business being found by strangers.
The preview directives nobody configures
max-snippet takes a character count, with -1 for no ceiling and 0 for no snippet at all.
max-image-preview takes none, standard or large. max-video-preview counts seconds. These
arrived together in 2019 as the response to European press publishers' rights, and most sites have
never touched them — which is itself a decision, because the default in the EU is that Google shows
what it likes within its own limits.
Two of them are worth deliberate thought. max-image-preview:large is what gets a picture shown at
full width in Discover, and a publisher who leaves it unset is turning down that surface. A tight
max-snippet on a paywalled article is how you stop the free preview from giving away the piece,
though set it too tight and the result becomes unclickable.
Bot-specific tags, and one trap in them
Any crawler token can replace robots in the tag name: googlebot, bingbot, googlebot-news.
The rule is that a crawler which finds a line addressed to it by name ignores the generic one
entirely — it does not merge the two. So a page carrying <meta name="robots" content="noindex">
alongside <meta name="googlebot" content="max-snippet:50"> is telling Google to index it with a
short snippet, and everyone else to stay away. That is almost never what the author meant.
When the page is finished, run it through the meta tag analyser to see the
head as a crawler would, and make sure the canonical on the page agrees
with it: a noindex page that other pages canonicalise to is a contradiction that resolves badly.