Robots.txt generator for Next.js and AI bots. In your browser.

Free, no sign-up. Computation, copy and downloads stay in your browser. There is no upload or processing endpoint. Usage analytics include only a fixed tool name and action, never your input or result.

No. It is a voluntary crawl policy, not authentication or noindex. Protect private routes with authorization. A blocked URL can still be known to a search engine. The tester never contacts your site or a bot.

Google-style most-specific product-token groups are used; equally specific groups are merged. Bot-specific groups replace wildcard rules. The longest matching path wins; allow wins ties. Paths are case-sensitive with * and terminal $ support. robots.txt itself is implicitly allowed.

They are content-use tokens, not separate HTTP crawlers. Google-Extended covers Gemini training and grounding; Applebot-Extended covers foundation-model training. Blocking them alone does not block ordinary search crawlers.

No. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity-User generally ignores it; Amzn-User may not follow every directive. A local rule result is policy evaluation, not a compliance guarantee. Read each provider source below.

Allow, Disallow, User-agent and Sitemap are parsed. Unsupported fields like Crawl-delay warn and are omitted from Next.js export. Fetch status, redirects, WAF and provider-specific fallback policies are not simulated. The selected documented roster is not exhaustive; sources were checked 2026-10-09 and should be reviewed monthly.