# Robots.txt generator for Next.js and AI bots. In your browser.

Generate a crawl policy, export app/robots.ts and test URL rules for a selected user-agent. Every listed AI bot has a dated official source — free, no sign-up.

Page: https://saascode.ai/tools/robots-txt-generator

## How to use

1. Enter excluded paths and an optional sitemap. Select only bot tokens you intend to block.
2. Generate robots.txt or paste an existing file. Test a URL or path with a crawler product token.
3. Review the matched group and rule. Export robots.txt or app/robots.ts for your own site's origin, then verify actual crawl behavior.

## Scope

Local crawl-rule simulation with Google-style group precedence. No fetching, access-control or indexing guarantee. Unsupported directives warn and are omitted from app/robots.ts.

## References

- [Robots Exclusion Protocol — RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)
- [Google robots.txt matching](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [Next.js app/robots.ts](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

## AI bot tokens — verified 2026-10-09

Review monthly; next due 2026-11-09. Selected documented tokens, not an exhaustive roster. Each setting is independent; sources can change.

| Token | Purpose / limitation | Verified | Official source |
| --- | --- | --- | --- |
| GPTBot | Training crawler | 2026-10-09 | [Official source](https://developers.openai.com/api/docs/bots) |
| OAI-SearchBot | Search discovery | 2026-10-09 | [Official source](https://developers.openai.com/api/docs/bots) |
| ChatGPT-User | User-requested access; robots.txt may not apply | 2026-10-09 | [Official source](https://developers.openai.com/api/docs/bots) |
| OAI-AdsBot | Submitted ad landing pages; not foundation-model training | 2026-10-09 | [Official source](https://developers.openai.com/api/docs/bots) |
| ClaudeBot | Training crawler | 2026-10-09 | [Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-SearchBot | Search discovery | 2026-10-09 | [Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-User | User-requested access | 2026-10-09 | [Official source](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| PerplexityBot | Search discovery; not foundation-model training | 2026-10-09 | [Official source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Perplexity-User | User-requested access; generally ignores robots.txt | 2026-10-09 | [Official source](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
| Google-Extended | Gemini training and grounding control token; no separate HTTP crawler | 2026-10-09 | [Official source](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| Applebot-Extended | Foundation-model training control token; no separate HTTP crawler | 2026-10-09 | [Official source](https://support.apple.com/en-us/119829) |
| Amazonbot | Amazon services; content may train AI models | 2026-10-09 | [Official source](https://developer.amazon.com/amazonbot) |
| Amzn-SearchBot | Search including Alexa; not generative-model training | 2026-10-09 | [Official source](https://developer.amazon.com/amazonbot) |
| Amzn-User | User-requested access; may not follow all robots.txt rules | 2026-10-09 | [Official source](https://developer.amazon.com/amazonbot) |
| CCBot | Common Crawl public web corpus, also used in model training | 2026-10-09 | [Official source](https://blog.commoncrawl.org/ccbot) |

## Build your SaaS

[Explore SaaSCode Growth](https://saascode.ai/growth)

## Related tools

- [Schema markup generator & validator](https://saascode.ai/tools/schema-markup-generator)
- [XML sitemap & Next.js sitemap generator](https://saascode.ai/tools/sitemap-generator)
- [Meta tag & Next.js metadata generator](https://saascode.ai/tools/meta-tag-generator)
- [All free tools](https://saascode.ai/tools)

## FAQ

### Is this free, and does my input leave the browser?

Free, no sign-up. Computation, copy and downloads stay in your browser. There is no upload or processing endpoint. Usage analytics include only a fixed tool name and action, never your input or result.

### Does robots.txt secure private pages or guarantee no indexing?

No. It is a voluntary crawl policy, not authentication or noindex. Protect private routes with authorization. A blocked URL can still be known to a search engine. The tester never contacts your site or a bot.

### How are conflicting rules selected?

Google-style most-specific product-token groups are used; equally specific groups are merged. Bot-specific groups replace wildcard rules. The longest matching path wins; allow wins ties. Paths are case-sensitive with * and terminal $ support. robots.txt itself is implicitly allowed.

### What do Google-Extended and Applebot-Extended control?

They are content-use tokens, not separate HTTP crawlers. Google-Extended covers Gemini training and grounding; Applebot-Extended covers foundation-model training. Blocking them alone does not block ordinary search crawlers.

### Will blocking user-requested AI agents always work?

No. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity-User generally ignores it; Amzn-User may not follow every directive. A local rule result is policy evaluation, not a compliance guarantee. Read each provider source below.

### Which directives and bot policies are covered?

Allow, Disallow, User-agent and Sitemap are parsed. Unsupported fields like Crawl-delay warn and are omitted from Next.js export. Fetch status, redirects, WAF and provider-specific fallback policies are not simulated. The selected documented roster is not exhaustive; sources were checked 2026-10-09 and should be reviewed monthly.