UpstreamKit

Generators

Robots.txt Generator

Build a robots.txt with explicit rules for search crawlers and AI agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

Allow rules override a broader Disallow for the same agent.

Google ignores this. Bing and others honour it.

Block search crawlers

Block AI crawlers

robots.txt

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /*?sort=

Training crawlers vs retrieval agents

These are two different decisions and they are constantly confused.

  • Training crawlers(GPTBot, Google-Extended, CCBot, Applebot-Extended) collect data to train models. Blocking them has no effect on whether you appear in today's answers.
  • Retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) fetch pages to answer a live question. Blocking these removes you from AI answers.

Blocking the second group to opt out of the first is the single most expensive mistake we see in AI visibility audits.

Before you ship it

  • robots.txt controls crawling, not indexing. To keep a page out of results, use noindex — a blocked page can still be listed without a snippet.
  • Save it at the domain root and test with the Robots.txt Checker.