Free AI Crawler Rules Builder & Auditor
Control which AI bots can crawl your site with intent-based presets, then audit any live robots.txt to see exactly which AI crawlers it allows or blocks - by search, training, and user intent.
1. Pick your policy
Cite me, don't train on me
Allow AI search & user bots (so you stay citable in AI answers) but block training crawlers. The recommended default for most businesses.Open to all AI
Allow every AI crawler, including training. Maximizes visibility; your content may be used to train models.Block all AI
Disallow every AI crawler. Protects content fully but removes you from AI answers and citations.Custom
Choose allow/block for each crawler individually.2. Fine-tune per crawler
3. Your robots.txt rules
# AI crawler rules - generated by Semark GEO tools # Blocked: these crawlers may not access the site User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: Meta-ExternalAgent User-agent: cohere-ai Disallow: / # Allowed: these crawlers may access the site User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Amazonbot User-agent: DuckAssistBot Disallow:
Why "block AI" is the wrong question
Most robots.txt tools treat AI as a single switch, but AI crawlers come in three intents. Search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index your pages so AI engines can cite you in answers. Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content to train models. User bots (ChatGPT-User, Claude-User, Perplexity-User) fetch a page only when a person shares its URL in a chat. Blocking training bots protects your content from being used in model training; blocking search bots removes you from AI answers entirely. The common, costly mistake is a single blanket rule that blocks the search bots by accident - so this tool separates the three and lets you decide each on purpose.
Build with intent-based presets
The builder starts from a policy, not a checklist. "Cite me, don't train on me" - the recommended default - allows AI search and user bots so you stay citable, while blocking training crawlers. "Open to all AI" maximizes visibility. "Block all AI" locks everything out. Pick a preset, fine-tune any individual crawler, and the tool generates copy-paste robots.txt rules with clear allow/block groups and an optional sitemap line. Everything runs in your browser, and the output is standard robots.txt any crawler understands.
Audit any live site
Switch to audit mode, enter a domain, and the tool fetches the live robots.txt and shows a matrix of every AI crawler with a plain Allowed or Blocked verdict - plus a headline warning if you are blocking the search bots that make you citable. It is the fastest way to sanity-check your own site or see how competitors have set their AI policy. Note that Google-Extended controls AI training use only and does not affect your Google Search ranking, and that honoring robots.txt is voluntary - reputable crawlers comply, but a rule is a request, not a firewall.
Frequently Asked Questions
Both are operated by OpenAI but do different jobs. GPTBot collects pages to train OpenAI models - block it if you do not want your content used for training. OAI-SearchBot indexes pages so ChatGPT search can find and cite you - keep it allowed if you want to appear in ChatGPT answers. Blocking GPTBot does not affect your ChatGPT-search visibility, and vice versa.
No. Google-Extended only controls whether your content can be used to train Google's AI models (Gemini, Vertex AI). It has no effect on Googlebot, Google Search indexing, or your rankings. You can block Google-Extended and remain fully indexed in regular Google Search.
robots.txt is a voluntary standard. Reputable operators - OpenAI, Anthropic, Google, Perplexity - publicly commit to honoring it, and they do. But it is a request, not enforcement: a bad actor can ignore it. For content you must protect absolutely, use authentication or server-side blocking; for declaring intent to legitimate AI companies, robots.txt is the right and expected tool.
For most businesses that want to be found in AI answers, the "cite me, don't train on me" policy is the sweet spot: allow AI search and user bots so ChatGPT, Perplexity, and others can cite you, while blocking pure training crawlers. This keeps you visible in the fast-growing AI-answer channel without handing your content over for model training.
In the robots.txt file at the root of your domain - yourdomain.com/robots.txt. If you do not have one yet, create it with the generated rules. If you already have a robots.txt, merge the new User-agent groups into it rather than replacing the file, so your existing rules (like Disallow for admin paths) are preserved.
Not Sure What to Allow?
Getting AI-crawler policy right is the difference between being cited in AI answers and being invisible. We set the strategy and implement it across your whole site.