Free AI Training Data Checker
Search the most recent Common Crawl indexes for pages from your domain to find out whether your content is in the open dataset that large language models learn from.
What is Common Crawl and why does it matter?
Common Crawl is a free, open archive of the web - billions of pages captured every month and published for anyone to use. It matters for AI because it is one of the largest public datasets that large language models train on: GPT, Llama, and many open models have drawn on Common Crawl data. So checking whether your pages appear in it is a concrete, verifiable answer to a question every brand now asks - "is my content part of what AI has learned from?" This tool queries Common Crawl's public index directly from your browser and reports how many of your pages it found across the most recent monthly crawls.
Being in the corpus cuts both ways
If your pages are in Common Crawl, that is a visibility signal - your content is in the raw material models are built from, which can help your brand and messaging surface in AI outputs. If you would rather not contribute to model training, you control future crawls through robots.txt: the crawler is called CCBot, and disallowing it stops Common Crawl from collecting new pages. One important caveat - robots.txt only affects future crawls. Pages already captured in past crawls remain in those historical datasets, which cannot be retroactively purged. This is why the decision to allow or block CCBot is worth making deliberately rather than by default.
How to read a "not found" result
A domain can be absent from recent Common Crawl indexes for several ordinary reasons: it is new or low-authority and has not been crawled yet, it is not well-linked from other sites (Common Crawl follows links), or it blocks CCBot in robots.txt. Common Crawl also does not capture every URL even on sites it does crawl, so a small sample here is normal. If being discoverable to AI matters to you, make sure CCBot is allowed, publish genuinely useful content, and earn links - the same fundamentals that help classic SEO also get you into the crawl.
Frequently Asked Questions
Not with certainty, but it is a strong indicator. Common Crawl is a major training source for many models, so pages in the corpus are in the pool of data models like GPT have drawn from. However, AI companies filter, deduplicate, and select from Common Crawl rather than using all of it, and they use other sources too. Presence means "in the candidate training data," not "definitely memorized."
Add a rule to your robots.txt blocking the CCBot user-agent: "User-agent: CCBot" followed by "Disallow: /". Common Crawl honors robots.txt, so this stops future crawls. Use our AI Crawler Rules builder to generate the exact lines. Remember that pages captured in past crawls stay in those historical datasets - robots.txt only prevents new collection.
Common Crawl publishes a new index roughly monthly and asks tools to query gently to keep the free service available for everyone. We search the most recent few monthly crawls, which is enough to answer "are you currently in the corpus" without hammering their API. Historical presence across older crawls follows the same pattern if recent ones show captures.
No. The lookup runs entirely in your browser and talks directly to Common Crawl's public index API. We do not proxy, log, or store the domain you check - there is no server of ours in the loop for this tool.
Not necessarily. Common Crawl samples the web rather than mirroring every page, so even large sites often appear with a partial set of URLs in any given crawl. A handful of captured pages confirms you are in the corpus. If you expected far more and see almost nothing, check that CCBot is not blocked and that your important pages are well-linked internally.
Want to Shape How AI Sees Your Brand?
Being in the training data is one signal. We build the full GEO picture - structured data, entity presence, and content AI engines actually cite.