How to Stop AI From Making Things Up About Your Own Documents
When Stanford researchers tested the leading AI legal research tools built specifically to answer questions from a trusted set of documents, those tools still invented facts, cited cases that did not exist, or misread the source material in 17% to 33% of queries (Stanford RegLab, Magesh et al., 2024). These were not generic chatbots. They were expensive, purpose-built systems from LexisNexis and Thomson Reuters, marketed as hallucination-free. If the specialists get it wrong a third of the time, the tool you bolted onto your own PDFs deserves a hard look.
Short answer: AI makes things up about your documents because a raw language model answers from its training and its own fluent guessing, not from your files. You fix this by "grounding" the model: retrieving the exact relevant passages from your documents and forcing the model to answer only from that text, with citations you can click and check. Grounding (usually called RAG) sharply cuts fabrication, but it never reaches zero, so the reliable setups also require citations, an honest "I don't know," and a human check on anything that matters.
Why AI invents facts about files you gave it
A large language model does not "read" your document the way you do and then look things up. On its own, it predicts the next most plausible word based on patterns it learned during training. When you paste a contract and ask "What is the termination notice period?", the model produces text that sounds like a correct answer. If your actual clause is in front of it, great. If it is not, the model will still produce a confident, well-formatted number, because producing fluent text is the only thing it was built to do. Douwe Kiela, the co-author of the original 2020 RAG paper, describes general-purpose models as trained to be versatile and creative, which is exactly the behavior we label "hallucination" when we need them to stick to a source.
This is why the problem feels so backwards to business owners. The information is in your file, yet the AI still gets it wrong. The model was never actually required to consult your file. It was asked to sound right, and it did.
Grounding (RAG): the fix that actually works
The technique that fixes this has an ugly name, retrieval-augmented generation, usually shortened to RAG. Strip away the jargon and it is a two-step process:
- Retrieve. When you ask a question, the system searches your documents and pulls out the handful of passages most relevant to the question. Your files are broken into small chunks and indexed in advance so this search is fast.
- Generate. The system hands those specific passages to the language model along with an instruction like "Answer only using the text below, and quote your source." The model now writes its answer from your real content instead of from memory.
For small and mid-sized businesses this matters because the "read our documents and answer" use case is everywhere: a chatbot that answers customer questions from your knowledge base, an assistant that pulls figures from your invoices, a tool that summarizes intake forms. Every one of those is a grounding problem. If you want the fuller picture of where these assistants fit, see our guides on AI agents for small businesses and the best AI tools for small business in 2026.
Grounding cuts hallucinations sharply, but never to zero
Here is the honest part most vendors skip. Grounding dramatically reduces fabrication; it does not end it. On Vectara's Hallucination Leaderboard, which measures how often models add unsupported facts when summarizing a document they were given, the best-performing models score in the low single digits, roughly 1% to 3% as of the May 2026 update (Vectara HHEM Leaderboard). That is excellent, and far better than an ungrounded chatbot. But it is measured on a clean, narrow task: summarize this one document faithfully.
Real business questions are messier, and accuracy drops accordingly. That is exactly what the Stanford study exposed. Even with retrieval built in, the leading legal tools "each hallucinate between 17% and 33% of the time," and the researchers concluded plainly that "the providers' claims are overstated" (Stanford RegLab, 2024). A grounded model can still misread an ambiguous passage, retrieve the wrong chunk, or blend two documents into a claim neither one makes. Grounding shrinks the surface area for fabrication; it does not install a truth detector.
The practical takeaway is not "give up." It is "design for the error rate you will actually have." A tool that is right 83% to 98% of the time is genuinely useful for drafting, triage, and search, and genuinely dangerous if you let it send answers to customers or into legal filings with no check.
Eight steps to keep AI honest about your documents
You do not need a data science team to get most of the benefit. In priority order:
- Ground it in the first place. If your AI setup just pastes a question to ChatGPT with no retrieval from your files, that is the root cause. Use a tool or build a flow that retrieves from your documents before answering. This single change removes the largest category of made-up answers.
- Demand citations, and make them clickable. Configure the system to quote the exact source passage and link to it. Citations do two things: they let a human verify in seconds, and they expose the failure mode where a model writes a plausible answer with a citation that does not actually support it. If you cannot click through to the source, treat the answer as unverified.
- Let it say "I don't know." The most valuable instruction you can give a document assistant is permission to refuse. Tell it, in the system prompt, to answer only from the provided text and to reply "I don't know" when the answer is not there. Kiela's own framing of grounded models is that they should stay strictly inside the provided context and admit when information is missing. A tool that abstains beats one that guesses.
- Clean and structure your source documents. Garbage in, confident garbage out. Scanned PDFs with bad text extraction, outdated versions, and duplicate files are a top cause of wrong retrieval. Keep one authoritative, current copy of each document and make sure the text is actually readable, not a photo of text.
- Get chunking right. Documents are split into pieces before indexing. If the pieces are too large, retrieval is imprecise; too small, and it loses context. If your assistant keeps missing answers that you know are in the file, bad chunking is a likely culprit and is worth having your provider tune.
- Keep the knowledge base current. A grounded answer is only as good as the document behind it. If you updated your refund policy but the old PDF is still indexed, the AI will confidently cite the outdated rule. Assign someone to remove and re-index stale files.
- Keep a human in the loop for anything consequential. For internal search and drafting, let the AI run. For anything that goes to a customer, a regulator, a patient, or a court, a person reviews before it ships. The Stanford numbers are the argument for this rule, not against automation.
- Test it before you trust it. Write 20 to 50 real questions with known correct answers and run them through the system. Count how often it is right, cites a real source, and admits when it should. Rerun this whenever you change documents or vendors. This is the only way you will know your actual error rate instead of the vendor's brochure rate.
A concrete example
Say you run a home services company and you want an assistant that answers "Is this repair covered under our workmanship warranty?" from your warranty PDF.
The ungrounded version reads the question, recognizes "workmanship warranty," and produces a confident, generic answer about what such warranties usually cover, which may or may not match your actual terms. The grounded version searches your specific warranty document, retrieves the paragraph defining covered workmanship and the exclusions, and answers "Yes, covered for 2 years on labor per Section 4," with a link to Section 4. Your dispatcher clicks the link, sees the clause, and relays it with confidence. When a customer asks about something genuinely not in the document, the good setup says "That is not addressed in the warranty," instead of inventing a policy you will have to honor later. If you are weighing whether to automate a task like this at all, our piece on AI versus hiring an employee walks through where the line usually falls.
What This Means for Your Business
The uncomfortable truth is that most AI adoption is still early and shallow; under 20% of the smallest US firms report using AI at all (US Census Bureau BTOS, 2026), which means many of the "AI reads our documents" tools now hitting the market are unproven in real use. When you deploy one, you are not choosing between hallucinations and no hallucinations. You are choosing how low you can push the error rate and how you catch what slips through.
Do three things. First, insist that any document AI you buy or build is grounded and shows clickable citations; if it cannot, it is guessing. Second, run your own 20-to-50 question test and demand to see the real accuracy before you rely on it. Third, keep a human check on any output that touches a customer, a payment, or the law. Grounding takes you from a coin-flip to something genuinely dependable; the review step and the honest "I don't know" close the remaining gap. In regulated fields the stakes are highest, which is why our legal and healthcare work leans hard on citations and human sign-off.
If you want AI that answers accurately from your own contracts, policies, or knowledge base, and you want it built with citations and a tested error rate instead of a vendor promise, talk to us about a grounded document AI build or get in touch. We would rather show you the real accuracy number than sell you a hallucination-free myth.
Scrub changelog
- kept-verified: Vectara HHEM ~1%-3% best-model range, May 2026 update, 7,700+ docs at temp 0 - Re-checked live GitHub leaderboard (Open Flag 2): last updated May 11, 2026; top three models at 1.8%, 3.1%, 3.3%; 7,700+ articles at temperature 0. Post's 'roughly 1% to 3%' for best-performing models is accurate. No edit needed.
- kept-verified: Stanford RegLab 17%-33% and 'providers' claims are overstated' - Re-confirmed via live search: Magesh et al. 2024 tested Lexis+ AI and Westlaw AI-Assisted Research, hallucination 17%-33%, researchers concluded providers' claims overstated. Stable preregistered study.
- kept-verified: Kiela conceptual attribution (no quotation marks) - Per Open Flag 1, framing is attributed conceptually, not as verbatim quotation. Post uses no quotation marks around Kiela's words, so no accuracy risk. Left intact.
- removed-contradicted: Kiela described as "now CEO of Contextual AI" - Wikipedia confirms Kiela departed Contextual AI in May 2026 (joined Google DeepMind under a Google-Contextual AI licensing agreement; Jay Chen became interim CEO). The present-tense "now CEO" is no longer accurate as of publication, so the current-role clause was excised from the intro and from reference 6. His 2020 RAG co-authorship and his grounding framing remain verified via the Data Exchange interview and Wikipedia.
- kept-verified: Menlo Ventures RAG second-most-common ranking - Cited qualitatively (Open Flag 3), no exact percentage stated. Ref confirmed. No edit needed.
- removed-contradicted (second pass): Federal Reserve corroboration of low small-firm AI adoption + "growth from Dec 2025 to May 2026 concentrated in firms with 20+ employees" - The FEDS Note (Monitoring AI Adoption in the U.S. Economy, Apr 3 2026) actually reports that by mid-2025 small businesses were adopting AI faster than large firms ("a reversal"), with nearly 40% exploring or using AI. That contradicts the "corroborates low adoption among small firms" framing, and the note's data runs only through year-end 2025, so the claimed Dec 2025-May 2026 growth concentration in 20+-employee firms is unsupported. Removed the "and Federal Reserve" inline attribution (the <20% smallest-firm figure remains fully supported by the US Census BTOS source) and dropped reference 5; the Kiela reference was renumbered from 6 to 5.