Confident, Cited and Wrong: Why AI-Written Documents Need a Verifier
In October 2025, Deloitte Australia agreed to repay part of a A$440,000 fee to the Australian Department of Employment and Workplace Relations. The report it had delivered, a review of a welfare compliance system, contained references to academic papers that do not exist and a quotation attributed to a federal court judgment that the judgment does not contain. The revised version disclosed that a generative AI system had been used in its preparation. The department said the substance of the review stood (CFO Dive, Fortune).
Two years earlier, in June 2023, a federal judge in New York fined two lawyers and their firm $5,000 for filing a brief that cited six cases. None of the six existed. ChatGPT had produced them, complete with reporter volumes and page numbers, and nobody had looked them up (Mata v. Avianca, Inc.).
These two stories are usually told as warnings about AI. They are better read as warnings about a missing step. In both cases a capable professional produced a document with a model's help, and in both cases the document went out without anyone checking its claims against the sources they were supposed to rest on. The model did what models do. The process had no verifier.
The failure is in the checking, not the writing
It is tempting to treat these as early mistakes that better models will fix. The evidence says otherwise.
In 2024, researchers at Stanford's RegLab and Institute for Human-Centered AI tested the AI research tools sold by LexisNexis and Thomson Reuters, products built specifically to ground their answers in a curated legal database and marketed as free of fabrication. They found that the tools "each hallucinate between 17% and 33% of the time," and concluded that "the providers' claims are overstated." The study was published as a preprint in May 2024 and, after peer review, in the Journal of Empirical Legal Studies in 2025 (Magesh et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools").
That is the important number. These were not chatbots asked to improvise. They were retrieval systems with the right documents in front of them, and between one answer in six and one in three still contained something false. Grounding a model in good sources lowers the rate. It does not make the output trustworthy without a check.
In September 2025, researchers at OpenAI published an explanation of why. Their paper argues that hallucinations persist because "the training and evaluation procedures reward guessing over acknowledging uncertainty": a model that guesses when unsure scores better on the benchmarks it is optimised against than one that says it does not know (Kalai, Nachum, Vempala and Zhang, "Why Language Models Hallucinate"). A confident answer is the trained behaviour. The fix they propose is to change how models are scored, which is a change to the industry's incentives, not something a buyer of a document can wait for.
So the writing side will keep producing plausible, well-formatted, occasionally false statements for the foreseeable future. What decides whether that matters is whether anything checks them before they reach a reader who will act on them.
Most of the time, nothing does
The largest study of how people actually use AI at work is the 2025 survey by KPMG and the University of Melbourne, which asked over 48,000 people across 47 countries between November 2024 and January 2025. Among employees who use AI at work, 66% said they rely on its output without evaluating its accuracy, 56% said they had made mistakes in their work because of AI, and 57% said they present AI-generated work as their own (KPMG, "Trust, attitudes and use of artificial intelligence: A global study 2025").
Put the two findings together. Grounded tools are wrong somewhere between a sixth and a third of the time, and two-thirds of the people using them do not check. The Deloitte report and the Avianca brief are not outliers. They are the cases where someone outside the process happened to look.
The reason people do not check is not carelessness. Checking a document by hand means finding every claim in it, finding the sentence in the source that should support each one, and reading the two side by side. For a forty-page board pack that is a day's work, and the whole point of using the model was to save the day. The step gets skipped because it is expensive, and it stays skipped because nothing forces it.
What a check has to be
A useful verifier has to do the expensive thing cheaply, and it has to do it in a way that a reader can audit afterwards. That rules out the obvious shortcut of asking the same kind of model whether the document is right. A model grading its own output has the same incentive to sound confident that produced the error in the first place, and its "looks fine" is exactly as unverifiable as the original claim.
Four properties matter.
It checks against documents you chose, not against the internet. A claim in a board pack should be checked against the annual report and the deck it was drawn from, not against whatever a search engine turns up. A verifier that retrieves its own evidence has quietly replaced your question, "does our source say this?", with a different one, "does anything say this?". The second question is how fabricated citations get laundered into real-looking ones.
Its verdict is relative, and says so. Supported means consistent with the documents you supplied. It never means true. A claim the documents do not speak to should come back as exactly that, insufficient evidence, rather than being rounded up to supported or down to contradicted. That distinction is the difference between a finding and a guess.
It cites the sentence, not the document. "See the annual report" is not a check. A verdict has to point at the specific sentence that supports or contradicts the claim, so that a reviewer can read that sentence and agree or disagree in seconds. If the verifier cannot name the sentence, it has not verified anything.
The decision is not made by the model. A model can read a sentence and measure whether it establishes a claim, whether it contradicts it, whether it speaks to it at all. Those measurements should then be turned into a verdict by fixed rules that are the same for every claim and every run, so the outcome cannot be argued into by a persuasive paragraph. Models measure. Code decides. When a result is questioned later, the rules that decided it are something you can show.
What this looks like in practice
This is the shape of Attest, the verification service inside the Quorum platform. It exists because Quorum turns board packs and company documents into briefings for the people who have to decide on them, and a briefing that is confidently wrong is worse than no briefing.
You give Attest a claim and the evidence it should be true against, either inline or as documents loaded once and checked against many times. It returns one of six verdicts, the ids and text of every sentence the verdict rests on, a reading of how the claim was measured, and a provenance block naming every version and hash that produced the answer, so the same result can be reproduced and audited.
Three things follow from the properties above.
Attest never retrieves. It has no search and no memory of other customers' documents. What you supply is the entire universe it judges against, and the verdict says which sentences of that universe it used.
Attest never stores. Documents are parsed in memory and held only until you delete them or they age out; the one durable trace is a SHA-256 hash of each file, which you can recompute from your own copy to prove what a verdict was checked against.
Attest reviews whole drafts, not just the claims you thought to list. Hand it the document and the sources and it finds the claims itself, then checks each one. The claims a writer leaves off a list are the ones they did not notice were claims, and in our experience those are where the errors live. The Deloitte report's fabricated quotation was a footnote. Nobody lists footnotes.
Applied to the two stories that opened this post, the check is unglamorous. The six Avianca cases would have come back as insufficient evidence against any real case database, because there was nothing to cite. The Deloitte quotation would have come back as contradicted or insufficient against the judgment it was attributed to, with the nearest real sentence shown beside it. Neither result requires the checker to be cleverer than the writer. It only requires the check to happen.
What to do on Monday
If your team produces documents with a model's help, three changes cover most of the risk.
Decide which documents each output is accountable to. A memo drawn from three sources should be checked against those three, not against the model's general knowledge. Write the list down before the draft exists.
Check claims, not vibes. Reading a draft for whether it sounds right is what everyone already does, and it is how the 66% got there. Check each factual statement against its source, or use something that does.
Keep the evidence of the check. A verdict with the cited sentence and the hash of the document it was checked against is what you show when a regulator, a board member or a court asks how you knew. "The model seemed confident" is what Avianca's lawyers had.
Attest is available to Quorum workspaces through the Enterprise API at developer.quorumtech.ch, as an MCP server for AI assistants, and on RapidAPI for anyone who wants to try it without an account. The models will keep guessing. The check is now cheap enough to run every time.