Why AI makes things up, and the five places it happens

Ask a language model for a quote from a book it has effectively memorised and you'll usually get it right. Ask for a quote from a book it saw twice in training and you'll get something quote-shaped, right register, right themes, right sort of sentence. Just not real.

The model cannot tell those two situations apart. It has no internal signal that says this next bit is well-supported and this bit is me filling in the pattern. That's the whole phenomenon, and once you see it that way the behaviour stops being mysterious.

The mental model

A language model has learned, in extraordinary detail, which words tend to follow which other words. When you give it your text, it continues that text in the most plausible way it can.

It is a plausibility engine. Everything else follows:

• It rarely says "I don't know", because the most plausible continuation of a question is an answer.
• It sounds equally confident whether it's right or wrong, because confidence is a property of the writing style, not of the underlying knowledge.
• It's much better at shapes than at facts. The shape of a project brief is a stable pattern. A specific company's Q3 revenue is not.

Where fabrication concentrates

It isn't evenly distributed. It clusters, reliably, in five places.

Specific numbers. Statistics, prices, dates, percentages, measurements. Anything where being approximately right looks identical to being right.

Citations and sources. Paper titles, authors, page numbers, URLs. This is the classic, career-damaging failure: a real author, a real journal, a plausible title, a plausible year — and the paper does not exist. It reads as more credible than a real citation, because it was generated to be maximally plausible.

Named people and small organisations. Anyone not extensively written about on the public internet.

Recent events. Anything near or past the training cutoff, unless the tool is genuinely searching and showing you links.

Anything where the honest answer is "there isn't one." It will invent a plausible answer rather than return nothing.

What this actually means for your work

It does not mean the tool is untrustworthy. It means trust is task-shaped.

Structuring, drafting, rephrasing, summarising text you supplied, brainstorming, explaining an established concept, critiquing your own writing, all low risk, because you supplied the substance or the substance is well-trodden.

Retrieving a specific fact you can't check, high risk, every single time.

The reframe that fixes most of it

Stop treating it as a source of knowledge and start treating it as a processor of knowledge you supply.

You paste the report and ask for the summary. You paste the figures and ask for the narrative. You paste the source and ask what it implies.

Substance from you. Transformation from it.

Nearly every fabrication problem you'll encounter disappears the moment you make that switch, and the tasks that remain on the risky side of the line are exactly the ones worth doing the checking on.