Why it makes things up

Workshop · 2 of 7 · Prediction, not lookup

Ask an AI about a book that doesn't exist and you may get a fluent summary of it. Ask for a citation and you may get a perfectly formatted reference to a paper nobody wrote. This is the failure that surprises people most — and it stops being surprising the moment you remember lesson one.

It completes; it doesn't consult

The model predicts what plausibly comes next, given what's on the desk. When the true answer is in the window — or strongly represented in what it learned — the most plausible continuation is also the correct one. When it isn't, the machinery doesn't stop and say “no record found.” It does the only thing it does: it produces the most plausible-sounding continuation. That's not lying — lying requires knowing the truth and choosing against it. It's completion running past the edge of knowledge without a warning light.

Where invention happens most

The narrower the fact, the higher the risk. Broad synthesis — “explain how mortgages work” — is usually solid, because a million consistent examples shaped it. Pinpoint recall — an exact quote, a specific citation, a phone number, a court case, a niche API flag, a link — is where invention thrives, because a plausible-shaped answer and a true answer look identical to the machinery. And the tone never changes: confidence is styling, not signal. The model writes right answers and wrong ones with the same calm authority.

The four defenses

Ground it. Give it the document and ask about that, instead of asking it to remember — the model quotes the desk far more reliably than its training. Make it doubt. Ask “how confident are you, and what would you check?” — the answer often surfaces the weak spots. Demand openable sources. A real citation can be clicked and read; ask for ones you can open, then open them. Keep the last check yours. For anything that leaves your hands — a number, a quote, a claim — the verification is your job, not the model's promise.

Used this way, invention stops being a trap and becomes a known property of the tool — like glare on a camera lens. You don't throw away the camera. You learn where the sun is.