How much of what an AI explained to you last week could you reconstruct on your own, with the screen off?
I ask because the feeling of having understood something and the fact of understanding it come apart more easily than most of us admit, and a fluent explanation is the quickest way to pry them apart.
In 2002 Leonid Rozenblit and Frank Keil ran a study that has since become a small classic. They asked people how well they understood ordinary machines: a zipper, a flush toilet, a cylinder lock. Confidence ran high. Then they asked the same people to write out, step by step, how each device works, and the confidence collapsed on contact with the blank page. The two named the gap the illusion of explanatory depth, and showed it was specific to explanatory knowledge. We do not overrate our memory of facts or our command of a procedure nearly as much as we overrate our sense of how a mechanism works. The illusion runs deepest where the parts feel visible, where it seems the explanation is sitting right there for the taking.
An AI assistant is engineered to make the parts feel visible. Ask one how a policy would ripple through a labor market, or how a class of drugs acts on the body, and it returns a clean chain of steps in confident prose that arranges the reasoning in tidy stages. The delivery is the whole point. It hands me the sensation Rozenblit and Keil measured, the warm certainty that the mechanism is now mine, and it does so faster and more smoothly than any textbook or colleague ever managed. The recent fashion for showing a model's step-by-step "reasoning" sharpens the effect, since a visible chain of thought reads as the machinery itself, whether or not it reflects how the answer was reached.
There is direct evidence the sensation misleads. In a pre-registered experiment reported in 2025 by Sunnie Kim and colleagues, 308 people used a language model to answer factual questions. When the model attached an explanation to its answer, people leaned on it more, and here is the part that stings, they leaned on it more whether the answer was right or wrong. The explanation raised trust without tracking truth. What pulled people back toward calibration on the wrong answers was the presence of sources, rather than more reasoning. Fluent justification and correctness are separate quantities, and the human eye keeps mistaking one for the other.
None of this is new to the species. Steven Sloman and Philip Fernbach argue that human knowledge has always lived in the group more than in the single head. I trust the pilot, the pharmacist, the person who drafted the standard, and I borrow their understanding without ever reproducing it. The arrangement holds until it has to be cashed out, and then the borrowed understanding either survives contact with the problem or does not. What shifts with a language model is the price and the pliancy. The feeling of understanding is now available on demand, at no cost, for any question I care to type. It arrives from something with no stake in whether I learn anything at all.
The same literature hands us a cheap countermeasure. In 2013 Fernbach, Rogers, Fox and Sloman found that asking people to explain, mechanically, how a policy they favored would produce its promised effects left them less confident and less extreme afterward. Asking instead for their reasons did nothing, and at times dug them in deeper. The demand to produce the steps is what punctures the illusion, because the steps are the very thing the illusion was covering for.
So the discipline writes itself, even if following it takes some nerve. When a model hands me a satisfying explanation, the explanation is a hypothesis about my own understanding, and not a receipt for it. The test is whether I can rebuild the mechanism with the window closed, or predict what it implies in a case the model never raised. A program officer who approves a sophisticated intervention off a fluent brief is running the Rozenblit and Keil experiment in reverse. The self-rating goes in high, and the blank page that would check it never arrives.
For an institution the fix is almost embarrassingly cheap. Two lines stapled to the foot of every AI-generated brief, filled in by the person who owns the decision before the brief is allowed to move it.
Reproduce the mechanism Close the tool. In three sentences, state the causal steps by which this produces its claimed effect. If you stall, the brief understood it and you did not. Predict a fresh case Name one situation the brief never mentioned, say what it implies there, and state the result that would prove the prediction wrong.Neither line asks for more analysis. Each asks the reader to produce the understanding the summary only implied, which is the one thing the fluent version quietly skipped.
I want to avoid overselling the effect. Some researchers have questioned how much of it reflects true overconfidence and how much reflects people misreading a vague rating scale, and the measured gap shifts with the way you probe it. A fluent explanation earns its keep even when it fails to transfer understanding, since it can prompt a sharper question or surface an assumption I had missed. My claim is the narrow one. The feeling an explanation produces is weak evidence of the understanding it appears to certify, and an AI is an unusually efficient engine for producing that feeling.
Before the next confident summary talks me into a decision, I would like an answer to one question. Would I still believe I understood it if the tool that explained it went dark, and the job of explaining it fell to me?
REFERENCES
Fernbach, P. M., Rogers, T., Fox, C. R., and Sloman, S. A. (2013). Political Extremism Is Supported by an Illusion of Understanding. Psychological Science, 24(6), 939-946.
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., and Russakovsky, O. (2025). Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems.
Rozenblit, L., and Keil, F. (2002). The Misunderstood Limits of Folk Science: An Illusion of Explanatory Depth. Cognitive Science, 26(5), 521-562.
Sloman, S., and Fernbach, P. (2017). The Knowledge Illusion: Why We Never Think Alone. Riverhead Books.