CogniqOS
Subscribe

TRAP #027 ·VOL I The Individual · Judgment & Evaluation ErrorsLEXICON ENTRY

Illusion of Validity

Discovered by Daniel Kahneman, 1973

The Coherent Story Feels Like Evidence The more consistent a pattern looks, the more confident we feel - even when the pattern has zero predictive power. *(Evidence: Strong)* A cognitive bias where the internal coherence of a narrative or dataset creates subjective confidence that is unrelated to the actual validity of the judgment. Identified by Kahneman and Tversky: people mistake "good fit" for "good prediction." A compelling story with clean data feels reliable even when the underlying signal is noise.
Category: BiasEvidence: ReplicatedTier 1Type: Mechanismⓘ what these mean

This is a Lexicon entry: the mechanism, when it shows up, and the countermeasure. The full correction, with real-world cases and the audit prompt, hasn't been written yet.

When it shows up

Hiring decisions based on structured interview narratives. Investment theses built on internally consistent but cherry-picked data. Medical diagnoses where symptoms form a neat pattern. Sports drafts where highlight reels create false certainty.

Failure mode

Confidence tracks coherence rather than evidence quality. Decision-makers reject superior statistical models because the model's output "doesn't feel right" compared to their intuitive narrative. Interview confidence bears near-zero correlation with actual job performance.

Countermeasure

Separate the question "Does this story make sense?" from "Does this story predict outcomes?" Demand out-of-sample validation. Use base rates and actuarial methods alongside narrative judgment.

Get the pre-decision checklist →

Related traps

Also connected in the mapThe Narrative Fallacy·5 more, locked

Read next

Want the full correction for Illusion of Validity?

This entry has the mechanism and countermeasure. The full write-up - real cases, sources, the audit prompt - comes later.

Entry #27 of 631 in The Lexicon · see the full Lexicon