This is a Lexicon entry: the mechanism, when it shows up, and the countermeasure. The full correction, with real-world cases and the audit prompt, hasn't been written yet.
Definition
Nassim Taleb's concept: people who are highly skilled or insightful in one domain fail to apply those skills in structurally similar situations in a different domain. A doctor who reasons probabilistically about medical evidence reasons no more carefully than the general public about financial risk. A chess grandmaster is no better than average at other cognitively demanding strategic games. Domain dependence limits the transferability of expertise and undermines the assumption that intelligence or skill in one area confers advantage in others.
When it shows up
When a researcher tests many things simultaneously; when a study has many outcome variables. In any expert field: when understanding your own calibration helps you communicate uncertainty to stakeholders. In research: when hypothesis pre-registration forces you to articulate your prior before seeing data.
Failure mode
A model that fits training data perfectly and fails on new data. Strategy built around patterns that were noise. Policy based on overfitted social research.
Countermeasure
Require replication before significantly updating on any finding. Be especially sceptical of models that fit historical data too well. Ask: "How does this perform on out-of-sample data?"
Related traps
Also connected in the map5 more, locked
Credit & first seen
DiscoveredDeveloped by Nassim Nicholas Taleb in The Black Swan (2007) and Antifragile (2012). source ↗
Also revealed
Entry #150 of 631 in The Lexicon · see the full Lexicon