This is a Lexicon entry: the mechanism, when it shows up, and the countermeasure. The full correction, with real-world cases and the audit prompt, hasn't been written yet.
Definition
The property of a belief system in which expressed confidence levels match actual frequencies of being correct. A well-calibrated person who says they are 80% confident is right about 80% of the time across such statements. Overconfidence (saying 90% when you are right 60% of the time) is the most common form of miscalibration. Philip Tetlock's superforecaster research found that calibration - not intelligence - is the single most important predictor of forecasting accuracy.
When it shows up
In any prediction task; in any domain where you express confidence in beliefs. In weather forecasting: when forecasters track their historical accuracy by confidence level to improve future predictions. In clinical medicine: when a doctor tracks their diagnostic accuracy on uncertain cases to calibrate over time.
Failure mode
Systematic over- or under-confidence producing bad bets, insufficient contingency planning or paralysis from excessive uncertainty.
Countermeasure
Track your predictions against outcomes. Use calibration training. Practice saying "I'm about 70% confident" and check whether you're right about 70% of the time on such claims.
Related concepts
Also connected in the mapHard-easy Effect·The Overconfidence Effect·Bayes’ Theorem·Intellectual Humility·Separation of Signal and Noise·Uncertainty Tolerance·False Precision·Ambiguity Tolerance
Origin
Also revealed
Entry #611 of 631 in The Lexicon · see the full Lexicon