Quotes from Philip E. Tetlock

Predictably, psychologists who test police officers' ability to spot lies in a controlled setting find a big gap between their confidence and their skill. And that gap grows as officers become more experienced and they assume, not unreasonably, that their experience has made them better lie detectors. As a result, officers grow confident faster than they grow accurate, meaning they grow increasingly overconfident.

Research on calibration—how closely your confidence matches your accuracy—routinely finds people are too confident.10 But overconfidence is not an immutable law of human nature. Meteorologists generally do not suffer from it. Neither do seasoned bridge players. That's because both get clear, prompt feedback.

In Germany's war academies, scenarios were laid out and students were invited to suggest solutions and discuss them collectively. Disagreement was not only permitted, it was expected, and even the instructor's views could be challenged because he "understood himself to be a comrade among others

Once we know the outcome of something, that knowledge skews our perception of what we thought before we knew the outcome: that's hindsight bias.

Everybody has said, 'I want push-back from you if you see something I don't,' " said Rosenthal. That made a difference. So did offering thanks for constructive criticism.

A forecaster who doesn't adjust her views in light of new information won't capture the value of that information, while a forecaster who is so impressed by the new information that he bases his forecast entirely on it will lose the value of the old information that underpinned his prior forecast. But the forecaster who carefully balances old and new captures the value in both—and puts it into her new forecast. The best way to do that is by updating often but bit by bit.

what is often forgotten is that the Nazis did not create the Wehrmacht. They inherited it. And it could not have been more different from the unthinking machine we imagine—as the spectacular attack on the Belgian fortress of Eben Emael demonstrated.

Amount of luck in tournament determines amount of regression to the mean from one year to the next.

How skillfully leaders perform this balancing act determines how successfully their organizations can cultivate superteams that can replicate the balancing act down the chain of command. And this is not something that one isolated leader can do on his own. It requires a wider willingness to hear unwelcome words from others—and the creation of a culture in which people feel comfortable speaking such words.

Special counsel Theodore Sorensen and the president's brother Bobby were designated "intellectual watchdogs," whose job was to "pursue relentlessly every bone of contention in order to prevent errors arising from too superficial an analysis of the issues," Janis noted.

readers of the New York Times opened the newspaper on the kitchen table and read the front-page headline: U.S. HELPS TRAIN AN ANTI-CASTRO FORCE AT SECRET GUATEMALAN AIR-GROUND BASE

As Moltke observed, "It shakes the trust of subordinates and gives the units a feeling of uncertainty if things happen entirely differently from what orders from higher headquarters had presumed."9

Need for cognition" is the psychological term for the tendency to engage in and enjoy hard mental slogs. [...] superforecasters score high in need-for-cognition tests.

Galen's writings were the indisputable source of medical authority for more than a thousand years. "It is I, and I alone, who has revealed the true path of medicine," Galen wrote with his usual modesty. And yet Galen never conducted anything resembling a modern experiment. Why

All who drink of this treatment recover in a short time, except those whom it does not help, who all die," he wrote. "It is obvious, therefore, that it fails only in incurable cases."5

members of any small cohesive group tend to maintain esprit de corps by unconsciously developing a number of shared illusions and related norms that interfere with critical thinking and reality testing.

The more you want to explain about a black swan event like the storming of the Bastille," wrote the sociologist Duncan Watts, "the broader you have to draw the boundaries around what you consider to be the event itself.

So finding meaning in events is positively correlated with wellbeing but negatively correlated with foresight. That sets up a depressing possibility: Is misery the price of accuracy?

Not until the twentieth century did the idea of randomized trial experiments, careful measurement, and statistical power take hold. "Is the application of the numerical method to the subject-matter of medicine a trivial and time-wasting ingenuity as some hold, or is it an important stage in the development of our art, as others proclaim it," the Lancet asked in 1921. The

The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.11

The idea of randomized controlled trials was painfully slow to catch on and it was only after World War II that the first serious trials were attempted. They

Physicians and the institutions they controlled didn't want to let go of the idea that their judgment alone revealed the truth, so they kept doing what they did because they had always done it that way—and they were backed up by respected authority. They didn't need scientific validation. They just knew. Cochrane despised this attitude. He called it "the God complex.

people realized that excessive politeness was hindering the critical examination of views, so they made special efforts to assure others that criticism was welcome. "Everybody has said, 'I want push-back from you if you see something I don't,' " said Rosenthal. That made a difference. So did offering thanks for constructive criticism. Gradually, the dancing around diminished.

What people didn't grasp is that the only alternative to a controlled experiment that delivers real insight is an uncontrolled experiment that produces merely the illusion of insight. Cochrane