Friday 27 March 2009
Hold ‘experts’ accountable for their predictions
Ever wonder how financial experts could lead the world over the economic cliff? One explanation is that so-called experts turn out to be, in many situations, a stunningly poor source of expertise. There’s evidence that what matters in making a sound forecast or decision isn’t so much knowledge or experience as good judgment - or, to be more precise, the way a person’s mind works.
Subscribe to:
Post Comments (Atom)
1 comment:
Hold ‘experts’ accountable for their predictions
By NICHOLAS D KRISTOF, NYT
27 March 2009
Ever wonder how financial experts could lead the world over the economic cliff? One explanation is that so-called experts turn out to be, in many situations, a stunningly poor source of expertise. There’s evidence that what matters in making a sound forecast or decision isn’t so much knowledge or experience as good judgment - or, to be more precise, the way a person’s mind works.
More on that in a moment. First, let’s acknowledge that even very smart people allow themselves to be buffaloed by an apparent ‘expert’ on occasion.
The best example of the awe that an ‘expert’ inspires is the ‘Dr Fox effect’. It’s named for a pioneering series of psychology experiments in which an actor was paid to give a meaningless presentation to professional educators. The actor was introduced as ‘Dr Myron L Fox’ (no such real person existed) and was described as an eminent authority on the application of mathematics to human behaviour. He then delivered a lecture on ‘mathematical game theory as applied to physician education’ - except that by design it had no point and was completely devoid of substance. However, it was warmly delivered and full of jokes and interesting neologisms.
Afterwards, those in attendance were given questionnaires and asked to rate ‘Dr Fox’. They were mostly impressed. ‘Excellent presentation, enjoyed listening,’ wrote one. Another protested: ‘Too intellectual a presentation.’ A different study illustrated the genuflection to ‘experts’ another way. It found that a president who goes on television to make a case moves public opinion only negligibly, by less than a percentage point. But experts who are trotted out on television can move public opinion by more than 3 percentage points, because they seem to be reliable or impartial authorities.
But do experts actually get it right themselves? The expert on experts is Philip Tetlock, a professor at the University of California, Berkeley. His 2005 book, Expert Political Judgment, is based on two decades of tracking some 82,000 predictions by 284 experts. The experts’ forecasts were tracked both on the subjects of their specialties and on subjects that they knew little about.
The result? The predictions of experts were, on average, only a tiny bit better than random guesses - the equivalent of a chimpanzee throwing darts at a board.
‘It made virtually no difference whether participants had doctorates, whether they were economists, political scientists, journalists or historians, whether they had policy experience or access to classified information, or whether they had logged many or few years of experience,’ Professor Tetlock wrote.
Indeed, the only consistent predictor was fame - and it was an inverse relationship; the more famous experts did worse than unknown ones. That had to do with a fault in the media. Talent bookers for television shows and reporters tended to call up experts who provided strong, coherent points of view, who saw things in black and white.
People who shouted - like, yes, Jim Cramer! Prof Tetlock called experts such as these the ‘hedgehogs’, after a famous distinction by the late Isaiah Berlin (my favourite philosopher) between hedgehogs and foxes. Hedgehogs tend to have a focused worldview, an ideological leaning, strong convictions; foxes are more cautious, more centrist, more likely to adjust their views, more pragmatic, more prone to self-doubt, more inclined to see complexity and nuance. And it turns out while foxes don’t give great sound-bites, they are far more likely to get things right.
This was the distinction that mattered most among the forecasters - not whether they had expertise. Over all, the foxes did significantly better, both in areas they knew well and in areas they didn’t. Other studies have confirmed the general sense that expertise is overrated. In one experiment, clinical psychologists did no better than their secretaries in their diagnoses. In another, a white rat in a maze repeatedly beat groups of Yale undergraduates in understanding the optimal way to get food dropped in the maze. The students overanalysed and saw patterns that didn’t exist, so they were beaten by the rodent.
The marketplace of ideas for now doesn’t clear out bad pundits and bad ideas partly because there’s no accountability. We trumpet our successes and ignore failures - or else we attempt to explain that the failure doesn’t count because the situation changed or that we were basically right but the timing was off.
For example, I boast about having warned in 2002 and 2003 that Iraq would be a violent mess after we invaded. But I tend to make excuses for my own incorrect forecast in early 2007 that the troop ‘surge’ would fail.
So what about a system to evaluate us prognosticators? Prof Tetlock suggests that various foundations might try to create a ‘trans-ideological Consumer Reports for punditry’, monitoring and evaluating the records of various experts and pundits as a public service.
I agree: Hold us accountable!
Post a Comment