Keeping tabs on the experts

by | Tuesday, September 08, 2009

In an age where experts are a dime a dozen, willing to pontificate at the drop of a pin, it is hard to tell whom to believe, and whom NOT to believe. In comes Phillip Tetlock, an academic who has made it his mission to evaluate the prognosticators! This is described in his book:

Tetlock, P.E. (2005). Expert political judgment: How good is it? How can we know? Princeton: Princeton University Press.

I recently came across a review written by him, titled Reading Tarot on K Street (in the September/October 2009 issue of The National Interest) and I thought it captured his work in this area quite nicely (and would be worth preserving).

When we score the accuracy of thousands of predictions from hundreds of experts across dozens of countries over twenty years, we find the best forecasters tend to be modest about their forecasting skills, eclectic in their ideological and theoretical tastes, and self-critical in their analytical styles.1 Borrowing from philosopher Isaiah Berlin, I call them foxes—experts who know many things and are not finicky about where they get good ideas. Paraphrasing Deng Xiaoping, they do not care if the cat is white or black, only that it catches mice.

Contrast this with what I call hedgehogs—experts who know one big thing from which likely future trends can be more or less directly deduced. The big thing might be any of a variety of theories: Marxist faith in the class struggle as the driver of history or libertarian faith in the self-correcting power of free markets, or a realist faith in balance-of-power politics or an institutionalist faith in the capacity of the international community to make world politics less ruthlessly anarchic, or an eco-doomster faith in the impending apocalypse or a techno-boomster faith in our ability to make cost-effective substitutes for pretty much anything we might run out of.

What experts think—where they fall along the Left-Right spectrum—is a weak predictor of accuracy. But how experts think is a surprisingly consistent predictor. Relative to foxes who are less encumbered by loyalties to an all-encompassing worldview, hedgehogs offer bolder forecasts and, although they hit occasional grand slams, they strike out a lot and wind up with decidedly poorer batting averages.

The implications for people who make projections about technology and schools and learning is quite obvious to me. It is the hedgehogs we need to be careful of, mainly because of the vehemence of their beliefs which can sometimes override our “foxy” nature. I say inherent because I think that educators, for the most part, are pragmatists, sensitive to the limits of arm-chair theorizing and big ideas. A hard nosed approach to reality, that recognizes its complexity, that demands multi-faceted problems solving approaches is what is needed, not being wedded to one, just one overarching idea.

A few randomly selected blog posts…

Curt becomes Bonk (and vice versa)

Curt Bonk is Professor of Instructional Systems Technology in the School of Education at Indiana University. Curt is one of the most fun academics I know. He is also a good friend. That's us at the COSN conference earlier this year. What I didn't remember was that...

Election 2.0

An op-ed in today's NYTimes by William Poundstone about how the web can be used during an election to prevent election fraud. As the article describes a computer scientist and a mathematician "have proposed an ingenious method that would combine paper ballots and a...

The value of school: Part 2

The value of school: Part 2

Note: This is the second of two posts on the value of school by Kevin Close and Punya Mishra. Read the first post: What value do schools bring? Revisiting Accountability In the previous post we argued that schools play a varied and rich role in the economic, social,...

Creativity in Las Vegas

I was recently invited to present a keynote address at the 21st Century Instructional Technology Conference (titled Elements of Technology) at the Clark County School District in Las Vegas, Nevada. Clark County is the 5th largest school district in the country with...

Happy Teacher’s Day (new ambigrams)

September 5 is Teacher's Day in India. It is celebrated on the birthdate of Sarvepalli Radhakrishnan, Indian philosopher and statesman who was also the first Vice-President and the second President of India. He famously said, "teachers should be the best minds in the...

Looking for IT in India

A few days ago Jack Schwille, assistant dean for international studies in education, sent an email out to all faculty and students at the college of education announcing a talk by me titled: "Help Punya find IT in India?" This presentation was to be fifth in the...

Blast from the past: Theories and memory

Ambigram for the word "Theory" by Punya Mishra My first real research study was one that I conducted back when I was a graduate student under the mentorship of Bill Brewer. It was designed as a classic educational psychology memory study and though I have done little...

On making computation visible

Here is a cool video about a "a mechanical, binary adding machine that uses marbles to flip the bits" - in other words a computer made of wood, that works at a pace that we can grasp! Marvelous. (HT: Collision Detection). Check out the video: [youtube width="425"...

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *