In an extremely roundabout way I just came across this: a paper written by Google’s Bo Cowgill and academics Justin Wolfers and Eric Zitzewitz and published on January 6th, studying the evidence from Google’s dabbling in prediction markets, described as means to “efficiently aggregate many employees’ information and augment existing forecasting methods”. Not everyone’s definition maybe (there’s prediction markets a bit wider) but a real practical example.
The study tracks the accuracy and biases of different internal prediction markets, following on quite a number of variables (from the professional relevance of the question posed to its dependence on the company’s execution, but also every other you can think of and some more). The results are intuitive in parts (physical proximity between employees correlates with their answers since they tend to share lots of information, extreme options are underestimated), less intuitive in others (the presumed role of optimism, an insight into some Google management principles, and the flow of information in closely “packed-in” groups of people).
It’s not exactly an easy or a fast chew, but my particular prediction is that it will be worth poring over. There are not too many detailes analyses on real prediction markets, so even such a narrow example should be relevant. And it is narrow, as the market participants appear to have been a very definite segment of employees. So narrow indeed that the dominant factor seems to have been physical proximity in the workplace (Google really tracked all sorts of data during the study).
Narrowness being a main driver in increasing clustering of opinions (reducing the variety of the sample decreases the differences in valuation since the variety of factors considered and data handled decreases too), the results of the study are hardly representative of multi-company or open prediction markets. But it does give rise to the question of the relative importance of some information flows that were apparently being underestimated in these “2.0” days of electronic communications.
And it does seem that those markets don’t do too bad a job at evaluating mainstream options (I’m guessing less clustering would improve the evaluation of the less mainstream ones). This is not enough to abandon scenario planning and other prediction tools, but can help to gauge lots of opinions in a fast and efficient way.
And it’s not all statistics. The talent management practices of Google peek out of the study in several ways.