Should we trust experts?

Retailers have a lot of incentives to accurately predict holiday sales. Why? Low sales forecast will leave their shelves empty. This will result in lost profits. On the other hand excess sales forecast will result in unsold inventory. This will put pressure on the profit margins. In order to predict sales accurately the retailers depend on the internal experts. Electronics giant Best Buy is one such retailer. In the book Think Twice – Michael Mauboussin writes

James Surowiecki, author of the best selling book The Wisdom of Crowds, strolled into Best Buy’s headquarters and delivered a startling message: a relatively uninformed crowd could predict better than the firm’s best seers. Surowiecki’s message resonated with Jeff Severts, an executive then running Best Buy’s gift-card business. Severts wondered whether the idea would really work in a  corporate setting, so he gave a few hundred people in the organization some basic background information and asked them to forecast February 2005 gift-card sales.

It turned out that the forecast of 200 respondents was 99.5 percent accurate. The experts response was only 95 percent accurate. After more experiments the company was convinced that the crowd can predict better than the experts. It developed a prediction market call TagTrade. It helped the company to predict the performance of a service package for laptop computers.

In January, Jeff Severts asked both his management team and the market to predict sales of a new service package the company was offering for laptop computers. A week before the company began selling the product, the market’s guess was 33% lower than the team’s official sales forecast. It subsequently sank further. When initial sales figures confirmed the market’s prediction, Mr. Severts ended the offer and began redesigning the service package. He listed a TagTrade stock to gauge the revised package’s chances of starting on time, in mid-September. The stock rose, and Mr. Severts says he took “great comfort from that.” On Sunday, Best Buy started offering the new package.

Harrah owns and operates 50+ casinos. The company was giving enormous attention to the high rollers. A high roller is a person who gambles a large amount of money. Harrah assumed that high rollers are the primary revenue generators for the business. Later the company analyzed all the data using the computers. It revealed that it was the middle aged and senior adults with free time and income who added most of the value to the business. Excerpt from the book Think Twice

The conventional wisdom that the expert executives had perpetuated – that high rollers were the highest value customers – was flat wrong, but only revealed by a new look at the data.

Looking at the above examples. I had the question – should we trust experts OR should we just depend on the wisdom of the crowds and the computer algorithms. Excerpt from the book Think Twice

As networks harness the wisdom of crowds and computing power grows, the ability of experts to add value in their predictions is steadily declining. I call this the expert squeeze, and evidence for it is mounting.

We need to understand the nature of the problem to decide if we can trust the decision of the experts. In the book the author tells that the problem an expert is trying to answer can fall in any of the four broad categories.

Domain Description Rule based limited range of outcomes Rule based wide range of outcomes Probabilistic limited range of outcomes Probabilistic wide range of outcomes
Expert Performance Worse than computers Generally better than computers Equal to or worse than collectives Worse than collectives
Expert Agreement High Moderate Moderate/Low Low
Examples Credit Scoring, Simple medical diagnosis Chess, Go (Board Game) Admissions Officers, Poker Stock Market, Economy

1. Rule based and limited range of outcomes

Experts are needed in the initial phase. They are the ones to define the rules and algorithms that solves the problem. Once that task is done then the computers can take it over. Computers do not get tired and are cheaper to scale. Unlike humans computers do not have any biases. Given below are the weights which FICO uses for calculating the credit score. Once these rules are defined the computers can do the rest.


2. Rule based and wide range of outcomes

Go is a 2 player board game that originated in China 2,500 years ago. The game is rich in strategy and humans still dominate the computers. The Go board is 19 * 19 which is larger than a chess board. This results in several combinations which takes a long time for the computers to calculate. Excerpt from the wikipedia

Given an average of 200 available moves through most of the game, for a computer to calculate its next move by exhaustively anticipating the next four moves of each possible play (two of its own and two of its opponent’s), it would have to consider more than 320 billion (3.2×1011) possible combinations. To exhaustively calculate the next eight moves, would require computing 512 quintillion (5.12×1020) possible combinations. As of June 2008, the most powerful supercomputer in the world, IBM’s “Roadrunner” distributed cluster, can sustain 1.02 petaflops.[105][106][107] At this rate, even given an exceedingly low estimate of 10 operations required to assess the value of one play of a stone, Roadrunner would require 138 hours, more than five days, to assess all possible combinations of the next eight moves in order to make a single play.

Overall experts do well in this case. Excerpt from the book Think Twice

Experts do well with rules-based problems with wide range of outcomes because they are better than computers at eliminating bad choices and making creative connections between bits of information.

3. Probabilistic limited range of outcomes

CEO of a company makes strategic decisions. He decides on the new markets to enter and sell the company products.  Decisions about how to challenge a competitor and how to create a new product are taken by him. But the effectiveness of these decisions are debatable. In certain cases experts are better. Excerpt from the book Think Twice

Computers and crowds fare poorly if they lack domain specific knowledge. For instance, an expert coach will probably create a better game plan than a computer because he can draw on the unique knowledge of his team and the competition. Similarly, an executive may be able to better shape strategy for corporation.

4. Probabilistic wide range of outcomes

Wide range of outcomes are possible in the stock market and the economy. Evidence shows that collectives outperform the experts in solving these problems. Excerpt from the book Think Twice

For instance, economists are extremely poor forecasters of interest rates, often failing to accurately guess the direction of rate moves, much less their correct level. Note, too, that not only are experts poor at predicting actual outcomes, they rarely agree with one another. Two equally credentialed experts may make opposite predictions and, hence, decisions from one another. One example is the forecasting of oil prices. In one camp are experts like Matthew Simmons, an investment banker and consultant specializing in energy, who argues that the world has reached its peak of oil extraction and that oil prices are likely to rise as a consequence. In the other camp are experts including Daniel Yergin, an economic researcher, who argues that technology will make it possible to find new sources of oil and to extract them profitably.

If the collectives are smart then why did the stock market crash of 1929 and 2000 happen? Some conditions must be in place for the collectives to be correct. The crowd must be diverse and each participant should form their own opinion. During the boom times everyone wants to get rich. Because of the greed and social proof most of them do not think and they copy each other. Hence market crash takes place and the collective wisdom is completely incorrect.


You can trust the experts in systems that are rule based, stable, and the cause-and-effect relationships are linear. If the outcomes of the system is probabilistic then you should take the expert advice with a pinch of salt.

I really enjoyed reading the book Think Twice and I highly recommend it. Watch the video to get the summary about the book.


One thought on “Should we trust experts?

Comments are closed.