Economic forecasting is a highly difficult process. More people get forecasting wrong than right.
However, being aware of the pitfalls and learning proven forecasting techniques could help you make a meaningful contribution to your organisation’s long-term strategies.Improving forecasting
In 2008, 1,000 multinationals took part in a study into economic forecasting. A staggering 98% of these organisations admitted that their budgets, which had taken an average of three months to prepare, were obsolete before they even published them.
More recently, a survey of over 500 multinationals showed that over a 10-year period, overall budget allocations between departments had changed by less than 5% - so yours is unlikely to change of its own accord - you will need to do things to make and to sustain any changes in it that to you need! This, despite regular public proclamations about having to cope with rapidly changing environments.
The struggle to forecast with any accuracy is not limited to corporations. Philip Tetlock, Annenberg University Professor at the University of Pennsylvania, has conducted a large-scale study of tens of thousands of forecasts made by hundreds of experts over decades. His conclusion is that, generally, expert forecasts have the predictive power of a dart-throwing monkey.
Worse, he found that the more confident and eye-catching the forecast, the less likely it was to come to fruition.
Clearly, good forecasting is a difficult exercise, beset by multiple challenges. But it is hard to plan without at least some reference points - no matter how flimsy - so it is necessary to forecast and re-forecast as best we can in order to help our businesses' to run themselves and in order to help us to run our teams.
Bias and fallacies in forecasting
As well as the natural bias that stems from self-interest and self-promotion, forecasters are prone to over 100 forms of cognitive bias. Among these are three that are particularly prominent:
- Linearity bias. Most people believe the future will be a linear extrapolation of the past. This belief rests on the assumption that the world operates like clockwork and obeys the laws of Newtonian physics. In reality, the principles of complexity and chaos theory are more relevant. Economic events are, in fact, influenced to a greater degree by sudden explosive growth spurts, such as technology adoption, and sudden structural collapses, such as the fall of communism, change of political leaders, referendum votes etc. This means the challenge is to look for in-flexion points, both positive and negative, when forecasting trends. Pursuing linear goals such as squeezing costs or chasing more share in the same market rapidly becomes a futile exercise with fast diminishing returns;
- Texas sharpshooter fallacy. A drunk cowboy shoots at the side of a barn. Having sprayed bullets all over the side, he takes a paint can and paints a target over a cluster of holes. He then claims to be an expert marksman by virtue of his accuracy on the target. In a similar way, many forecasters randomly assemble large quantities of data, then claim insight when, by chance, some subset fits a pattern or theory they favoured to begin with. So, when dealing with underlying data, it’s important to start with reasonable, analytical hypotheses and then progress to testing them; and
- The what-you-see-is-all-there-is (WYSIATI) fallacy. WYSIATI is the notion that people jump to firm conclusions based on the information that is immediately at hand, no matter how fragmentary or incomplete. Then, once they’ve settled on this initial judgement, they’re slow to change their minds (a fact that politicians, lobby groups and campaigning/tabloid media often play with). Indeed, they’ll actively screen out information that contradicts their initial thinking. Managers are prone to this fallacy because it supports the idea that instant decision-making is a signifier of strength and confidence. It’s also intuitive and comfortable as it makes people appear confident in their work. However, this self-imposed myopia is dangerous, especially when dealing with complex inter-related issues such as business strategy – or the economy as a whole. So, at the outset, forecasters need to consider all the factors that may affect their forecast, even if they discard some later on because their impact is negligible.
Limitations of big data
The current business culture regards those able to wield large quantities of statistical data as modern Einsteins. So with this - and the cognitive errors our brains are prone to – in mind, should we not put all our faith solely in computers?
The current hype around big data and advanced analytics suggests that ever-larger data sets and increasingly sophisticated statistical techniques will give us infallible predictions.
These approaches do have some clear benefits. However, they’re also only as good as the cognitive framework we apply to them. In particular, they are prone to four major drawbacks:
- Ramsey theory. Some executives believe that constructing a superior forecast is simply a matter of collecting more data and identifying previously unseen correlations. However, humans are, by nature, pattern recognition machines. We look for patterns or correlations where none exists simply to impose a sense of order. British mathematician Frank P Ramsey (1903–1930) noted that this was why early humans made connections between stars billions of miles apart from each other. He also believed the need to impose order was the reason people claimed they found hidden predictions in the Bible by rearranging the format of the words;
- Crabtree’s Bludgeon. Another result of our compulsion to impose pattern and order is an ability to construct explanations when no good reason exists. A companion to Occam’s Razor (the simpler explanation is usually better), Crabtree’s Bludgeon states that ‘there is no set of data, however contradictory or absurd, from which the human brain cannot conjure up a plausible narrative.’ So, be suspicious of overly complex explanations, however much they might explain observed data;
- The overfitting problem. Where a forecaster has large amounts of data and powerful computing systems, it’s tempting to build a sophisticated forecasting program and factor in dozens of variables to model that data. However, these models frequently become over-sensitive to very small changes in the environment. They lose their predictive power because they’re overfitted with variable calculations. The Max Planck Institute for Human Development has demonstrated that a one-variable model can outperform a ten-variable model when predicting which consumers will repeat purchase music. This is down to the level of compound error introduced by each additional variable and a lower signal to noise ratio; and
- Variable omission problem. Any variable included in a forecasting model will have significant effects on its predictive ability. The problem here, though, is that factors such as human psychology are not easily expressible in statistical models. This means that even minor changes in uncaptured, or omitted, variables will have huge negative effects on the model’s predictive power. This is true even if nothing changes in the variables that have been modelled.
The super-forecaster approach
Don’t despair. Not all hope is lost. During his studies, Tetlock identified a small cluster of individuals with superior forecasting ability. Because these people consistently achieved results that were significantly better than random chance, Tetlock called this group the super-forecasters.
Tetlock examined the super-forecasters’ methods. He discovered that, by being aware of, and able to eliminate the impact of the biases described above, anyone can become a better forecaster by adopting this six-point approach:
- Break large questions down into smaller constituent parts. For example, if you’re looking to forecast demand in a particular country, consider GDP growth, the rate of inflation, growth in wages and other factors individually in a granular manner.
- For each factor identified in step 1, determine the base rate. In this context, base rate means the natural frequency or historical incidence of a particular event.
- Adjust the base rate for each factor and note how it differs from historic precedents.
- Identify areas of uncertainty and areas where you can work with experts to develop and improve your granular understanding. Choose experts who have a strong track record in these areas.
- Synthesise clashing viewpoints. This will increase the reliability of estimates for each component of your forecast.
- Correct past mistakes or any previous under- or over-shooting.
It’s also important to measure your forecasting as this will help you evaluate your own judgement and that of the experts you consult. Learn from success as well as failure. Being right is not enough – you need to understand why you were right and how much of your forecasting success was down to your process and how much was down to luck.
Finally resist the temptation to:
- Under- or overreact to emerging evidence in relation to your forecast;
- Fall prey to hindsight bias; and
- Overcompensate for past mistakes.
These points apply at the level of an economy, at the level of a company and at the level of forecasting and budgeting for your own team's resources and needs!
Conclusion
Economic and business forecasting (at national, corporate and at your team level) is fraught with difficulties and pitfalls. Some of these, such as linearity bias, Texas sharpshooter fallacy, WYSIATI fallacy and Crabtree’s Bludgeon are psychological while others, such as over-fitting, are down to the models used. The most effective forecasters are those who remain aware of these threats, work to minimise them and follow a set of proven techniques.