Forecast accuracy is not enough because businesses are paid for decisions, not predictions. Data Department, an applied machine learning consultancy, builds forecasting systems for retailers, distributors and manufacturers that report uncertainty, price in the different costs of being wrong in each direction, and test policies by simulation before they reach planners. A slightly less accurate forecast with honest uncertainty often produces better stock, pricing and staffing decisions.
What is wrong with judging a forecast by accuracy alone?
Most forecasting projects are judged by one number: MAPE (mean absolute percentage error) or a similar error metric. It is easy to compute and easy to compare. It also hides three problems.
First, it averages over items that matter very differently. A 20% error on a fast-moving, low-value item and a 20% error on a slow, expensive item have very different consequences.
Second, it treats over-forecasting and under-forecasting as equally bad. In almost every business they are not. A stockout of a critical spare part can stop a customer's vehicle for a week. An extra unit on the shelf costs a little holding cost.
Third, it ignores what happens after the forecast. A forecast is an input to an order, a price, a shift plan. If the rule that turns the forecast into a decision is poor, a better forecast barely helps.
Why does uncertainty matter more than the point forecast?
A point forecast says "we expect 12 units next week". A probabilistic forecast says "there is a 50% chance of 12 or fewer, a 90% chance of 19 or fewer". The second statement is what a planner needs, because the right stock level depends on how much risk the business accepts.
Two items can have the same expected demand and very different spreads. A stable item with expected demand of 12 might need 15 units for a 95% service level. A volatile item with the same expected demand might need 25. A point forecast treats them identically. A probabilistic forecast does not.
Uncertainty also builds trust. Planners override forecasts they do not believe. When they see a confidence band that widens for new or erratic items and narrows for stable ones, they learn when to rely on the model and when to apply judgement.
How do asymmetric costs change the decision?
Every forecast-driven decision has two kinds of error with two different prices:
| Decision | Cost of too little | Cost of too much |
|---|---|---|
| Spare-parts stock | Lost sale, emergency shipment, idle customer vehicle | Holding cost, obsolescence |
| Fresh food production | Lost sale, unhappy customer | Waste |
| Store staffing | Long queues, lost sales | Idle labour cost |
| Price setting | Margin left on the table | Lost volume |
The optimal decision is not the most likely outcome. It is the quantity at which the expected cost of being short equals the expected cost of having too much. When shortage costs ten times more than excess, the right stock level sits well above the average forecast. When the costs are equal, it sits near the middle. This logic only works if the forecast tells you the full range of outcomes, not just the centre. Our article on spare-parts inventory with probabilistic forecasting works through a numeric example.
What is the simulation layer between model and decision?
Between the forecast and the decision there is a policy: reorder points, safety stock rules, price ladders, staffing templates. Simulation tests those policies before they go live.
A simulation replays thousands of possible demand paths drawn from the probabilistic forecast, applies the proposed policy and records the outcomes: stockouts, holding cost, waste, revenue. You can then compare policies on the metrics finance cares about, not on forecast error. Typical questions simulation answers:
- What happens to service level if we cut safety stock by 15% on slow movers?
- How much extra inventory does moving from a 95% to a 98% service level cost?
- Which price change rule protects margin when demand drops?
Simulation is also where operational constraints enter: supplier lead times, minimum order quantities, shelf space, budget limits. A forecast cannot know about these. The policy has to.
What does this look like in practice?
In a project for a large automotive distribution group with more than 100,000 active spare parts, the legacy system produced point forecasts that ignored uncertainty. We moved them to probabilistic forecasting with prediction intervals per part and separate models for frequent, intermittent and high-value parts. Forecast accuracy improved by about 30% over the legacy method. The more important number was the result of using those forecasts in stocking decisions: about 10% lower inventory holding cost. Planners also got interpretable confidence bands. The full story is in the spare-parts case study.
The same principle applies outside inventory. In dynamic pricing, the measure of success is profitability, not price-elasticity model error. For an e-commerce retailer, this approach raised profitability by about 4% (pricing case study). In automotive customer retention, renewal and brand-churn models were judged by which customers were contacted with which offer, and the result was about an 8% revenue gain (vehicle renewal case study).
In each case the model mattered, but the gain came from connecting it to a decision and measuring the decision.
Which five questions should you ask before an ML project?
Before starting any ML (machine learning) project that involves forecasting, answer these five questions. They take an afternoon and save months.
- Which decision will the forecast feed, and who makes it? Name the order, price or plan, and the person accountable.
- What does it cost to be wrong in each direction? Even rough numbers, such as "a stockout costs us about five times what an extra unit costs", change the design.
- Do we need a range or a single number? If the decision depends on risk, you need a probabilistic forecast.
- Which constraints sit between the forecast and the action? Lead times, minimum order quantities, budgets, capacity.
- Which business metric will prove success? Holding cost, service level, waste, margin, revenue. Agree it on day one and measure the current baseline.
If you cannot answer the first and fifth question, the project is not ready. A free 30-minute AI opportunity review is a reasonable place to test these answers with someone outside the team.
Frequently asked questions
Is forecast accuracy useless then?
No. Accuracy is a useful diagnostic and a better model usually helps. It is just not the goal. Track it alongside the business metric the forecast is meant to improve.
What is a probabilistic forecast in simple terms?
It is a forecast that gives a range with probabilities instead of one number, for example "90% chance demand stays below 19 units". This lets you choose a stock level or plan that matches how much risk you accept.
Do we need simulation for every forecasting project?
Not always. For simple decisions with clear costs, a direct calculation is enough. Simulation earns its place when policies interact with lead times, capacity or budgets, or when you want to compare several policies before changing how planners work.
Which error metric should we use instead of MAPE?
Use a metric that matches the decision. For stocking decisions, quantile loss or service-level achievement is more informative; for financial impact, simulate the cost of the decisions the forecast produces.