Two numbers ran the weekly review of a demand forecasting system I worked on: a volume weighted error and an aggregate bias. On the synthetic panel in this article, built so that the answer is known before the metrics are computed, those two numbers read 7.6 percent and plus 0.2 percent. I have never seen a review challenge a pair of numbers that look like that. The same forecast has a higher mean absolute error than a seasonal naive on 77 of the 116 slow moving series where the comparison is defined, and it is 47.1 percent long on them.
The strongest argument against this article is that the two aggregates are correct. If the cost of a forecast error is roughly proportional to units, then weighting an error by units is the honest weighting, and a percentage error averaged evenly over series is the distortion: it lets an item selling 0.2 units a week shout as loudly as one selling several hundred. Rob Hyndman’s note on WAPE makes the point that WAPE is exactly a percentage error with volume weights, “where the weights are given by w_t = |y_t| / sum_t |y_t|”, and that unlike MAPE it stays defined when actuals are zero. That is a real property, not an accident, and it is why the metric survives contact with an intermittent assortment at all.
The counterargument I take least seriously is the one about metric sprawl, and I take it somewhat seriously. Every metric added to a dashboard is a number someone can select on, and a dashboard with a dozen panels is one nobody reads. The case below is not for more metrics. It is for three specific ones that answer questions the first two structurally cannot answer, and for computing all of them at the grain where the decision is taken rather than at the grain where the number is prettiest.
The two numbers, stated precisely
The volume weighted error is the sum of absolute errors divided by the sum of actuals across every series and every period in the evaluation window. The aggregate bias is the same sum with the absolute value removed, so positive and negative errors cancel before the division. In the language of residual diagnostics, a bias is a non zero mean of the residuals, and the textbook remedy is a shift: “Adjusting for bias is easy: if the residuals have mean m, then simply add m to all forecasts.”
Both numbers are sums before they are ratios. That is the whole problem. A sum answers one question and destroys the information needed to answer the next one. The volume weighting destroys the identity of the low volume series. The signed sum destroys the direction of the error. Neither is a bug in the formula, and neither can be fixed by choosing a different aggregate.
A panel built so the failure is known in advance
The reproduction is one file, standard library only, seed 20260919. It builds 136 weekly series with 156 weeks of history and evaluates a 13 week horizon:
- 8 high volume series with annual seasonality, with a base level of about 90 to 270 units a week.
- 120 intermittent series, each with a per week probability of selling anything between 0.06 and 0.48, and one to three units when it does.
- 8 items with no sales history at all, which start selling inside the evaluation window.
The forecast is then written by hand rather than fitted, so that the failure is a known quantity and the metrics are the thing under test. The high volume series get their own actual value multiplied by lognormal noise with a 1.3 percent systematic shortfall. The intermittent series get a flat forecast equal to 1.62 times their trailing yearly rate, which is the plausible failure shape for a series the model has almost no signal on: smooth, positive and too large. The new items get a flat 0.35 units a week against demand that does arrive.
The three metrics are eight lines of arithmetic:
def wape(actual, pred):
return sum(abs(a - p) for a, p in zip(actual, pred)) / sum(actual)
def bias(actual, pred):
return sum(p - a for a, p in zip(actual, pred)) / sum(actual)
def signed_halves(actual, pred):
over = sum(max(p - a, 0.0) for a, p in zip(actual, pred))
under = sum(max(a - p, 0.0) for a, p in zip(actual, pred))
return over / sum(actual), under / sum(actual)Running the script prints the two reported numbers, then the three that were missing:
the two numbers the programme reported
volume weighted error WAPE 7.6 percent
aggregate bias 0.2 percent
the three numbers it did not
pooled relative MAE, all 0.683 (1.000 = the seasonal naive)
signed bias, over half +3.9 percent of units
signed bias, under half -3.7 percent of unitsThe panel sold 22,410 units in the horizon. The 8 high volume series account for 21,768 of them, or 97.1 percent. The 120 intermittent series account for 559 units, 2.5 percent, and the new items for 83 units, 0.4 percent. So the volume weighted error is, to within a rounding error, the error on eight series.
The aggregate is a decision to not look at a distribution
Split the same errors by segment and the picture separates cleanly. The high volume segment has a 3.4 percent weighted error. The intermittent tail has 160.3 percent. The new items have 91.6 percent. A single 7.6 percent sits on top of a range spanning two orders of magnitude, and it sits there legitimately, because the weights say so.
The useful comparison is not the raw error but the error against something a human would accept without a model. I used a seasonal naive: the forecast for week t is the actual value from week t minus 52. Forecasting: Principles and Practice defines it in one line, it costs nothing to compute, and it is the benchmark that scaled errors are built on. The interpretation is the one thing everyone in the room already agrees on: below one is better than the naive, above one is worse.
Figure 1. The forecast beats the seasonal naive on all 8 high volume series and loses to it on 77 of the 116 intermittent series where the ratio is defined. The failure is concentrated exactly where the volume weighted aggregate assigns almost no weight.
Per segment, the pooled ratio of summed absolute errors is 0.455 on the head and 1.120 on the tail. On 4 of the 120 intermittent series the seasonal naive was exactly right in all 13 weeks, so no ratio exists at all, and the honest move is to report that count rather than silently drop those series from the denominator.
Here is the part I did not expect when I first ran this, and it is the reason a baseline alone is not the fix. The pooled relative error over the whole panel is 0.683. Carried to monitoring as one number, the baseline comparison passes too. It is volume weighted, like everything else, and the hand written forecast is close to an oracle on the head by construction, which flatters the pooled ratio exactly where the volume sits. A baseline is necessary. Segmentation is what makes it load bearing.
A net bias of zero is an accounting identity
The aggregate bias of plus 0.2 percent is not an approximation of anything. It is an exact sum: the head is 175 units short, the tail is 263 units long, the new items are 47 units short, and minus 175 plus 263 minus 47 is plus 41 units on 22,410.
Figure 2. The same errors, read seven ways. The reading that says the forecast is calibrated is the only one computed by summing signed errors across the whole assortment first.
Two repairs, both cheap. The first is the directional split: stop summing signed errors and sum the positive part and the negative part separately. On this panel that is plus 3.9 percent and minus 3.7 percent of units. Those two halves are not a decomposition of the bias into causes, and it is worth being clear about that, because part of each half is ordinary noise on the high volume series rather than systematic direction. What the split does buy is that the two halves cannot cancel, so a large mutual offset is visible before anyone asks for it.
The second repair is to change the weight. Units are one denominator among several, and it is the one that treats a cheap consumable and an expensive item as interchangeable. Price the same errors at each item’s unit value, where the intermittent items in this panel are the expensive ones, and the aggregate bias moves from plus 0.2 percent to plus 14.9 percent, with a 39.4 percent over forecast half against a 24.5 percent under forecast half. Nothing about the forecast changed. The question changed, from how many units are we off to how much stock value are we committing, and those are different conversations with a replenishment team.
Magnitude, direction and cost are three questions. One number answers one of them.
Why I would not reach for MASE or RMSSE here
Scaled errors are the principled answer to comparing across series, and they are not a free lunch on a sparse assortment. MASE divides each series’ error by the in sample mean absolute error of a naive forecast on that series, which makes it scale free and gives the same below one or above one reading as a direct baseline comparison. RMSSE, the squared error variant used in the M5 competition, is built the same way.
The catch is the denominator. On this panel, 8 of the 136 series have a scaling denominator of exactly zero, because an item with no history has no naive error to scale by, and the metric is undefined for them. That is not an artefact of a synthetic panel: the M5 organisers had to restrict the denominator to “the periods following the first non-zero demand observed for the series”, precisely because products start selling at different times. Hyndman introduced MASE in 2006 partly to escape the failure mode where a relative error metric on an intermittent series would “involve division by zero”, and the escape is partial rather than complete.
Over the 128 series where it is defined, the median MASE is 1.15, the 90th percentile is 1.79 and the maximum is 4.37. That distribution is more informative than the aggregates, and it is also a per series ratio, which means whoever averages it inherits a heavy tail and an arbitrary decision about the eight undefined entries. The pooled relative error avoids that specific trap by summing numerator and denominator before dividing, which is well defined as long as the baseline makes at least one error in the segment.
There is a deeper problem with absolute error on intermittent series, and it applies to MASE, WAPE and the pooled relative error alike. The M5 paper states it plainly: absolute errors “are optimized for the median”, so on a series that is mostly zeros they “assign lower scores (better accuracy) to forecasting methods that derive forecasts close to zero”. The optimal forecast under the metric is a flat zero. That is a correct median and a useless replenishment instruction. If the decision needs an expected quantity, the evaluation has to include something optimised for the mean, or a bias term, or both. This is why I would rather carry a relative error plus a signed bias than a single scaled score: two numbers that disagree about what good means are harder to satisfy by accident.
What this does not show
The panel is synthetic and the failure was injected. It demonstrates that these metrics can hide this failure, not how often real systems contain it, and not that any particular production forecast does.
One panel, one horizon, one set of generator constants. Raising the tail’s share of volume by an order of magnitude would change every number in the article. The conclusion does survive redrawing the panel, which the script checks rather than asserts:
seed WAPE bias tail rel tail series losing to the naive
20260919 7.6% +0.2% 1.120 77 of 116
1 8.6% -0.4% 1.087 72 of 114
7 8.1% +1.0% 1.123 70 of 114
42 8.9% +0.3% 1.084 69 of 112Across those four draws the aggregate error stays between 7.6 and 8.9 percent, the aggregate bias stays between minus 0.4 and plus 1.0 percent, and the tail keeps losing to the seasonal naive with a pooled ratio between 1.084 and 1.123. That is four draws from one generator, not an estimate of any population.
The segmentation into head, tail and new items is a choice I made, and it is the choice that does the work. A different cut, by lead time or by supplier or by store cluster, would expose a different failure and hide this one. There is no segmentation that exposes everything, which is the honest version of the whole argument: you see the failure you decided to look for.
Value weighting needs a price, and in a real system that price is itself an input with its own quality problems. It moves the metric’s denominator into the blast radius of a second data source.
The checklist
- Compute every headline metric at the grain where the decision is taken, and publish the segment table next to the aggregate rather than behind it.
- Carry a naive baseline all the way into monitoring, not just into the model selection notebook. A model that cannot beat a seasonal naive on a segment is a finding, not a rounding error.
- Report the count of series where the baseline comparison is undefined. That count is a data quality signal about new and dead items.
- Split signed bias into its positive and negative halves. Keep the net if people are used to it, and never let it be the only one.
- Publish bias under at least two weights, one in units and one in value, and state which one the target is set against.
- Before adopting a scaled metric, count the series where its denominator is zero or near zero on your own data, and decide what happens to them.
What is still unresolved
I do not have a defensible rule for choosing the segmentation. Volume deciles are easy to compute and easy to game, because the deciles move when the forecast moves. A fixed classification by demand pattern is more stable and goes stale. I have used both and prefer the second, without being able to show that the preference is right.
The harder open question is which target the tail should be held to. Telling a team that their intermittent segment scores 1.120 against a seasonal naive invites the answer that intermittent demand is unforecastable, which is sometimes true. Distinguishing a segment that is genuinely at the noise floor from one that is merely neglected needs an estimate of the achievable error on that segment, and a naive baseline is a floor rather than a ceiling. That estimate is the piece I still do not have a clean way to produce.
Primary sources
- Rob J. Hyndman, WAPE: Weighted Absolute Percentage Error, for the weighted percentage error identity and the behaviour of WAPE at zero actuals.
- Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice, 3rd edition, section 5.8, for scale dependent errors, percentage errors, scaled errors and the below one or above one reading of a scaled error.
- Hyndman and Athanasopoulos, section 5.2, for the naive and seasonal naive definitions used as the baseline here.
- Hyndman and Athanasopoulos, section 5.4, for bias as a non zero residual mean and its correction.
- Spyros Makridakis, Evangelos Spiliotis and Vassilios Assimakopoulos, The M5 Accuracy competition: results, findings and conclusions, for the RMSSE definition, the restricted denominator, the dollar sales weighting of WRMSSE, and the argument about absolute errors and the median on intermittent series.
- Rob J. Hyndman, Another look at forecast accuracy metrics for intermittent demand, Foresight issue 4, June 2006, for why relative error metrics break on intermittent series and what MASE was introduced to fix.