Forecast Metrics

These are my notes on forecast metrics, written in three parts while working on a weekly sales forecast for a retailer.

Definitions

The 3 core metrics are:

Scale Dependent Metrics

MAE

MAE = (1/n) Σ |Actual - Forecast|

calculates the average errors [sales]

ex: if mae=3 -> on average, we miss/overforecast 3 sales per forecast

Advantages: very simple and understandable

Disadvantages: doesn’t reflect the fact that misses on high sellers are more important

BIAS

Bias or Mean error

BIAS = (1/n) Σ (Forecast - Actual)

-> gives additional information about the bias but not intended to evaluate accuracy because positive and negative errors cancel out

Technical Characteristics:

• Signed error: Positive = systematic over-forecasting, Negative = systematic under-forecasting
• Volume-weighted: Same limitations as WAPE
• Critical for inventory management: Direct impact on stock levels

Interpretation:

• Bias = 0: Perfectly balanced (but could have high variance!)
• Bias = +10%: Over-forecasting by 10% on average (excess inventory risk)
• Bias = -10%: Under-forecasting by 10% on average (stock-out risk)

Percentage Metrics

MAPE

MAPE = (1/n) Σ |(Actual - Forecast) / Actual|

calculates the average percentage error [%]

ex: if mape=0.2 -> on average, we forecast 20% more or less sales per forecast

Advantages: simple and understandable

Disadvantages: undefined for 0 sales (not an issue if 0 sales aren’t your primary concern)

Declinations

sMAPE: more robust against 0 sales, does not penalise positive and negative forecast errors equally (Hyndman)

-> 0 sales and 0 forecast are still a problem. if forecast <1 it explodes

sMAPE = (2/n) Σ |Actual - Forecast| / (|Actual| + |Forecast|)

MAAPE: is defined even when sales =0 (maape=pi/2)

-> doesn’t offer any value cause you can’t aggregate the value pi/2 with other values

MAAPE = (1/N) Σ arctan( |(A_t - F_t) / A_t| )

WAPE

WAPE: scaled to the sum of sales

WAPE = Σ_s Σ_t |Actual_s,t - Forecast_s,t|  /  Σ_s Σ_t Actual_s,t

s = 1..C series, t = 1..H weeks

Advantages:

• Almost always defined
• When aggregating: Gives more weight to high sellers than low sellers

Disadvantages:

• Meaningless when comparing different scopes, different time series (high vs low)

Technical Characteristics:

• Naturally volume-weighted: The denominator (total sales) means larger-volume SKUs contribute proportionally more to the error
• Scale-dependent: Cannot be compared across different product categories or markets without normalization
• Aggregation property: WAPE of aggregated groups ≠ average of individual WAPEs

Critical Limitation:

WAPE masks small-volume SKU performance. If 80% of your SKUs are low-volume but only represent 20% of total sales, their forecast quality is essentially invisible in WAPE. You must segment by volume cohorts or selling class to get meaningful insights.

How to use:

• Always in comparison on an equivalent perimeter in terms of sales volume
• Try to use on perimeters of equal sales volume

When NOT to use:

• Comparing forecast quality across different volume segments

Relative Metrics (relative to a naive forecast)

Each of these divides the forecast’s error by the error of a naive forecast. The literature calls MASE and RMSSE scaled errors: they divide by the naive’s error on the training history (Hyndman and Koehler 2006, FPP3 5.8). It calls RelMAE and RelMSE relative measures: they divide by the naive’s error over the same test weeks (Hyndman and Koehler 2006, Hewamalage, Ackermann and Bergmeir 2023).

MASE

MASE = MAE / MAE_in-sample,naive

MASE = (1/n) Σ_i |y_i - ŷ_i|  /  ( (1/(T-m)) Σ_{t=m+1..T} |y_t - y_{t-m}| )

The denominator is the MAE of the seasonal naive forecast (the same week one season earlier) over the T weeks of the training set, with m the season length, 52 for weekly data.

Technical Characteristics:

• Can be aggregated/disaggregated across any dimension
• Unweighted by default: Each SKU contributes equally regardless of volume
• Baseline comparison: Normalized against naive seasonal forecast performance
• Interpretable: 100 = same error as the in-sample naive, 50 = half the error, 200 = twice the error
• unit [-]
• <1 -> forecast errors smaller than the naive forecast’s errors on the training set
• >1 -> forecast errors larger than the naive forecast’s errors on the training set

Advantages:

• MASE can be meaningfully averaged across SKUs, time periods, and hierarchies. Unlike WAPE, you can compute MASE at SKU level, aggregate to category level, and the interpretation remains valid.
• Allows comparison between different scopes and different times series (high vs low)
• Gives functional insight into our model’s performance, are our errors smaller than those of a seasonal naive on the training set? by how much?
• “a consistent estimator provided the series is difference stationary, which is a much weaker condition than stationarity” R. Hyndman -> meaning it remains relevant if the series has a trend or a seasonality for example, contrary to the WAPE
• ex: you can still compare beginning of the season MASE to the middle and end of the season MASE, you can’t with WAPE

Disadvantages:

• Compares to the scale of naive forecast -> important to define that scale correctly
• New sellers
• “optimising the MASE will lead to the median forecast, which is probably not what you want, especially when you have intermittent demand time series” R. Hyndman
• For us, neither the mean nor the median is a good enough solution, the only solutions are to separate high or low sellers (or initialization/normalization)

RMSSE

RMSSE = sqrt( (1/n) Σ MSE / MSE_naiveModel )

RMSSE = sqrt( (1/n) Σ_j (y_j - ŷ_j)²  /  ( (1/(T-m)) Σ_{t=m+1..T} (y_t - y_{t-m})² ) )

Technical Characteristics:

• Penalizes large errors quadratically: One 20-unit error worse than four 5-unit errors
• Variance detector: Sensitive to outliers and inconsistent performance
• Scale-independent: Same aggregation properties as MASE

Advantages:

• Optimizing it leads to the mean rather than the median
• comparing RMSSE to MASE shows if the model is robust to outlier predictions, the more the model produces outlier prediction the higher the RMSSE compared to the MASE

Disadvantages:

• Same as MASE

RelMAE

RelMAE = MAE / MAE_naive = Σ_j |y_j - ŷ_j|  /  Σ_j |y_j - ŷ_naive,j|

The MAE of the forecast divided by the MAE of the naive forecast, both over the same test weeks j. <1 -> our errors are smaller than the naive’s on those weeks, >1 -> larger. unit [-]

RelMSE

RelMSE = MSE / MSE_naive = Σ_j (y_j - ŷ_j)²  /  Σ_j (y_j - ŷ_naive,j)²

The same ratio with squared errors, so large misses weigh more, as with RMSSE compared to MASE. There is no square root, so it moves further from 1 than a ratio of RMSEs would.

Weighted Metric : wRMSSE / wMASE

Weighted metrics allow the aggregation of the metric across multiple time series using a certain weight, the higher the weight of a given time series, the more it influences the aggregated value of the metric

for example for the M5 competition, they used:

w_i = Σ_{t=n-27..n} (p_t,i × Y_t,i)  /  Σ_j Σ_{t=n-27..n} (p_t,j × Y_t,j)

WRMSSE = Σ_{i=1..42,840} w_i × RMSSE_i

where p(t,i) and Y(t,i) are the price and units sold of item i at day t

so for products that are very rarely sold or are very cheap the weights are very close to 0

In our case, we’ve identified 3 types of weights that could be of use: price, volume and turnover (price x volume). These weights are computed as the average value on the last 2 years of historical data (or whatever duration is available)

Interpretations & Examples

MAE vs MAPE vs WAPE

WAPE favors the high sellers when aggregating over high and low sellers

MASE vs WAPE

High seasonal sellers

• Bias doesn’t inform regarding the accuracy of the prediction
• MAE and MAPE are useful and easy to interpret
• WAPE is relevant here since we’re computing for one time series
• MASE/RMSSE shows clearly how bad our model is compared to the naive forecast

Low seasonal sellers

MASE: the naive does not get a score of exactly 1 (here 1.1017) cause the denominator is calculated on the training set

RMSSE vs MASE

Case 1: RMSSE > MASE

Possible interpretations:

• High error variance (inconsistent performance)
• Presence of outliers or edge cases
• Model struggles with specific conditions (promotions, new products, etc.)
• Risk indicator: Some forecasts are very wrong

Case 2: RMSSE <= MASE

Possible interpretations:

• Low error variance (consistent performance)
• Fewer outliers than naive baseline
• Quality indicator: Predictable error behavior

Weighted vs Unweighted

MASE_volume < MASE_unweighted (or RMSSE)

• Forecast performs better on products that matter (product with high volume)
• Good business outcome

MASE_turnover < MASE_unweighted (or RMSSE)

• Forecast performs better on products that matter (product with high turnover, either high volume or high price or both)
• We can evaluate the improvement of the business outcome

Week-by-Week Progression

Why Track MASE/RMSSE Week-by-Week? Unlike WAPE, MASE and RMSSE are unweighted by volume, making weekly tracking meaningful:

MASE/RMSSE weekly trends show pure forecast quality evolution, independent of business volume changes. This is critical for model monitoring and A/B testing.

MASE/RMSSE are scale free meaning Each time series MASE is normalized by its own baseline so the interpretation remains consistent across different levels (high vs low season)

Aggregating

Below is an example where 4 types of time series are studied: mix of flat/seasonal and high/low axes

For the remainder of the plots, our forecast (in red) is average value over the training set

Filtering on the high sellers seasonal

• same denominator for all time series -> wape consistent with MAE and MAPE
• errors of the naive forecast vary on the training set from one high sellers to the next -> the trend is less consistent for MASE

MASE reflects that our forecast is far from the ideal forecast, wape doesn’t very much

Filtering on the high flat

MASE reflects that our forecast is very close to the basic (naive) forecast.

Filtering on all the dataset

4 different blobs are distinguishable in the MASE plot, the 2 blobs with the lowest MAE and MASE correspond to flat series when our average forecast provides a pretty good accuracy. The other 2 blobs correspond to the seasonal series for which the denominator is high due to varying naive forecast errors. The blob with the highest MASE and MAE is for high seasonal and the other for low seasonal.

Notice that for the WAPE, we also have 4 blobs but they are inverted, the ones with the highest WAPE are for flat series while the lowest WAPEs are for seasonal. This is counterintuitive, in fact, from the plot we can clearly see that our forecast (mean over training set) is way better for flat series than for seasonal but the WAPE doesn’t reflect that.

Conclusion

Metrics differ in sensitivity: Absolute ones (e.g., MAE) treat a missed sale equally regardless of volume, which might undervalue high-seller impacts, while relative ones (e.g., MAPE) amplify errors on low sellers but can distort with zeros. WAPE, which you’re using, weights by volume, making errors on high sellers more prominent

• MAE for individual cases and comparison across time series
• MAPE can be useful for comparing high volumes series
• MASE (wMASE) very useful for both individual cases and comparisons across different perimeters
• WAPE (wWAPE) as a business facing metric

For our data science use case, we believe that MASE, RMSSE, BIAS and WPE are essential tools that allow us to evaluate the efficacy of a new modeling strategy. WAPE remains a business facing metrics and can also be used as a data science metric when comparing similar scopes.

Decision matrix

Below is a set of criteria to verify during each modeling experiment

Replacing volume with turnover in the table above is equally valuable as it gives a more direct business value estimation.

Moreover, we can also examine the following:

• week over week MASE/RMSSE progression
• RMSSE on seasonality and key item cohorts
• MASE on high sales cohorts

Business KPIs

In order to evaluate our performance from a point of view of business, we’re suggesting computing bias metrics

We would like to quantify the improvement of our modeling strategy from a business perspective. We want to use turnover information to conduct an analysis of potential gains unlocked with the forecast provided

Based on experts’ estimates from operational experience and supply chain economics, the cost incurred by an equivalent magnitude of underforecasting is estimated to be 3 to 4 times higher than the cost generated by overforecasting (represented primarily by the costs of the additional stock that must be kept and managed).

This fundamental asymmetry necessitates the processing of under and over forecasting differently so that we may evaluate each part. In conclusion:

• we need to augment our bias metrics with the ability to compute in terms of turnover as well as volume
• we need to revise and augment the definition of Bias to separate the under and over forecasting components

Beyond Bias

The current Bias is aggregated on the entire dataset and compensates under and over forecast to give a final value which represents the tendency of the entire forecastable time series. Even with the addition of cohorts, which allow to see a bit clearer the tendencies of bias for the different cohorts. We’re still missing the under and over forecasting components essential to evaluate the business impact but also to evaluate the weaknesses of the model (ex: underforecasting a specific family while overforecasting when we aggregate on the department level)

For these reasons, we propose the definition of SignedBias which allows:

• The computation of over and under or total bias components separately.
• The computation in the unit of volume or in the unit of turnover (by multiplying by the average reference price across the training set for a given sku, store)
• The computation of these metrics in terms of % of the total sales (normalised) or in terms of absolute value (not normalised)

The over component keeps only the errors where the forecast is above sales, the under component only those where it is below.

ex:

Censored Demand & Models Backtesting

During the testing of a new modeling strategy (model2) vs an existing model (model1) we face an immediate issue for a fair comparison. model1 has been used (partially or entirely) for the definition of available stock for items x stores. This is the issue of censored demand:

• Past forecasts influenced inventory → influenced observed sales
• observed_sales = min(true demand, inventory) with inventory = fct(model1 forecast)
• Naive comparison of model1 and model2 predictions vs historical sales is invalid as it has a bias showing that model1 underforecasts less than it actually does.

This is particularly apparent after the definition in the section above of under and over forecast components.

Therefore we have 2 solutions:

• Compare on reconstructed sales: This present has some drawbacks related to the fact that reconstruction isn’t as accurate as true demand.
• Compare periods of available stocks: precisely periods where stocks > sales. This also has some drawbacks because items that have stocks > sales are a group that doesn’t necessarily represent the entire population (it might over represents low sellers for example)

No solution is perfect, however the evaluation on available stocks seems closer to reality. We recommend the use of this solution (in addition to the first solution if we want a bigger picture) for cases of backtesting of a new modeling strategy

We compared the results between normalization and the previous model using the 2 methods (sales reconstructed and true demand) the result confirms that there is a reduced underforecast when using sales reconstructed for the prod model but it’s a very slight difference because we’re using sales reconstruction. If raw sales are used, the difference should be larger

Improved MASE / RMSSE

See MASE and RMSSE above for the definitions and implementation of the current mase/rmsse metrics.

We’ve observed that in some cases, the values of mase/rmsse are significantly higher than 100 which would mean that our forecast is really bad compared to a naive forecast. However, upon examination of the forecasts in question, it’s not the case.

Example (a made-up series, not the real one):

Here our forecast is not perfect, or more particularly, sales were difficult to predict since the peak is totally unexpected.

As we observe, the real problem is that during the test phase in question, we observe an unusual peak (a non seasonal peak that hasn’t been witnessed during training). This makes the numerator (calculated on the test set) of the mase/rmsse very high compared to the denominator (calculated on the train set). Since the denominator is used to gauge the performance of the naive forecast against which we compare our model forecast, this makes our model appear worse than the naive one when in fact it’s the test phase that is harder than the train set.

To remedy this issue, we propose to calculate the mase/rmsse for the naive forecast as well so that we can compare the performance of our forecast & that of the naive in the test phase compared to the performance of the naive on the train set.

• mase_naive > 100: The test phase is harder to predict than the train phase
• mase_naive <= 100: The test phase is as difficult to predict as the train phase
• mase < mase_naive: Good Forecast: Our forecast is better than naive forecast
• mase >= mase_naive: Bad Forecast: Our forecast is worse than naive forecast

This analysis is the same for Mase/Rmsse with or without weights.

Rq:

• we’re using last year’s sales averaged over the 3 weeks centred on the same week at horizon 1 (or, when missing, the average of last month’s sales) as the naive forecast for the train set but this is an approximation since for each h we should use the corresponding horizon h column
• Since the normalization, we’re denormalizing columns like last year’s centred 3-week average with mean and std of sales which should be quite close to the mean and std of the column in question but isn’t exactly equal.

I’ve improved the implementation of MASE/RMSSE by calculating the naive statistics per sku, store, cutoff, horizon. I then calculated the MASE/RMSSE by aggregating on (series, horizon) first then on all horizons second.

MASE/RMSSE usefulness

mase naive and rmsse naive tell us how the test set is compared to the training set, however in practice, often we find that mase and rmsse naive are:

• lower for low selling sku/store and higher for high selling sku/store because naive is better at predicting when it’s most random or least predictable (low sales, high sparsity)
• Most of the time rmsse naive or mase naive aren’t exactly that close to 100 (the naive’s test error can be lower or higher than its training error) with mase highest and rmsse lowest.. they do tell in specific cases something interesting but you’d have to dig deep to find that
• comparing mase with mase naive and rmsse with rmsse naive tells u the same thing as relMae and relMse
• what are we looking for with mase naive and mase is cases where mase naive < mase but also mase naive <100 -> those would indicate a problem with our forecast cause the naive is doing better and the test phase is as easy or easier than the train however in practice this is almost never realized
  • what we do have is
    • mase100
    • mase < or > mase_naive and mase_naive <100

therefore having just relMae & relMse is good enough

First MASE table, each model relative to the naive (naive = 100):

cases where naive forecast <100 and naive forecast > prod model (metrics computed per family x country x sales cohort x horizon x cutoff)

scatter plot of ranks of the worst performances of prod model computed using both metrics -> correlation ~= 70%

I implemented SignedBias class & RelMse and RelMae

SignedBias allows to compute the cost of overforecast & underforecast independently

RelMAE (Relative Mean Absolute Error): mae(forecast)/mae(naive)

RelMAE relative to the no weather model (no weather = 100):

RelMSE (Relative Mean Squared Error): mse(forecast)/mse(naive)

RelMSE relative to the no weather model (no weather = 100):

All three models are below the naive in every row, for both metrics.

Both produce a single percentage. < 100 means your model beats the naive baseline; > 100 means it’s worse.

• Scale-agnostic. Unlike MAE or MSE which are in units of sales quantity or WAPE which depends on the volume of sales.
• RelMAE penalizes all errors linearly, RelMSE squares errors first penalizing larger errors quadratically
• Replaces MASE/RMSSE which required computing the naive’s training MAE/MSE over the full training window, a separate preprocessing step with its own SQL, seed data, and maintenance burden. RelMAE/RelMSE compute the naive error directly against the same prediction window, using the naive forecast already present in every forecast output.

We compared the gains of Wape & RelMse of weather implementation vs no weather (metrics computed per product category x sales cohort x horizon x cutoff, compared per product category)

as a result of its Scale independence, RelMse gives more homogeneous gains across categories (fewer outliers) than Wape when aggregating across time series with different sales volumes

References

• A Comprehensive Survey of Time Series Forecasting
• Measuring forecast accuracy: The complete guide
• Ultimate Guide to Forecast Accuracy Metrics
• WAPE vs MAPE for Forecast Accuracy Reporting
• Another Look at Forecast-Accuracy Metrics for Intermittent Demand
• Measuring Intermittent Demand: Forecast Accuracy and Inventory Metrics
• TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods
• Rob Hyndman, WAPE
• Time series forecast error metrics you should know
• Amazon Forecast, evaluating predictor accuracy
• Rob Hyndman, sMAPE
• Hyndman and Koehler (2006), Another look at measures of forecast accuracy
• Hyndman and Athanasopoulos, Forecasting: Principles and Practice (3rd ed.), 5.8 Evaluating point forecast accuracy
• Hewamalage, Ackermann and Bergmeir (2023), Forecast evaluation for data scientists: common pitfalls and best practices