Skip to content

Testing Tabular Foundation Models on a Real Sales Forecast

Updated:
•25 min read

I tested tabular foundation models (TFMs) against the LightGBM model we use to forecast weekly store sales, in two tests: a small experiment, then a backtest (forecasting from past dates and comparing with what sold) against the model in production. A third test runs a simpler version of the comparison on public data, so anyone can rerun it.

In the backtest, TabICLv2 (Inria’s TFM, the one I tested most) came within one WAPE point (weighted absolute percentage error) of LightGBM on every sales band (products grouped by how much they sell) but the slowest sellers, and never beat it. The only thing I tuned was the choice of context rows, and it mattered: with random rows, TabICLv2’s forecasts ran 9% of sales lower than LightGBM’s on the best sellers (a bias gap, not an accuracy one). On public data it was a draw. It is much slower to serve and harder to explain.

How these models predict, step by step, is in the companion post, How a tabular foundation model predicts, step by step. This one is about using them on a real case.

The models

With LightGBM, fit() is where the work happens. With a TFM, fit() mostly stores your table, and the work happens in predict(), where the stored rows (the context) and the rows you want answered go through the network together. The weights never change. In short (each point is explained in the companion post):

How much context they take, for TabPFN:

The TabICLv2 paper runs million-row tables too, with CPU or disk offloading.

Kumo-Tabular is NVIDIA’s model, released on 29 September 2026 in three sizes (28M to 215M parameters) by a team that includes TabICL’s first author; its name comes from Kumo AI, whose purchase by NVIDIA was reported in June 2026. It combines the TabPFN and TabICL designs (how, in the companion post), was pretrained on contexts of up to 60,000 rows, and NVIDIA warns accuracy “may degrade on tables far beyond the training ranges or when the query rows come from a different distribution than the context rows”, which is the normal case when forecasting.

It ships in NVIDIA’s structured-data-models library, next to TabICLv2, TabFM and a relational model (KumoRelational, built on KumoRFM-2).

Licences

In practice, for a commercial forecast, that leaves TabICL, TabPFN v2 and Kumo-Tabular, unless you buy a commercial TabPFN licence.

Leaderboard

The reference is TabArena, a continuously updated benchmark of tabular models.

What to expect before testing

Accuracy

From the first TabPFN paper: TabPFN learns smooth decision boundaries. On small classification datasets (up to 1,000 training rows, 100 numeric features and 10 classes), that paper reports TabPFN clearly beating boosted trees and performing on par with state-of-the-art AutoML systems (such as Auto-sklearn 2.0, which searches pipelines with Bayesian optimisation and ensembles the best), up to 230 times faster. TFMs learn a lot from few rows: the Nature paper shows TabPFN matching default CatBoost’s accuracy with half the training samples.

The papers show this with decision boundaries on toy datasets. I recomputed that picture with TabICLv2 instead of TabPFN (TabPFN’s current weights are gated behind a licence click-through and a login); TabICL in these two figures means TabICLv2. Every panel below is a model trained for this post, seed 0:

Decision boundaries of a decision tree, LightGBM, nearest neighbours, an MLP and TabICL on moons, circles and linearly separable toy data. The tree and LightGBM draw axis-aligned steps, the MLP draws smooth curves, and TabICL's boundary follows the data but with ragged, streaky edges rather than a smooth curve.

Toy data. The number in each panel is accuracy on held-out points; with 40 to 80 held-out points, a difference of 0.03 is one or two points.

The TabPFN papers also show it fitting a very broad class of functions, where traditional methods struggle to fit some of them. I ran that too, on six toy curves:

Four models fitted to six toy 1-D curves. TabICL follows all six shapes, including the step and the kink, and its 10 to 90 percent band widens where the noise grows. LightGBM follows them with small steps, the MLP smooths the step and the kink, and the linear model only fits straight lines.

TabICL follows every shape, including the step and the kink of |x|, with no tuning. LightGBM gets there as a staircase that also chases the noise. The neural net rounds off the step. TabICL’s 10% to 90% band widens on the last panel, where the noise grows with x: it returns a distribution, not just a number.

A smooth model suits continuous effects and small tables. Trees cut sharp thresholds, such as a promotion flag, a price break or a stock-out, exactly; TabICL draws them slightly rounded (the step panel in the second figure). Retail has both, so the two kinds of model could make different mistakes. In my backtest their forecasts mostly moved together.

Speed

TFMs are much slower at inference than LightGBM. In a credit-risk benchmark by Mission Lane, a US credit-card company, LightGBM on a 64-vCPU machine was 100 to 1,000 times faster than the TabICL version they tested on T4 and A100 GPUs, and the gap widened for single-record predictions. In my tests LightGBM was at least 100 times faster than TabICL; Test 3 measures the gap on public data. In production you will probably want a GPU: TFMs are much faster there, though even on a GPU they come up short against LightGBM on CPU. On CPU, the tabpfn package refuses runs above 1,000 rows (5,000 for TabPFN-3 and later) unless you set TABPFN_ALLOW_CPU_LARGE_DATASET=1 or ignore_pretraining_limits=True.

Most of the time goes into reading the context, not the rows you ask about. And the more context you have, the slower it gets, faster than linearly: the attention between context rows grows with the square of their number (see How the cost grows).

Between the two TFMs, TabICL was 3 times slower than TabPFN on CPU in my Test 1 notes; the TabICLv2 paper reports the opposite for TabICLv2 against TabPFN-2.5, on all the hardware it tried at 100 features, CPU included.

Features

The feature set we built for LightGBM works unchanged with a TFM, and Test 1 shows that some choices, like the year feature, matter for it too, and so does normalising the target.

Text columns work if you embed them first with an LLM embedding model and pass the embedding columns as numeric features. Without that, TabPFN-3.5, the package default, expands a pandas string column with more than 30 distinct values into up to 30 features (tf-idf over character n-grams, reduced by SVD; its checkpoint turns this on), while an object column is still ordinal-encoded. Older TabPFN checkpoints ordinal-encode free text as a category and warn, and only TabPFN-3-Plus, through the Prior Labs API or an enterprise licence, reads string columns natively. TabICL treats strings as categories too. Text-aware TFMs also exist: ConTextTab and TabSTAR.

Time series

Use a time-series extension (TabPFN-TS, or TabICL’s own TabICLForecaster, which its README calls heavily inspired by TabPFN-TS: pip install "tabicl[forecast]"), or add time-based features (we already do week of year, year, month). TabPFN-3.5 also expands a column with a datetime dtype (not a date string) into calendar features itself.

Two ways to forecast time series with a tabular foundation model. One: put rows from all series in one table with time features such as week of year, month, year, last week's sales and weather, and fit once, so series can inform each other. Two: TabPFN-TS takes one series at a time, builds features from its dates (a running index, calendar features, seasonality found by a Fourier transform, plus covariates known ahead) and fits TabPFN on that series alone, repeated for every series, with no learning across series.

Time-based features. Put every series in one table, with calendar columns and anything else you know, and fit once. This is what we do for LightGBM, and how I ran all three tests below (Test 3 has a single series). With many series, the context mixes their rows, so one series can inform another.

TS extension. TabPFN-TS uses a general TabPFN model underneath (the paper used plain TabPFN v2; the package defaults to TabPFN-3.5 as of October 2026, with an optional checkpoint fine-tuned for time series), and does some signal processing on top. The pipeline:

  1. For each of the N time series, it builds a table from that series’ own history: a running index, calendar features, seasonal features whose periods are detected with a Fourier transform (FFT), and your covariates.
  2. It calls TabPFNRegressor.fit(X_series_i, y_series_i) for each series.

So it is N separate fits of the same network, each seeing only its own (store, product) history: no cross-series learning at all, unlike regular tabular models.

Two more limits from its documentation: it only uses covariates whose future values you supply (past-only and static covariates are dropped), and a multivariate target is split into independent univariate forecasts.

In forecasting terms, TabPFN-TS is a local model (one per series) and LightGBM or TabICL on the full table is a global one. Time-series foundation models such as Chronos-2 are a third kind: pretrained on many series, and able to learn across the series they are given at prediction time (cross-learning). That is for a separate post.

Explainability

Use SHAP. For TabPFN, the separate tabpfn-extensions package has a SHAP integration (pip install "tabpfn-extensions[interpretability]"); TabICL has its own tabicl.shap module (pip install "tabicl[shap]"), which masks a feature by setting it to missing (TabICL then imputes it) instead of drawing it from background rows. For a tree model SHAP is exact and near instant; for a TFM each explanation costs many full predictions; caching the context (kv_cache) cuts that cost.

How I measured

Test 1: a small experiment

Scope:

TFM out of the box. Without choosing the training samples, it can perform better than LightGBM in some cases and worse in others.

Training samples (the context). More training samples do not necessarily perform better (chosen randomly). This choice may depend on the size of the initial dataset. I ran TabPFN with growing numbers of random context rows on two datasets of different sizes: in run 1 the context grew from 1,000 to 91,000 rows, in run 2 from 1,000 to 9,000.

Two runs of Test 1. In run 1, from 1,000 to 91,000 context rows, the indexed error falls from 100 to about 80 at 31,000 rows and then stays between about 82 and 86. In run 2, from 1,000 to 9,000 rows, it rises from 100 to about 114.

Error (MAE) indexed so 100 is each run’s error with 1,000 rows.

In the first run the error fell up to roughly 31,000 rows, then stopped improving. In the second, almost every step up from 1,000 rows made it a little worse.

TabPFN with a smart context (the same 10-week period last year plus recent weeks, what Test 2 calls a seasonal context) gave about 5% less error than LightGBM on the first forecast week, but it under-forecast more than LightGBM.

Problematic features. The year and week of month features (to a lesser degree) held the forecast down for the high-sales week. Effect of removing year, on the first forecast week, in a separate run from the smart-context one above, so its numbers differ:

Error with yearError withoutBias with yearBias without
LightGBM10084too lowtoo low, half as much
TabPFN88100too low, a little less than LightGBMtoo high

Error is WAPE indexed so LightGBM with year = 100 (TabPFN without year happens to equal it). Lower is better.

With year, TabPFN had less error than LightGBM. Without it, LightGBM improved a lot, while TabPFN lost accuracy and swung from too low to too high. In this comparison, TabPFN forecast higher than LightGBM both with and without year.

Effect of normalisation. Normalising the target improves TabPFN considerably. Without it, TabPFN missed the jump in sales in the peak week. With it, TabPFN was closer than LightGBM in the peak week, though it overshot the week after. TabPFN already standardises the target as a whole internally, so a plain global rescale changes nothing; the normalisation that helps has to do more than that.

The same holds for the input features: these models already rescale each column themselves, and their ensemble members vary the preprocessing or the feature order, so a global rescale of an input changes nothing either. Only a normalisation the model cannot do itself, such as scaling each product’s sales by that product’s own level, can add something. I have not tested raw inputs; normalising the target is what I tested, and it helped.

Test 2: a backtest against LightGBM in production

The larger test: a backtest of TabICLv2 against the LightGBM model in production. A backtest means standing at a past date, forecasting the following weeks, and comparing with what really sold.

Both models saw the same features, built for LightGBM, in these groups: long-term trend (a baseline sales level); short-term trend (last weeks’ sales, last month’s average); seasonality (month, week of month, week of year, the same weeks last year); product attributes (where the product sits in the catalogue); store and geography; weather; holidays.

Backtest perimeter:

Context sample choices, the only thing I tuned:

Results over the first four weeks of each forecast (horizons 1 to 4):

Left: TabICLv2's extra WAPE against LightGBM by sales band, zero or one point everywhere except the lowest band, where it is one point with a random context and four with a seasonal one. Right: bias relative to LightGBM. With a random context TabICLv2 forecasts lower than LightGBM on every band, by 9 points on the highest; with a seasonal context the gap on the highest band is 2 to 3 points, and on the lowest band it forecasts 4 to 6 points higher.

Products are grouped into five bands by average weekly sales; the lowest band sells less than one unit a week on average. Error and bias are shown relative to LightGBM, in points.

Two products: the TabICLv2 and LightGBM forecasts compared with each other, indexed so 100 is the product’s average week. Actual sales are not shown, for confidentiality.

The TabICLv2 and LightGBM forecasts for two products, indexed, for three forecast dates and up to eight weeks ahead. The two forecasts follow the same shape, with levels up to about a fifth apart. Actual sales are not shown, for confidentiality.

The two forecasts follow the same shape; their levels can differ by up to about a fifth, as on Product A’s first forecast.

Test 3: a simpler comparison on public data

The store data is confidential, so I repeated the comparison on a public demand dataset anyone can rerun: hourly bike rentals in Washington DC (OpenML 42712: 17,379 hours with calendar and weather columns, 2011 and 2012). Each of three months, July, September and November 2012, is forecast from every hour before it. Both models get the same 12 columns, with the observed weather, as if the weather forecast were perfect, and no lagged rentals, and run with their defaults (LGBMRegressor() and TabICLRegressor(), seed 0): no tuning, no context selection. TabICLv2 runs on an 8-thread CPU machine, no GPU, with tabicl 2.2.0 and lightgbm 4.7.0.

Per forecast month, TabICLv2's WAPE minus LightGBM's: July 2012 +1.5 points, September 2012 minus 1.0, November 2012 minus 2.6. Bias, LightGBM then TabICLv2: July minus 7.0% and minus 12.0%, September minus 6.8% and minus 6.7%, November +5.5% and +2.8%.

WAPE and bias as defined in How I measured. Each month has about 720 hours to forecast and 13,000 to 16,000 hours of history.

On a GPU these times should shrink (not measured here), but the shape stays: TabICLv2 costs one pass over the context, LightGBM almost nothing.

Conclusion

The Mission Lane team reached the same verdict on credit-risk data: “In summary, while TabICL delivers exceptional performance parity without tedious tuning, it currently comes with a substantial operational latency cost that must be carefully considered for high-speed, low-batch inference environments.”

What next

To test next

Ideas for going further

Choosing the context:

Combining with LightGBM:

Adapting the model and its inputs:

Serving cost:

Baselines:

Further reading

Related PFN work covers prompt-tuning (TuneTables, NeurIPS 2024, learns a compact context instead of passing the whole table), data generation, density estimation and outlier detection (the Nature paper and the tabpfn-extensions unsupervised module). There are also separate PFN models for data that drifts over time (Drift-resilient TabPFN, NeurIPS 2024, relevant to forecasting), counterfactual fairness (FairPFN) and interpretable GAM output (GAMformer).

Sources

From my notes:

Primary sources for the facts above:

Newsletter subscriptions are not currently available.