How Far Should You Tune LightGBM? A Budget for the Last Fraction of a Point

Six LightGBM tuning rows in High Per Parameter reported the same median improvement: 0.0%. TabArena, meanwhile, moved tuned LightGBM from 1197 to 1446 Elo while reporting approximately 212.745 times the training time per 1,000 rows. Those results only look incompatible when “tuning” is treated as one experiment.

The validation bill is equally concrete. In the proposed five-outer-fold design below, an early-stopped baseline requires 30 fits, a 16-trial search requires 255, and a 64-trial search requires 975. A fraction of a point has to repay that growing bill on untouched data.

My approach starts with a deployment-faithful split and metric, sane tree capacity, and early stopping. A small targeted budget comes next. Expansion depends on blinded inner evidence, while aggregate outer results decide whether the complete procedure earns deployment. The knobs are real, and so is their validation cost.

The local protocol is a measurement design with no results yet. Published evidence supplies the opposing cases; the experiment determines which case resembles the problem at hand.

The expensive question is not whether tuning can win

An optimizer will nearly always return a winner. That is its contract. Give it 64 noisy estimates and it returns the largest one, even when the candidates have identical true performance. The maximum contains model quality plus validation noise, and increasing the search budget gives noise more chances to look like signal.

Cawley and Talbot made this point precise in their work on selection overfitting: variance in the model-selection criterion can itself be exploited, with degradation sometimes comparable to the reported differences between algorithms. Tuning is therefore not only model fitting. It is an adaptive procedure whose entire output - including the search - needs evaluation.

The useful question is not “did trial 47 beat trial 3 in inner CV?” It is “what distribution of outer-fold deltas did the complete procedure produce?” That procedure includes preprocessing, split logic, objective, metric, early stopping, search space, trial count, pruning policy, seed policy, candidate selection, refitting, and prediction aggregation. Change any of them and the estimand changes.

There is a product question underneath the statistical one. A small AUC movement can be valuable in a high-volume ranking system and irrelevant in a low-volume workflow. Flat accuracy can hide a useful calibration or latency change. Conversely, a benchmark gain can disappear when the split respects time, customers, hospitals, devices, or other repeated entities. No universal fraction of a point is worth buying. The threshold must be declared in the units that make the decision.

This is why “HPO worked” is an incomplete result. Against what baseline? Under which split? At what search and validation cost? On an inner score or an untouched estimate? The rest of the article keeps those questions attached to every number.

Four baselines hidden inside one word

“Default LightGBM” can refer to at least four regimes:

1. Raw package defaults. These are a reproducible control. They are not a claim about the best deployment configuration.
2. A sensible baseline. It uses the correct objective and metric, a deployment-faithful split, reasonable tree complexity, a high iteration ceiling, and early stopping.
3. A targeted search. It gives a small budget to interacting capacity, sampling, and regularization controls.
4. Broad HPO and ensembling. It pursues peak benchmark performance and pays a correspondingly larger selection and computation bill.

The claim I care about is the marginal value of steps three and four after step two. Comparing step four only with step one answers a different and still legitimate question: how much can careful optimization repair an untailored package configuration?

This separation also stops early stopping from becoming a hidden advantage. If the tuned model can discover a suitable boosting horizon but the baseline is forced to stop at an arbitrary n_estimators, the search is partly buying iteration selection. A fair sensible baseline receives the same opportunity. Set a high ceiling, watch the declared validation metric, freeze the best iteration by a predeclared rule, and refit without peeking at the outer holdout.

The official LightGBM early-stopping callback stops when the validation metric fails to improve by at least min_delta for stopping_rounds, and records the best iteration. One boundary matters: early stopping does not apply to boosting_type="dart". The official parameter-tuning guide identifies num_leaves as the main complexity control and min_data_in_leaf as an important guard against overfitting; a max_depth limit must remain consistent with the leaf count.

Once iteration selection belongs to the baseline, the remaining search has a cleaner job. It should test whether capacity, row or feature sampling, and regularization produce an outer-fold improvement large enough to matter.

What six zero medians do - and do not - say

The High Per Parameter paper, with an accessible arXiv version, reports LightGBM rows across a large experimental grid. The study covered 26 algorithms, 250 datasets, six metrics, 96,192 replicates, and 28,857,600 reported algorithm runs. For each algorithm-dataset pair it used 30 random 70/30 splits, an IID-style evaluation rather than evidence that the data-generating process was IID. Each metric received 50 Optuna TPE trials evaluated by five-fold CV. For the package-default arm, 50 evaluations were run and the model with the best CV score was retained.

Its comparison is narrow 50-trial HPO against package defaults - not HPO after the sensible early-stopped baseline proposed here. Every reported LightGBM median was 0.0%. The accompanying spreads are standard deviations, not confidence intervals, so they preserve the possibility of substantial dataset-specific wins and losses. For classification, the published search space lists bagging_fraction but no bagging_freq activation control; activation is undocumented and bagging_fraction may therefore have been inactive.

The classifier means were 1.2% ± 11.5% for accuracy, 1.0% ± 11.9% for balanced accuracy, and 0.9% ± 12.1% for F1, each over 4,287 replicates. The regressor means were 0.0% ± 25.6% for R², −1.2% ± 34.4% for adjusted R², and 0.4% ± 2.1% for complement RMSE, each over 3,179 replicates.

Figure 1. High Per Parameter used a narrow 50-trial package-default comparison with random 70/30 IID-style splits, retained the best-CV model from 50 default evaluations, and left classifier bagging activation undocumented. This is not the proposed sensible baseline, and the dispersion leaves room for dataset-specific gains and losses.

The plot makes two bad summaries difficult. “The medians are zero, therefore search cannot help” ignores the tails. “The means are positive, therefore search usually pays” ignores the zero medians and very large dispersion. The defensible inference is conditional: across these LightGBM experiments, the typical reported marginal improvement was zero, while particular datasets could behave very differently.

The 144 classification datasets had at most 10,992 rows and 100 features; the 106 regression datasets had at most 8,192 rows and 100 features. The classifier search used log-scaled n_estimators from 10 to 1,000, linear learning_rate from 0.01 to 0.2, and linear bagging_fraction from 0.5 to 0.95. The regressor search used log-scaled lambda_l1 and lambda_l2 from 1e-8 to 10 and integer num_leaves from 2 to 256.

That scope excludes temporal shift, grouped entities, very large data, and many parameter interactions. The study supports a capped first budget without dictating where every search must stop.

The countercase: 200 configurations win

TabArena supplies the counterexample that any serious stopping rule must survive. The benchmark selected 51 datasets from 1,053 candidates, evaluated 16 models, and fitted roughly 25 million model instances. Tunable models received one default plus 200 random configurations; TabICL and TabDPT were excluded from that search. Evaluation used repeated three-fold CV: ten repeats below 2,500 rows and three repeats otherwise.

Default LightGBM scored 1197 Elo, 0.088 normalized score, and 25.3 average rank at 1.96 reported train seconds per 1,000 rows. Tuned LightGBM scored 1446 Elo, 0.359, and 13.2 at 416.98 seconds; tuned plus ensembled reached 1527 Elo, 0.448, and 9.9 at the same reported 416.98 seconds. The tuned-to-default time ratio is approximately 212.745×.

Figure 2. TabArena’s 200-configuration search shows a strong predictive-performance-reported-training-time tradeoff. These are public IID benchmark regimes, not a local cumulative-trial curve, and the result is not a universal production budget.

This is decisive evidence against dismissing the knobs. A 249-point Elo gain and a 12.1-place average-rank improvement are not rounding errors. It is also evidence against pretending that peak benchmark performance is free. The time figure is reported training time per 1,000 rows under the benchmark’s setup, not a hardware-independent law and not a direct quote for a different pipeline.

The x-axis is logarithmic and the Elo axis is truncated from 1100 to 1600, so neither visual dimension should be read as a zero-based effect size. The tuned and tuned-plus-ensembled points share the same reported time in the source table. Most importantly, the three points are discrete regimes. They do not reveal what happened at trial 16, 25, or 64.

TabArena-Lite presents 25 configurations as a cheaper optional screen for a new model when the full benchmark is impractical. That does not make 25 a general LightGBM policy. A small budget still needs an expansion gate.

A sensible baseline is already a selection procedure

Early stopping turns the baseline into a modest model-selection procedure. For every outer fold, the baseline uses only the outer-training partition. Within that partition, the same three inner folds used by the search estimate a best iteration. The median of those three best iterations becomes a fixed round count. The configuration is then refitted on all outer-training data with three declared seeds, its predictions are averaged, and the untouched outer holdout is scored once.

That symmetry matters. If the search receives three refit seeds while the baseline receives one, variability can masquerade as a tuning gain. If baseline iteration count is selected on the outer holdout, the control is contaminated before the first trial begins. If seeds themselves are searched, they become another hyperparameter and must be charged to the budget.

The baseline should also reflect the data-generating process. Repeated entities call for group-aware folds. Forecast-like deployment calls for rolling-origin evaluation. Preprocessing learned from data belongs inside each training partition. The inner folds keep the same causal or grouping restriction as the outer folds. Another hundred trials cannot repair a split that answers the wrong question.

Metric choice belongs here rather than at the end. If probabilities drive a decision, an accuracy-only search can select a model with poor probability quality. Report log loss, Brier score, and reliability alongside the search objective; boosted methods can need calibration, as shown by Niculescu-Mizil and Caruana. Ranking systems need the metric at their operating depth. Thresholded decisions need expected cost or utility at the chosen threshold. Latency, model size, subgroup behavior, and fold or seed variability can all veto a tiny primary-score gain.

This setup costs more than a flat train/validation split because it estimates the procedure honestly. That is exactly why a tuning budget should count validation fits, not only optimizer trials.

Better defaults are candidates, not recipes

The NeurIPS 2024 paper Better by Default asks whether cross-dataset meta-tuning can produce stronger starting configurations. It learned on 118 datasets and evaluated on a disjoint 90-dataset benchmark containing 1,000 to 500,000 samples. Strong defaults were competitive with per-split HPO on the meta-training suite, but the gap reopened on the disjoint meta-test data. Selecting among strong default models offered an attractive time-performance tradeoff, while transfer remained imperfect.

Those candidate defaults were evaluated with LightGBM 4.1.0, ten random 60/20/20 train-validation-test splits per dataset, 50 random HPO configurations per split, one validation split per train-test split, accuracy for classification, RMSE for regression, and numerical rows with missing values removed.

The published classification/regression candidates set num_leaves to 50/100, learning_rate to 0.04/0.05, subsample to 0.75/0.70, bagging_freq to 1/1, colsample_bytree to 1.0/1.0, min_data_in_leaf to 40/3, min_sum_hessian_in_leaf to 1e-7/1e-7, max_bin to 255/255, n_estimators to 1000/1000, and early_stopping_rounds to 300/300.

The bagging_freq=1 row is essential. In LightGBM, a subsample below one is inactive when the frequency remains zero. Copying the fractions without their activation control silently changes the proposed candidate. The exact PyTabKit v1.0.0 snapshot also pins the dependency and split setup; a table of parameter values alone is not a reproduction.

The separate LightGBM 4.7.0 raw sklearn control uses num_leaves=31, max_depth=-1, learning_rate=0.1, n_estimators=100, min_child_weight=0.001, min_child_samples=20, subsample=1, subsample_freq=0, colsample_bytree=1, reg_alpha=0, reg_lambda=0, and max_bin=255; its objective is inferred by the estimator and early stopping is not enabled by that raw default.

Keep those regimes separate. The 4.1.0 values are candidate strong defaults under the paper’s controls. The 4.7.0 values are a current raw control. The sensible baseline is a proposed procedure that still needs a dataset-specific objective, metric, split, and early-stopping implementation. None is a universal recipe.

The experiment I would run

I would compare three budgets on identical frozen preprocessing and outer splits: the sensible early-stopped baseline, 16 targeted configurations, and an expansion to 64 configurations when the inner evidence clears a predeclared useful-effect threshold. A 200-trial regime can remain a research upper bound for high-value cases, but the first measurement should reveal whether the improvement curve deserves that expense.

This local protocol has not been run: dataset, target, split strategy, primary metric, useful-effect threshold, constraints, and hardware are unspecified; every local delta remains null rather than zero.

Use a prefix-stable random search with seed 20260808. The exact PRNG and sampler implementation remains unspecified and must be pinned before a run. Draw learning_rate log-uniformly from [0.01, 0.2]; num_leaves as a log-uniform integer from [15, 255]; min_data_in_leaf as a log-uniform integer from [3, 256]; feature_fraction uniformly from [0.6, 1.0]; and bagging_fraction uniformly from [0.6, 0.95], with bagging_freq fixed at 1. Draw lambda_l2 from a zero-inflated log-uniform distribution: 20% exact zero, otherwise [1e-8, 10].

Generate 64 configurations once. The 16-trial regime uses the first 16 exactly; expansion adds the remaining 48 without redrawing.

The frozen design uses five outer folds and three inner folds, trial budgets of 16 and 64, no pruning, an iteration ceiling of 10,000, early_stopping_rounds=300, min_delta=0, refit seeds 17, 29, and 47, and no seed search.

Candidate selection converts the declared metric to higher-is-better utility, chooses the largest arithmetic mean across three inner folds, breaks ties within 1e-12 by lower trial index, freezes the median of three best iterations, and averages the three refit predictions before scoring. The outer holdout is evaluated once per procedure per fold and cannot control budget expansion.

For classification, the three refits produce an arithmetic mean of predicted probabilities; regression averages predictions. That aggregation is frozen before any holdout score exists.

Across five outer folds, the sensible baseline costs 5 × (3 + 3) = 30 fits; 16 trials cost 5 × (16 × 3 + 3) = 255; 64 trials cost 5 × (64 × 3 + 3) = 975.

outer_folds = 5
inner_folds = 3
refit_seeds = (17, 29, 47)

baseline_fits = 5 * (3 + 3)
targeted_16_fits = 5 * (16 * 3 + 3)
expanded_64_fits = 5 * (64 * 3 + 3)

assert outer_folds == 5 and inner_folds == 3 and len(refit_seeds) == 3
assert (baseline_fits, targeted_16_fits, expanded_64_fits) == (30, 255, 975)

Figure 3. This proposed protocol has not been run. Across five folds it requires 30, 255, and 975 fits for the baseline, 16-trial, and 64-trial procedures; only inner-CV evidence opens the expansion branch, and the outer holdout never opens the budget.

Only inner-CV evidence may expand the search from 16 to 64 trials. The outer holdout is scored once after the adaptive inner procedure finishes. After all outer folds complete, paired outer deltas - not the best inner score - drive the final decision.

What the untouched holdout is protecting

The gap between selected inner improvement and outer improvement estimates selection optimism. If a search reports a large inner lift and little outer lift, the optimizer may have found a validation accident. Plotting those values by outer fold or by dataset makes the debt visible instead of hiding it inside an average.

Varma and Simon’s nested-validation study supports the qualitative conclusion that including feature and parameter selection inside a nested procedure can remove selection bias. Its public SVM values disagree between the abstract and table, so no exact SVM percentage belongs here. The conclusion is useful without choosing a number from an unresolved discrepancy.

Nested CV also deserves a counterweight. Wainer and Cawley tested 12 algorithms on 115 real binary datasets and found that flat CV generally selected a practically similar algorithm when only a few hyperparameters were optimized. Their paper makes nested evaluation the safer estimator of an adaptive procedure without requiring it for every low-stakes choice among a handful of configurations.

Figure 4. This is a future diagnostic with no local result. Null means not-run or not-specified and never zero: the dataset, metric, split, threshold, constraints, and hardware still need to be declared before any point can exist.

The empty frame is more honest than a synthetic curve. A fabricated diminishing-return plot would look persuasive while answering nothing about a real dataset. Once results exist, the comparison should show inner and outer deltas together, paired at the outer-window or dataset level. For a multi-dataset claim, aggregate within each dataset before bootstrapping datasets so that a dataset with more folds does not quietly acquire more voting weight.

The outer result should also be evaluated against a smallest worthwhile effect, not merely a null-hypothesis threshold. Nonsignificance does not prove equivalence. Weerts and colleagues demonstrate an equivalence-style analysis over 59 OpenML datasets. Their study reinforces the need to declare the margin that would change the decision before observing the result.

When to spend more

A 16-trial cap is a starting budget, not a law. Expansion is justified when the baseline clearly underfits or overfits, early stopping repeatedly hits the ceiling, the first trials show consistent inner improvements above the useful-effect threshold, or a small ranking movement has unusually high value. Unusual class imbalance, sparse or noisy data, hard monotonicity or fairness constraints, and a tolerable ensemble footprint can also make a deeper search rational.

The search should be diagnostic rather than indiscriminate. Hitting the iteration ceiling points toward learning rate or capacity. A wide train-validation gap points toward leaves, minimum leaf size, sampling, or regularization. A latency violation points toward leaves, depth, iteration count, and model size. Changing everything at once may produce a winner while revealing little about why it won.

Adjacent evidence warns against treating a strong baseline as the end of tunability. Probst, Boulesteix, and Bischl’s JMLR study concerns XGBoost, not LightGBM. Their cross-dataset defaults removed some improvement over package defaults while leaving measurable tunability. The model boundary prevents copying its numerical result into a LightGBM claim, but the experimental lesson transfers cautiously: better defaults can reduce, not necessarily erase, the value of search.

Search can also be worthwhile when the primary metric is flat but a secondary constraint improves. A configuration with similar AUC may calibrate better, occupy less memory, or meet a latency ceiling. That is still a successful optimization if the constraint and measurement were declared in advance. It is not evidence that another hundred unconstrained accuracy trials will help.

The stopping rule

Before the first trial, write down the primary metric, deployment-faithful split, smallest useful improvement, secondary constraints, maximum search budget, and the inner evidence required to expand it. Then follow a short sequence:

1. Build the sensible early-stopped baseline and verify that its iteration ceiling and split behave as intended.
2. Diagnose the likely parameter family instead of opening a broad space by reflex.
3. Run 16 targeted, prefix-stable configurations inside each outer-training partition.
4. Stop at 16 when inner evidence cannot clear the useful-effect threshold under the deployment-faithful folds.
5. Expand to 64 or beyond only inside the inner procedure; never let an outer result reopen the search.
6. Complete every outer fold, aggregate paired deltas, and accept HPO only when the lift survives the declared constraints.

In product units, the final inequality is simple even when its inputs are not:

probability-adjusted lifetime value of lift
>
search compute + engineering + validation + added monitoring cost

The bill decides. Continue when paired outer lift clears the declared margin and survives the secondary constraints. Stop when it does not. Either outcome prices the last fraction of a point on untouched data.

Primary sources

• Moshe Sipper, “High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms”, with the arXiv full text.
• Holzmüller et al., “Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data”, and the pinned PyTabKit v1.0.0 repository.
• Erickson et al., “TabArena: A Living Benchmark for Machine Learning on Tabular Data”.
• LightGBM, early-stopping callback and parameter-tuning guide.
• Cawley and Talbot, “On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation”.
• Varma and Simon, “Bias in Error Estimation When Using Cross-Validation for Model Selection”.
• Wainer and Cawley, “Nested Cross-Validation When Selecting Classifiers Is Overzealous for Most Practical Applications”.
• Probst, Boulesteix, and Bischl, “Tunability: Importance of Hyperparameters of Machine Learning Algorithms”.
• Niculescu-Mizil and Caruana, “Predicting Good Probabilities with Supervised Learning”.
• Weerts et al., “Importance of Tuning Hyperparameters of Machine Learning Algorithms”.