Do Fancy Training Goals Beat Plain Error for Table Data?
This preprint fairly retested 18 clever training goals for spreadsheet prediction on 41 datasets. Most did not consistently beat the plain, well-tuned standard.
In one paragraph
Neural networks that predict numbers from tables are usually trained to minimise mean squared error. This preprint retested 18 alternative objectives on 41 datasets with careful tuning and 15 seeds. Most did not consistently beat tuned MSE; quantile loss gave the most consistent gains across four architectures, though median gains were under 1%. The authors also show that noisy and clean data regions peak at different epochs and propose AdaQR, which raised median gains to as much as +0.97% on one architecture.
How were the regression objectives compared?
Training a regression model means choosing what it minimises. Mean squared error (MSE) is the default, but many alternatives exist: robust losses, binning approaches that predict distributions over bins, quantile loss, likelihood-based losses and regularisers. Earlier studies tested them on different datasets and setups, so they were hard to compare.
The authors trained an MLP with 18 objectives on 41 BeyondArena datasets, tuning hyperparameters for each and retraining the best configuration with 15 random seeds. Promising objectives were then tested on three stronger tabular architectures: TabPack, TabM-mini and RealMLP. They also used synthetic data with known noise, and uncertainty estimates on real data, to study when different regions reach their best performance during training.
Which training objective works best for tabular regression?
Most objectives did not provide consistent improvements over a carefully tuned MSE baseline. Two binning approaches and quantile loss stood out on the MLP, and quantile loss was the most consistent across architectures, with median gains from +0.40% to +0.58% on the advanced backbones and more wins than losses on each.
Switching from the AdamW optimizer to Muon improved plain MSE by a median of +1.50%, a larger effect than most objectives.
Noisy regions began overfitting before clean regions finished learning; choosing checkpoints separately for uncertainty groups improved median RMSE by 0.18% on the seven largest datasets. The authors' AdaQR objective raised median gains over MSE to +0.66% on MLP and +0.97% on TabM.
Why does this regression benchmark matter?
Predicting numbers from tabular data underpins many practical systems, from pricing to logistics. A fair benchmark shows that many proposed objectives add little once the baseline is tuned properly, which can save teams effort chasing small or illusory gains.
The study also highlights an overlooked issue: a single early-stopping point is a compromise when parts of the data differ in noise. AdaQR adds no cost at prediction time, because its extra smoothing runs only during training, and its learned noise levels agreed closely with a separate reference model.
A fair reading is that careful tuning and attention to training dynamics matter more than the choice of a new objective, though small gains can still be worthwhile at scale.
What are the limits of this tabular regression study?
Read before you quote it
This is a preprint that has not been peer reviewed, and its limits are clear:
- Small gains. Improvements are often well under 1% and vary by dataset and backbone.
- Measurement tool only. Per-group early stopping is not practical in real use.
- Partial fix. AdaQR does not fully remove the stopping conflict, and its main tests exclude the smallest datasets.
- Scope. Pretrained tabular foundation models were not included.
The study does not show a single best objective for every dataset. It also does not compare these neural approaches with gradient-boosted trees, which are widely used for tabular data in practice.
- It does not show that any objective is best for every dataset.
- It does not show how these objectives compare with gradient-boosted trees or pretrained tabular models.
- Gains are small, often well under 1%, and vary across datasets and backbones.
- Per-group early stopping is not practical; it was used only to measure the problem.
- AdaQR's main evaluation excludes the smallest (Tiny) datasets, and it does not fully remove the stopping conflict.
- Pretrained tabular foundation models were outside the study's scope.
Takeaways
- saysMost of eighteen alternative training objectives did not consistently beat a well-tuned mean squared error on forty-one table datasets.
- saysQuantile loss gave the most consistent gains across architectures, but they were around half a percent.
- saysNoisy and clean parts of the data peak at different training times, and the authors' AdaQR objective partly eases that.
- explainsAn objective is the quantity a model tries to minimise during training, and it shapes what the model learns.
- interpretsA fair reading is that tuning the baseline and optimizer well matters more than adopting a new objective.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Is MSE still a good loss for tabular regression?
Yes: most of 18 alternatives did not consistently beat a carefully tuned MSE baseline. Hear the full answer in the app →
Which loss works best for tabular deep regression?
Quantile loss gave the most consistent gains across architectures, but they were small, under 1%. Ask in the app →
What is the early stopping problem in this study?
Noisy parts of the data start overfitting before clean parts finish learning, so one stopping point is a compromise. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY-SA 4.0. Corrections: support@papersays.com.