Tiny Data, Big Gains: Small Unlabeled Subsets Slash the Cost of Tabular Self-Supervised Learning

Self-supervised learning has become one of the defining techniques of modern artificial intelligence, powering systems such as GPT and SimCLR by extracting useful patterns from oceans of unlabeled text, images and audio. But a new large-scale study suggests that for tabular data — the spreadsheets, patient records and financial tables that dominate industry and science — feeding a model everything you have may be exactly the wrong approach. Researchers report that pretraining on carefully chosen subsets of as few as 100 unlabeled examples can match, and often beat, training on the entire unlabeled pool, while cutting training time by more than half.

The study, published open access in the journal Machine Learning by Sintija Stevanoska, Christian L. Camacho Villalón, Sašo Džeroski and Katharina Dost, tackles a well-known pain point in applied machine learning. Labels are expensive: healthcare systems accumulate millions of patient records, yet expert-annotated outcomes remain scarce. Self-supervised learning promises to bridge that gap by defining pretext tasks from the intrinsic structure of data — masking features and reconstructing them, corrupting values and asking a model to spot the damage, or learning invariant representations across contrastive views. For tables, however, these methods inherit an awkward problem. Unlike images or sentences, tabular rows lack spatial or sequential structure, mix numerical and categorical features, and often contain missing values and tangled inter-dependencies. And pretraining on very large unlabeled pools is computationally expensive while potentially importing noisy or unrepresentative samples that actively degrade learning.

The researchers asked a deceptively simple question: is a small, well-chosen subset of unlabeled data sufficient for effective pretraining? To answer it, they ran what they describe as the first large-scale evaluation of unlabeled data subsampling for tabular self-supervised learning, spanning 25 benchmark datasets drawn from the OpenML-CC18 suite and covering domains from biology and finance to medicine and climate science. Four representative self-supervised models were tested — TabNet, VIME, SCARF and SAINT — deliberately chosen to span the field’s three main paradigms. TabNet and VIME learn by masking or corrupting features and reconstructing them, SCARF relies on contrastive views generated by resampling features from their empirical distributions, and SAINT combines contrastive and denoising objectives inside a transformer that attends over both rows and columns. This diversity of pretext tasks and encoder architectures ensured the conclusions would not hinge on one particular model family.

At the heart of the study lies a family of selection strategies borrowed and adapted from active learning, the branch of machine learning concerned with identifying the most informative examples to label. The team compared random sampling against three principled approaches. Disagreement sampling, based on the classic query-by-committee method, trains a committee of 100 decision trees on bootstrap samples of the labeled data, restricting each tree to a random subset of features in the manner of random forests, and scores every unlabeled point by the normalized entropy of the committee’s votes — a value from 0, indicating complete agreement, to 1, indicating uniform disagreement. Diversity sampling applies k-means clustering to the unlabeled pool and picks one representative per cluster, the point closest to each centroid, enforcing a minimum cluster size to avoid crowning representatives of tiny outlier groups. Optimal transport sampling treats the labeled and unlabeled sets as empirical distributions, computes an entropically regularized transport plan between them with the Sinkhorn algorithm, and assigns each unlabeled point its expected transport cost to the labeled distribution — a low score signaling strong alignment with the labeled data, a high score signaling underrepresented regions of the feature space.

Crucially, all of these strategies are deliberately model-agnostic. They are computed before any self-supervised pretraining begins, using no gradients, losses, embeddings or augmentations from the downstream model. Score-based methods partition the unlabeled pool into percentile brackets — low, medium and high — and samples are then drawn from a chosen bracket, either uniformly at random or via the diversity-based cluster representatives. This design allowed the authors to evaluate the sampling approaches consistently across all four pretext objectives, isolating the effect of data selection from the idiosyncrasies of any particular architecture.

The experimental scale was formidable. Labeled subsets of 100, 500 and 1000 examples were drawn under both unbiased, stratified conditions and a deliberately biased regime in which labeled examples were preferentially selected from one branch of a decision stump, simulating real-world selection bias. Unlabeled subsets of 100 to 1000 examples were drawn under every combination of strategy, subset size, bracket and five random seeds. Performance was measured through three lenses: downstream classification quality using macro-averaged area under the precision-recall curve, total wall-clock time including both selection and training, and representation quality assessed by linear probing — training a simple logistic regression classifier on frozen pretrained embeddings to test whether class information is linearly encoded in the learned latent space.

The headline finding is striking. Subsampling only 100 to 1000 unlabeled examples often matched or exceeded full-data pretraining across both labeled-data regimes. In the low-label, unbiased setting, the best sampling strategies reached up to 119 percent of the full-data baseline and 114 percent of a purely supervised baseline that skipped pretraining entirely. Under biased labeled selection the gains grew even more pronounced, climbing to 123 percent of the full-data baseline and 115 percent of supervised learning. The effect was strongest exactly where it matters most: when labels were scarcest, and the choice of pretraining data had the greatest opportunity to shape what the model learned. Optimal transport distance and disagreement-based sampling performed most consistently, but the humble random sampler was surprisingly competitive, hinting that for small labeled sets, simply reducing redundancy and avoiding overfitting to the unlabeled distribution may itself act as a form of regularization.

The computational story is equally compelling. Pretraining on the full unlabeled pool consistently demanded 60 to 70 percent more time than the sampling strategies, and the selection step itself added a negligible share of the total runtime. Random sampling typically finished in 29 to 42 percent of full-data time, with optimal transport close behind; diversity-based selection was the most expensive to compute, consuming up to 70 percent of the full-data baseline’s runtime without delivering consistent accuracy dividends. In other words, competitive or better predictive performance was achieved at roughly 30 to 40 percent of the computational cost — a margin that alone would justify subsampling in resource-constrained environments even without accuracy improvements. Linear probing reinforced the picture: targeted subsampling consistently matched or exceeded the representation quality of full-data pretraining, and in many configurations full-data self-supervision actually fell below the supervised baseline, an adverse effect the selective strategies appeared to mitigate, particularly under label bias.

To understand when and why subsampling pays off, the team conducted a meta-analysis linking dataset characteristics to observed gains. They extracted statistical and complexity-related metafeatures for each dataset — among them the performance of a nearest-neighbor classifier built on the single most important feature, the stability of local neighborhood structure across cross-validation folds, the minimum support of decision-tree leaves, the median absolute deviation of the features, and trimmed means capturing distributional asymmetry. The patterns that emerged were model-specific rather than universal. TabNet and SCARF gained most from subsampling on datasets where a single dominant feature carries the predictive signal, while SAINT and VIME tended to benefit more on structurally complex tasks; datasets containing a feature with strongly asymmetric central mass favored using all the data. No single metafeature predicted gains across all architectures, pointing to a decisive interplay between dataset geometry and each model’s inductive bias.

The authors are candid about limitations and open questions. The number and size of unlabeled subsets and the granularity of the score brackets were fixed by design, so the study does not determine an optimal data budget for a given dataset, and the meta-analysis, based on 25 datasets, is exploratory rather than confirmatory. Future directions include adaptive budget selection guided by validation performance or meta-learned dataset properties, curriculum-style ordering of the unlabeled pool, incorporating self-supervised model feedback into the selection loop, and training a meta-learner to recommend strategy and subset size automatically. Even so, the message for practitioners is already actionable and, in its own way, subversive: in tabular self-supervised learning, the brute-force assumption that more unlabeled data is always better does not survive scrutiny. A few hundred well-chosen — or even randomly chosen — examples can deliver richer representations and better predictions at a fraction of the cost, a result that could reshape how resource-conscious teams approach pretraining on the structured data that quietly runs the world.

Subject of Research: Unlabeled data selection for efficient self-supervised learning on tabular data

Article Title: Less is More: Unlabeled Data Selection for Efficient Tabular Self-supervised Learning

Article References: Stevanoska, S., Camacho Villalón, C. L., Džeroski, S., & Dost, K. (2026). Less is More: Unlabeled Data Selection for Efficient Tabular Self-supervised Learning. Machine Learning, 115(9), Article 208. https://doi.org/10.1007/s10994-026-07133-8

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07133-8

Keywords: self-supervised learning, tabular data, data selection, unlabeled data, active learning, optimal transport, query-by-committee, computational efficiency, representation learning, machine learning benchmarks, TabNet, VIME