Two interpretable supervised algorithms reveal feature aggregation in nonlinear systems

Researchers at Politecnico di Milano have unveiled a pair of new machine learning algorithms that shrink massive datasets while keeping every feature interpretable, extending dimensionality reduction into the nonlinear territory where most real-world data lives. The algorithms, called NonLinCFA and GenLinCFA, are described in a study published in the open-access journal Data Mining and Knowledge Discovery, and they promise to make high-dimensional analysis more transparent in fields ranging from climate science to genomics and image recognition.

Dimensionality reduction is one of the oldest and most practical problems in machine learning. Modern datasets can contain thousands or even tens of thousands of variables, and feeding all of them into a model invites trouble: computations slow down, memory demands balloon, and models risk overfitting, memorizing noise rather than learning genuine patterns. The classical solutions come in two flavors. Feature extraction methods such as Principal Component Analysis compress many variables into a handful of latent components, while feature selection simply keeps the most informative variables and throws away the rest. Both come with a catch. PCA-style components are dense linear combinations of nearly all the original features, weighted by coefficients that a domain expert cannot easily interpret, and feature selection discards information that might have been valuable.

The new work, led by Paolo Bonetti together with Alberto Maria Metelli and Marcello Restelli, builds on an earlier technique the same group introduced, known as LinCFA, which took a third path: instead of extracting latent variables or discarding features, it merges correlated groups of features by replacing them with their average. Averaging is a transformation any scientist can understand without consulting a machine learning specialist, and no feature is ever lost entirely, because each one contributes to some mean. The catch with the original method was its assumption that the relationship between features and targets was linear, an assumption that rarely holds in practice and that reduced the technique to a heuristic whenever real data behaved nonlinearly.

NonLinCFA removes that restriction. The researchers consider a general nonlinear function linking the features to the target, corrupted by additive Gaussian noise, and they carry out a rigorous asymptotic bias-variance analysis of what happens when two features are replaced by a single aggregated variable. Their analysis yields precise expressions for both sides of the trade-off. On one side, merging features shrinks the hypothesis space, cutting the variance of the fitted model by exactly the noise variance divided by the number of samples. On the other side, aggregation increases bias, by an amount that depends on the correlations between the two features, their aggregation, and the true underlying function. Combining the two results produces an elegant criterion: merging two features is guaranteed not to increase the mean squared error if and only if the variance reduction, sigma squared divided by n minus one, exceeds the difference between how well the target is predicted by the two features jointly and how well it is predicted by the aggregated version alone. Intuitively, aggregation pays off when the noise is large, the sample is small, the aggregated feature shares plenty of information with the target, and the individual features contribute little unique information of their own.

The second algorithm, GenLinCFA, pushes the framework even further into generalized linear models, the statistical workhorses behind logistic regression, Poisson regression, and many other techniques. Here the target’s distribution is assumed to belong to the canonical exponential family, which covers the Normal, Exponential, Poisson, Bernoulli, and Binomial distributions, and the expected value of the target passes through a link function before being connected to the features. Because the mean squared error no longer provides a natural goodness-of-fit measure in this setting, the researchers instead analyze the deviance, a quantity that measures how far the fitted model’s likelihood falls short of the best achievable one. Through a second-order approximation, they derive an upper bound on how much the expected deviance can increase when two features are merged, and this bound becomes the merging criterion of GenLinCFA. Crucially, this makes the method applicable to classification problems, which the original linear approach could not handle at all.

In both algorithms the workflow is the same at heart. The procedure iteratively examines candidate pairs of features, applies a method-specific merge test derived from theory, and replaces a pair with its average whenever the test indicates this is beneficial. The process repeats until no further aggregation is worthwhile, producing a reduced set of features, each of which is the mean of a group of original variables. The computational cost remains quadratic in the number of features and linear in the number of samples, the same as the earlier linear method, and no more memory is required than storing the original dataset. A single hyperparameter called epsilon governs how aggressive the algorithms are: large values encourage more merging, small values keep the method conservative.

The theoretical guarantees were validated through an extensive experimental campaign. On synthetic regression problems with 100 and 1000 features, both algorithms achieved strong predictive scores using far fewer reduced features than a wrapper feature-selection baseline needed to match the same performance. The earlier LinCFA method, applied to the same data, produced 39 reduced features with an R-squared score of about 0.866 in the smaller setting, and nearly 194 reduced features with a score of about 0.707 in the larger, noisier one, confirming that the nonlinear extensions extract more compact and more accurate representations. When the sign of the target was applied to create binary classification tasks, GenLinCFA matched or beat the wrapper baseline while shrinking the dimensionality dramatically.

Real-world benchmarks reinforced the picture. The team tested their methods on four datasets from Kaggle and the UCI Machine Learning Repository, including a finance dataset, a corporate bankruptcy dataset, a Parkinson’s disease classification dataset, and a gene expression dataset with nearly 20,000 features but only about 800 samples, a notoriously challenging regime. Against ten established baselines, including PCA, Linear Discriminant Analysis, Kernel PCA, Isomap, Locally Linear Embedding, UMAP, t-SNE, autoencoders, supervised PCA, and Neighborhood Components Analysis, the new algorithms were competitive across the board and outperformed their linear predecessor in almost every case. More complex nonlinear projections such as Kernel PCA and autoencoders sometimes achieved better raw scores, but at the cost of producing features whose meaning is opaque.

Perhaps the most striking demonstration came from climate science, the motivating application behind the research. When predicting the vegetation state of a sub-basin of the Po River in Northern Italy from temperature and precipitation measurements taken at many locations, the algorithms automatically discovered spatial sub-regions within which measurements could be safely averaged. Because climatologists routinely aggregate neighboring measurements by hand, the data-driven partitions produced by NonLinCFA are immediately meaningful to them: a map of merged temperature readings over a sub-region reads like a familiar climatological summary rather than an inscrutable mathematical object. Additional experiments on MNIST and Fashion-MNIST image datasets, including a regression task predicting a central pixel from its neighbors and binary classification of visually similar digits, confirmed that the methods remain competitive with state-of-the-art dimensionality reduction while preserving interpretability.

The researchers acknowledge limitations, including the sensitivity of averaging to outliers, which can be mitigated with robust preprocessing, and the fact that their exact theoretical results currently cover linear models within the nonlinear framework. They also note that when features represent heterogeneous quantities, the average of a merged group may lack a concrete physical meaning, even though it remains transparent and empirically useful. Still, the study demonstrates that interpretability and nonlinear performance need not be opposing goals. By grounding a simple, human-readable operation, the mean, in formal bias-variance and deviance analysis, the Milan team has shown that it is possible to compress data aggressively without losing the ability to explain what the compressed features actually are. The authors suggest that future theoretical work could extend the guarantees to broader classes of supervised models, and that large-scale applications could deepen the empirical contribution, with the code and datasets made freely available to the community.