Dimensionality reduction is one of the oldest and most practical problems in machine learning. Modern datasets can contain thousands or even tens of thousands of variables, and feeding all of them into a model invites trouble: computations slow down, memory demands balloon, and models risk overfitting, memorizing noise rather than learning genuine patterns. The classical solutions come in two flavors. Feature extraction methods such as Principal Component Analysis compress many variables into a handful of latent components, while feature selection simply keeps the most informative variables and throws away the rest. Both come with a catch. PCA-style components are dense linear combinations of nearly all the original features, weighted by coefficients that a domain expert cannot easily interpret, and feature selection discards information that might have been valuable.

The new work, led by Paolo Bonetti together with Alberto Maria Metelli and Marcello Restelli, builds on an earlier technique the same group introduced, known as LinCFA, which took a third path: instead of extracting latent variables or discarding features, it merges correlated groups of features by replacing them with their average. Averaging is a transformation any scientist can understand without consulting a machine learning specialist, and no feature is ever lost entirely, because each one contributes to some mean. The catch with the original method was its assumption that the relationship between features and targets was linear, an assumption that rarely holds in practice and that reduced the technique to a heuristic whenever real data behaved nonlinearly.

NonLinCFA removes that restriction. The researchers consider a general nonlinear function linking the features to the target, corrupted by additive Gaussian noise, and they carry out a rigorous asymptotic bias-variance analysis of what happens when two features are replaced by a single aggregated variable. Their analysis yields precise expressions for both sides of the trade-off. On one side, merging features shrinks the hypothesis space, cutting the variance of the fitted model by exactly the noise variance divided by the number of samples. On the other side, aggregation increases bias, by an amount that depends on the correlations between the two features, their aggregation, and the true underlying function. Combining the two results produces an elegant criterion: merging two features is guaranteed not to increase the mean squared error if and only if the variance reduction, sigma squared divided by n minus one, exceeds the difference between how well the target is predicted by the two features jointly and how well it is predicted by the aggregated version alone. Intuitively, aggregation pays off when the noise is large, the sample is small, the aggregated feature shares plenty of information with the target, and the individual features contribute little unique information of their own.

The second algorithm, GenLinCFA, pushes the framework even further into generalized linear models, the statistical workhorses behind logistic regression, Poisson regression, and many other techniques. Here the target’s distribution is assumed to belong to the canonical exponential family, which covers the Normal, Exponential, Poisson, Bernoulli, and Binomial distributions, and the expected value of the target passes through a link function before being connected to the features. Because the mean squared error no longer provides a natural goodness-of-fit measure in this setting, the researchers instead analyze the deviance, a quantity that measures how far the fitted model’s likelihood falls short of the best achievable one. Through a second-order approximation, they derive an upper bound on how much the expected deviance can increase when two features are merged, and this bound becomes the merging criterion of GenLinCFA. Crucially, this makes the method applicable to classification problems, which the original linear approach could not handle at all.

In both algorithms the workflow is the same at heart. The procedure iteratively examines candidate pairs of features, applies a method-specific merge test derived from theory, and replaces a pair with its average whenever the test indicates this is beneficial. The process repeats until no further aggregation is worthwhile, producing a reduced set of features, each of which is the mean of a group of original variables. The computational cost remains quadratic in the number of features and linear in the number of samples, the same as the earlier linear method, and no more memory is required than storing the original dataset. A single hyperparameter called epsilon governs how aggressive the algorithms are: large values encourage more merging, small values keep the method conservative.

The theoretical guarantees were validated through an extensive experimental campaign. On synthetic regression problems with 100 and 1000 features, both algorithms achieved strong predictive scores using far fewer reduced features than a wrapper feature-selection baseline needed to match the same performance. The earlier LinCFA method, applied to the same data, produced 39 reduced features with an R-squared score of about 0.866 in the smaller setting, and nearly 194 reduced features with a score of about 0.707 in the larger, noisier one, confirming that the nonlinear extensions extract more compact and more accurate representations. When the sign of the target was applied to create binary classification tasks, GenLinCFA matched or beat the wrapper baseline while shrinking the dimensionality dramatically.

Real-world benchmarks reinforced the picture. The team tested their methods on four datasets from Kaggle and the UCI Machine Learning Repository, including a finance dataset, a corporate bankruptcy dataset, a Parkinson’s disease classification dataset, and a gene expression dataset with nearly 20,000 features but only about 800 samples, a notoriously challenging regime. Against ten established baselines, including PCA, Linear Discriminant Analysis, Kernel PCA, Isomap, Locally Linear Embedding, UMAP, t-SNE, autoencoders, supervised PCA, and Neighborhood Components Analysis, the new algorithms were competitive across the board and outperformed their linear predecessor in almost every case. More complex nonlinear projections such as Kernel PCA and autoencoders sometimes achieved better raw scores, but at the cost of producing features whose meaning is opaque.

Perhaps the most striking demonstration came from climate science, the motivating application behind the research. When predicting the vegetation state of a sub-basin of the Po River in Northern Italy from temperature and precipitation measurements taken at many locations, the algorithms automatically discovered spatial sub-regions within which measurements could be safely averaged. Because climatologists routinely aggregate neighboring measurements by hand, the data-driven partitions produced by NonLinCFA are immediately meaningful to them: a map of merged temperature readings over a sub-region reads like a familiar climatological summary rather than an inscrutable mathematical object. Additional experiments on MNIST and Fashion-MNIST image datasets, including a regression task predicting a central pixel from its neighbors and binary classification of visually similar digits, confirmed that the methods remain competitive with state-of-the-art dimensionality reduction while preserving interpretability.

The researchers acknowledge limitations, including the sensitivity of averaging to outliers, which can be mitigated with robust preprocessing, and the fact that their exact theoretical results currently cover linear models within the nonlinear framework. They also note that when features represent heterogeneous quantities, the average of a merged group may lack a concrete physical meaning, even though it remains transparent and empirically useful. Still, the study demonstrates that interpretability and nonlinear performance need not be opposing goals. By grounding a simple, human-readable operation, the mean, in formal bias-variance and deviance analysis, the Milan team has shown that it is possible to compress data aggressively without losing the ability to explain what the compressed features actually are. The authors suggest that future theoretical work could extend the guarantees to broader classes of supervised models, and that large-scale applications could deepen the empirical contribution, with the code and datasets made freely available to the community.

Groundwater serves as the backbone of global freshwater reserves, supplying drinking water to billions and irrigating vast agricultural expanses. However, the complex interrelations between climatic variables, geological formations, hydrological processes, and human activities pose significant challenges to accurately modeling groundwater levels. Traditional statistical and numerical simulation models often fall short in capturing this intricate web of dependencies, inspiring the integration of machine learning tools which promise enhanced predictive power and adaptability.

Yet, machine learning models themselves are not immune to challenges. The selection and optimization of input features — the parameters or variables fed into these algorithms — greatly influence the reliability and robustness of groundwater level predictions. Feeding models with redundant or noisy data can obscure meaningful patterns and degrade performance. Hence, dimensionality reduction methods, which condense high-dimensional datasets into more informative and manageable representations, are invaluable tools in this context.

Principal Component Analysis (PCA) has long been a cornerstone technique for reducing dimensionality, offering a linear transformation that projects data onto orthogonal components explaining maximum variance. In contrast, t-Distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear technique adept at preserving local similarities in the data, often employed for visualizing complex datasets in low dimensions. However, the suitability of these methods for preparing input features specifically for groundwater simulation models had not been rigorously compared until this study.

The researchers embarked on an extensive evaluation involving multiple groundwater datasets encompassing diverse climatic zones and geological conditions. Their approach triangulated between hydrological records, meteorological information, soil characteristics, and anthropogenic factors, creating a rich but complex feature space. They then applied PCA and t-SNE separately to these datasets, extracting optimized feature sets to be used as inputs for several state-of-the-art machine learning architectures, including random forests, support vector machines, and neural networks.

A key finding of the study was that PCA generally offers superior interpretability and computational efficiency due to its linear nature, making it more accessible for integration into real-time groundwater monitoring frameworks. PCA’s principal components often captured dominant hydrological trends, allowing domain specialists to infer physical meanings behind model decisions. However, PCA’s linear assumption limited its ability to fully capture nonlinear intricacies inherent in groundwater systems influenced by heterogeneous subsurface structures and nonlinear climate interactions.

On the other hand, t-SNE excelled at unveiling subtle, nonlinear relationships within the data, uncovering clusters and patterns that PCA overlooked. Such nuanced embeddings enabled machine learning models to gain deeper insights into complex groundwater behaviors, especially in regions where localized geological anomalies significantly affected water table dynamics. Nevertheless, t-SNE’s computational cost and parameter sensitivity posed practical challenges, particularly concerning scalability and reproducibility — vital considerations for operational groundwater forecasting.

Interestingly, the study identified a hybrid strategy that harnesses PCA’s strengths for initial dimensionality reduction, followed by a selective application of t-SNE for in-depth feature transformation on critical subsets. This approach balanced efficiency with the richness of information extraction, demonstrating improved predictive accuracy over using either technique alone. The researchers emphasized the importance of tailoring dimensionality reduction techniques to the specific characteristics of groundwater datasets and the intended application scenario.

Beyond the technical performance, the paper discusses the broader implications of optimized feature selection for water resource management. Accurate groundwater level predictions enable better decision-making in drought preparedness, aquifer recharge planning, and environmental conservation. With climate change exacerbating hydrological variability, the integration of advanced machine learning pipelines supported by robust dimensionality reduction is increasingly essential.

The authors also critically highlighted the need for comprehensive data collection and standardization to fully capitalize on these analytical advancements. The availability of high-resolution spatiotemporal data feeds into machine learning models’ power, while gaps or inconsistencies can propagate errors even in sophisticated frameworks. Thus, investments in monitoring infrastructure and open data initiatives complement algorithmic progress.

Delving deeper into methodological nuances, the study meticulously evaluated hyperparameter tuning for both PCA and t-SNE, elucidating their impact on feature extraction quality. For instance, the choice of perplexity in t-SNE and the number of principal components in PCA directly influenced the balance between dimensionality reduction and information loss. These insights serve as a valuable guide for practitioners aiming to deploy similar techniques in environmental modeling contexts.

Furthermore, the research underscores the value of cross-validation and ensemble learning in reinforcing model robustness. By testing optimized feature sets across multiple machine learning models and geographic regions, the researchers ensured that their conclusions held under varied conditions, minimizing overfitting and maximizing generalizability. This rigorous validation framework sets a high standard for future studies intersecting data science and hydrology.

The study also ventured into interpretability aspects, a pressing concern in machine learning applications within environmental sciences. By juxtaposing linear PCA-derived features and nonlinear embeddings from t-SNE, the authors revealed distinct pathways toward explaining model predictions and fostering trust among stakeholders. Transparent models grounded in understandable features enhance collaboration between data scientists, hydrologists, policymakers, and community managers.

This work arrives at a time when sustainable water management is thrust to the forefront of global challenges. As populations swell and climate patterns become more erratic, the pressure on groundwater reserves intensifies. Leveraging artificial intelligence not only to predict but also to optimize the utilization of these critical resources is essential. The insights provided by Wang and colleagues mark a significant step toward integrating advanced data science methodologies into practical groundwater management tools.

Looking ahead, the authors advocate for extending their comparative framework to incorporate emerging dimensionality reduction techniques such as Uniform Manifold Approximation and Projection (UMAP) and autoencoder-based methods. Such expansions can further enhance feature extraction fidelity and model adaptability. Moreover, coupling these approaches with remote sensing data and Internet of Things (IoT) sensors promises real-time, high-resolution groundwater monitoring capabilities previously unattainable.

In conclusion, this seminal study bridges hydrology, environmental science, and machine learning by rigorously assessing PCA and t-SNE for optimizing input features in groundwater level simulation. Its comprehensive evaluation framework, innovative hybrid strategies, and pragmatic recommendations advance both scientific understanding and practical applications. As water scarcity challenges mount globally, such interdisciplinary endeavors spotlight the transformative potential of data-driven solutions for securing our planet’s precious groundwater resources.

Subject of Research: Evaluation of PCA and t-SNE for optimizing input features in groundwater level simulation using machine learning models.

Article Title: Evaluating the efficacy of PCA and t-SNE in optimizing input features for groundwater level simulation using machine learning models.

Article References:
Wang, N., Zhou, Q., Gao, J. et al. Evaluating the efficacy of PCA and t-SNE in optimizing input features for groundwater level simulation using machine learning models. Environ Earth Sci 84, 336 (2025). https://doi.org/10.1007/s12665-025-12357-3

Image Credits: AI Generated