Greedy Deep Kernel Method Outperforms Neural Networks in Approximation Tests

The work, published as an open access article in the journal Computational Science and Engineering, extends an existing algorithm called the Vectorial Kernel Orthogonal Greedy Algorithm, or VKOGA, by equipping it with kernels that are up to eight layers deep.

Kernel methods are a family of machine learning techniques built on a mathematical object called a positive definite kernel, which implicitly measures similarity between data points in a high-dimensional space known as a reproducing kernel Hilbert space, or RKHS. This so-called kernel trick allows linear algorithms to capture complex nonlinear relationships while resting on a solid theoretical foundation of representer theorems, error analysis and convergence statements. The catch, the authors note, is that classical kernel methods are tied to a fixed feature map: choosing the best kernel function and tuning its parameters remains a difficult task that strongly influences approximation quality, and for large datasets the cost of computing, storing and inverting the kernel matrix can quickly become prohibitive.

Neural networks, by contrast, learn hierarchical features automatically during training and handle high-dimensional data flexibly, but they typically need large training sets, offer limited interpretability and demand extensive hyperparameter tuning. The new approach aims to take the best of both worlds. Greedy kernel methods, such as VKOGA, build surrogate models as sparse expansions by iteratively selecting representative points, called greedy centers, from the training data. This sparsity makes evaluation computationally cheap, and the method comes with rigorous, sometimes optimal, convergence guarantees. Notably, the associated error bounds contain a dimension-independent factor, which helps mitigate the curse of dimensionality. The classical theory, however, assumes the target function actually lives in the RKHS of a fixed kernel, an assumption that is rarely verifiable in practice when the target is only known through data samples.

The deep kernel architecture relaxes that requirement by adapting the kernel itself to the data, thereby inducing a data-dependent RKHS. The authors formalize an L-layer deep kernel as a composition of kernel-induced feature maps, alternating between two kinds of layers. Odd-indexed layers are linear kernel layers, which enforce purely linear relationships between components and can be trained as weight matrices. Even-indexed layers are kernel activation layers, which apply a componentwise, block-diagonal kernel to individual input dimensions, introducing the nonlinearities needed for expressive function approximation. Unlike a standard neural network, where the activation function is chosen in advance, here the activation itself is a trainable kernel. The authors restrict the depth to even numbers, since an odd final linear layer would not increase expressiveness.

Training proceeds in two stages. First, the trainable parameters of the deep kernel are optimized over several stochastic batch epochs using a specifically tailored objective: a leave-one-out cross-validation error computed on each batch, made efficient through a classical algorithm due to Rippa that avoids explicitly inverting the kernel matrix. Gradients are obtained by automatic differentiation and used in a batch gradient-descent scheme with the Adam optimizer. Second, the trained deep kernel is treated as a fixed kernel inside the greedy VKOGA procedure, which selects greedy centers and computes the final expansion coefficients from interpolation conditions. The inner centers of the deep kernel are not free parameters; they are propagated recursively through the layer maps, a choice the authors describe as optimal according to a deep kernel representer theorem.

To probe how depth affects performance, the researchers first tested three model problems with two, three and four input dimensions, using 5,000 training and 1,000 test samples per function. Across the experiments, the deep VKOGA models achieved significantly smaller relative test errors than ReLU neural networks of comparable depth, and the most accurate VKOGA model beat the most accurate neural network for all three target functions. For the simplest function, which is essentially a Gaussian kernel itself, a two-layer model sufficed, while the harder functions benefited from four or more layers, indicating that deeper kernels help for higher-dimensional targets with heterogeneous characteristics. Beyond four layers, however, test errors stagnated while runtimes grew. In a controlled comparison with a nearly constant number of trainable parameters, depth proved clearly important for the deep kernels on the harder functions, whereas added depth brought the neural networks little or no benefit.

The second application concerned breakthrough curves, which describe how chemical species transported through reactive flow emerge from three-dimensional porous geometries such as catalyst structures. The dataset comprised 59 samples, each a 3D voxel geometry of about 3.4 million voxels paired with a breakthrough curve evaluated at 500 time steps. Because the raw geometry data are high-dimensional, the team first reduced them to six features using a principal component analysis. They compared discrete-time models, which predict the whole curve at once, with continuous-time models that predict the concentration at a single time point. Across three random train-test splits of 47 training and 12 test samples, the deep VKOGA models achieved smaller relative test errors than the neural networks in essentially all cases, and even the single-layer kernel model outperformed all neural networks. The discrete-time VKOGA models were also the most efficient in both training and prediction. A graph attention network, included as a modern spatio-temporal competitor, was less accurate than the kernel models and by far the most expensive, with online runtimes of several seconds and offline runtimes exceeding roughly 2,800 seconds.

The third problem class involved surrogate models for parameterized ordinary differential equations, namely the Lotka-Volterra predator-prey system and the Brusselator model of autocatalytic chemical reactions, with training sets of 36, 64 or 100 parameter samples and 100 test samples. Here the continuous-time VKOGA models were the most accurate surrogates for both systems, generally surpassing the discrete-time models and the graph neural networks by more than an order of magnitude, and even on the smallest training sets they captured the numerical solutions well, with only small deviations in the worst cases. In terms of efficiency, the discrete-time approaches were cheapest overall, while the continuous-time VKOGA models were usually slower than their neural network counterparts because the greedy iterations become expensive on larger datasets.

The authors are careful about the limits of their study. For very large datasets, deep kernel training, the greedy iterations and the online evaluation all demand more computational resources than the tested neural networks, since the leave-one-out loss is costlier than a mean-squared-error loss and each greedy step requires evaluating the training residual on the whole training set. They also note that only a limited set of architectures and hyperparameters was explored, so better configurations may exist for both model families. Still, across all three problem classes the deep VKOGA models consistently outperformed the neural networks in accuracy and, in many cases, in efficiency, supporting the authors’ conclusion that deeper kernel architectures are advantageous for complex approximation tasks and pointing toward future work such as convolutional deep kernels that could process 3D geometry data directly, without the PCA preprocessing step.

Subject of Research: Deep kernel greedy approximation methods and their comparison with neural networks for function approximation tasks

Article Title: Solving approximation tasks with greedy deep kernel methods

Article References: Klink, M., Ehring, T., Herkert, R., Lautenschlager, R., Göddeke, D., & Haasdonk, B. (2026). Solving approximation tasks with greedy deep kernel methods. Computational Science and Engineering, 3(1), Article 6. https://doi.org/10.1007/s44207-026-00012-z

Image Credits: AI Generated

DOI: 10.1007/s44207-026-00012-z

Keywords: kernel methods, deep kernels, VKOGA, greedy approximation, neural networks, surrogate modeling, reproducing kernel Hilbert space, breakthrough curves, porous media, parameterized ODEs, graph neural networks, machine learning