The study, published in Frontiers in Manufacturing Technology, addresses a fundamental inefficiency in how production processes are traditionally optimized. Conventional experimental design methodologies, including full and fractional factorial designs, one-factor-at-a-time approaches, the Taguchi Method, and response surface modeling, are static and non-adaptive. They separate experiment design from execution and analysis, meaning that the entire experimental plan is laid out before a single trial is run. If early results suggest a more promising direction, the methodology offers no mechanism to adapt. In an environment where physical experimentation is time-consuming, costly, and resource-intensive, this rigidity represents a significant bottleneck for process optimization across the entire life cycle of a production process, from the design and development of new processes through failure analysis and optimization to adaptation and calibration in series production.

Bayesian Optimization, or BO, offers a fundamentally different approach. The technique, which traces its origins to the work of Močkus in the 1970s, is a sequential decision-making strategy particularly suited for optimizing expensive-to-evaluate black-box functions that lack a closed-form representation, do not provide function derivatives, and only allow for point-wise evaluation. BO consists of two core components: a surrogate model that approximates the input-output behavior of the system being optimized, and an acquisition function that guides the selection of parameter sets to evaluate next. After each experiment, the Bayes theorem is applied to update the surrogate model from a prior to a posterior belief, incorporating the new observation into the model’s understanding of the process. The acquisition function then uses the uncertainty quantification of the updated surrogate to maximize the information gain of each subsequent experiment while balancing the exploration-exploitation trade-off. When BO is applied to sequential experimental design in scientific and engineering experimentation, the resulting methodology is termed Bayesian experimental design, or BED.

Despite its promise, BED has not achieved widespread adoption in production engineering practice, and the reason is configuration complexity. Setting up a BO algorithm for a specific optimization problem requires extensive knowledge of both the algorithm and the process characteristics. The mean and covariance functions of the surrogate model, the choice of acquisition function, and the size of the initial data sampling must all be individually configured, and each of these choices significantly influences overall optimization performance. For Gaussian process surrogate models, the covariance function, also known as the kernel, is of particularly high criticality for model fidelity and the sample path behavior of the experimentation. Previous benchmark studies in materials science and computational fluid dynamics have explored the impact of BED configuration, but typically focusing on only two components in combination. None had systematically examined the combined interdependencies among acquisition function, surrogate model, and initial design in the context of production processes.

To close this gap, the research team designed a benchmark study of remarkable scope. They selected eight engineering-focused synthetic test functions with different characteristics, complexities, and known optimal solutions. Four of these, designated F1 through F4, are purely mathematical problems. The remaining four, AdaptedBranin, Borehole, OTLCircuit, and WingWeight, are physically motivated optimization problems drawn from a virtual library of simulation experiments. Each test function was evaluated in both a noiseless and a noisy variant, with the noisy version introducing homoscedastic random noise at a level of 0.1 relative to the standard deviation of the function’s objective. In the configuration space, the team tested a total of 96 BO configurations obtained by combining eight kernels, four acquisition functions, and three initial sampling sizes in a full-factorial manner. The kernels included RBF, Matern05, Matern15, and Matern25, each investigated both as an isotropic version and as an anisotropic counterpart utilizing automatic relevance detection, or ARD, which implicitly determines the relevance of input parameters. The acquisition functions comprised expected improvement, probability of improvement, upper confidence bound, and Noisy Expected Improvement. Initial sampling sizes were set at small, five trials; medium, 10 trials; and large, 30 trials, all within a fixed total budget of 35 trials. Each configuration was run ten times with different random seeds applied to a pseudo-random Sobol sampling strategy, ensuring that all configurations started with the same initial data and that no configuration benefited from random fluctuations.

For performance evaluation, the team employed the dominated hypervolume metric, a multi-objective measure that integrates both robustness, quantified as the probability of failure across runs, and efficiency, quantified as the running time of successful experiments. Higher hypervolume values indicate superior overall performance, combining high success rates with fast convergence to the known optimum. A run was considered successful if the algorithm found a solution within a tolerance range corresponding to 95 percent optimization of the known optimal solution. The team then applied a nuanced classification approach that considered both performance level and variability across combinations, dividing hypervolume values into non-qualified, undetermined, and qualified areas for each test function. This approach was necessary because simple aggregation techniques, such as computing the median hypervolume across all runs, could distort actual performance when a configuration performs well in some combinations but poorly in others.

The results revealed significant performance differences between individual BO algorithm configurations, and the test functions themselves grouped naturally into three categories of optimization difficulty. Group A, comprising F1, F2, and OTLCircuit, proved relatively simple, with median hypervolume values of approximately 0.7 and interquartile ranges below 0.12, indicating that most BO configurations could achieve similar optimization potential. These functions showed low sensitivity to noise, with noisy results similar to noiseless ones, and F1 and F2 achieved a 100 percent success rate across almost all configurations. Group B, comprising F4 and WingWeight, presented moderate challenges with median hypervolume values around 0.5 and higher variability, though no significant differences between noiseless and noisy cases. Group C, comprising F3, AdaptedBranin, and Borehole, proved the most complex, with lower efficiency levels, significant differences between noiseless and noisy cases, and a pronounced impact of noise on robustness. For AdaptedBranin and Borehole, the median hypervolume dropped from 0.56 and 0.45 in noiseless cases to 0.2 in noisy cases, with probability of failure rising from 0.1 and 0.05 to 0.5 respectively, underscoring that careful configuration selection is essential when dealing with complex and noisy optimization problems.

Among the kernels, the Matern05 kernel was consistently classified as non-qualified across the test functions, indicating poor performance in optimizing the selected problems, and Matern05-ARD performed poorly in all test functions and was never classified as the best option. RBF showed decent performance for simple functions but struggled in more complex and noisy scenarios. RBF-ARD and Matern25-ARD exhibited similar behaviors, performing well on noiseless functions while presenting acceptable performance on noisy ones. Matern15 and Matern15-ARD showed promising performance, classified as qualified for most test functions. The isotropic Matern15 kernel outperformed the anisotropic version in group B, while the anisotropic version performed better in group C. As a general conclusion, group B seemed to be optimized better by isotropic kernels and group C by anisotropic ones, with Matern15 and Matern15-ARD appearing suitable for optimizing a wide range of problems.

Among the acquisition functions, probability of improvement consistently performed worse than all other options, a finding consistent with previous works that explained its poor performance due to its greedy nature. The upper confidence bound performed better but still encountered difficulties in several functions, potentially due to its fixed exploration parameter, which may have prioritized exploitation over exploration. Expected improvement and Noisy Expected Improvement stood out as the best performing options, with NEI showing the best performance across all test functions in both noiseless and noisy cases. Regarding initial sampling sizes, the small initial sampling size clearly outperformed the medium one in groups A and C, with the medium size only representing a better option in the noisy cases of group B. The large initial sampling size proved universally inefficient, with no experiment surpassing a hypervolume value of 0.15 across all test functions, a mathematically inevitable consequence of the minimum running time achievable by large experiments occurring at trial 30 out of a total budget of 35 trials.

From these findings, the team derived actionable guidelines for applying Bayesian experimental design in manufacturing. There is no one-fits-all solution, as different optimization problems place different demands on BO configuration. RBF presents a reasonable choice for simple test functions, while Matern15-ARD is recommended as a reasonable default option for complex optimization problems, with anisotropic kernels advisable for the more complex problems typically encountered in manufacturing. Since noise negatively impacts optimization performance, process and measurement noise should be minimized by precisely calibrating both actuators and sensors of the manufacturing process. Expected improvement and Noisy Expected Improvement are recommended as qualified default acquisition functions, while probability of improvement and the investigated upper confidence bound configuration are not suggested. Randomly generated initial experiments should be kept small, leaving the search for the optimum to a sufficiently exploratory BO algorithm, though already existing datasets should be utilized to initialize the algorithms. Screening and characterization trials prior to optimization are decisive for determining the most critical parameters and associated parameter ranges, keeping the dimensionality of the optimization problem as small but also as influential as possible. The study advocates BED as an adaptive, data-efficient tool for optimizing process parameters, achieving 95 percent precision within a budget of 35 iterations using the best-qualified configurations at various levels of complexity. The team acknowledges that the study did not unravel a clear relationship between the characteristics of optimization problems and the performance of BO configurations, suggesting that current characteristics do not adequately capture the inherent patterns of response to optimization. Further research focuses on the collaboration between BED and domain experts, the integration of expert knowledge into BED, and the extension to multi-objective optimization cases, all aimed at increasing comprehensibility, facilitating user acceptance, and promoting widespread adoption in industry applications.

Subject of Research: Bayesian experimental design in production engineering: a comprehensive performance and robustness study

Article Title: Bayesian experimental design in production engineering: a comprehensive performance and robustness study

Article References: Leyendecker, L., Gonzalez Degetau, A. M., Bata, K., Emonts, J., Schmitz, A., & Schmitt, R. H. (2026). Bayesian experimental design in production engineering: a comprehensive performance and robustness study. Frontiers in Manufacturing Technology, 5, Article 1614335. https://doi.org/10.3389/fmtec.2025.1614335

Image Credits: AI Generated

DOI: 10.3389/fmtec.2025.1614335

Keywords: Bayesian optimization, Bayesian experimental design, production engineering, manufacturing, process optimization, benchmark study, Gaussian processes, Frontiers in Manufacturing Technology, Bayesian, experimental, design, production