The core of the work is an algorithm the authors call MonteCarloDQN. It builds on the Deep Q-Network, a reinforcement learning method in which an agent learns by trial and reward: at each moment it observes the state of the equipment, chooses a diagnostic action, and receives a positive or negative reward depending on whether that decision was correct. Over many episodes the agent converges on a policy that maximizes its expected long-term reward. The researchers framed equipment diagnosis as a Markov decision process with three possible actions, Monitor, Alert and Repair, and three risk levels ranging from low to high probability of failure. A carefully designed reward table penalizes wrong calls heavily, for example punishing the agent with a score of minus fifty if it merely watches while the equipment sits at high risk, and rewarding it with plus forty for correctly recommending repair.

What makes the system novel is how its hyperparameters are chosen. The performance of a Deep Q-Network depends critically on settings such as the learning rate, batch size, dropout rate and the number of neurons in the hidden layer. Traditional approaches like Grid Search exhaustively test every combination and become computationally explosive as dimensions grow, while Random Search ignores what previous trials have already revealed. The team instead turned to Monte Carlo Tree Search, which adaptively explores the hyperparameter space by repeatedly selecting promising branches, expanding the tree with new configurations, simulating outcomes, and backpropagating the results. A mathematical criterion called UCT, the Upper Confidence bounds applied to Trees, balances the exploration of untested settings against the exploitation of known good ones. After twenty search iterations, the method settled on a learning rate of 0.001, a hidden layer of 64 neurons, dropout of 0.1 and a batch size of 16.

To prove the concept on real hardware rather than a purely simulated playground, the researchers built a laboratory test bench at the Industrial Automation Lab in Almaty. It pairs a Schneider Electric Altivar 630 frequency converter, the kind of drive that controls motor speed in conveyors, pumps, fans and compressors across industry, with a Modicon M340 programmable logic controller. A Python data logger polled the drive’s internal registers over Modbus TCP roughly five times per second, capturing output frequency, motor current, DC bus voltage, thermal state and fault codes. Because genuine equipment failures are rare and dangerous to provoke, the team generated faults using physics-informed synthesis: by abruptly flipping the analog reference voltage from plus eight to minus eight volts, they forced the drive to reverse direction within a single controller scan, reliably triggering realistic overcurrent faults whose signatures matched the expected physical behavior.

The resulting dataset contained 25,754 records across 37 features, covering six equipment states from normal running through acceleration, deceleration and stopping, to the crucial pre-fault phase in which a developing malfunction shows its first signs, and finally the fault state itself. Four fault families were simulated: overcurrent, DC bus overvoltage, undervoltage and overtemperature. The researchers added Gaussian noise and sinusoidal perturbations during training to make the model robust to the measurement noise that plagues real factories, and used a sliding window of five observations so the agent could learn from sequences of states rather than isolated snapshots, an advantage over classifiers like support vector machines that judge each moment in isolation.

The results were striking. Across five independent runs with different random seeds, the MonteCarloDQN with noise augmentation achieved an accuracy of 0.9609, balanced accuracy of 0.9368, precision of 0.9617, recall of 0.9609, F1-score of 0.9596 and G-mean of 0.9571, outperforming baseline Deep Q-Network, support vector machine, K-nearest neighbors and Informer Transformer models. Just as important, the small standard deviations showed the method was stable regardless of initialization. Statistical testing backed this up: the Friedman test found significant differences between models, and Nemenyi post-hoc analysis confirmed that the proposed method significantly beat the basic DQN on balanced accuracy, precision, recall and F1-score, and also outperformed the Informer Transformer on those metrics.

Perhaps the most consequential part of the paper is not the algorithm at all but how it is bolted onto the factory. The researchers integrated their AI module into the GEMMA model, a guide for organizing the operating modes of automated systems originally developed by the French agency ADEPA. GEMMA divides a machine’s life into zones covering shutdown, normal operation and failure. The team modified Zone D, the failure and emergency zone, adding three new signals: AI_ready, indicating the analysis is complete; AI_fault, flagging a diagnosed problem; and AI_validation, requiring an operator to confirm the AI’s verdict. When the agent detects a fault and the operator confirms, the GEMMA state machine executes the emergency transition from normal operation to emergency stop, then walks through recovery steps back to production.

Critically, the AI never overrides the existing safety logic. If the operator fails to respond, or if communication with the AI module is lost entirely, the standard protection chain in the programmable controller continues to function independently, so a critical failure still triggers an emergency stop. Timing measurements showed the whole loop, from reading the drive’s registers to returning a diagnostic result to the controller, takes up to 45 milliseconds over a Modbus TCP link, comfortably within the 100-millisecond scanning cycle during which the PLC completes about five of its own 20-millisecond scan cycles. The AI therefore acts as an advisory layer that adds intelligence without injecting stochastic behavior into the deterministic control loops that keep workers and equipment safe.

The authors are candid about limitations. The technology was validated on one class of equipment under laboratory conditions, and transferring it to other drives or controllers requires retraining on new data, even though the underlying standards, including IEC 61131-3 for PLC programming and the IEC 61800 series for power drive systems, mean the architecture itself can be reused across manufacturers. Growing datasets will demand more computing resources, and success on synthesized fault scenarios does not guarantee coverage of every failure mode a real plant will encounter. Still, the demonstration that a game-search-inspired optimizer can sharpen reinforcement learning for fault diagnosis, while slotting cleanly into a decades-old industrial safety framework, offers a compelling template for Industry 4.0: intelligent, self-improving diagnostics that earn the trust of the factory floor by knowing exactly where their authority ends. The team’s next goal is a full production-environment evaluation, weighing the economic and computational costs of running such systems at scale.

Subject of Research: Reinforcement learning-based fault diagnosis of industrial equipment integrated with the GEMMA control model

Article Title: Monte Carlo-driven reinforcement learning for smart manufacturing systems

Article References: Samigulina, G., Samigulina, Z., Dyussenkulova, B., Ben Yahia, S., & Permitin, B. (2026). Monte Carlo-driven reinforcement learning for smart manufacturing systems. Discover Artificial Intelligence, 6(1), Article 1285. https://doi.org/10.1007/s44163-026-02302-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02302-z

Keywords: reinforcement learning, Monte Carlo Tree Search, Deep Q-Network, smart manufacturing, fault diagnosis, GEMMA model, Industry 4.0, programmable logic controller, frequency converter, predictive maintenance, hyperparameter optimization, industrial automation