The research addresses a class of systems that engineers find particularly awkward: unknown, non-affine, discrete-time systems. In plain terms, these are processes sampled at fixed time steps, whose mathematical equations are either unknown or too complicated to write down, and in which the control input does not appear in a convenient, separable form. Most industrial equipment behaves this way in practice. Friction, saturation, and nonlinear electrical characteristics mean that the tidy models used in textbooks rarely match the hardware on the factory floor. Rather than trying to identify the system’s equations, the new approach is model-free: it learns directly from the data the system produces, adjusting its behavior online as measurements arrive.

The central insight of the work is that safety cannot be judged by the output alone. When a DC motor is commanded to follow a position trajectory, the variable that actually determines whether the machine survives is the motor current, an internal state that depends on the output and on the control action. If the controller drives the position aggressively, the current can spike, overheating the windings or tripping protective hardware. The study therefore treats constraints on these dependent internal states as first-class citizens in the control design, explicitly incorporating them into the learning objective rather than hoping that good output tracking will keep them in check as a side effect.

To achieve this, the author built the controller around a fuzzy-rule emulated actor-critic architecture. Actor-critic methods are a staple of reinforcement learning: one network, the actor, proposes control actions, while a second network, the critic, estimates how good those actions are and guides the actor toward better ones. The novelty here lies in the structure of these networks. Instead of conventional neural layers, they are built from fuzzy rules, which partition the input space into overlapping regions and respond with locally tuned outputs. This gives the architecture a piecewise, interpretable character that suits systems sampled in discrete time, and it allows the learning laws to be derived with clear analytical guarantees rather than relying purely on gradient heuristics.

Safety enters through a barrier function embedded in a hierarchical reward function. Barrier functions are a classical control-theory tool: they grow without bound as a state variable approaches a forbidden boundary, so any strategy that minimizes them is naturally repelled from unsafe regions. By folding such a barrier into a layered reward, the scheme makes the agent feel escalating penalties as the motor current nears its limits, while the upper levels of the hierarchy reward accurate tracking of the commanded position. The result is a controller that balances two competing goals in a principled way: it wants to follow the reference signal, but not at the cost of violating the constraints that keep the hardware healthy.

Learning in the scheme is carried out by two distinct online laws, each employing time-varying learning rates. Rather than adjusting the actor and critic with fixed step sizes, the algorithm modulates how aggressively it adapts depending on the operating conditions, which helps the networks converge quickly at the start of a task and then settle into stable, fine-grained refinement. The paper goes beyond simulation and empirical tuning: the effectiveness of these learning laws is theoretically demonstrated, with analysis showing that they ensure robust closed-loop performance. In the control community, where reinforcement learning proposals are often criticized for lacking formal guarantees, this combination of online adaptability and theoretical backing is a meaningful contribution.

The experimental validation is refreshingly concrete. Rather than testing only on numerical benchmarks, the author implemented the method on a real DC motor position control system, exactly the kind of electromechanical plant found in robotics, manufacturing lines, and automation equipment across industry. The results showed that the controller achieved sufficient tracking of the position reference while producing a significant reduction in the accumulating reward function compared with alternative approaches. In reinforcement learning terms, a lower accumulated reward cost means the controller found a way to do the job while incurring far less penalty, which here translates directly into gentler, safer behavior of the physical machine.

One of the most striking findings comes from an information-theoretic lens. The study compared the spectral entropy of the motor current, the constrained internal state, under the proposed scheme and under competing controllers. Spectral entropy measures how spread out and unpredictable a signal’s frequency content is. Under the new method, the spectral entropy of the current was significantly lower, and the probability distribution of the current was noticeably narrower. In practical terms, the motor current stayed in a tight, predictable band instead of wandering across a wide range of values. Narrow, low-entropy currents mean less thermal stress, less electrical noise, and smoother operation, which is precisely what a safety-oriented controller should deliver.

The implications extend well beyond a single motor on a laboratory bench. Model-free safe control of unknown discrete-time systems is a recurring need in fields as varied as wind turbine generators, electro-hydraulic servo systems, permanent magnet motor drives, and chemical process control, all of which appear in the paper’s extensive engagement with recent literature on adaptive dynamic programming, control barrier functions, and safe reinforcement learning. Many of those approaches either require a system model, handle constraints only on the input or the output, or offer no formal learning guarantees. By combining fuzzy-rule-based function approximation, hierarchical reward design, barrier-function safety, and provable online learning laws in a single model-free framework, the study offers a template that could be adapted to any plant where an observable output hides safety-critical internal states.

There are, of course, the usual caveats that accompany any single-author experimental study. The validation platform is a DC motor position control system, and extending the guarantees to higher-dimensional, multi-input systems with more complex constraint structures will require further work. The published article does not report external funding, and the author declares no conflict of interest. Still, the core message is likely to resonate widely: reinforcement learning controllers can be made genuinely safe, not by restricting them so heavily that they stop learning, but by building the physics of the constraints into the reward they optimize. As machines increasingly learn their own control policies online, approaches like this one, which watch the hidden variables that keep hardware alive, may become the standard grammar of trustworthy autonomous control.

Subject of Research: Model-free hierarchical reinforcement learning for safe output tracking with internal state constraints in unknown discrete-time systems

Article Title: Hierarchical reinforcement learning for safe output tracking: managing constraints of dependent internal states in unknown discrete-time systems

Article References: Treesatayapun, C. (2026). Hierarchical reinforcement learning for safe output tracking: managing constraints of dependent internal states in unknown discrete-time systems. Neural Computing and Applications, 38(18), Article 739. https://doi.org/10.1007/s00521-026-12456-7

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12456-7

Keywords: reinforcement learning, safe control, actor-critic, fuzzy rules, barrier function, discrete-time systems, output tracking, DC motor control, state constraints, spectral entropy, model-free control, adaptive control

The core difficulty with conventional grid optimization is that it depends on knowing the network intimately. Optimal Power Flow, the workhorse of distribution management, requires accurate line impedances, converter parameters, and complete network states to compute the best operating point. In practical low-voltage hybrid AC/DC systems, that information is frequently incomplete, outdated, or simply unavailable, particularly as topologies change with new connections and distributed energy resources fluctuate with the weather. Artificial-intelligence approaches and metaheuristic algorithms such as particle swarm optimization have been proposed as alternatives, yet most still lean on accurate system models or extensive offline training. The researchers behind the new work argue that this model dependence is the fundamental bottleneck, and their solution dispenses with it entirely.

Their method, called online feedback optimization, treats the hybrid grid as a black box. Rather than solving power-flow equations inside the optimizer, the controller probes the network with small, carefully timed perturbations to its converter setpoints, measures how voltages, currents, and imported power respond, and estimates the input-output sensitivities directly from those responses. A projected augmented-Lagrangian update then nudges the active- and reactive-power commands of eight voltage source converters toward an operating point that minimizes power drawn from the upstream grid while keeping every monitored bus voltage and feeder current within its limits. The approach requires no analytical network model, no explicit network Jacobian, and no offline system identification, relying exclusively on real-time measurements from the physical system.

The novelty, the authors stress, lies not in any single algorithmic ingredient, since finite-difference sensitivity estimation, projected-gradient updates, and augmented-Lagrangian constraint handling are all established tools. The breakthrough is their unified extension to a coupled multi-converter hybrid AC/DC low-voltage network. Previous feedback-optimization frameworks were developed for single-domain AC or DC microgrids with limited controllable devices. The new formulation coordinates sixteen control dimensions across eight converters spanning both AC and DC subsystems, enforcing a vectorized structure of voltage limits at six critical buses, feeder-current limits on three feeders, and converter capability bounds, all within a single optimization layer.

To test the framework, the team built a hybrid AC/DC version of the CIGRE European low-voltage benchmark network, adding an 800-volt DC grid with underground cables and deploying voltage source converters at eight strategic buses, one operating in grid-forming mode and the rest in grid-following mode. The entire network was implemented in PyDAE, an open-source Python environment for differential-algebraic equation modeling, integrated with a one-millisecond time step and a sparse Newton-Raphson solver. Three 24-hour load scenarios were examined: a baseline conventional AC case, an AC-overload stress test peaking at 20:00, and a hybrid scenario in which 40 percent of total demand is shifted to the DC grid. The scenarios were deliberately chosen to progress from familiar operation to the most technically challenging conditions a modern feeder is likely to face.

The results are compelling. Under the AC-overload scenario, the uncontrolled network suffered numerous voltage deviations and current stresses, and a conventional particle swarm optimizer running under the same 15-iteration budget failed to contain them, leaving the commercial feeder 16.6 amperes over its limit. The proposed feedback optimizer, by contrast, kept every bus voltage within the 0.95 to 1.05 per-unit band and every feeder current safely below its maximum. In the hybrid scenario with 40 percent DC penetration, the gap widened further: the swarm optimizer still exceeded the commercial feeder limit by 26.1 amperes, while the model-free approach eliminated all violations. Constraint satisfaction, not energy savings, is the method’s defining achievement, and it delivered that security without ever seeing a network equation.

Energy and reactive-power benefits followed as secondary consequences of the improved operating point. Because load demand was identical in controlled and uncontrolled cases, the reductions in imported energy reflect lower network losses: 125.5 kilowatt-hours in the base case, 161.2 in the AC-overload case, and 121.0 in the hybrid case, corresponding to savings of 1.2, 1.1, and 0.8 percent. More dramatic were the reactive-power results. The optimizer cut the 24-hour net reactive-energy exchange with the upstream grid by 68.2 percent under AC overload and 26.0 percent in the hybrid case, as the bidirectional converters strategically injected or absorbed reactive power near voltage-sensitive buses, at times even exporting reactive support back to the main grid.

Robustness testing reinforced the picture. The algorithm converged within 8 to 10 iterations across a tenfold range of perturbation steps, a three-decade range of penalty parameters, and measurement noise levels reaching 2 percent. Twenty independent randomized initializations under the harshest scenario all converged successfully, averaging 8.6 iterations. Computationally, the optimizer needed roughly 165 seconds per hourly control step on an ordinary workstation, occupying just 4.6 percent of the available supervisory interval. A fully tuned swarm optimizer with a much larger budget achieved near-identical energy savings, but only by exploiting full access to the network model, a luxury unavailable in real deployments.

The researchers are careful about what model-free means here. The optimizer needs no analytical power-flow equations, line parameters, or network Jacobian, but it does require predefined knowledge of converter locations, measurement channels, capability limits, and operational bounds. The quasi-steady-state RMS representation also does not capture converter switching harmonics, and communication delays are assumed negligible at the hourly timescale. These caveats frame the boundaries of the current work rather than undermining it, and the authors identify harmonic-aware models, delay-aware optimization, protection coordination for perturbation injection, and integration with energy-storage systems as future directions.

The implications extend well beyond one benchmark network. As distribution grids absorb ever more converters, electric-vehicle chargers, and distributed generation, the cost of maintaining accurate network models grows in step, and model-based tools become increasingly fragile. A controller that learns the grid’s behavior from its own responses, adapts continuously to changing loads, and scales linearly with the number of converters offers a practical path to resilient operation without infrastructure investment. The 44 to 59 megawatt-hours of annual loss savings per feeder are a welcome bonus, but the deeper promise is simpler: a grid that can keep itself safe even when nobody holds its blueprint.

Subject of Research: Model-free online feedback optimization for coordinating converters in hybrid AC/DC low-voltage distribution networks

Article Title: Model-free online feedback optimization for multi-converter hybrid AC/DC low-voltage distribution networks

Article References: Model-free online feedback optimization for multi-converter hybrid AC/DC low-voltage distribution networks. (n.d.). https://doi.org/10.1016/j.rineng.2026.113217

Image Credits: AI Generated

DOI: 10.1016/j.rineng.2026.113217

Keywords: online feedback optimization, hybrid AC/DC networks, low-voltage distribution, voltage source converters, model-free control, augmented Lagrangian, CIGRE benchmark, smart grid, distributed energy resources, voltage regulation, PyDAE simulation, optimal power flow