AI Learns From a Model Teacher to Keep Multi-Microgrid Frequencies Stable

As solar panels and wind turbines take over more of the world’s power supply, the electricity grid is quietly losing something invisible but essential: inertia. Synchronous generators, the spinning masses that have kept grid frequency steady for a century, are being displaced by inverter-based renewables that contribute far less natural resistance to sudden power imbalances. For microgrids—small networks combining distributed generation, energy storage, and local loads—the result is a frequency-stability problem that grows harder with every megawatt of renewable capacity added. A new study published in Results in Engineering proposes a way out: teach a fleet of reinforcement-learning agents to coordinate frequency control across interconnected microgrids, but let them learn from a model-based teacher first.

The research, led by Yantao Lou and Jing He with colleagues Tonghui Wang and Ruiqian Liu, tackles the load frequency control (LFC) problem in a multi-microgrid system (MMS). In such systems, several microgrids are linked through tie lines, and power disturbances in one network propagate through the connections, superimposing on the random fluctuations already present in its neighbors. The frequency dynamics become strongly nonlinear and substantially harder to control. Electric vehicles, which can act as mobile energy storage through vehicle-to-grid (V2G) capability, add flexible regulation resources but also new uncertainties: their state of charge and the randomness of user behavior complicate the control problem further.

Traditional LFC approaches, from PID controllers to fuzzy logic and model predictive control (MPC), rely on explicit models of the system. They work well when the model matches reality, but under strong stochastic and intermittent disturbances they can suffer from model mismatch and limited adaptability. Machine learning methods, particularly deep reinforcement learning (RL), offer an alternative: agents learn control policies by interacting with the environment. Deep Q-Networks (DQN) have been applied to isolated microgrid frequency control with discrete action spaces, while Deep Deterministic Policy Gradient (DDPG) extends this to continuous control. But single-agent approaches rely on centralized information and decision-making, which is less suitable for a multi-microgrid system requiring local autonomy and privacy preservation, and may introduce communication bottlenecks and single-point failures.

Multi-agent reinforcement learning (MARL) addresses these limitations by giving each microgrid its own agent, achieving distributed coordinated control. Yet MARL brings its own problems: Q-value overestimation, inefficient early exploration, and slow convergence. The new study’s key insight is that the physical constraints and coordination information embedded in expert experience can be systematically exploited to improve training efficiency and policy stability. Rather than using model information only as auxiliary guidance, the researchers let MPC generate high-quality, constraint-satisfying expert trajectories through model-based rolling optimization, then use these demonstrations to guide the learning process.

The proposed framework combines a model of the multi-microgrid system—diesel generators and EV clusters as controllable frequency-regulation resources, photovoltaic generation and local loads as exogenous disturbances—with an MPC controller that solves a constrained optimization problem at each sampling instant. The MPC generates control sequences through receding-horizon optimization, explicitly considering constraints on control inputs, control increments, and system outputs. These state-action pairs are stored as an expert knowledge library, providing physically interpretable and constraint-satisfying prior knowledge for the subsequent MARL training.

The learning architecture is a Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MA-TD3), which mitigates Q-value overestimation through dual-critic networks, nonsmooth control actions through target policy smoothing, and policy oscillations through delayed policy updates. Each agent’s state vector includes the current and previous frequency deviations of all interconnected microgrids, the current and previous tie-line power variations, and the current and previous net source-load disturbances of its own microgrid. The action is a three-dimensional continuous vector corresponding to the regulation commands of the DG and two EV clusters in the microgrid. The reward function integrates local frequency regulation, inter-microgrid coordination, and penalties for excessive frequency deviations, with episode-dependent exploration and collaboration factors that gradually shift the agents from local stabilization to inter-microgrid coordination.

The training strategy is phased. In the imitation learning stage, agents learn state-action mappings from the MPC expert knowledge through behavior cloning, without environmental interaction. In the hybrid training stage, environmental interaction is introduced based on the policy obtained from imitation learning, with a proportion of expert experiences preloaded into the experience replay buffer. As environmental interaction data accumulate, their relative proportion increases while the influence of expert experiences gradually decreases. The agents transition smoothly from expert guidance to autonomous optimization, mitigating abrupt policy changes and reducing the risk of training instability and policy forgetting.

Simulation results on a two-microgrid validation platform, with parameters based on operational data from actual microgrids, demonstrate the effectiveness of the approach. Under a random load disturbance scenario, the proposed method achieves the lowest mean absolute frequency deviation, maximum absolute frequency deviation, and frequency-deviation standard deviation of 0.0019585 Hz, 0.0053834 Hz, and 0.0019452 Hz respectively, with a pass rate of 99.010 percent. Under a two-sided source-load random disturbance scenario, it again achieves the lowest values of the three deviation indices and a pass rate of 99.519 percent. In a DG fault exit scenario simulating the sudden loss of key frequency regulation resources, the proposed method limits the frequency deviation to approximately 0.005 Hz, achieving a pass rate of 99.704 percent. In scenarios where EV regulation capability is restricted to charging only or subject to time-varying limits, the proposed method consistently outperforms comparison methods including MA-TD3, MA-A2C, MPC, behavior cloning, and fixed-ratio expert replay.

A multi-seed statistical analysis across five independent training runs confirms the consistency of the proposed method. Across the five operating scenarios, it achieves mean absolute frequency deviations of 0.001667 to 0.002200 Hz, maximum absolute frequency deviations of 0.005793 to 0.007209 Hz, and pass rates of 97.2910 to 99.4330 percent, with generally lower across-seed variability than the comparison methods. A preliminary extension to a three-microgrid configuration, with each agent’s observation dimension increased from 8 to 10, maintains coordinated frequency regulation under disturbance, providing evidence that the framework can scale beyond the two-microgrid case.

The study acknowledges limitations: the simulations assume ideal and reliable communication links among microgrids, so communication delays, measurement noise, and packet loss are not explicitly considered. Future research will incorporate these uncertainties and investigate robustness under nonideal communication conditions. The researchers also note that because constraints are enforced during expert-data generation rather than directly on the learned policy during online execution, the guidance does not provide a formal safety guarantee during deployment; mechanisms such as distributional safety critics and risk-constrained policy optimization could be incorporated as complementary safety layers. Nevertheless, the framework demonstrates that systematically exploiting expert knowledge—rather than using it only as auxiliary guidance—can improve the efficiency and stability of multi-agent learning for coordinated frequency control in renewable-rich power systems.

Subject of Research: Reinforcement learning guided by model predictive control keeps multi-microgrid frequencies stable under renewable-rich conditions.

Article Title: Coordinated frequency control of multi-microgrids: A phased MARL guided by constraint-informed expert knowledge

Article References: Lou, Y., Wang, T., Liu, R., & He, J. (2026). Coordinated frequency control of multi-microgrids: A phased MARL guided by constraint-informed expert knowledge. Results in Engineering, 32, Article 113255. https://doi.org/10.1016/j.rineng.2026.113255

Image Credits: AI Generated

DOI: 10.1016/j.rineng.2026.113255

Keywords: microgrid, reinforcement learning, model predictive control, frequency stability, multi-agent systems, electric vehicles, renewable energy, load frequency control, Coordinated, frequency, control, multi-microgrids