The appeal of the approach lies in what it leaves out. Deep reinforcement learning methods such as Deep Q-Networks have demonstrated impressive results in building control, learning nonlinear strategies directly from sensor data without an explicit model of the building’s thermodynamics. Yet their adoption in real building energy management systems has been hampered by practical friction: training instability, high computational cost, and hardware demands that exceed what typical controllers installed in commercial buildings can provide. A tabular learner, by contrast, stores its knowledge as an explicit table mapping situations to action values, which makes it transparent, compact, and fast. The Tanta team’s best-performing controller made decisions in about 45 microseconds per zone and required a Q-table of only 1.6 megabytes, figures that place the method squarely within the reach of resource-constrained building automation hardware.

SARSA, which stands for State-Action-Reward-State-Action, belongs to the family of on-policy temporal-difference learning algorithms. Unlike its off-policy cousin Q-learning, which learns the value of the greedy action regardless of what the agent actually does, SARSA evaluates the policy it is currently following, updating its action values based on the action it actually takes next. This distinction matters in a building context, where exploration must be gentle and where an overly aggressive update can translate into uncomfortable temperature swings for occupants. The on-policy character of SARSA tends to produce more conservative, stable learning, which the authors identified as a key reason for choosing it over deep alternatives for a safety-sensitive application like climate control.

The central technical challenge was inter-zone coupling. In an open-plan office, warm air drifting from a sunlit zone near the facade alters the cooling load of the interior zone beside it, so a controller that observes only its own local temperature is fundamentally flying blind. The researchers addressed this with a coupling-aware observation design: each zone agent receives, in addition to its local conditions, an aggregate summary of the temperatures in neighboring zones. This lightweight information sharing allows each tabular learner to compensate for the heat exchanged across the invisible boundaries of the open floor, without requiring a centralized controller that would scale poorly as the number of zones grows. The result is a decentralized architecture in which simple agents, each with a modest memory footprint, collectively manage a thermally entangled space.

Training took place in a simulated six-zone office environment driven by TMY3 typical meteorological year weather data from the EnergyPlus weather library, ensuring that the agents experienced a realistic annual profile of outdoor temperatures and solar conditions. Occupancy was modeled with stochastic hybrid-work schedules, reflecting the irregular attendance patterns that have become common in post-pandemic offices and that routinely defeat fixed timetables. The reward function was deliberately multi-objective, balancing thermal comfort against energy consumption and penalizing excessive equipment switching, since frequent ON/OFF cycling of HVAC components accelerates wear and shortens equipment life. The best-trained policy settled at an average of 4.1 switching actions per zone per day, a level the authors suggest is compatible with practical equipment longevity.

The headline result, a 39.39 percent reduction in annual HVAC energy use relative to a clearly defined schedule-based baseline, comes with an important caveat of scientific honesty: the comparison target is a conventional scheduled controller, not a state-of-the-art model predictive control system. Even so, the robustness of the finding is strengthened by replication. Across ten independent training runs, the final energy savings converged to 38.0 plus or minus 0.9 percent, indicating that the learning process is reliable rather than the product of a lucky random seed. The authors also conducted a sensitivity analysis over the reward weights, mapping out the trade-off curve between comfort and energy and showing how the balance can be tuned to a building operator’s priorities.

Thermal comfort was assessed against established standards for human occupancy, and the trained controller held violations to 2.85 percent of occupied hours, a figure that suggests the energy savings did not come at the expense of shivering or sweltering occupants. This comfort-energy balance is the perennial tension of building automation: the cheapest way to save energy is simply to switch everything off, but the value of a learning controller lies in discovering subtler strategies, such as pre-conditioning zones ahead of occupancy, exploiting thermal inertia, and modulating output in response to predicted heat flows between zones. The coupling-aware observations appear to be what enables these strategies, since agents that understand their neighbors’ states can coordinate implicitly through the physics of the shared air volume.

The study situates itself within a rapidly expanding literature on reinforcement learning for building energy management, a field that has produced deep learning controllers demonstrated in real buildings and multi-agent frameworks for allocating energy across zones. What distinguishes this contribution is its deliberate minimalism. Where much of the field has pursued ever-larger neural function approximators, the Tanta researchers pursued the opposite direction, asking how much intelligence can be extracted from the simplest possible learner if the state representation is designed thoughtfully. Their answer is that a tabular method, often dismissed as incapable of handling high-dimensional control problems, becomes competitive when the problem is decomposed sensibly and the agents are given just enough information about coupling to act intelligently.

The practical implications are significant for the building industry, where retrofitting advanced control into existing stock is often blocked by cost and complexity. A controller whose entire learned knowledge fits in 1.6 megabytes and whose decision latency is measured in microseconds can run on the embedded processors already present in many building energy management systems, without cloud connectivity or GPU acceleration. That said, the authors are explicit that their results are simulation-based, and embedded field validation remains future work. The gap between simulated and real buildings, with unmodeled sensor noise, actuator degradation, and occupant behavior, is where many promising control algorithms have stumbled, and the transition from the EnergyPlus simulation to a physical open-plan office will be the true test.

Still, the study offers a compelling counterpoint to the assumption that progress in building intelligence must ride on ever-deeper networks. As buildings account for a substantial share of global energy demand and HVAC represents one of the largest end uses in commercial spaces, even fractional improvements compound into meaningful climate impact. If a decades-old, on-policy learning rule, given a clever observation design and a well-balanced reward, can deliver nearly 40 percent savings with hardware that costs almost nothing extra, the barrier to adoption drops dramatically. The work suggests that for many practical control problems, the smartest move may not be a bigger model, but a better-posed question, and it hands building engineers a lightweight, stable, and interpretable tool that could bring learning-based control out of the laboratory and into the ordinary office floor.

Subject of Research: On-policy reinforcement learning with SARSA for energy-efficient multi-zone HVAC control in open-plan office buildings

Article Title: Enhancing energy management in multi-zone buildings using the on-policy reinforcement learning algorithm SARSA

Article References: Attia, M. A., Abdelaal, M. A., & Sallam, E. A. (2026). Enhancing energy management in multi-zone buildings using the on-policy reinforcement learning algorithm SARSA. Neural Computing and Applications, 38(17), Article 717. https://doi.org/10.1007/s00521-026-12445-w

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12445-w

Keywords: SARSA, reinforcement learning, HVAC control, building energy management, open-plan offices, multi-zone buildings, thermal comfort, energy optimization, tabular learning, smart buildings, EnergyPlus simulation, decentralized control

The core difficulty lies in the mismatch between how standard reinforcement learning agents learn and how a moving charging vehicle actually behaves. In the classic SARSA algorithm, an agent learns by trial and error, updating the value of state-action pairs as it experiences rewards and penalties. That works reasonably well when the world is static, because the consequences of a given action in a given cell of a grid map stay consistent across episodes. But a mobile charging vehicle follows irregular motion patterns that are difficult to predict, which means the reward landscape itself shifts beneath the learner. The researchers identify three intertwined failure modes in this setting: exploration is inefficient because the agent wastes effort sampling regions of the environment that no longer matter, the vehicle’s motion is irregular enough to defeat simple models, and prediction errors about where the charger will be grow large enough to poison the learned policy.

ASDE, which stands for adaptive SARSA for dynamic environments, attacks all three problems at once. The first ingredient is an explicit motion model of the mobile charging vehicle woven directly into the learning framework, so that the drone’s planner has a structured expectation of how the charger moves rather than treating its position as an arbitrary, unknowable quantity. The second is a restructured reward function that reshapes the feedback the agent receives, steering it toward trajectories that are not merely successful but also efficient and smooth. Together these changes give the learning process a much better-shaped objective, reducing the amount of random wandering the agent must do before it discovers useful behavior.

The third ingredient is perhaps the most conceptually interesting: a time-varying epsilon-greedy strategy. In textbook reinforcement learning, epsilon-greedy exploration means the agent takes a random action with a fixed probability epsilon and otherwise exploits its current knowledge. A fixed epsilon is a blunt instrument. Too high, and the agent never settles into the good policy it has found; too low, and it stops exploring before it has found one. ASDE instead adapts the exploration rate continuously according to environmental feedback, exploring more aggressively when the situation is uncertain or the charger’s motion has invalidated prior assumptions, and exploiting more heavily when the environment appears stable and the learned value estimates are trustworthy. This dynamic balance is what allows the algorithm to remain responsive in a setting where the target of the entire mission is itself in motion.

The fourth component extends the algorithm’s memory of its own trajectory. Standard SARSA is a one-step temporal difference method: each update looks only one step into the past. ASDE employs a hybrid temporal difference lambda mechanism with multi-step backtracking, which propagates credit backward across a stretch of recent states and actions rather than a single transition. Crucially, the effective backtracking horizon is not fixed. It adjusts dynamically based on the mobile charging vehicle’s instantaneous motion, stretching out when the charger’s behavior is predictable and contracting when it changes abruptly. The researchers report that this adaptive backtracking enhances both predictive accuracy and responsiveness, allowing the drone’s value estimates to track a moving target without the lag that plagues fixed-horizon methods.

To find out whether these design choices actually matter, the team ran simulation experiments across grid environments ranging from a compact 20 by 20 layout to a more demanding 50 by 50 space, comparing ASDE against benchmark algorithms including conventional SARSA. The results, as summarized in the paper, are consistent across scales. ASDE improved task success rates by between 4.4 and 10.4 percent relative to the benchmarks, a meaningful margin in a domain where a failed rendezvous can mean a drone stranded far from its base. The learned paths were also substantially shorter, with reductions of 21.2 to 30.7 percent in path length, which translates directly into energy saved and mission time recovered.

One of the most practically significant findings concerns path quality rather than raw performance. ASDE reduced the number of inflection points along planned trajectories by 33.0 to 46.0 percent compared with the benchmark algorithms. Inflection points are the sharp corners where a path changes direction abruptly, and for a flying vehicle each one costs energy and imposes maneuvering loads. A smoother path is easier for a flight controller to track accurately, gentler on the airframe, and more predictable for any surrounding traffic. That a learning algorithm can produce paths that are simultaneously shorter, more reliable, and smoother suggests the restructured reward function and adaptive exploration are doing real work in shaping the geometry of the solutions, not just the success statistics.

The convergence result deserves particular attention because learning speed is often the hidden bottleneck in deploying reinforcement learning on real hardware. An agent that needs millions of training episodes to converge is impractical to train in simulation for every new environment a drone might face. By reporting that ASDE reaches convergence approximately 45 percent faster than conventional SARSA, the authors are making a claim about deployability as much as about benchmark performance. Faster convergence means the planner can adapt more quickly when the mobile charging vehicle’s behavior shifts, and it lowers the computational cost of retraining, both of which matter for operations where conditions change from mission to mission.

The broader context makes the work timely. Drones are being pressed into service for search and rescue, infrastructure inspection, agricultural monitoring, and delivery, and in many of these roles the endurance limit of batteries is the binding constraint on how the technology can be used. Charging infrastructure that moves with the mission, rather than waiting at a fixed pad, is one of the more elegant proposed answers, and the concept has close cousins in research on electric vehicles and mobile robotic refueling. What has been missing is a planning layer robust enough to handle the uncertainty of a charger that is itself navigating traffic, terrain, and its own constraints. The Shenyang team’s contribution is a concrete, quantitatively evaluated step toward that layer, grounded in one of the workhorse algorithms of reinforcement learning rather than requiring an entirely new theoretical apparatus.

There are, of course, the usual caveats that separate simulation from the sky. The reported gains come from grid-world experiments of up to 50 by 50 cells, and real flight adds wind, sensing noise, communication delays, and safety constraints that no grid abstraction fully captures. The authors state that the datasets generated and analyzed in the study are available from the corresponding author on reasonable request, and the work was published in the International Journal of Aeronautical and Space Sciences on 14 July 2026 under the auspices of the Korean Society for Aeronautical and Space Sciences. Even with those caveats, the pattern of results is striking enough to suggest that the adaptive machinery at the heart of ASDE, the feedback-driven exploration schedule and the motion-aware temporal difference backtracking, could generalize beyond charging rendezvous to any drone task that involves intercepting a moving target. For a field where the difference between a 60 percent and a 70 percent success rate can determine whether an autonomous mission is viable at all, an algorithm that delivers higher success, shorter paths, smoother trajectories, and faster learning in a single package is the kind of incremental engineering advance that quietly enables the next generation of flying robots.

Subject of Research: Reinforcement learning-based path planning for UAV dynamic charging with mobile charging vehicles

Article Title: ASDE Algorithm-Based UAV Dynamic Charging Path Planning Method

Article References: Shan, D., Zhang, M., He, J., Zhang, T., & Li, Y. (2026). ASDE Algorithm-Based UAV Dynamic Charging Path Planning Method. International Journal of Aeronautical and Space Sciences. https://doi.org/10.1007/s42405-026-01253-0

Image Credits: AI Generated

DOI: 10.1007/s42405-026-01253-0

Keywords: UAV, reinforcement learning, SARSA, ASDE algorithm, mobile charging vehicle, path planning, dynamic environments, temporal difference learning, epsilon-greedy exploration, drone battery charging, autonomous flight, Shenyang Jianzhu University