The shepherding problem, as the authors describe it, involves designing the movement law of steering agents, called shepherds, to navigate another set of agents, called sheep, which are driven by repulsive forces from the shepherds. The concept is bio-inspired, drawing on the way real sheepdogs herd livestock, and it has attracted attention for applications in robotics, group dynamics, and even nanochemistry, where active particles could potentially steer passive ones. Most existing algorithms, however, assume a central coordinator that can observe the entire system and communicate instructions to every shepherd, an assumption the Osaka team argues is not realistic enough for emerging applications where such infrastructure may be unavailable or unreliable.

To understand what makes the new approach different, it helps to look at the underlying sheep model. Each sheep agent moves in discrete time steps on a two-dimensional plane, its movement vector composed of five forces: separation, alignment, cohesion, repulsion from shepherds, and a random disturbance. The first three forces are a modified implementation of the classic Boid model introduced by Craig Reynolds in 1987, which has become a standard framework for simulating flocking behavior. Separation ensures agents do not collide, alignment steers them toward the average heading of local neighbors, and cohesion pulls them toward the average position of nearby agents. The repulsion force from shepherds uses a diverging potential function so that the closer a shepherd gets, the stronger the sheep’s urge to move away, which is precisely the mechanism that makes herding possible in the first place.

The shepherds themselves operate under strict information constraints. Each has a finite recognition range and can detect only its relative position to the goal, the relative positions of sheep and other shepherds within that range, and nothing else. The authors note that the first two types of sensing could be realized with proximity sensors, while goal-relative positioning could be achieved with recent methodologies such as single beacon-based positioning systems. Crucially, no shepherd ever needs to know what another shepherd intends or plans, which is what distinguishes this work from earlier decentralized algorithms that still implicitly required communication, such as one that needs shepherds to understand each other’s intentions or another that requires multiple rounds of messaging to reach consensus on which sub-flock each shepherd should chase.

The algorithm builds on an existing single-shepherd method called the Farthest-Agent Targeting algorithm, or FAT. In FAT, the shepherd chases the sheep farthest from the goal, maintaining an appropriate distance before pushing it toward the goal region. While effective for a single shepherd, FAT has two problems when generalized to multiple shepherds: all shepherds would target the same sheep, and the flock tends to scatter when a shepherd chases a sheep on the opposite side of the flock. The Osaka team’s extension introduces a parameter alpha that modifies target selection. Each shepherd chooses as its target the sheep maximizing a weighted difference between the sheep’s distance from the goal and its distance from the shepherd. When alpha is zero, the algorithm reduces to FAT. When alpha is large, each shepherd simply targets the closest sheep, preventing the scattering phenomenon. A moderate value balances both objectives.

Cooperation between shepherds emerges not from any messaging but from the spatial distribution induced by inter-shepherd repulsion built into the algorithm. Each shepherd’s movement vector is a weighted sum of four components: chasing its target sheep, maintaining an appropriate distance from neighboring sheep, pushing toward the goal region, and repelling other shepherds to avoid competition. The repulsion between shepherds is weighted by the shepherd’s distance from the goal, because shepherds need to be relatively close to each other at the final stage of navigation. The result is that although each shepherd independently chases its own target, the collective behavior resembles coordinated herding, a form of emergent cooperation arising purely from local rules and physical proximity.

The authors evaluated the algorithm through extensive numerical simulations across three initial flock configurations, a small flock of twenty sheep, a large flock of fifty, and two separate flocks of twenty and thirty, combined with three shepherd placement patterns. They compared the proposed method against FAT, FAT with occlusion, and an online-target switching algorithm. The proposed method achieved almost one hundred percent success rate regardless of the number of shepherds or placement pattern, while also outperforming the baselines in completion time and average path length. Both completion time and path length steadily decreased as more shepherds were added, suggesting stable and synergistic coordination. Notably, the online-target switching algorithm failed entirely at herding two separate flocks, while the proposed method handled them successfully.

The team also tested the algorithm under two additional parameter sets, one where sheep were more sensitive to shepherd movements and one where they were harder to navigate. The proposed method consistently showed higher success rates, lower completion times, and shorter path lengths than the baselines across all scenarios. A slight decrease in success rate appeared under the sensitive scenario, which the authors attributed to interference behaviors when multiple shepherds coincidentally chose the same target and drove it further from the goal, and to the difficulty of simultaneously containing all sheep within the goal region when the flock parameters made them more reactive.

Perhaps most striking was the comparison with centralized algorithms. The authors tested point-offset circling control, where shepherds form an arc formation directed by a central controller, and potential-based caging, where shepherds occupy the vertices of a regular polygon surrounding the flock. Both centralized methods showed increasing completion times and decreasing success rates as the number of shepherds grew, and neither could handle two separate flocks at all. The authors analyzed that the poor performance stemmed from the large interaction distance between shepherds and the flock, and that centralized algorithms were neither robust nor resilient enough to adapt to changes in flock shape caused by sheep movement. The communication-free approach, by contrast, thrived precisely where the centralized methods struggled.

The robustness and resilience tests round out the picture. When the shepherds’ sensing range was varied from fifty to four hundred units, success rates dropped only when the range was too short, suggesting the algorithm needs shepherds with reasonably long sensing capabilities but tolerates substantial variation otherwise. When shepherds were prone to random failures with probabilities up to forty percent per time step, with recovery probability of ten percent, the algorithm maintained high success rates, with completion times rising but traversal distances remaining almost unchanged. Even with additive sensing noise weighted at ten units, the algorithm tolerated relatively large sensing errors across all initial configurations. The authors attribute this fault tolerance to the decentralized mechanism itself, since no single shepherd is indispensable and each operates independently based on its own observations. Future work, they note, includes investigating whether the communication-free coordination mechanism can extend to other navigation tasks and establishing mathematical properties such as stability and convergence, which remain challenging due to the nonlinearity of the underlying swarm model.

Subject of Research: Communication-free decentralized shepherding navigation of passive agent flocks with multiple steering agents

Article Title: Communication-free shepherding navigation with multiple steering agents

Article References: Li, A., Ogura, M., & Wakamiya, N. (2023). Communication-free shepherding navigation with multiple steering agents. Frontiers in Control Engineering, 4, Article 989232. https://doi.org/10.3389/fcteg.2023.989232

Image Credits: AI Generated

DOI: 10.3389/fcteg.2023.989232

Keywords: shepherding, swarm guidance, decentralized control, multi-agent systems, flocking, bio-inspired robotics, communication-free navigation, emergent coordination, robustness, resilience, control engineering, agent-based modeling

The appeal of the approach lies in what it leaves out. Deep reinforcement learning methods such as Deep Q-Networks have demonstrated impressive results in building control, learning nonlinear strategies directly from sensor data without an explicit model of the building’s thermodynamics. Yet their adoption in real building energy management systems has been hampered by practical friction: training instability, high computational cost, and hardware demands that exceed what typical controllers installed in commercial buildings can provide. A tabular learner, by contrast, stores its knowledge as an explicit table mapping situations to action values, which makes it transparent, compact, and fast. The Tanta team’s best-performing controller made decisions in about 45 microseconds per zone and required a Q-table of only 1.6 megabytes, figures that place the method squarely within the reach of resource-constrained building automation hardware.

SARSA, which stands for State-Action-Reward-State-Action, belongs to the family of on-policy temporal-difference learning algorithms. Unlike its off-policy cousin Q-learning, which learns the value of the greedy action regardless of what the agent actually does, SARSA evaluates the policy it is currently following, updating its action values based on the action it actually takes next. This distinction matters in a building context, where exploration must be gentle and where an overly aggressive update can translate into uncomfortable temperature swings for occupants. The on-policy character of SARSA tends to produce more conservative, stable learning, which the authors identified as a key reason for choosing it over deep alternatives for a safety-sensitive application like climate control.

The central technical challenge was inter-zone coupling. In an open-plan office, warm air drifting from a sunlit zone near the facade alters the cooling load of the interior zone beside it, so a controller that observes only its own local temperature is fundamentally flying blind. The researchers addressed this with a coupling-aware observation design: each zone agent receives, in addition to its local conditions, an aggregate summary of the temperatures in neighboring zones. This lightweight information sharing allows each tabular learner to compensate for the heat exchanged across the invisible boundaries of the open floor, without requiring a centralized controller that would scale poorly as the number of zones grows. The result is a decentralized architecture in which simple agents, each with a modest memory footprint, collectively manage a thermally entangled space.

Training took place in a simulated six-zone office environment driven by TMY3 typical meteorological year weather data from the EnergyPlus weather library, ensuring that the agents experienced a realistic annual profile of outdoor temperatures and solar conditions. Occupancy was modeled with stochastic hybrid-work schedules, reflecting the irregular attendance patterns that have become common in post-pandemic offices and that routinely defeat fixed timetables. The reward function was deliberately multi-objective, balancing thermal comfort against energy consumption and penalizing excessive equipment switching, since frequent ON/OFF cycling of HVAC components accelerates wear and shortens equipment life. The best-trained policy settled at an average of 4.1 switching actions per zone per day, a level the authors suggest is compatible with practical equipment longevity.

The headline result, a 39.39 percent reduction in annual HVAC energy use relative to a clearly defined schedule-based baseline, comes with an important caveat of scientific honesty: the comparison target is a conventional scheduled controller, not a state-of-the-art model predictive control system. Even so, the robustness of the finding is strengthened by replication. Across ten independent training runs, the final energy savings converged to 38.0 plus or minus 0.9 percent, indicating that the learning process is reliable rather than the product of a lucky random seed. The authors also conducted a sensitivity analysis over the reward weights, mapping out the trade-off curve between comfort and energy and showing how the balance can be tuned to a building operator’s priorities.

Thermal comfort was assessed against established standards for human occupancy, and the trained controller held violations to 2.85 percent of occupied hours, a figure that suggests the energy savings did not come at the expense of shivering or sweltering occupants. This comfort-energy balance is the perennial tension of building automation: the cheapest way to save energy is simply to switch everything off, but the value of a learning controller lies in discovering subtler strategies, such as pre-conditioning zones ahead of occupancy, exploiting thermal inertia, and modulating output in response to predicted heat flows between zones. The coupling-aware observations appear to be what enables these strategies, since agents that understand their neighbors’ states can coordinate implicitly through the physics of the shared air volume.

The study situates itself within a rapidly expanding literature on reinforcement learning for building energy management, a field that has produced deep learning controllers demonstrated in real buildings and multi-agent frameworks for allocating energy across zones. What distinguishes this contribution is its deliberate minimalism. Where much of the field has pursued ever-larger neural function approximators, the Tanta researchers pursued the opposite direction, asking how much intelligence can be extracted from the simplest possible learner if the state representation is designed thoughtfully. Their answer is that a tabular method, often dismissed as incapable of handling high-dimensional control problems, becomes competitive when the problem is decomposed sensibly and the agents are given just enough information about coupling to act intelligently.

The practical implications are significant for the building industry, where retrofitting advanced control into existing stock is often blocked by cost and complexity. A controller whose entire learned knowledge fits in 1.6 megabytes and whose decision latency is measured in microseconds can run on the embedded processors already present in many building energy management systems, without cloud connectivity or GPU acceleration. That said, the authors are explicit that their results are simulation-based, and embedded field validation remains future work. The gap between simulated and real buildings, with unmodeled sensor noise, actuator degradation, and occupant behavior, is where many promising control algorithms have stumbled, and the transition from the EnergyPlus simulation to a physical open-plan office will be the true test.

Still, the study offers a compelling counterpoint to the assumption that progress in building intelligence must ride on ever-deeper networks. As buildings account for a substantial share of global energy demand and HVAC represents one of the largest end uses in commercial spaces, even fractional improvements compound into meaningful climate impact. If a decades-old, on-policy learning rule, given a clever observation design and a well-balanced reward, can deliver nearly 40 percent savings with hardware that costs almost nothing extra, the barrier to adoption drops dramatically. The work suggests that for many practical control problems, the smartest move may not be a bigger model, but a better-posed question, and it hands building engineers a lightweight, stable, and interpretable tool that could bring learning-based control out of the laboratory and into the ordinary office floor.

Subject of Research: On-policy reinforcement learning with SARSA for energy-efficient multi-zone HVAC control in open-plan office buildings

Article Title: Enhancing energy management in multi-zone buildings using the on-policy reinforcement learning algorithm SARSA

Article References: Attia, M. A., Abdelaal, M. A., & Sallam, E. A. (2026). Enhancing energy management in multi-zone buildings using the on-policy reinforcement learning algorithm SARSA. Neural Computing and Applications, 38(17), Article 717. https://doi.org/10.1007/s00521-026-12445-w

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12445-w

Keywords: SARSA, reinforcement learning, HVAC control, building energy management, open-plan offices, multi-zone buildings, thermal comfort, energy optimization, tabular learning, smart buildings, EnergyPlus simulation, decentralized control