The study, published in Frontiers in Control Engineering by Keishu Utimula and colleagues at the Japan Advanced Institute of Science and Technology, together with Trevor J. Bihl of the Air Force Research Laboratory, demonstrates a form of machine learning that plans what the authors call information acquisition strategies through collective behavior. Rather than passively observing a malfunctioning agent, the system actively choreographs group maneuvers that force the competing explanations of the fault to produce visibly different outcomes. The work addresses a stubborn problem in swarm robotics: diagnosing failures in real time, when the very act of diagnosis must be planned as part of the mission itself.

The core of the dilemma is deceptively simple. Suppose a command base notices that one agent in a cooperative task has stopped changing position. Two hypotheses fit the observation. Under the first, the actuator has failed and the robot genuinely cannot move. Under the second, the robot is moving but its sensor no longer registers the movement, so the base sees a frozen position that does not reflect reality. Each hypothesis demands a different repair strategy, yet neither can be confirmed by talking to the agent, because the agent’s own reports are exactly what may be corrupted. Human operators intuitively resolve this by nudging: push the suspect and see whether it moves. If the position changes, the actuator works and the sensor is at fault; if nothing shifts, the actuator is broken.

What the researchers achieved was to turn that intuition into an autonomous planning problem. They formulated a value function that measures the difference between the states the group would reach under each hypothesis after executing the same command sequence. A good diagnostic plan is one that maximizes this difference, because only a large divergence between the predicted outcomes makes the true hypothesis identifiable. In their prototype, three agents move on a plane under the supervision of a command base, each tasked with carrying an item to a goal post. Agent #2 is assumed to have lost mobility along one axis, and the base must determine whether the actuator or the sensor is responsible before it can revise the mission plan.

The mathematical structure of the problem is what makes it interesting — and hard. For the overwhelming majority of possible action sequences, the two hypotheses lead to indistinguishable outcomes, so the value function is flat almost everywhere. Only a tiny fraction of plans, chiefly those involving physical contact between agents, produce a nonzero difference. This is the classic sparse reward problem: gradient-based optimization, which relies on a slope pointing toward better solutions, finds nothing to climb when nearly the entire landscape is level. Reinforcement learning, which samples actions and accumulates value through the Bellman equation rather than following gradients, is far better suited to searching such a barren landscape for the rare pockets of signal.

The researchers implemented their system using the PPO2 algorithm, proximal policy optimization version 2, from the OpenAI Gym library, running on a Linux server with a single processor. The learning phase took roughly three hours. They benchmarked two network architectures, a multilayer perceptron and a long short-term memory network, and found that the simpler multilayer perceptron performed better, achieving a 94 percent success rate in generating trajectories that distinguish the hypotheses, compared with 78 percent for the LSTM. The authors attribute this to the LSTM’s more complex internal structure, which would require more iterations to reach comparable quality. Notably, the LSTM still produced nearly identical agent behaviors, suggesting the learned strategy is robust to the choice of architecture.

The striking result is what the learned plan looks like. Without any rule-based instruction to collide, the reinforcement learning agent converged on the optimal solution: steer the healthy agents so that one of them rams into the malfunctioning robot. The collision is the diagnostic instrument. If the suspect agent is displaced, its actuator works and the fault lies in its sensor; if it stays put despite the impact, the actuator is dead. The branching between the two predicted trajectories occurs exactly at the moment of contact, and the value function registers the earned difference, confirming that the learning captured the discriminating event on its own.

Diagnosis, however, was only half the mission. Once the base identified the actuator failure, it launched a second learning phase to plan the actual delivery task under the agent’s reduced capability. Here the emergent behavior took on an almost social quality. Agents #1 and #3 completed their own deliveries and then headed toward the crippled teammate. Agent #2 struggled along its one working axis, and at the closest point of approach all three agents coalesced, with the healthy pair pushing the malfunctioning one toward its goal. The authors are careful to note that this apparent altruism is simply the consequence of maximizing the collective value function — the team earns more when the mission succeeds — yet the result reads as if the group autonomously decided to assist its fallen member.

The framework generalizes beyond the three-agent prototype. The authors describe a formulation in which the state is a vector of agent coordinates, each hypothesis imposes its own constraints on how commands update the state, and the objective is to find a plan that maximizes the summed difference across all pairs of hypotheses. This means the same machinery could handle scenarios with multiple simultaneous faults across several sensors and actuators, where diagnosis becomes far more tangled. Prior approaches, such as the immune-system-inspired fault diagnosis of O’Keeffe and colleagues, learn from past diagnostic episodes but struggle with unpredictable or simultaneous faults. The new method sidesteps that limitation by designing experiments on the fly rather than matching symptoms to a library.

Challenges remain before such systems leave the simulation. Training runs that failed to reach the target reward were reset to escape local optima, and the authors note that planning with more malfunctioning agents, or with a larger hypothesis space, will produce more rejected trials and demand more learning. Computational cost also grows: evaluating pairwise distances between agents scales with the square of the agent count, though domain decomposition methods that pre-register voxel distances can reduce this to linear scaling. The authors themselves caution that additional comparisons with more baselines and multiple trials are needed to substantiate the conclusions. Still, the demonstration marks a compelling step: an AI that, faced with a broken teammate and two plausible explanations, invents the shove-and-see test that a human engineer would devise — and then organizes the whole team to carry the mission home.

Subject of Research: Autonomous fault diagnosis in multi-agent robot teams using reinforcement learning to plan information-gathering collective behaviors

Article Title: Using reinforcement learning to autonomously identify sources of error for agents in group missions

Article References: Utimula, K., Hayaschi, K.-T., Bihl, T. J., Hongo, K., & Maezono, R. (2024). Using reinforcement learning to autonomously identify sources of error for agents in group missions. Frontiers in Control Engineering, 5, Article 1402621. https://doi.org/10.3389/fcteg.2024.1402621

Image Credits: AI Generated

DOI: 10.3389/fcteg.2024.1402621

Keywords: reinforcement learning, multi-agent systems, fault diagnosis, swarm robotics, sparse rewards, PPO2, actuator failure, sensor failure, collective behavior, autonomous agents, machine learning, robotics