The work, published in the journal Cluster Computing by Wenlong Ke, Ying Wang and Yong Zhang, addresses a blind spot in what is known as Adaptive Bitrate, or ABR, streaming. Traditional ABR algorithms exist to solve one central problem: network throughput fluctuates constantly, especially on cellular connections, so a fixed video quality would either stutter when the connection dips or waste capacity when it surges. By dynamically adjusting the bitrate of each successive video chunk, ABR systems such as the buffer-based BOLA approach or control-theoretic methods like the one behind DASH keep playback smooth. What those classic designs largely ignore, the researchers argue, is that downloading and playing video is one of the most power-hungry activities on a mobile device, and that the energy cost of a streaming session depends on far more than just the bitrate the client selects.

ESM reframes the adaptation problem in two important ways. First, instead of driving its decisions with raw bitrate figures, it evaluates candidate chunks using perceptual quality metrics, specifically VMAF, the Video Multimethod Assessment Fusion score that estimates how good a video actually looks to a human viewer. This matters because the relationship between bitrate and perceived quality is nonlinear and content-dependent: a fast-moving sports scene may need far more bits than a static talking-head shot to reach the same visual quality. By feeding VMAF values for the next chunk at each available bitrate level directly into the decision process, the agent can target perceptual quality rather than a proxy number.

Second, the system explicitly models energy. In the researchers’ formulation, the total energy consumed for each video chunk is decomposed into two components: the energy spent acquiring the data over the network and the energy spent playing the video back locally. The energy cost of data acquisition depends on the size of the chunk and the actual throughput the network delivers, because a slow, strained radio link keeps the transmitter active longer per byte. Local playback energy, in turn, scales with the video bitrate being decoded and rendered. Crucially, the energy model incorporates user experience preferences through dynamic scaling factors, allowing the trade-off between quality and battery life to be tuned rather than hard-coded.

To actually learn a policy under this richer objective, the team built ESM on deep reinforcement learning, and in particular on an improved version of Proximal Policy Optimization, or PPO. PPO is an actor-critic algorithm: a policy network, the actor, learns to select bitrate decisions, while a value network, the critic, estimates how good each state is so that the actor’s updates are guided by an advantage signal rather than raw rewards. PPO’s signature trick is clipped surrogate optimization, which constrains how far the policy can move in a single update and thereby stabilizes training. The ESM authors go further with a dual clipping parameter, extending the bounds on policy updates in both directions, which they combine with an entropy term weighted toward a target entropy to keep the agent exploring rather than collapsing prematurely onto a mediocre strategy.

The state that the agent observes at each decision point is deliberately rich. It includes throughput measurements and download times from past observations, the sizes and VMAF values of the next video chunk at every candidate bitrate level, the current playback buffer occupancy, the VMAF score of the previously downloaded chunk, and the number of chunks remaining in the video. The action space is the choice of bitrate for the next chunk. Rewards combine the perceptual quality achieved with penalties for rebuffering time and, distinctively, the energy consumed, with penalty coefficients weighting the components. Because the QoE weights themselves are dynamic scaling factors drawn from a tunable range, the same trained framework can express different user priorities, from quality-maximal viewing to battery-sipping endurance mode.

Evaluating such a system honestly requires more than a toy setup. The researchers ran trace-driven simulations using network traces and video content configured to match diverse real-world conditions, drawing on the kind of bandwidth recordings gathered from mobile and cellular environments in prior measurement studies. Against the average performance of all evaluated baseline algorithms, which include both rule-based ABR schemes and learning-based methods developed in recent years, ESM’s gains of 9.97 percent in QoE and 14.44 percent in model-calculated relative energy consumption represent a meaningful margin. The authors are careful to note a caveat that is often glossed over in machine learning papers: these figures are aggregated statistically from the specific network traces and video samples used in the study, and the optimization gains will fluctuate somewhat across different combinations of content and network conditions.

The broader significance of the work lies in where streaming is heading. Mobile video already dominates global internet traffic, and short-video platforms have made continuous, hours-long consumption the norm, multiplying the energy bill paid in batteries and, upstream, in data centers and radio networks. Researchers studying quality of experience have begun quantifying the trade-off between viewer satisfaction and carbon emissions across current and future network generations, and dedicated measurement datasets for content-consumption energy are now emerging. Earlier energy-aware streaming efforts tended to bolt energy considerations onto existing ABR logic or to sacrifice quality aggressively in exchange for power savings. ESM’s contribution is to fold energy directly into the learning objective alongside perceptual quality, so the agent discovers the Pareto frontier itself rather than following a hand-tuned heuristic.

The choice of PPO as the learning backbone also reflects lessons the field has absorbed since deep reinforcement learning first entered networking with systems like Pensieve in 2017. Value-based methods such as deep Q-networks can be brittle when the action space and state complexity grow, and streaming presents a partially observable, nonstationary environment in which the network behaves almost adversarially. Policy-gradient methods with trust-region-style constraints, entropy regularization and advantage estimation have proven more robust in exactly these settings, and variants of soft actor-critic and PPO now appear across the learning-based streaming literature. ESM’s dual-clipped, entropy-targeted refinement is a natural evolution of that trend, aimed at making training converge reliably across the wildly different bandwidth regimes a phone encounters over a single commute.

The authors have released their code and data in a public GitHub repository, which matters for a subfield where reproducibility has been a persistent complaint; randomized in-situ experiments have shown that simulation results can diverge from live deployments in surprising ways. The next frontier for systems like ESM is validation on physical devices, where Wi-Fi and 5G power behavior, thermal throttling and vendor-specific decoding pipelines complicate any analytical energy model. Still, the direction is clear. As the researchers demonstrate with their trace-driven results, an agent that understands both what viewers see and what their batteries pay can deliver better streaming on both fronts at once, and that multi-objective balance under hardware constraints is precisely what the next billion hours of mobile video will demand.

Subject of Research: Energy-efficient adaptive bitrate video streaming using deep reinforcement learning

Article Title: ESM: energy-efficient bitrate adaptation for video streaming with deep reinforcement learning

Article References: Ke, W., Wang, Y., & Zhang, Y. (2026). ESM: energy-efficient bitrate adaptation for video streaming with deep reinforcement learning. Cluster Computing, 29(13), Article 746. https://doi.org/10.1007/s10586-026-06576-x

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06576-x

Keywords: adaptive bitrate, video streaming, deep reinforcement learning, energy efficiency, quality of experience, PPO, VMAF, mobile devices, QoE, network throughput, Cluster Computing, machine learning

To understand why compound disturbances are so corrosive to learning, it helps to recall how policy-gradient reinforcement learning works. In methods descended from the classic REINFORCE algorithm, a neural network policy is nudged in the direction of actions that produced higher-than-expected returns, where the return is the cumulative reward collected over a trajectory. The crucial phrase is higher than expected. Because the raw return varies wildly from episode to episode, even for identical behavior, practitioners subtract a baseline from the return to compute an advantage, a quantity that estimates how much better an action fared than average. Subtracting a baseline does not change the expected direction of the gradient, but it can dramatically reduce the variance of the estimate, and variance reduction is often the difference between a policy that converges and one that thrashes. The trouble, as Wang and Zhao point out, is that most standard methods use a single batch-mean return as the baseline for every state in the batch.

Under compound disturbances, that shortcut breaks down. Training samples within a batch differ enormously in their states and environmental conditions: one drone may be fighting a strong crosswind with a lagging controller while another glides through calm air with pristine sensors. A single scalar average of returns is simply an inadequate reference for the expected return at each individual state. The result is advantage estimates polluted by environmental noise, gradients that point in misleading directions, and training curves that are unstable across repeated runs. The problem is not that the learning signal is absent, but that it is measured against the wrong yardstick, and the mismatch grows precisely in the conditions where robust navigation matters most.

GDPB’s answer is to make the baseline state-dependent and, crucially, to give it information the policy itself will never see at deployment. The method trains a separate baseline branch that predicts Monte Carlo returns from two sources of input: the onboard observation features that the policy uses, and privileged disturbance parameters supplied by the simulator, such as the true wind field, actuation noise characteristics, control delay, and sensor degradation levels. Because the simulator knows exactly what adversity each drone is facing, the baseline branch can learn a far more accurate map from situation to expected return than any function of onboard observations alone. This idea of exploiting privileged information during training while keeping the deployed system lean belongs to a lineage of asymmetric actor-critic methods and teacher-student approaches that have proven successful in quadrupedal locomotion and agile flight, but GDPB applies it specifically to the variance-reduction machinery of policy gradients.

The second key ingredient is a gradient-decoupling trick that keeps the two branches from interfering with each other. When computing advantages for the policy update, a stop-gradient operation is applied to the baseline prediction, which means the policy loss cannot directly update the baseline branch through backpropagation. Instead, the baseline branch is trained separately through return regression, fitting its predictions to the actual Monte Carlo returns observed during training. This separation matters because entangled gradients can create a feedback loop in which the baseline shifts to flatter the policy rather than to accurately predict returns, degrading the very variance reduction it was meant to provide. By decoupling the gradients, GDPB preserves the theoretical property that a state-dependent baseline leaves the expected policy gradient unchanged while still delivering the practical benefit of lower-variance estimates. The technique echoes a broader lesson from recent work on REINFORCE-style optimization, including its prominent role in training large language models, that careful handling of baselines and gradient flow is central to stable policy learning.

Perhaps the most elegant feature of the design is what happens at deployment: the baseline branch and all privileged inputs are simply removed. The trained policy relies only on onboard observations, exactly as a real drone would, so GDPB adds no computational burden to the deployed system whatsoever. The privileged information and the extra network capacity are pure training-time scaffolding, discarded once their job of shaping a clean learning signal is done. This stands in contrast to approaches that attempt to estimate disturbance parameters online or carry recurrent memory of past disturbances, both of which impose runtime costs and can introduce their own failure modes. For weight- and power-constrained aerial platforms, a method that improves robustness without adding a single floating-point operation at flight time is a meaningful engineering advantage.

The empirical evaluation was conducted on a compound-disturbance navigation task in Isaac Sim, NVIDIA’s physics-accurate robotics simulator, where wind fields, actuation noise, control delay and sensor degradation were applied simultaneously. The results are striking. GDPB achieved a composite success rate of 96.47 percent, outperforming four established baselines: Proximal Policy Optimization, the workhorse of robotic reinforcement learning; Group Relative Policy Optimization, the method that has attracted attention for its role in reasoning models; Soft Actor-Critic, a maximum-entropy off-policy algorithm; and Twin Delayed Deep Deterministic Policy Gradient, a staple of continuous control. Just as important as the headline number is the consistency: GDPB exhibited a smaller standard deviation across repeated runs, indicating that its advantage is not a lucky seed but a reproducible property of the training process.

The gains were not uniform across conditions, and that pattern is itself informative. Wang and Zhao report that GDPB’s advantage over PPO became more pronounced under moderate-to-strong disturbances, exactly the regime where the batch-mean baseline is most miscalibrated and where state-dependent, disturbance-aware baselines should shine. In calm conditions, when all samples in a batch face similar environments, a simple average is a reasonable reference and the sophisticated baseline has less to correct. As the disturbance distribution widens, the spread of returns within a batch grows, the single-yardstick assumption fails harder, and GDPB’s per-state reference pulls further ahead. This dose-response relationship between disturbance intensity and method advantage lends credibility to the authors’ mechanistic explanation rather than suggesting an incidental benchmark win.

The work sits within a rapidly maturing research landscape. Deep reinforcement learning has already produced champion-level drone racing pilots and high-speed flight through unstructured wild environments, and robustness techniques such as domain randomization, dynamics randomization, adaptive adversaries and rapid motor adaptation have each chipped away at the sim-to-real gap. Yet most of these efforts address disturbances one at a time or treat robustness as a property to be baked into the policy through exposure to varied conditions. GDPB reframes the problem as one of statistical estimation: under compound disturbances, the learning signal itself must be denoised before a policy can extract reliable lessons from experience. That perspective connects drone navigation to a deep current in reinforcement learning theory, where variance reduction techniques for gradient estimates have been studied for two decades, and suggests that ideas from the estimation side of the field may be underexploited in robotics.

For the field at large, the study offers both a practical tool and a conceptual nudge. Practically, GDPB provides a drop-in improvement for policy-gradient training of aerial vehicles destined for messy, disturbance-rich environments, with the reassuring property that deployment-time behavior is unchanged. Conceptually, it demonstrates that privileged simulation information, so often used to shape policies directly, can be just as valuable when routed through the humble baseline that calibrates the learning signal. As drones move from curated test arenas into gusty cities, dusty fields and degraded sensor conditions, methods that make training stable under exactly those compound stresses may prove to be the quiet enabler of the next generation of autonomous flight. The authors note that the data supporting their findings are available from the corresponding author upon reasonable request, and the full details appear in Applied Intelligence, volume 56, article 482.

Subject of Research: Deep reinforcement learning for robust UAV navigation under compound disturbances

Article Title: GDPB: a gradient-decoupled privileged baseline for UAV navigation under compound disturbances

Article References: Wang, X., & Zhao, J. (2026). GDPB: a gradient-decoupled privileged baseline for UAV navigation under compound disturbances. Applied Intelligence, 56(15), Article 482. https://doi.org/10.1007/s10489-026-07533-2

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07533-2

Keywords: UAV navigation, deep reinforcement learning, compound disturbances, policy gradient, privileged information, variance reduction, PPO, GRPO, Isaac Sim, robust control, sim-to-real transfer, drone autonomy

The problem the researchers tackle is what they describe as a form of error propagation, closely related to hallucination in language models. When a tool-augmented pipeline invokes an OCR engine, most existing frameworks simply fold the returned text into the reasoning process, regardless of whether that text is noisy, spatially inconsistent, or plainly wrong. This becomes dangerous under realistic document conditions—low resolution, blur, occlusion, stylized fonts, and complex layouts—where OCR and layout parsers degrade substantially. The authors draw on the established distinction between intrinsic hallucinations, which contradict source evidence, and extrinsic hallucinations, which introduce unsupported information. Tool over-trust, they argue, behaves like extrinsic hallucination propagation: the model incorporates tool-generated text that is weakly supported by, or directly inconsistent with, what the image actually shows.

Why do powerful models fall for bad tool output? The study points to the limited grounding of large language models. Because language models operate largely through statistical pattern matching over textual co-occurrences rather than grounded semantic understanding, multimodal systems may privilege textual tool outputs simply because they arrive in linguistic form—even when visual evidence contradicts them. Existing tool-learning frameworks such as Toolformer and ToolLLM demonstrate that models can learn to invoke APIs effectively, but they primarily evaluate whether a tool was used and whether final performance improved. They rarely ask the more fundamental question: is the tool’s answer actually trustworthy for this particular document region?

The proposed answer is a conflict-aware uncertainty alignment framework with three working parts. First, a lightweight visual uncertainty estimation head—two convolutional layers producing a dense map over the image—predicts spatial regions where the model’s internal perception is likely to be unreliable. The head is trained with binary cross-entropy against region-level error targets derived from whether predictions were actually correct. Second, a spatial alignment mechanism converts scalar tool confidences into spatially grounded maps: OCR character-level confidences are projected onto the image plane through their bounding boxes, then transformed into tool uncertainty by subtracting from one, so that visual and tool reliability live on the same semantic scale and in the same coordinate system. Third, a conflict module computes the pointwise absolute difference between the two uncertainty maps, yielding a localized disagreement signal and a global conflict score for each tool.

On top of this representation sits a decision policy trained with reinforcement learning. Given the image, the query, the visual uncertainty map, and the aligned tool uncertainty maps, the policy chooses among three actions: trust the model’s internal reasoning, invoke a specific external tool, or reject an unreliable tool output before it contaminates downstream reasoning. The reward function is deliberately rejection-aware—it grants a bonus for correctly rejecting incorrect tool outputs, rewards correct predictions, and penalizes each tool invocation by a configurable cost. The policy is optimized using Proximal Policy Optimization (PPO), though the authors are careful to note that PPO is a practical optimization mechanism here, not a methodological contribution in itself, and that supervised gating or contextual-bandit approaches could in principle learn similar arbitration policies.

The empirical results are striking. Across four document-understanding benchmarks—DocVQA, FUNSD, SROIE, and PubLayNet—the framework achieved absolute gains of 5.8 percent on DocVQA, 4.9 F1 points on FUNSD, and 4.9 percent on SROIE over an always-call baseline, while cutting average external tool invocation by more than half. On DocVQA, the full framework reached 81.7 percent accuracy. The efficiency finding is perhaps the most counterintuitive: calling tools more often did not produce better answers. Unconditional tool usage introduced substantial error propagation, because faulty OCR text flowed directly into reasoning. The conflict-aware system instead settled at roughly one tool call per sample, invoking parsers only for genuinely ambiguous structures like nested tables and dense multi-column layouts, and skipping them for clean headers and well-aligned text.

Rejection behavior proved to be the framework’s signature capability. Measured by the proportion of incorrect external tool outputs correctly suppressed, the learned policy outperformed both heuristic threshold gating and supervised gating across all four datasets. Qualitative case studies show the mechanism in action: when OCR misreads a degraded text region, the visual uncertainty map flags the area, the aligned tool uncertainty diverges, and the resulting conflict spike prompts the policy to discard the OCR output and rely on internal multimodal reasoning. When the document is clean and the tool agrees with the image, conflict stays low and the tool’s contribution is welcomed. The system learns context-dependent trust calibration rather than blanket acceptance or rejection.

Robustness testing added a further layer of rigor. The researchers built a controlled tool-failure benchmark using Gaussian blur, motion blur, occlusion, geometric distortion, compression artifacts, low resolution, and stylized-font substitution—and crucially held out some corruption types from training to test generalization. The framework retained its arbitration advantage under these unseen degradations, suggesting the policy learned general reliability cues rather than memorizing specific corruption patterns. Ablation studies confirmed that every component matters: replacing spatial uncertainty with a scalar signal dropped accuracy to 78.9 percent, removing explicit conflict modeling dropped it to 79.6 percent, removing the rejection-aware reward dropped it to 79.4 percent, and abandoning learned policy optimization altogether fell to 74.6 percent.

The authors are candid about limitations. The conflict measure captures spatial disagreement but not semantic contradiction—two sources can agree spatially while disagreeing at a higher conceptual level. The approach assumes tools expose spatially localizable confidence, which is natural for OCR and layout parsers but harder for legacy systems returning only global scores. And controlled perturbations cannot fully reproduce real-world failures such as heterogeneous illumination, handwritten annotations, or multilingual content. Still, the practical implications reach well beyond the lab. Document automation, receipt processing, healthcare documentation, financial workflows, and archival digitization all depend on OCR and layout tools that fail in localized, hard-to-predict ways. A system that knows when to doubt its own assistants—and pays only for the help it actually needs—represents a meaningful step toward multimodal AI that is not just capable, but reliably self-aware about the limits of its evidence.

Subject of Research: Uncertainty-guided tool arbitration for tool-augmented multimodal document understanding

Article Title: Conflict-aware uncertainty alignment for tool-augmented document-centric multimodal intelligence

Article References: Muthusami, R., & Saritha, K. (2026). Conflict-aware uncertainty alignment for tool-augmented document-centric multimodal intelligence. Discover Artificial Intelligence, 6(1), Article 1359. https://doi.org/10.1007/s44163-026-02409-3

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02409-3

Keywords: vision-language models, tool augmentation, uncertainty estimation, OCR, document understanding, conflict-aware arbitration, reinforcement learning, PPO, hallucination, multimodal AI, DocVQA, selective inference

The framework, described in the journal Applied Intelligence, is called HALSI, short for Hierarchical Adaptive Latent Skill Inference. Its central insight is that the temporal structure of behavior matters as much as the behavior itself. Earlier skill-based hierarchical methods decompose complicated tasks into reusable behavior primitives, which eases the burden of long-horizon decision-making, but they typically fix the length of each skill in advance and compress behavior into oversimplified latent representations. That rigidity limits both temporal flexibility and the diversity of behaviors an agent can express. HALSI removes the fixed horizon entirely, learning skills of variable duration from data and adjusting them online as the task unfolds.

At the heart of the system sits a Transformer encoder that reads variable-length trajectories of state-action pairs and distills them into a compact latent embedding. The architecture is causal, meaning each token corresponds to a state-action tuple, and positional encodings preserve the sequential order so that self-attention can capture long-range dependencies across time. A special classification token at the output summarizes the overall temporal and behavioral pattern of the skill into a single vector. Crucially, during pretraining the system preserves the natural lengths of trajectory segments rather than chopping demonstrations into fixed windows, which allows the encoder to learn duration-agnostic representations that retain the genuine variability of different behaviors.

To turn those latent skills into actual motion, HALSI pairs the encoder with a conditional diffusion policy. Denoising diffusion probabilistic models, which have transformed image generation in recent years, work here by iteratively refining Gaussian noise into a coherent action sequence, conditioned on both a timestep embedding and the concatenation of the latent skill vector with the current state. This generative approach lets the decoder model complex, multimodal action distributions and produce temporally consistent behaviors that a simpler policy class would struggle to capture. Once pretrained on offline demonstration data, the encoder and decoder remain frozen during online reinforcement learning, serving as a stable library of skills that the higher levels of the hierarchy can draw upon.

The first major innovation addresses the fixed-horizon problem directly: an adaptive duration policy that predicts a discrete execution length for each skill it selects. Rather than committing to a predetermined number of steps, the high-level controller outputs both a latent direction and a duration, and the diffusion decoder generates a matching action sequence of exactly that length before a new skill is sampled. The researchers found that different tasks induce strikingly different patterns of skill durations. In a slippery pushing task, durations cluster tightly at short values, reflecting the need for frequent fine-grained corrections on low-friction surfaces. In a tool-use task involving a hook, the distribution becomes broad and multimodal, as the agent alternates between brief reactive skills and longer sustained motions for aligning, inserting, and pulling. A pick-and-place task showed a bimodal pattern corresponding to its two distinct sub-goals.

The second innovation is a multi-scale optimization scheme built on Proximal Policy Optimization. A high-level modulation policy refines latent skills with task-aware residuals, adding a learned direction to the encoded latent mean so that the skill embedding itself is adjusted rather than the raw actions. Because directly perturbing behavior early in training would destabilize learning, the team introduced a curriculum: a scheduling factor follows a logistic curve over training steps, starting near zero so the agent initially relies almost entirely on the pretrained skill encoder, then gradually rising to give the residual policy increasing influence. A low-level soft-blending controller complements this by mixing the offline-decoded skill action with an online-learned adaptive actor at every step, allowing moment-to-moment corrections while preserving the structure of the skill prior.

The ablation studies reveal how sensitive this balance is. When the blending coefficient was set low, the agent largely ignored its pretrained skills and fell back on the reactive actor alone, producing unstable and inefficient behavior in long-horizon tasks such as pyramid stacking, where structured skill priors are essential for guidance. When the coefficient was pushed very high, the agent over-trusted the offline decoder and lost the ability to respond to distribution shifts, a weakness most visible in the slippery pushing environment where precise reactive adjustments are frequently needed. The best overall performance emerged consistently at a coefficient of 0.8, a setting that retains the semantic consistency of the offline skills while leaving room for online refinement.

Evaluated on two demanding benchmark suites, the framework delivered substantial gains. On four long-horizon manipulation tasks from the Reskill benchmark, built on MuJoCo-based Fetch environments, the agent had to transfer skills learned from roughly 40,000 trajectories collected with deliberately suboptimal scripted controllers in simplified settings, then apply them in downstream environments featuring slippery surfaces, cluttered distractor objects, multi-stage stacking goals, and contact-rich tool use. On the D4RL Kitchen benchmark, where a Franka arm operates household appliances across episodes of roughly 280 steps with rewards granted only when subtasks are completed, the sparse-reward structure makes credit assignment notoriously difficult. Across these tests, HALSI surpassed state-of-the-art hierarchical baselines, achieving up to 30 percent higher returns and converging 25 percent faster.

Beyond the headline numbers, the work offers a broader lesson about what makes hierarchical learning succeed. The duration analysis showed that a fixed-duration scheme would inevitably fragment some behaviors while leaving others insufficiently abstract, whereas a learned duration model allocates temporal granularity according to context and task stage. The authors suggest that jointly modeling latent skills and their adaptive temporal scope is the key ingredient, rather than improving either component in isolation. The source code has been released publicly, and the datasets are available from the corresponding author on reasonable request, lowering the barrier for other groups to build on the approach.

The implications reach well beyond simulated kitchens and block-stacking. Long-horizon, sparse-reward problems pervade robotics, autonomous driving, and industrial automation, and any method that lets agents reuse skills flexibly across tasks with differing dynamics could accelerate progress in all of them. By combining the sequence-modeling power of Transformers, the expressive generative capacity of diffusion models, and a principled curriculum for handing control from imitation to online learning, HALSI points toward a future in which robots do not merely execute preprogrammed routines, but decide for themselves which skill fits the moment and how long to commit to it.

Subject of Research: Hierarchical reinforcement learning with adaptive latent skill inference for temporal abstraction in long-horizon robotic manipulation

Article Title: HALSI: Hierarchical adaptive latent skill inference for temporal abstraction in reinforcement learning

Article References: Si, X., Ping, Y., Wang, T., Chen, K., & Gao, Y. (2026). HALSI: Hierarchical adaptive latent skill inference for temporal abstraction in reinforcement learning. Applied Intelligence, 56(15), Article 444. https://doi.org/10.1007/s10489-026-07485-7

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07485-7

Keywords: reinforcement learning, hierarchical RL, skill abstraction, temporal abstraction, diffusion policy, transformer encoder, robotic manipulation, PPO, D4RL Kitchen, Fetch benchmark, machine learning, latent skills

The problem the team set out to address is known as actuator loss-of-effectiveness, or LoE. In a quadrotor, four rotors share the work of stabilizing the vehicle in roll, pitch, yaw, and altitude. If a single rotor degrades, whether through motor wear, a damaged propeller, or a partial power failure, the control authority available to the flight computer shrinks asymmetrically. A controller tuned for a healthy aircraft, with fixed gains calculated once before takeoff, may respond too weakly or too aggressively to the resulting imbalance, and the vehicle can destabilize. Fault-tolerant control strategies exist, ranging from robust backstepping designs to model predictive control and hardware redundancy such as tilting rotors, but many require substantial redesign of the control stack or additional actuators.

The Algerian team’s approach is deliberately conservative. They kept the standard cascaded PID structure, the underlying control mixer, and the rigid-body model of the quadrotor untouched. On top of that, they added a bounded online gain-adaptation layer implemented with reinforcement learning. Four decentralized agents operate in parallel, each responsible for one control channel: roll, pitch, yaw, and altitude. Each agent observes the state of the vehicle and adjusts the PID gains in its own channel in real time, but only within preset limits. This bounding is a critical safety feature, because it prevents the learning system from ever commanding gains that could make the aircraft unstable, a concern that has historically limited the acceptance of learning-based controllers in safety-critical flight applications.

To test the idea rigorously, the researchers built a six-degree-of-freedom Newton–Euler quadrotor model with first-order motor dynamics in MATLAB/Simulink. This level of modeling captures both the full rigid-body motion of the aircraft and the lag with which real motors respond to commands, which matters greatly when a rotor’s effectiveness is dropping. The adaptation layer was instantiated with Soft Actor-Critic, or SAC, a reinforcement learning algorithm known for balancing exploration and stability during training. The simulations subjected the controller to nominal flight conditions and to multiple transient and sustained single-rotor LoE profiles, meaning scenarios in which a rotor’s effectiveness dropped either briefly or permanently during flight.

A key strength of the study is its comparison set. The authors did not simply benchmark their learning controller against a naive baseline. They included two other prominent reinforcement learning algorithms, Twin Delayed Deep Deterministic Policy Gradient (TD3) and Proximal Policy Optimization (PPO), evaluated under identical observation and action definitions, the same reward structure, the same gain limits, and the same 500-episode training budget. An additional 1000-episode run of PPO was included to check whether the results were sensitive to how long the algorithms were allowed to train. This kind of controlled comparison is rare and valuable, because reinforcement learning results can vary dramatically with small changes in setup.

The team also addressed a subtler question: how much of the benefit comes from online adaptation itself, rather than from clever static tuning? To separate the two effects, they created offline-optimized fixed-gain PID baselines using two metaheuristic optimization methods, particle swarm optimization (PSO) and grey wolf optimization (GWO). These baselines represent the best that conventional, non-adaptive tuning can achieve before the flight even begins. Including two different metaheuristics also guards against the criticism that the comparison depends on which optimization algorithm happened to be chosen for the baseline.

The results tell a clear story. In nominal flight, with all rotors healthy, the methods performed comparably: the learning-augmented controller did not sacrifice accuracy in ordinary conditions, and step responses and three-dimensional trajectory-tracking simulations showed similar performance across the board. The differences emerged under degradation. When a rotor began to lose effectiveness, the fixed-gain controllers, even those tuned by sophisticated metaheuristics, showed larger post-fault deviations from the desired trajectory. The online gain adaptation layer, by contrast, produced smaller deviations after the fault, because the agents could shift the controller’s aggressiveness to compensate for the lost control authority as the degradation unfolded.

Among the three reinforcement learning algorithms tested, SAC provided the most consistent fault accommodation in the evaluated configurations. This finding aligns with SAC’s design philosophy: the algorithm optimizes both the expected reward and the entropy of its policy, encouraging robust behavior rather than overfitting to a narrow set of training conditions. TD3 and PPO remained competitive under the same budget, but SAC’s consistency across the various LoE profiles made it the standout. The additional 1000-episode PPO check helped the authors assess whether longer training would change the picture, addressing a common concern that reinforcement learning comparisons may simply reflect training-budget artifacts.

What makes this work notable for the drone industry is its integration cost, or rather its lack of one. Many fault-tolerant control approaches demand that engineers abandon the PID controllers their teams know well and adopt entirely new architectures, with all the certification, testing, and retraining burdens that implies. The approach demonstrated here treats learning as a thin adaptation layer on top of existing infrastructure. The mixer, the outer-loop structure, and the physical model all remain unchanged, and the learning agents act only within preset gain bounds. For operators of delivery drones, inspection platforms, and other commercial quadrotors, that means a path to greater resilience against rotor degradation without rewriting the flight stack from scratch.

The study is simulation-based, and the authors are careful about the scope of their claims: in the tested setup, the results support bounded online gain adaptation as a low-integration-cost way to improve robustness to rotor loss-of-effectiveness. Real-world deployment would bring additional challenges, including sensor noise, wind, computational constraints on embedded flight controllers, and the well-known sim-to-real gap that affects all learning-based control methods. Still, the work adds to a growing body of evidence that reinforcement learning can serve flight control best not as a wholesale replacement for classical control theory, but as a disciplined assistant that fine-tunes proven controllers when the aircraft’s condition changes. As drones take on ever more demanding missions, that kind of graceful degradation under failure may prove to be one of machine learning’s most practical contributions to aviation safety.

Subject of Research: Reinforcement-learning-based online PID gain adaptation for fault-tolerant quadrotor control under rotor loss-of-effectiveness

Article Title: Reinforcement-Learning-Based Online PID Gain Adaptation for Fault-Tolerant Quadrotor Control Under Rotor Loss-of-Effectiveness

Article References: Lahmar, O., Abdou, L., & Ghiloubi, I. B. (2026). Reinforcement-Learning-Based Online PID Gain Adaptation for Fault-Tolerant Quadrotor Control Under Rotor Loss-of-Effectiveness. International Journal of Aeronautical and Space Sciences. https://doi.org/10.1007/s42405-026-01269-6

Image Credits: AI Generated

DOI: 10.1007/s42405-026-01269-6

Keywords: quadrotor UAV, fault-tolerant control, actuator loss-of-effectiveness, reinforcement learning, PID gain adaptation, Soft Actor-Critic, TD3, PPO, MATLAB/Simulink simulation, trajectory tracking, drone safety, adaptive control

The research, led by Hima Soni of Gujarat Technological University together with colleagues at Vishwakarma Government Engineering College, Tata Consultancy Services, MultiQoS and MDI Gurgaon, introduces a framework that combines variational autoencoders with actor-critic deep reinforcement learning to estimate remaining useful life sequentially, refining its estimate step by step rather than producing a single one-shot prediction. The result, according to the authors, is a method that outperforms conventional supervised sequence-learning baselines, including long short-term memory networks and multilayer perceptrons, by 26.7 to 40.3 percent in root mean square error, with the best-performing variant reaching an RMSE of 0.1262.

To understand why this is notable, it helps to unpack the machinery underneath. Remaining useful life estimation is fundamentally a problem of reading degradation. Industrial machines are fitted with sensors that stream measurements of temperature, vibration, pressure and dozens of other signals. As components wear, these signals drift in characteristic ways, and the art of prognostics lies in mapping those drifts onto a timeline of future failure. Classical approaches feed sensor histories into supervised models that learn the mapping directly from labeled run-to-failure data. The trouble is that such data is scarce, noisy and expensive to collect, and the models often struggle to generalize across different degradation patterns.

The new framework attacks the problem in two stages. First, a variational autoencoder compresses the raw sensor streams into a compact latent representation. A variational autoencoder is a generative neural network that learns to encode data into a probabilistic latent space and then decode it back, forcing the latent variables to capture the essential structure of the data. In this application, the latent space preserves the underlying degradation dynamics while discarding redundant or irrelevant sensor noise. Crucially, the trained autoencoder then serves as a surrogate environment, a simulated world in which a reinforcement learning agent can act without touching real machinery or requiring vast amounts of fresh field data.

Inside that surrogate environment, the researchers deployed two actor-critic deep reinforcement learning algorithms: Twin Delayed Deep Deterministic Policy Gradient, known as TD3, and Proximal Policy Optimization, known as PPO. Both algorithms learn a policy, a rule for choosing actions, while simultaneously learning a value function that estimates how good those actions are. The agent’s task is to iteratively refine degradation trajectories within the learned latent space, adjusting its estimates of remaining useful life in a sequence of decisions rather than a single leap. This sequential formulation turns prognostics into a control problem, allowing the model to correct course as new evidence accumulates.

One of the study’s most interesting findings concerns reward shaping, the practice of designing the feedback signal that guides reinforcement learning. The researchers compared multiple reward formulations, including a weighted cosine similarity reward that measures how closely the agent’s predicted trajectory aligns with the true degradation path. Their comparative evaluation showed that the choice of optimization strategy meaningfully shapes both learning stability and downstream prediction accuracy. The on-policy approach, PPO, yielded more consistent convergence in 11 of 12 experimental configurations, a result that speaks to the well-known stability advantages of on-policy methods, which evaluate and update the same policy they use to collect experience. Meanwhile, the weighted cosine similarity reward produced comparable prediction accuracy, with large Cohen’s d effect sizes exceeding 0.8 in 10 configurations, indicating that the reward design produced differences large enough to matter in practice, not merely statistical noise.

The performance numbers are striking. An RMSE of 0.1262 on remaining useful life estimation, achieved while beating LSTM and MLP baselines by margins approaching 40 percent, suggests that the latent-space policy optimization is doing genuine work rather than simply repackaging existing sequence-learning tricks. The authors attribute this to the synergy between generative representation learning and sequential policy optimization: the autoencoder gives the agent a clean, information-dense state space, while the reinforcement learning loop gives the model a mechanism for iterative refinement that supervised models lack. Because the agent learns within a surrogate environment, the architecture is also computationally scalable and data efficient, qualities that matter enormously in industrial settings where labeled failure data is the bottleneck.

The broader context makes the contribution clearer. Predictive maintenance has accumulated a rich toolkit over the past decade, from deep convolutional neural networks for prognostics to Bayesian neural networks that quantify uncertainty, from regularized LSTM models for rolling bearings to ensemble deep learning with multi-objective optimization for rotating machinery. Reinforcement learning has already appeared in maintenance research, notably in scheduling and optimization roles, such as deep reinforcement learning for predictive aircraft maintenance using probabilistic prognostics. What distinguishes the new work is its modular marriage of a generative surrogate environment with policy optimization directly in latent space, a design the authors argue is adaptable to a variety of degradation patterns and therefore suited to challenging industrial scenarios where one-size-fits-all models falter.

There are, of course, caveats worth keeping in mind. The study reports that no datasets were generated or analysed during the current study in its data availability statement, and the evaluation rests on the configurations the authors describe, so practitioners will want to validate the approach on their own fleets before deploying it. Reinforcement learning remains notoriously sensitive to hyperparameters and reward design, and although the comparative evaluation across 12 configurations is a step toward taming that sensitivity, the finding that PPO converges more consistently than TD3 in most settings is itself a reminder that algorithm choice is not a detail but a first-order design decision. The authors also emphasize that the framework is modular, which means each component, the autoencoder, the policy optimizer and the reward function, can in principle be swapped as better versions emerge.

Still, the direction is compelling. If machines could reliably tell operators how much life they have left, maintenance could shift from reactive scrambling and wasteful over-servicing toward precisely timed intervention, saving money and preventing failures before they happen. By reframing remaining useful life estimation as sequential policy optimization in a learned latent space, this work offers a template for how generative models and reinforcement learning might together make that vision practical. The authors thank Gujarat Technological University and Bacancy DataProphets Private Limited for supporting the research, and their results suggest that the intersection of representation learning, reward design and control may be one of the more productive frontiers in reliability engineering for years to come.

Subject of Research: A VAE-based deep reinforcement learning framework with reward shaping for sequential estimation of machinery remaining useful life.

Article Title: Sequential Estimation of Remaining Useful Life Through VAE-Based Deep Reinforcement Learning With Reward Shaping

Article References: Sequential Estimation of Remaining Useful Life Through VAE-Based Deep Reinforcement Learning With Reward Shaping. (n.d.). https://doi.org/10.1007/s10994-026-07152-5

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07152-5

Keywords: predictive maintenance, remaining useful life, deep reinforcement learning, variational autoencoder, reward shaping, TD3, PPO, prognostics, latent space, reliability engineering, machine learning, sensor data