To understand why compound disturbances are so corrosive to learning, it helps to recall how policy-gradient reinforcement learning works. In methods descended from the classic REINFORCE algorithm, a neural network policy is nudged in the direction of actions that produced higher-than-expected returns, where the return is the cumulative reward collected over a trajectory. The crucial phrase is higher than expected. Because the raw return varies wildly from episode to episode, even for identical behavior, practitioners subtract a baseline from the return to compute an advantage, a quantity that estimates how much better an action fared than average. Subtracting a baseline does not change the expected direction of the gradient, but it can dramatically reduce the variance of the estimate, and variance reduction is often the difference between a policy that converges and one that thrashes. The trouble, as Wang and Zhao point out, is that most standard methods use a single batch-mean return as the baseline for every state in the batch.

Under compound disturbances, that shortcut breaks down. Training samples within a batch differ enormously in their states and environmental conditions: one drone may be fighting a strong crosswind with a lagging controller while another glides through calm air with pristine sensors. A single scalar average of returns is simply an inadequate reference for the expected return at each individual state. The result is advantage estimates polluted by environmental noise, gradients that point in misleading directions, and training curves that are unstable across repeated runs. The problem is not that the learning signal is absent, but that it is measured against the wrong yardstick, and the mismatch grows precisely in the conditions where robust navigation matters most.

GDPB’s answer is to make the baseline state-dependent and, crucially, to give it information the policy itself will never see at deployment. The method trains a separate baseline branch that predicts Monte Carlo returns from two sources of input: the onboard observation features that the policy uses, and privileged disturbance parameters supplied by the simulator, such as the true wind field, actuation noise characteristics, control delay, and sensor degradation levels. Because the simulator knows exactly what adversity each drone is facing, the baseline branch can learn a far more accurate map from situation to expected return than any function of onboard observations alone. This idea of exploiting privileged information during training while keeping the deployed system lean belongs to a lineage of asymmetric actor-critic methods and teacher-student approaches that have proven successful in quadrupedal locomotion and agile flight, but GDPB applies it specifically to the variance-reduction machinery of policy gradients.

The second key ingredient is a gradient-decoupling trick that keeps the two branches from interfering with each other. When computing advantages for the policy update, a stop-gradient operation is applied to the baseline prediction, which means the policy loss cannot directly update the baseline branch through backpropagation. Instead, the baseline branch is trained separately through return regression, fitting its predictions to the actual Monte Carlo returns observed during training. This separation matters because entangled gradients can create a feedback loop in which the baseline shifts to flatter the policy rather than to accurately predict returns, degrading the very variance reduction it was meant to provide. By decoupling the gradients, GDPB preserves the theoretical property that a state-dependent baseline leaves the expected policy gradient unchanged while still delivering the practical benefit of lower-variance estimates. The technique echoes a broader lesson from recent work on REINFORCE-style optimization, including its prominent role in training large language models, that careful handling of baselines and gradient flow is central to stable policy learning.

Perhaps the most elegant feature of the design is what happens at deployment: the baseline branch and all privileged inputs are simply removed. The trained policy relies only on onboard observations, exactly as a real drone would, so GDPB adds no computational burden to the deployed system whatsoever. The privileged information and the extra network capacity are pure training-time scaffolding, discarded once their job of shaping a clean learning signal is done. This stands in contrast to approaches that attempt to estimate disturbance parameters online or carry recurrent memory of past disturbances, both of which impose runtime costs and can introduce their own failure modes. For weight- and power-constrained aerial platforms, a method that improves robustness without adding a single floating-point operation at flight time is a meaningful engineering advantage.

The empirical evaluation was conducted on a compound-disturbance navigation task in Isaac Sim, NVIDIA’s physics-accurate robotics simulator, where wind fields, actuation noise, control delay and sensor degradation were applied simultaneously. The results are striking. GDPB achieved a composite success rate of 96.47 percent, outperforming four established baselines: Proximal Policy Optimization, the workhorse of robotic reinforcement learning; Group Relative Policy Optimization, the method that has attracted attention for its role in reasoning models; Soft Actor-Critic, a maximum-entropy off-policy algorithm; and Twin Delayed Deep Deterministic Policy Gradient, a staple of continuous control. Just as important as the headline number is the consistency: GDPB exhibited a smaller standard deviation across repeated runs, indicating that its advantage is not a lucky seed but a reproducible property of the training process.

The gains were not uniform across conditions, and that pattern is itself informative. Wang and Zhao report that GDPB’s advantage over PPO became more pronounced under moderate-to-strong disturbances, exactly the regime where the batch-mean baseline is most miscalibrated and where state-dependent, disturbance-aware baselines should shine. In calm conditions, when all samples in a batch face similar environments, a simple average is a reasonable reference and the sophisticated baseline has less to correct. As the disturbance distribution widens, the spread of returns within a batch grows, the single-yardstick assumption fails harder, and GDPB’s per-state reference pulls further ahead. This dose-response relationship between disturbance intensity and method advantage lends credibility to the authors’ mechanistic explanation rather than suggesting an incidental benchmark win.

The work sits within a rapidly maturing research landscape. Deep reinforcement learning has already produced champion-level drone racing pilots and high-speed flight through unstructured wild environments, and robustness techniques such as domain randomization, dynamics randomization, adaptive adversaries and rapid motor adaptation have each chipped away at the sim-to-real gap. Yet most of these efforts address disturbances one at a time or treat robustness as a property to be baked into the policy through exposure to varied conditions. GDPB reframes the problem as one of statistical estimation: under compound disturbances, the learning signal itself must be denoised before a policy can extract reliable lessons from experience. That perspective connects drone navigation to a deep current in reinforcement learning theory, where variance reduction techniques for gradient estimates have been studied for two decades, and suggests that ideas from the estimation side of the field may be underexploited in robotics.

For the field at large, the study offers both a practical tool and a conceptual nudge. Practically, GDPB provides a drop-in improvement for policy-gradient training of aerial vehicles destined for messy, disturbance-rich environments, with the reassuring property that deployment-time behavior is unchanged. Conceptually, it demonstrates that privileged simulation information, so often used to shape policies directly, can be just as valuable when routed through the humble baseline that calibrates the learning signal. As drones move from curated test arenas into gusty cities, dusty fields and degraded sensor conditions, methods that make training stable under exactly those compound stresses may prove to be the quiet enabler of the next generation of autonomous flight. The authors note that the data supporting their findings are available from the corresponding author upon reasonable request, and the full details appear in Applied Intelligence, volume 56, article 482.

Subject of Research: Deep reinforcement learning for robust UAV navigation under compound disturbances

Article Title: GDPB: a gradient-decoupled privileged baseline for UAV navigation under compound disturbances

Article References: Wang, X., & Zhao, J. (2026). GDPB: a gradient-decoupled privileged baseline for UAV navigation under compound disturbances. Applied Intelligence, 56(15), Article 482. https://doi.org/10.1007/s10489-026-07533-2

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07533-2

Keywords: UAV navigation, deep reinforcement learning, compound disturbances, policy gradient, privileged information, variance reduction, PPO, GRPO, Isaac Sim, robust control, sim-to-real transfer, drone autonomy

A new study published in Machine Learning with Applications addresses this challenge with a reasoning-centric training methodology for multimodal large language models, or MLLMs. Led by Mohamed Bayan Kmainasi of Qatar Computing Research Institute and colleagues including Mucahid Kutlu, Ali Ezzat Shahroor, Abul Hasnat, and Firoj Alam, the work demonstrates that reinforcement learning with chain-of-thought supervision can push explainable meme moderation to state-of-the-art performance on both English hateful memes and Arabic propagandistic memes. The research represents, according to the authors, the first systematic study of group relative policy optimization, known as GRPO, in multimodal reasoning under cross-lingual, fine-grained, and self-training settings.

The team’s central insight is that thinking-based MLLMs, which generate explicit intermediate reasoning steps before committing to an answer, are ideally suited to memes because their meaning depends on image-text interaction rather than unimodal cues. But whether the reasoning capabilities of such models, typically honed on mathematics and code generation, transfer to subjective, culturally situated tasks like meme moderation remained an open question. Standard supervised fine-tuning alone provides limited control over the balance between prediction correctness and rationale faithfulness, since cross-entropy loss treats all output tokens equally regardless of their functional role.

To resolve this, the researchers designed a three-stage training pipeline. The first stage is a supervised fine-tuning warm-up that aligns the model with gold labels, natural language explanations, and distilled reasoning traces produced by GPT-4.1. The second stage applies GRPO with a composite reward function that jointly optimizes classification correctness, output-format compliance, reference-based explanation similarity measured by METEOR, explanation length, and a novel thinking-length reward called Rthink. The third stage, self-training GRPO or ST-GRPO, extends the approach to unlabeled data using consensus-based pseudo-labels derived from the model’s own majority-vote predictions.

The composite reward is carefully weighted so that the two binary objectives, label correctness and format compliance, each receive 0.35 and together dominate the three auxiliary components, which contribute 0.30 combined. The thinking-length reward is particularly significant. Without it, the authors observed a consistent reward-hacking pattern: the model learned to compress or empty its reasoning traces while still collecting high reward, a length-shortening bias that was especially pronounced on the harder Arabic dataset. Rthink penalizes only reasoning traces shorter than a minimum threshold of 150 words, discouraging degenerate outputs without incentivizing verbosity.

The evaluation spans two distinct tasks and two languages. The English benchmark is the Facebook Hateful Memes dataset, containing roughly 11,000 memes where classification requires joint multimodal understanding. The Arabic benchmark, ArMeme, contains about 5,700 memes with four labels covering propaganda, not-propaganda, not-meme, and other. Because no fine-grained propaganda annotations existed for ArMeme, the team built a dual-annotator pipeline using GPT-4.1 and Llama-4-Scout as independent labelers of 23 propaganda techniques, consolidated by Gemini-3-Pro as an arbiter. Human validation on 584 memes showed the consolidated annotations aligned better with human reference labels than either single-model source.

The results are striking. On the Hateful Memes benchmark, the best supervised GRPO configuration with thinking-length regularization achieved 82.0 percent accuracy and 0.80 macro-F1, outperforming prior reported results and strong sequence-classification baselines such as Qwen3-VL-8B-Instruct and Gemma-3-12B-IT. On ArMeme, self-training GRPO reached 0.612 macro-F1, improving over previous work by 7.6 points and over the original ArMeme benchmark by 6.1 points. Notably, unimodal baselines lagged far behind: the best text-only model on ArMeme reached only 0.509 macro-F1, and image-only models averaged just 0.267, confirming that cross-modal reasoning captures signals that neither modality provides alone.

The self-training stage showed a telling asymmetry. On ArMeme, where unlabeled data was collected from the same social media sources as the labeled set, ST-GRPO improved macro-F1 by 1.5 points with gains concentrated in minority classes. On the English dataset, it slightly degraded performance, a result the authors attribute to distribution mismatch between the unlabeled pool and the benchmark, and to majority-vote bias amplification in binary classification, where small prediction biases can produce near-unanimous consensus that reinforces rather than corrects class skew. The finding suggests that consensus-based pseudo-labeling requires both distribution alignment and sufficient label-space diversity to provide reliable supervision.

Beyond raw accuracy, the model produces natural language explanations alongside its predictions, a property that sequence classifiers lack. Modality ablations showed the model genuinely depends on both inputs, with image removal hurting the English task most and OCR text removal crippling Arabic propaganda detection. An LLM-as-judge evaluation using GPT-4.1 and Gemini-2.5-Pro found the trained models substantially outperformed the zero-shot backbone on grounding, correctness, and usefulness for moderation, approaching the quality of human-verified reference explanations. The team has released all code, data extensions, prompting templates, and evaluation resources on GitHub.

The implications extend well beyond memes. The study shows that fine-grained supervision and distilled chain-of-thought rationales complement reinforcement-learning-based optimization, that multi-LLM annotation pipelines can scale fine-grained labeling to previously unlabeled domains, and that even a simple thresholded reasoning-length penalty can stabilize RL training against reward hacking in multimodal settings. For content moderation at scale, where subjective interpretation of culturally embedded imagery is the daily reality, the work offers a concrete template for building systems that not only flag harmful content but also articulate why, enabling meaningful human review rather than opaque automated verdicts.

Subject of Research: Reinforcement learning with chain-of-thought supervision for explainable detection of hateful and propagandistic memes

Article Title: Adapting reinforcement learning with chain-of-thought supervision for explainable detection of hateful and propagandistic memes

Article References: Kmainasi, M. B., Kutlu, M., Shahroor, A. E., Hasnat, A., & Alam, F. (2026). Adapting reinforcement learning with chain-of-thought supervision for explainable detection of hateful and propagandistic memes. Machine Learning with Applications, 26, Article 101003. https://doi.org/10.1016/j.mlwa.2026.101003

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101003

Keywords: reinforcement learning, chain-of-thought, multimodal large language models, hateful memes, propaganda detection, content moderation, GRPO, self-training, explainable AI, Arabic memes, reward hacking, fine-grained annotation

The core idea behind MR-UIE is a shift from treating large language models as passive extractors to training them as active reasoners. Rather than simply asking a model to map input text onto a fixed output structure, the framework teaches it to consider multiple analytical perspectives, weigh different reasoning strategies, and select the path most likely to yield a correct, faithful extraction. The researchers describe this as enabling models to understand not only what to extract but also how to reason, a distinction they argue is critical when texts are riddled with ambiguity, implicit causal links, and context-dependent information that simple pattern matching cannot resolve.

Technically, the framework rests on four pillars. The first is a unified schema representation, drawing on function-call semantics, that converts task-specific schemas for entities, relations, and events into a common hierarchical JSON structure with class identifiers, property-defining arguments, and semantic descriptors. This abstraction allows a single model to handle named entity recognition, relation extraction, and event extraction within one consistent output grammar, and it lets schemas for unseen tasks be projected into the same space at inference time, supporting zero-shot generalization without task-specific adapters.

The second pillar is the construction of a multi-perspective reasoning dataset, modeled on a human cognitive cycle of brainstorming, generating multiple solutions, and converging on the best one. For each input text and schema, a large language model acting as a strategy generator produces fine-grained thinking strategies along three dimensions: cognitive perspective, professional role, and heuristic rules. These raw strategies are then clustered into keyword-defined paradigms, ranked by TF-IDF embeddings and cosine similarity, and sampled to balance distinctiveness against representativeness. DeepSeek-R1 is then prompted to produce step-by-step chain-of-thought rationales together with structured predictions for each retained strategy, and rejection sampling discards any strategy whose prediction disagrees with the ground truth. Samples that fail to accumulate a minimum number of correct strategies are withheld from supervised training and routed instead to the reinforcement learning stage, ensuring the model learns from cases where reasoning demonstrably helps.

Training proceeds in three stages. The model, built on Qwen3-8B, first undergoes supervised fine-tuning on a curated corpus of roughly one million instances drawn from the bilingual IEPile collection, teaching it to interpret instructions and emit outputs that strictly conform to the unified schema. In the second stage, multi-perspective reasoning fine-tuning injects strategy prefixes into inputs and introduces special thinking tokens that segment reasoning steps, with a composite objective that optimizes both the chain-of-thought rationales and the structured predictions. To prevent overfitting to any single strategy, the team employs a regularization trick called strategy hiding, in which ten percent of training examples instruct the model to skip reasoning entirely. The final stage applies reinforcement learning using a rule-based Group Relative Policy Optimization procedure, in which the model generates multiple candidate reasoning chains per question and learns from a multi-grained reward function that scores both results and process.

The reward design is notable for its granularity. The result reward decomposes correctness into category-level and argument-level components, reweighted to reflect the fact that category prediction is easier than argument extraction, and combined as a weighted harmonic mean to encourage joint optimization. The process reward evaluates faithfulness, checking whether the reasoning strictly follows the schema, effectively uses the source text, and reflects a sound strategy. A convex combination of the two guides the policy, allowing the model to internalize effective reasoning patterns rather than memorize specific templates. During reinforcement learning, difficult samples are deliberately emphasized to raise the ceiling of model capability while leaving reasoning formats unconstrained, encouraging autonomous exploration of optimal strategies.

The experimental results are striking. Across eleven supervised benchmarks spanning newswire, finance, biomedicine, and general science, MR-UIE set new state-of-the-art Micro-F1 scores on five datasets, and on the remaining six never trailed the best published result by more than 3.2 F1 points. The wins span the full spectrum of extraction tasks: the framework topped the mit-restaurant dataset with its eight colloquial entity types and ambiguous boundaries, led on the terminology-dense biomedical AnatEM corpus despite limited training data, achieved the best overall scores on the skewed eighteen-category KBP37 relation benchmark, and ranked first on argument extraction for DUEE-fin, a financial event dataset with thirteen event types, ninety-one argument roles, and severe role overlap.

Zero-shot performance tells perhaps the more important story. Evaluated on datasets from domains and schemas never seen during training, spanning literature, music, law, and social media in both English and Chinese, MR-UIE achieved top-two results on most tasks and beat GPT-4 in five experimental settings, including a new state-of-the-art for zero-shot named entity recognition on CrossNER. The researchers attribute this generalization to the unified schema, which lets supervised capabilities transfer without structural adaptation, and to adaptive chain-of-thought reasoning that dynamically decomposes problems into finer-grained sub-steps, helping the model distinguish easily confused elements on schema-heavy datasets.

Ablation studies confirm that each component earns its place. Removing the unified schema cost between 0.42 and 0.72 F1 points across subtasks, while dropping multi-perspective reasoning training caused a maximum decline of 2.48 F1 points, demonstrating that explicit multi-angle fine-tuning is essential rather than decorative. Reinforcement learning alignment proved to internalize the diverse strategies rather than memorize them: when compared directly against static alternatives, the reinforcement-learned dynamic strategy beat random template selection, which actually degraded performance below a no-strategy baseline, and outperformed similarity-based template matching, whose effectiveness varied by task. Intriguingly, the training dynamics showed that as reinforcement learning progressed, generated reasoning chains shortened and converged to roughly 300 tokens while reward scores steadily climbed, suggesting that concise, efficient thinking benefits extraction performance more than lengthy elaboration.

The authors are candid about limitations, noting that the framework still struggles to dynamically integrate diverse reasoning paths and to handle highly noisy text, and they point toward noise-aware reasoning modules, contrastive path selection, refined reward design, and extension to multimodal settings as future directions. Even so, the work makes a compelling case that the road to robust structured understanding runs through reasoning itself. By rewarding models for faithful, efficient thinking and giving them multiple analytical lenses to think through, MR-UIE demonstrates that teaching language models how to reason may matter as much as teaching them what to know, a conclusion likely to shape how the next generation of extraction systems is built.

Subject of Research: Multi-perspective reasoning with reinforcement learning for universal information extraction

Article Title: MR-UIE: multi-perspective reasoning with reinforcement learning for universal information extraction

Article References: Li, Z., Wang, S., Fang, R., Bao, M., Wu, Z., Song, S., Li, Y., & He, Z. (2025). MR-UIE: multi-perspective reasoning with reinforcement learning for universal information extraction. Vicinagearth, 2(1), Article 17. https://doi.org/10.1007/s44336-025-00019-8

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00019-8

Keywords: information extraction, multi-perspective reasoning, reinforcement learning, large language models, chain-of-thought, universal information extraction, named entity recognition, relation extraction, event extraction, GRPO, zero-shot generalization, schema representation