How Brains and Machines Learn to Avoid Disaster: The New Science of Safe Reinforcement Learning

Learning from experience is a dangerous business. To master a new environment, an agent—whether a mouse, a human, or a robot—must try things out, and some trials end in catastrophe. A single severe injury can end an animal’s ability to learn anything at all, and a single catastrophic fault can destroy a robot. This paradox, in which the very process of learning demands the mistakes that learning must avoid, sits at the heart of a new review published in Communications Psychology by Pranav Mahajan and Ben Seymour of the University of Oxford. The authors argue that the question of how intelligent systems learn safely is not merely an engineering puzzle for artificial intelligence, but a fundamental problem that evolution has been solving for hundreds of millions of years—and that the solutions engineered in silicon and those sculpted by natural selection are beginning to converge on a shared computational logic.

The framework the authors use is reinforcement learning, the mathematical formalism that describes how agents learn to act by maximising cumulative reward. In its canonical form, an agent observes a state, selects an action, and receives a reward, gradually refining a policy that maps states to actions. The celebrated Q-learning algorithm, for example, updates the value of state-action pairs using a temporal-difference prediction error—a signal with a famous neural counterpart in the phasic activity of dopamine neurons in the ventral tegmental area. But standard reinforcement learning has a blind spot: it optimises expected return without regard for the catastrophic costs that certain errors can impose. An agent that learns only from average outcomes will happily walk off a cliff if the view is, on balance, rewarding.

To organise the field, Mahajan and Seymour adopt a taxonomy, originally proposed by Javier García and Fernando Fernández, that divides safe reinforcement learning methods into two broad families. The first modifies the optimisation criterion itself, embedding safety directly into the learning objective. The second modifies the exploration process, controlling how the agent gathers experience so that dangerous actions are never taken in the first place. The authors pair this with Chris Watkins’s classic catalogue of six forms of innate knowledge—from initial value functions to tendencies to experiment—creating a three-dimensional map on which any safe-learning method can be located by what it changes, what prior knowledge it exploits, and which part of the canonical reinforcement-learning scaffold it touches.

Within the first family, the simplest strategy is pessimism. Worst-case methods, such as β-pessimistic Q-learning, interpolate between the best and worst possible future outcomes, with the parameter β quantifying how much weight an agent places on the worst case. This abstract knob turns out to have striking clinical relevance. Work by Zorowitz, Momennejad and Daw showed that a high pessimism parameter—a belief that future avoidance will fail—causes fear to propagate excessively to situations far removed from the actual threat, offering a computational account of the exaggerated threat appraisals and persistent avoidance seen in anxiety disorders. A single pessimistic assumption, encoded in an update rule, can generate the signature of pathological worry.

A more refined approach targets the lower tail of the return distribution directly. Conditional value at risk, or CVaR, optimises the expected return conditional on outcomes falling in the worst ζ-quantile—in effect, formalising an agent’s focus on tail events rather than the full spectrum of possibilities. Empirical studies show that human behaviour in sequential decision tasks is often well captured by CVaR-style objectives, and that individual differences in risk sensitivity can model the spectrum from ‘brave’ to ‘timid’ exploration in mice confronting novel objects. Intriguingly, the brain may possess the machinery to compute such distributional quantities: recent evidence for distributional reinforcement learning in dopaminergic neurons of the basal ganglia and in prefrontal cortex suggests that neural circuits represent not just the mean value of an outcome but its entire probability distribution, making tail-sensitive learning biologically plausible.

A third strategy keeps safety costs entirely separate from reward. Constrained Markov decision processes maximise expected return subject to an explicit budget on expected cost, and Mahajan and Seymour suggest the brain may approximate such solutions through multiple interacting control systems. In one model, an instrumental reward-seeking system operates alongside a distinct Pavlovian fear system that acts as a dedicated controller, weakly minimising constraint violations such as cumulative pain. This architecture captures an important psychological intuition: some harms are fungible with reward and can be traded off, while others—severe tissue damage, loss of viability—are better treated as non-negotiable bounds that no amount of reward should be allowed to breach.

The second family of methods modifies exploration rather than the objective, and here the authors argue that biology offers the richest lessons. One engineering technique biases agents toward predictable, ‘controllable’ regions of the environment by rewarding low prediction-error variance. Neuroscience supplies a striking parallel in Pearce-Hall associability, a running average of absolute prediction errors that, in models proposed by Mahajan and colleagues, arbitrates between instrumental control and a specialised Pavlovian fear system. When outcome uncertainty is high, the Pavlovian system gains influence, enforcing cautious withdrawal; as uncertainty resolves, control cedes to the more efficient instrumental system. A failure to down-regulate this Pavlovian influence may help explain persistent avoidance in generalised anxiety and in the fear-avoidance model of chronic pain.

Other biological solutions concern what agents bring to learning before any experience at all. Animals are not blank slates: neophobia, the wariness of novel stimuli, functions as a pessimistic value initialisation, and has been linked to dopamine signals in the tail of the striatum. Recent models propose that approach-retreat bouts during exploration arise from dynamically composing multiple value functions with different initialisations, gated by an inferred threat-belief state. Engineering mirrors this through apprenticeship learning and meta-learned initialisations, which bootstrap agents from demonstrations to reduce dangerous random exploration. For embodied agents, safety also demands self-monitoring: robots use forward models and fault detection to spot discrepancies between predicted and actual sensor readings, a strategy that parallels the brain’s interoceptive predictive models of the body’s internal state. Recent work even shows that competing homeostatic needs—hunger versus pain—converge on neuropeptide-expressing neurons in the parabrachial nucleus, producing endogenous analgesia when survival priorities shift.

Underlying all of these approaches, the review identifies a fundamental safety-efficiency trade-off. Stricter safety constraints inevitably slow reward learning, and theoretical proofs across computer science now formalise what evolution seems to have known all along: an agent cannot simultaneously be maximally safe and maximally sample-efficient. Risk-sensitive objectives solve the approximate problem exactly, while hard-wired Pavlovian biases apply approximate solutions to the correct problem—a distinction the authors suggest maps onto a key question in computational psychiatry: is a patient optimising a pathological objective, or applying a maladaptive heuristic to a normal one?

The converging picture is one in which safety emerges from a dynamic interplay between reactive instincts and deliberative planning, whether those instincts are encoded in amygdala circuits or in safety layers bolted onto a learning algorithm. The authors close with a set of open challenges: designing tasks that distinguish pessimistic beliefs about control from lower-tail risk sensitivity, determining when harms should be modelled as constraints rather than costs, and—perhaps most importantly—building frameworks that jointly handle external threats and internal bodily integrity. As artificial agents move into the physical world, the lessons of biological survival may prove to be the most valuable training data of all.

Subject of Research: Safe reinforcement learning and its computational parallels in behavioural and systems neuroscience

Article Title: Natural and engineered safe reinforcement learning

Article References: Mahajan, P., & Seymour, B. (2026). Natural and engineered safe reinforcement learning. Communications Psychology, 4(1), Article 138. https://doi.org/10.1038/s44271-026-00542-4

Image Credits: AI Generated

DOI: 10.1038/s44271-026-00542-4

Keywords: safe reinforcement learning, computational neuroscience, risk sensitivity, conditional value at risk, constrained Markov decision processes, anxiety, Pavlovian control, exploration, interoception, computational psychiatry, dopamine, artificial intelligence