Spiking Neural Networks Learn Faster When They Can Predict What Happens Next

…

…

The study, published in Frontiers in Neurorobotics by researchers at Chiba Institute of Technology, arrives at a moment when the gap between the theoretical promise of spiking neural networks and their practical learning performance has become one of the more persistent obstacles in neuromorphic robotics. Spiking neural networks differ from conventional artificial neural networks in a fundamental way: rather than propagating continuous-valued activations between layers, they communicate through discrete, transient electrical pulses known as spikes. This event-driven mode of communication mirrors the behavior of biological neurons, which fire only when their membrane potential crosses a threshold, and it means that computation and energy are consumed only when meaningful events occur. On neuromorphic hardware, which is designed specifically to exploit this sparsity, SNNs can deliver order-of-magnitude improvements in power efficiency compared with conventional deep networks running on standard processors. For autonomous mobile robots, which must carry their own batteries and perform decision-making continuously under strict energy budgets, this efficiency advantage is not merely an academic curiosity but a practical necessity.

Yet the efficiency of SNNs has historically come at a cost in learning capability. When continuous-valued information must be converted into spike trains—a process often accomplished through rate coding, in which a value is represented by the firing frequency of a neuron or population of neurons over a time window—some of the fine-grained magnitude information is inevitably lost or distorted. Furthermore, the internal dynamics of spiking neurons, typically modeled with membrane potential states that integrate incoming spikes over time, impose constraints on how features can be composed and represented. A spiking actor network therefore tends to be at a disadvantage when it comes to forming the rich, continuous-valued internal features that policy gradient methods and actor-critic architectures rely upon for smooth, precise control. This is precisely the challenge that the Chiba Institute of Technology team—Hiroto Takigasaki, Takahiro Iwata, Seankein Yoshioka, and Daisuke Miki—set out to address, not by redesigning the spiking network itself, but by changing what information is fed into it.

The conceptual foundation of their approach draws on a growing body of work in representation learning for reinforcement learning. Representation learning, broadly speaking, is the practice of training auxiliary models to extract compact, informative features from raw observations, so that the downstream policy and value functions operate on inputs that are already structured in useful ways. In deep reinforcement learning, this idea has proven especially valuable when observations are high-dimensional, such as raw camera images, where convolutional encoders learned through auxiliary prediction tasks can dramatically accelerate policy learning. What is perhaps less widely appreciated is that representation learning can also yield substantial benefits when observations are low-dimensional state vectors, as is common in classical continuous control benchmarks. Ota and colleagues demonstrated this with OFENet, an online feature extraction network that learns representations of states and state-action pairs by predicting the next observation from the current one. The intuition is that a network forced to model the environment’s dynamics must internalize which aspects of the state actually matter for predicting what happens next, and those same aspects tend to be the ones that matter for control.

Fujimoto and colleagues extended this line of thinking with State-Action Learned Embedding, or SALE, which learns embeddings for both states and state-action pairs by modeling environmental dynamics in a latent space. Rather than merely predicting the next observation, SALE constructs representations that capture the relationship between the action taken and the resulting transition, embedding information about how the environment responds to the agent’s choices. When these learned embeddings are substituted for raw observations in both the policy and the action-value function, improvements have been reported not only in final performance but also in the speed with which rewards are acquired during the early stages of training. This early-stage acceleration is particularly significant for real-world robotics, where every interaction with the physical environment carries a cost in time, wear on hardware, and potential safety risk. An algorithm that reaches competent behavior in fewer episodes directly reduces the burden of real-world training.

The contribution of the new study lies in combining this transition-aware representation learning with a hybrid spiking architecture. In such hybrid frameworks, the actor—the component that selects actions—is implemented as a spiking neural network, while the critic, which estimates the value of state-action pairs and provides the learning signal, remains a conventional deep network. This division of labor is deliberate: the actor is the component that must run continuously on the robot’s onboard hardware during deployment, so placing the energy-efficient spiking network there yields the greatest practical benefit, while the critic, which is needed primarily during training, can afford the higher computational cost of a conventional architecture. The most prominent example of this design philosophy is the Population-Coded Spiking Actor Network, or PopSAN, proposed by Tang and colleagues, in which continuous values are encoded through the firing rates of populations of spiking neurons. PopSAN demonstrated that such hybrid architectures can achieve competitive performance on continuous control tasks while consuming dramatically less energy than deep actor networks, but subsequent studies have consistently noted that the learning performance of these spiking actors lags behind their conventional counterparts.

By integrating SALE-style transition-aware embeddings into this hybrid framework, the researchers effectively give the spiking actor a richer, more informative input than the raw state vector alone. The embeddings, learned through an auxiliary dynamics-modeling objective, condense information about the current state and the action being considered into a form that highlights the aspects of the transition most relevant to control. This compensates, at least in part, for the representational limitations of spike-based computation: even if the spiking network cannot easily form certain continuous-valued features internally, it can receive features that have already been shaped by the dynamics model into a form that is easier to map onto good actions. The critic likewise benefits, since action-value estimation on well-structured embeddings tends to be more accurate and more stable than estimation on raw observations.

The empirical evaluation was conducted on four continuous control tasks drawn from the OpenAI Gym benchmark suite, a standard testing ground for reinforcement learning algorithms that includes simulated robotic manipulation and locomotion problems with continuous action spaces. The results showed that the proposed method outperformed conventional SNN-based deep reinforcement learning approaches on both of the metrics the researchers emphasized: the maximum average reward achieved during training, which reflects the ceiling of performance the method can reach, and the rate of reward acquisition in the early stages of learning, which reflects sample efficiency. The authors also examined how the dimensionality of the learned embeddings influences performance, an important practical consideration because embedding dimensionality determines the amount of information that must pass through the spike encoding stage and thus affects both the representational capacity of the system and its computational cost.

These findings sit within a broader research landscape that includes several distinct strategies for bringing spiking networks into deep reinforcement learning. One strategy, known as ANN-to-SNN conversion, trains a conventional deep network first and then transfers its weights into a spiking equivalent for inference. Patel and colleagues showed that a trained Deep Q-Network could be converted in this way with gains in energy efficiency and robustness, although a recurring observation in the literature is that the converted spiking network rarely surpasses the original artificial network in accuracy. A second strategy trains spiking networks directly from the start, using surrogate gradient methods to approximate the derivatives of the non-differentiable spike generation function so that standard backpropagation-based optimization can proceed. Liu and colleagues reported that a directly trained Deep Spiking Q-Network could outperform both the artificial DQN and converted spiking variants on Atari benchmarks, and Chen and colleagues proposed representing Q-values directly in membrane potentials, improving training stability. However, these direct-training approaches have been developed primarily for discrete action spaces, such as the finite button-press choices in video games, and extending them to the continuous action spaces that dominate robotics remains difficult. The hybrid spiking-actor approach pursued in the current study represents a third path, one that sidesteps some of these difficulties by keeping the critic conventional and reserving the spiking network for the actor.

Within this third path, several refinements have accumulated. Zhang and colleagues proposed the multiscale dynamic coding-improved spiking actor network, which combines population coding at the network scale with neurons that incorporate second-order dynamics at the individual neuron scale, aiming to capture richer spatiotemporal state representations. Iwata and colleagues, whose work overlaps with the present authorship, introduced a non-recursive spiking actor structure designed to eliminate output delays—a real concern in control, where recurrent spiking dynamics can cause the network’s output to lag behind the current state—and adopted the burn-in strategy from recurrent replay distributed DQN to address the problem of stale membrane potentials stored in the replay buffer, thereby improving the accuracy of Q-function estimation. The present study’s embedding-based approach is complementary to these efforts: rather than modifying the spiking neuron model or the network topology, it improves the quality of the information entering the network, which suggests that the two lines of improvement could potentially be combined in future systems.

The significance of the work for the robotics community lies in what it implies about the path toward energy-efficient autonomous agents. If spiking actors can be brought closer to the learning performance of conventional deep actors without sacrificing their efficiency advantage, then the case for deploying neuromorphic hardware on real robots becomes considerably stronger. The demonstration that transition-aware embeddings accelerate early learning is especially relevant to real-world deployment scenarios, where the number of physical trials available for training is limited by cost, safety, and hardware wear. The researchers note that their results establish the effectiveness of leveraging dynamics-learned representations in the spiking setting, and they frame future work around further refinement of the integration between representation learning and spiking policy optimization. As neuromorphic processors continue to mature and become more accessible, methods of this kind—those that treat the spiking network not as an isolated curiosity but as one component within a carefully engineered learning system—are likely to define the practical frontier of energy-efficient robot intelligence.

Subject of Research: …

Article Title: Improving performance in SNN-based deep reinforcement learning via transition-aware embeddings

Article References: Improving performance in SNN-based deep reinforcement learning via transition-aware embeddings. (n.d.). https://doi.org/10.3389/fnbot.2026.1929177

Image Credits: AI Generated

DOI: 10.3389/fnbot.2026.1929177

Keywords: spiking neural networks, deep reinforcement learning, robotics, neuromorphic computing, representation learning, continuous control, sample efficiency, energy efficiency, MuJoCo, TD3, transition-aware embeddings, neurorobotics