In a paper published in Neural Processing Letters, Jongwoo Seo and Wuhyun Koh demonstrate that neural networks can learn successfully even when every weight and bias in the entire model is set to zero, provided the right mechanisms are in place. Their experiments span multilayer perceptrons (MLPs), convolutional neural networks (CNNs), residual networks (ResNets), vision transformers (ViTs), and multilayer perceptron mixers (MLP-Mixers), evaluated on MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet. In several settings, zero initialization matched — and in some cases exceeded — the performance of conventional random initialization, with MLP-Mixers delivering comparable results despite having no randomly initialized parameters at all.

The key insight emerged not from exotic mathematics but from a careful rereading of the standard backpropagation equations that underpin virtually all modern deep learning. The authors manually implemented forward and backward propagation from scratch using NumPy, covering architectures from basic MLPs to Transformers. In doing so, they noticed something mathematically nontrivial about the simplest case: for a linear transformation y = wx, the gradient with respect to the weight w is simply the input x. Crucially, this local gradient does not depend on the weight w itself. While the value of w can influence upstream error signals flowing from higher layers, it vanishes from the local chain rule computation.

This observation has profound consequences for how symmetry breaking is understood. In formal terms, the gradient of the loss with respect to a weight is the product of two factors: the upstream error signal from the layer above and the input activation from the layer below. The traditional argument against zero initialization assumes that if weights are identical, gradients will also be identical, freezing the network in a state of permanent uniformity. But the researchers point out that gradient diversity is determined primarily by the input values flowing into each weight — and those inputs do not need to come from randomly initialized weights. If the input activations differ across nodes, then the weight gradients will differ as well, breaking symmetry without any randomness in the weight values themselves.

So why does training fail in the textbook case of a fully zero-initialized network? The authors show that failure requires two consecutive zero-initialized layers. Under that condition, both essential functions of a layer during backpropagation break down simultaneously: weight updates halt because input activations are zero, and gradient transmission to lower layers halts because the weights themselves are zero. Learning comes to a complete standstill. But this deadlock is remarkably easy to break. Injecting random noise into node outputs during the forward pass gives the inputs the diversity they need, allowing gradients to differentiate and weights to begin diverging. Alternatively, residual connections can carry diverse input values directly to the next layer, achieving the same effect.

Perhaps most striking is the conceptual reframing this implies. When noise is injected into zero-initialized nodes, the resulting random activations produce gradient variation much as randomly assigned weights would. The authors therefore argue that conventional random initialization can be reinterpreted as a special case of zero-centered symmetry breaking — a form of randomness applied at the node level rather than the weight level. In their colorful analogy, random initialization splashes vibrant colors onto a canvas, while zero initialization begins with a pencil sketch that gradually acquires structure. The two strategies, long considered fundamentally different, may simply occupy different points on a continuum of initialization approaches.

The experimental results bear this out. When all weights and biases were set to zero, training failed as expected — but with residual connections or node-level noise added, learning proceeded successfully. Beyond the pure zero case, the team examined mixed initialization schemes in which only some layers are zero-initialized. On MNIST, MLPs achieved their highest accuracy under a “zero-alternate” setting applied to specific layers. For CNNs on CIFAR-10, models with roughly 20 percent of parameters randomly initialized tended to outperform fully random baselines, and zero initialization in the final layer produced faster early gains in accuracy across benchmarks.

The picture for contemporary architectures is more nuanced but still remarkable. On CIFAR-10 and CIFAR-100, ResNets and ViTs showed some statistically significant performance decreases under certain zero initialization conditions, yet other conditions maintained or improved performance. MLP-Mixers proved especially resilient, with several settings preserving accuracy despite extensive zero initialization. The location of zero-initialized layers mattered greatly: applying the scheme to layers closer to the input within the MLP module consistently produced significant performance drops, suggesting that where symmetry is broken may be as important as how.

Not every component cooperates with zero initialization, and the study is candid about the pitfalls. Batch normalization, which follows each layer operation in standard ResNets, becomes pathological when inputs are zero or uniform: the variance collapses, the normalization denominator shrinks to a tiny epsilon value, and gradients explode to enormous magnitudes — learning technically proceeds, but in a manner completely contrary to the layer’s design intent. For this reason, the authors restricted zero initialization in ResNets to the fully connected layers at the network’s end. Layer normalization behaves differently: because it normalizes across features within each sample rather than across channels, zero-initialized weights paired with nonzero biases still permit healthy output distributions. Architectures like ViTs and MLP-Mixers sidestep the issue entirely by applying residual connections before layer normalization.

Activation functions impose their own constraints. Popular functions like ReLU output a gradient of zero when their input is zero, blocking downward gradient flow — so the framework requires activation functions whose derivative is nonzero at the origin, such as Leaky ReLU or the Gaussian Error Linear Unit (GELU). Optimizers matter too: Adam, which dominates modern training, discards raw gradient magnitudes in its first update, effectively assigning gradients of approximately zero, one, or negative one based on direction alone, and preserving gradient information in subsequent steps.

To systematize these findings, the authors propose the “Seo Integrated Zero Initialization: Foundational Scheme (SIZIFS)” — a unified conceptual framework, its name deliberately echoing Sisyphus, that classifies initialization strategies into three categories: weight-level random initialization, node-level random initialization (in which all weights and biases are zero and randomness is injected into nodes), and context-dependent methods that fit neither category. Because learning is possible even without any weight initialization at all, the very concept of neural network initialization expands to encompass node-level and architectural approaches.

The work is not without limitations. The authors acknowledge that they provide no rigorous mathematical proof for why zero initialization can rival random initialization, that hardware constraints prevented experiments on full ImageNet, and that detailed statistical analyses of weight distributions and gradient magnitudes across training steps remain future work. They also raise tantalizing questions: whether zero initialization mimics synaptic resetting observed in biological brains during sleep, and whether new architectures could be designed specifically to exploit it. Still, they draw an evocative historical parallel — just as the same celestial observations once supported both geocentric and heliocentric models, the same backpropagation equations that justified avoiding zero initialization can, on closer inspection, justify embracing it.

The authors describe their work as a potential “Copernican shift” in how the field understands initialization. Whatever the ultimate practical impact, the message is clear: one of deep learning’s oldest certainties deserves a fresh look, and the humble zero may hold more power than anyone supposed.

Static networks, exemplified by artificial neural networks (ANNs) and wireless sensor networks, feature units arranged in fixed topologies. Each unit maintains stable, consistent relationships with its neighbors, allowing for robust, low-noise communication pathways and predictable interaction patterns. This structure facilitates well-established computational paradigms like gradient-based optimization and backpropagation in ANNs, where learning is embedded in the persistent connectivity and adjusted synaptic weights between units. Similarly, sensor networks use their stable links to reliably aggregate environmental data over time, leveraging the near-permanence of spatial and informational relationships.

Contrast this with systems composed of highly mobile units, such as biological swarm robotics or natural social insect colonies, where individual agents are in constant flux. Mobile units often encounter each other fleetingly, with ephemeral interactions that lack the longevity found in fixed networks. This fluidity demands radically different strategies for collective learning and computation. Without reliance on stable neighbors, mobile collectives cannot use classical gradient descent or persistent weight matrices. Instead, they must exploit alternative modes of plasticity intrinsic to single units, rapidly forming transient groupings, or ingeniously modifying their environments to encode information and influence subsequent interactions.

One intriguing strategy employed by mobile collectives involves environmental modifications—essentially embedding memory and computational signals in the surroundings. Ant colonies, for instance, lay pheromone trails that serve as ephemeral communication channels, guiding the swarm’s behavior and facilitating decentralized problem-solving. Similarly, mobile robot swarms can manipulate environmental markers or leverage spatial configurations to enact a form of stigmergy, encoding collective decisions and enhancing coordination without requiring constant unit-to-unit communication.

Understanding these fluid mechanisms provides a valuable lens for re-examining static systems. Although fixed networks are optimized for stable topologies, analogues to environmental modifications appear in forms such as maintaining global state variables or incorporating spatially distributed memory units. Recognizing these parallels invites cross-pollination of ideas, potentially inspiring hybrid architectures that leverage the benefits of both fixed and fluid characteristics, particularly in enhancing learning robustness and adaptability.

One of the most compelling insights emerges when considering resource efficiency. Mobility not only changes how information is processed but can fundamentally reduce the number of units necessary to achieve a desired performance threshold. By dynamically repositioning and aggregating, mobile units orchestrate collective actions that static arrangements require significantly more nodes to replicate. This principle challenges conventional assumptions in network design and artificial intelligence, suggesting that incorporating controlled mobility or movement-inspired signaling could yield lighter, more cost-efficient systems without sacrificing computational power.

To illustrate this concept, researchers draw an analogy between robot swarms tasked with reaching a consensus and convolutional neural networks (CNNs) employed in image classification. Both systems process distributed information through local interactions, yet swarms achieve coordination through transient networking and spatial reconfiguration, while CNNs rely on fixed receptive fields and hierarchical feature detection. Insights from the fluid dynamics of swarms could thus inform CNN designs, enabling smaller static networks with comparable expressiveness or improved training efficiency through adaptive connectivity schemes.

Conversely, infusing aspects of static topologies into mobile collections can enhance computational capabilities. Temporarily immobilizing units or imposing predictable movement patterns transforms a fluid system into one with quasi-static neighborhoods. This enables the application of richer, more complex algorithms typically exclusive to static networks, such as iterative consensus protocols and distributed optimization, thereby expanding the range of tasks mobile units can perform effectively.

Viewing collective intelligence through the dual perspectives of mobility and stability not only deepens theoretical understanding but also sparks practical innovation. Emerging proposals suggest dynamic hybrid networks wherein nodes alternate between stationary and mobile states, or where environmental modifications serve as persistent memory anchors for fluid interactions. Such fluid-statical blends promise novel architectures capable of seamlessly adapting to changing computational demands and operational conditions, bridging the gap between biological inspiration and technological implementation.

The implications of this paradigm stretch across disciplines. In robotics, adopting principles from biological swarms may improve autonomous coordination in unpredictable environments, such as disaster zones or extraterrestrial landscapes. For machine learning, incorporating mobility-inspired adaptability could yield more efficient model architectures tailored to edge computing constraints. Moreover, ecological and social sciences stand to benefit by modeling collective behavior with newfound fidelity, elucidating the emergent intelligence of natural collectives and their evolutionary advantages.

Nonetheless, significant challenges remain in fully harnessing mobility’s potential. Designing algorithms that robustly exploit transient contacts and environmental cues demands sophisticated modeling of temporal dynamics and uncertainties. Additionally, ensuring stability and fault tolerance amid shifting topologies calls for rigorous theoretical frameworks and experimental validation. These hurdles underscore the need for interdisciplinary collaboration blending insights from physics, computer science, biology, and engineering.

The synthesis of fluid and static viewpoints also provokes philosophical questions about the nature of intelligence and learning itself. Is stable memory a prerequisite for complex cognition, or can ephemeral interactions combined with environmental scaffolding suffice? Answering such questions could redefine foundational assumptions about how intelligence emerges in distributed systems, shifting emphasis from individual processing power toward relational dynamics and collective plasticity.

In sum, recognizing the centrality of unit mobility reshapes our conceptualization of collective intelligence. This recognition unlocks pathways toward more flexible, scalable, and efficient systems capable of tackling diverse computational challenges. By bridging static and fluid topologies, future research stands to innovate computational architectures that harness the best of both worlds, driving breakthroughs in artificial intelligence, robotics, and beyond.

As the field matures, experimental platforms integrating mobile and static components will be critical testbeds. Using physical robot swarms augmented with environmental modifications alongside simulations of neural network variants may clarify trade-offs and guide practical applications. The convergence of empirical data, theoretical models, and cross-disciplinary insights promises a fertile ground for transformative advances.

Ultimately, the paradigm of fluid thinking about collective intelligence heralds a new era. It challenges entrenched divisions between network types, unveiling richer forms of learning shaped by mobility and environment. This frontier invites scientists and engineers to rethink what it means to compute collectively in a world where units are not merely nodes in place but dynamic agents navigating complex spatial and temporal landscapes.

Subject of Research: Collective intelligence in distributed systems with emphasis on the impact of unit mobility on learning and computation.

Article Title: Fluid thinking about collective intelligence

Article References: Werfel, J. Fluid thinking about collective intelligence. Nat Mach Intell 8, 506–516 (2026). https://doi.org/10.1038/s42256-026-01211-1

Image Credits: AI Generated

DOI: https://doi.org/10.1038/s42256-026-01211-1

Keywords: collective intelligence, mobile units, static networks, swarm robotics, neural networks, environmental modification, learning algorithms, network topology, fluid topology, resource efficiency