Deep Learning from Zero: How Initialization Shapes Symmetry Breaking and Gradient Flow

For nearly as long as artificial neural networks have existed, one rule has been treated as almost sacrosanct: never initialize all weights and biases to zero. From foundational textbooks to standard online tutorials, students of machine learning have been taught that zero initialization causes every neuron in a layer to compute identical outputs, receive identical gradients, and therefore learn nothing — a phenomenon known as the failure of symmetry breaking. A new study from researchers at the Institute for Basic Science in South Korea now argues that this decades-old conviction is, at best, an oversimplification, and at worst, outright wrong.

In a paper published in Neural Processing Letters, Jongwoo Seo and Wuhyun Koh demonstrate that neural networks can learn successfully even when every weight and bias in the entire model is set to zero, provided the right mechanisms are in place. Their experiments span multilayer perceptrons (MLPs), convolutional neural networks (CNNs), residual networks (ResNets), vision transformers (ViTs), and multilayer perceptron mixers (MLP-Mixers), evaluated on MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet. In several settings, zero initialization matched — and in some cases exceeded — the performance of conventional random initialization, with MLP-Mixers delivering comparable results despite having no randomly initialized parameters at all.

The key insight emerged not from exotic mathematics but from a careful rereading of the standard backpropagation equations that underpin virtually all modern deep learning. The authors manually implemented forward and backward propagation from scratch using NumPy, covering architectures from basic MLPs to Transformers. In doing so, they noticed something mathematically nontrivial about the simplest case: for a linear transformation y = wx, the gradient with respect to the weight w is simply the input x. Crucially, this local gradient does not depend on the weight w itself. While the value of w can influence upstream error signals flowing from higher layers, it vanishes from the local chain rule computation.

This observation has profound consequences for how symmetry breaking is understood. In formal terms, the gradient of the loss with respect to a weight is the product of two factors: the upstream error signal from the layer above and the input activation from the layer below. The traditional argument against zero initialization assumes that if weights are identical, gradients will also be identical, freezing the network in a state of permanent uniformity. But the researchers point out that gradient diversity is determined primarily by the input values flowing into each weight — and those inputs do not need to come from randomly initialized weights. If the input activations differ across nodes, then the weight gradients will differ as well, breaking symmetry without any randomness in the weight values themselves.

So why does training fail in the textbook case of a fully zero-initialized network? The authors show that failure requires two consecutive zero-initialized layers. Under that condition, both essential functions of a layer during backpropagation break down simultaneously: weight updates halt because input activations are zero, and gradient transmission to lower layers halts because the weights themselves are zero. Learning comes to a complete standstill. But this deadlock is remarkably easy to break. Injecting random noise into node outputs during the forward pass gives the inputs the diversity they need, allowing gradients to differentiate and weights to begin diverging. Alternatively, residual connections can carry diverse input values directly to the next layer, achieving the same effect.

Perhaps most striking is the conceptual reframing this implies. When noise is injected into zero-initialized nodes, the resulting random activations produce gradient variation much as randomly assigned weights would. The authors therefore argue that conventional random initialization can be reinterpreted as a special case of zero-centered symmetry breaking — a form of randomness applied at the node level rather than the weight level. In their colorful analogy, random initialization splashes vibrant colors onto a canvas, while zero initialization begins with a pencil sketch that gradually acquires structure. The two strategies, long considered fundamentally different, may simply occupy different points on a continuum of initialization approaches.

The experimental results bear this out. When all weights and biases were set to zero, training failed as expected — but with residual connections or node-level noise added, learning proceeded successfully. Beyond the pure zero case, the team examined mixed initialization schemes in which only some layers are zero-initialized. On MNIST, MLPs achieved their highest accuracy under a “zero-alternate” setting applied to specific layers. For CNNs on CIFAR-10, models with roughly 20 percent of parameters randomly initialized tended to outperform fully random baselines, and zero initialization in the final layer produced faster early gains in accuracy across benchmarks.

The picture for contemporary architectures is more nuanced but still remarkable. On CIFAR-10 and CIFAR-100, ResNets and ViTs showed some statistically significant performance decreases under certain zero initialization conditions, yet other conditions maintained or improved performance. MLP-Mixers proved especially resilient, with several settings preserving accuracy despite extensive zero initialization. The location of zero-initialized layers mattered greatly: applying the scheme to layers closer to the input within the MLP module consistently produced significant performance drops, suggesting that where symmetry is broken may be as important as how.

Not every component cooperates with zero initialization, and the study is candid about the pitfalls. Batch normalization, which follows each layer operation in standard ResNets, becomes pathological when inputs are zero or uniform: the variance collapses, the normalization denominator shrinks to a tiny epsilon value, and gradients explode to enormous magnitudes — learning technically proceeds, but in a manner completely contrary to the layer’s design intent. For this reason, the authors restricted zero initialization in ResNets to the fully connected layers at the network’s end. Layer normalization behaves differently: because it normalizes across features within each sample rather than across channels, zero-initialized weights paired with nonzero biases still permit healthy output distributions. Architectures like ViTs and MLP-Mixers sidestep the issue entirely by applying residual connections before layer normalization.

Activation functions impose their own constraints. Popular functions like ReLU output a gradient of zero when their input is zero, blocking downward gradient flow — so the framework requires activation functions whose derivative is nonzero at the origin, such as Leaky ReLU or the Gaussian Error Linear Unit (GELU). Optimizers matter too: Adam, which dominates modern training, discards raw gradient magnitudes in its first update, effectively assigning gradients of approximately zero, one, or negative one based on direction alone, and preserving gradient information in subsequent steps.

To systematize these findings, the authors propose the “Seo Integrated Zero Initialization: Foundational Scheme (SIZIFS)” — a unified conceptual framework, its name deliberately echoing Sisyphus, that classifies initialization strategies into three categories: weight-level random initialization, node-level random initialization (in which all weights and biases are zero and randomness is injected into nodes), and context-dependent methods that fit neither category. Because learning is possible even without any weight initialization at all, the very concept of neural network initialization expands to encompass node-level and architectural approaches.

The work is not without limitations. The authors acknowledge that they provide no rigorous mathematical proof for why zero initialization can rival random initialization, that hardware constraints prevented experiments on full ImageNet, and that detailed statistical analyses of weight distributions and gradient magnitudes across training steps remain future work. They also raise tantalizing questions: whether zero initialization mimics synaptic resetting observed in biological brains during sleep, and whether new architectures could be designed specifically to exploit it. Still, they draw an evocative historical parallel — just as the same celestial observations once supported both geocentric and heliocentric models, the same backpropagation equations that justified avoiding zero initialization can, on closer inspection, justify embracing it.

The authors describe their work as a potential “Copernican shift” in how the field understands initialization. Whatever the ultimate practical impact, the message is clear: one of deep learning’s oldest certainties deserves a fresh look, and the humble zero may hold more power than anyone supposed.