Deep Belief Networks Trained Without Labels Converge on Matching Representations

Deep belief networks, the layered generative models that helped ignite the modern deep learning revolution, have revealed a striking property in a new study: when trained entirely without labels, they spontaneously organize their internal representations so that images from the same category drift together and images from different categories drift apart. The finding, published in Frontiers in Complex Systems by Patrick Krauss, Achim Schilling, Andreas K. Maier, Thomas Kinfe, and Claus Metzner of Friedrich-Alexander-Universität Erlangen-Nürnberg and collaborating institutions, shows that a hierarchy built purely from input statistics discovers class structure that no one ever told it about. Even more remarkably, networks that start from different random initializations end up with closely matching internal geometries layer by layer, suggesting that the learned hierarchy is strongly constrained by the data and the learning procedure rather than by the accidents of initialization.

The team trained stacks of Bernoulli-Bernoulli restricted Boltzmann machines, the classic building blocks of deep belief networks, on three benchmark datasets: MNIST with its handwritten Arabic digits, Fashion-MNIST with ten clothing categories, and Kuzushiji-MNIST with cursive Japanese characters. Each network consisted of a 784-dimensional visible layer followed by ten hidden layers of 100 units, trained greedily from bottom to top with one-step contrastive divergence, no momentum, no weight decay, and no end-to-end fine-tuning. Class labels were withheld throughout training and introduced only afterward as diagnostic tools. Five independent seeds defined genuinely separate networks rather than repeated evaluations of a single model, and these networks served as the statistical replicates for all inferential summaries.

To measure class organization, the researchers used the Generalized Discrimination Value, a scalar that compares typical distances between points of the same class with distances between points of different classes. The measure is invariant under translations, component-wise rescaling, and permutations of neuron indices, making it comparable across layers of different dimensionality. Across all three datasets, the value became progressively more negative with depth, meaning that the unknown classes formed increasingly distinct regions of representation space. The effect appeared at every investigated layer width and was particularly pronounced in narrower networks. Because the trend held for Fashion-MNIST and Kuzushiji-MNIST as well as MNIST, it cannot be attributed to any peculiarity of handwritten digits; greedy unsupervised learning appears to repeatedly extract the regularities that occur reliably across many examples, and those regularities often coincide with the distinctions that define human-labeled classes.

The improvement, however, is not unlimited. In the narrowest MNIST networks, class separation reached its optimum after roughly eight hidden layers and then deteriorated, while medium-width networks settled onto a broad plateau and the widest networks continued improving over much larger depths. Network width therefore determines how long the hierarchy can keep forming useful new structure before extreme compression begins to erase it. Alongside this geometric reorganization, two other quantities moved in predictable directions: reconstruction error rose steadily with depth, showing that higher layers retain progressively less information about the precise input image, and effective dimensionality, quantified by the participation ratio of the covariance spectrum, declined consistently, in extreme cases collapsing an entire hidden layer to nearly a one-dimensional representation.

Before accepting the spontaneous class separation as genuine, the team ran a battery of controls. Untrained networks with random weights produced only a small initial shift before their representations collapsed toward near-constant activity and their discrimination values returned toward zero. A weight-shuffling control, which preserved each neuron’s exact weight values, norms, and biases but destroyed the learned assignment of weights to input coordinates, retained only a modest fraction of the separation and showed little systematic improvement with depth. Increasing contrastive divergence from one to two Gibbs steps left the trajectories almost unchanged. Evaluating preactivations before the sigmoid nonlinearity revealed that the learned weighted projection already creates the class-related geometry, with the sigmoid often weakening it slightly. Replacing deterministic activation probabilities with sampled binary states substantially masked the effect, as noise accumulated across layers. The decisive ingredient, the authors conclude, is the structured set of affine features learned at each layer and passed upward through the hierarchy.

The dimensional collapse itself turned out to have a specific origin. When the researchers recalculated the effective dimension after removing all correlations between neurons while preserving each unit’s variance, the resulting variance-only dimension remained far above the actual dimension, ruling out silent neurons as the main explanation. Instead, activity correlations between neuron pairs grew steadily with depth, and the incoming weight vectors told the same structural story: in deeper layers, almost every neuron acquired another neuron with a nearly parallel or antiparallel weight vector. The network was learning multiple copies of similar features rather than using its full width for distinct ones. A targeted pruning intervention confirmed the functional consequence: when near-duplicate neurons, identified by a label-free score combining weight cosine similarity and activation correlation, were removed from layers five or ten, the pruned representation retained a higher effective dimensionality than after removing the same number of random neurons, in some cases nearly preserving the intact layer’s dimensionality despite halving its width.

Yet a more ordered representation proved not to be uniformly more informative. The global class geometry continued to improve with depth while the accuracy of supervised probes, both a linear logistic classifier and a flexible radial-basis support vector machine fitted to the frozen representations, progressively declined in all three datasets. Resolving the global measure into all 45 pairwise class relations for MNIST explained the paradox: most pairs became geometrically better separated, but a small minority, including visually similar digit pairs such as 4 versus 9 and 5 versus 8, deteriorated both geometrically and in mutual confusion. These difficult boundaries contribute little to an average taken over all class relations but generate a disproportionate share of actual classification errors. Global geometric organization and practical class accessibility, the study shows, are related but non-identical properties, and early hidden layers in MNIST and Kuzushiji-MNIST actually improved both relative to raw pixels.

The convergence between independently trained networks went beyond matching summary curves. Using centered kernel alignment, which compares the relational geometry of representations while remaining invariant to orthogonal transformations and neuron permutations, the researchers found remarkably high similarity between corresponding layers of different networks, already strong at the first hidden layer and close to its maximum throughout the hierarchy. Randomly permuting the correspondence between images reduced the alignment almost to zero, and a stricter control that preserved class membership while breaking the identity of individual images remained clearly below the trained comparison. Cross-layer comparisons concentrated the similarity near the matching depth, allowing the correct hierarchy level to be identified almost perfectly from the representation alone. An optimal one-to-one matching of neuronal response profiles, computed with the Hungarian algorithm, revealed substantially more similar matched neurons than arbitrary pairs, with agreement increasing toward the highest layers, and at the first hidden layer even the incoming weight patterns could be matched closely.

The generative side of the hierarchy developed in the opposite direction from its reconstructive side. When truncated stacks were used for free generation, with only the top restricted Boltzmann machine equilibrating through Gibbs sampling, samples from deeper truncations became substantially more recognizable, covered a broader effective number of classes, and moved closer to the real data distribution in the feature space of an external convolutional evaluator. Reconstruction, by contrast, grew steadily worse: deeper representations lost not only fine image details but eventually the information distinguishing the original class. Depth therefore redistributes complementary capabilities rather than improving everything at once, and no single layer is optimal for every purpose.

The authors are careful to bound their claims. The datasets are regular and low-resolution, all networks share one architecture and learning rule, and the observed convergence does not establish an architecture-independent universal code. The results are consistent, they write, with a local form of the Platonic Representation Hypothesis, the idea that sufficiently constrained learning systems may repeatedly recover privileged representations of underlying structure, but they do not demonstrate one universal representation of the visual world. The apparent freedom of a neural system, the study suggests, can greatly exceed the freedom of the representations it is likely to learn: the vast space of possible internal codes collapses onto a much smaller family of viable ones, and unsupervised depth follows a reproducible path through that space in which class geometry, accessibility, compression, reconstruction, and generation remain distinct but interacting properties of the same learned hierarchy.

Subject of Research: Unsupervised class organization, dimensional collapse, and representational convergence in deep belief networks

Article Title: Convergent evolution in neural representation space: emergent order in deep belief networks

Article References: Krauss, P., Schilling, A., Maier, A. K., Kinfe, T., & Metzner, C. (2026). Convergent evolution in neural representation space: emergent order in deep belief networks. Frontiers in Complex Systems, 4, Article 1960494. https://doi.org/10.3389/fcpxs.2026.1960494

Image Credits: AI Generated

DOI: 10.3389/fcpxs.2026.1960494

Keywords: deep belief networks, restricted Boltzmann machines, unsupervised learning, representation learning, class separation, effective dimensionality, convergent representations, centered kernel alignment, contrastive divergence, generative models, MNIST, emergent order