Neural Networks Trained on the Same Task Cluster in Matching Weight Structures

When two artificial neural networks learn the same problem from scratch, do they end up with the same internal wiring? The question is harder to ask than it sounds, because the coordinates of a network are treacherous. Hidden neurons can be shuffled freely, and in networks with rectified linear units their incoming and outgoing weights can be rescaled without changing a single output. Two networks implementing nearly identical solutions can therefore appear far apart in a naive weight comparison. Now a team at Friedrich-Alexander-Universität Erlangen-Nürnberg and partner institutions in Mannheim, Erlangen and Ludwigshafen has removed those disguises and found something striking: networks trained independently on the same dataset fall into reproducible, task-dependent clusters in weight space, and the clustering begins within the first handful of training updates. The study, published in Frontiers in Cognition, turns a philosophical puzzle about learning into a measurable phenomenon.

The experiment was deliberately simple and exhaustively controlled. The researchers trained thirty shallow multilayer perceptrons, ten each on MNIST handwritten digits, Fashion-MNIST clothing images and KMNIST cursive Japanese characters. Each network had 784 inputs, one hidden layer of 64 ReLU neurons and 10 outputs, trained for ten epochs with plain stochastic gradient descent. Performance was reliable and tightly grouped: final test accuracy averaged 97.37 percent for MNIST, 86.37 percent for Fashion-MNIST and 85.99 percent for KMNIST, with run-to-run variation of less than one percentage point. But similar accuracy alone proves nothing about internal structure, since many different mechanisms can support the same input-output mapping. The real question was whether the thirty networks, despite different random initializations and different shuffling histories, had written their solutions in related coordinates once the arbitrary neuron numbering was taken into account.

To make the weights comparable, the team aligned hidden neurons for every one of the 435 unordered network pairs at every checkpoint, using two independent matching procedures: an established method called Git Re-Basin, which searches for the permutation that maximizes agreement between complete weight tensors, and a homegrown Hungarian-assignment procedure based on permutation-invariant neuron features. They also eliminated the ReLU rescaling freedom by placing every hidden neuron in a canonical gauge in which its incoming and outgoing weight norms are equal. Their primary dissimilarity measure, called Alpha, is an importance-weighted sign-disagreement score: it asks whether corresponding parameters disagree in sign, and whether those disagreements are concentrated among weights that are large in both networks. At initialization, the measure found no trace of dataset grouping, with a between-minus-within contrast of just 0.0003, indistinguishable from zero. The random starting points contained no pre-existing organization.

Training changed that relation almost immediately and permanently. After only half an epoch, networks trained on the same dataset had become far more similar to each other than to networks trained on different datasets, with a contrast of 0.1596. At the final checkpoint, within-dataset dissimilarity was 0.0789 while between-dataset dissimilarity was 0.2379, roughly three times as large, and the contrast of 0.1590 carried a confidence interval bounded well clear of zero. The effect was not carried by a few unusual runs: all thirty networks contributed positively to the separation, and the result held in both weight matrices, across several dissimilarity measures, and after canonicalization. Crucially, the within-dataset distances never collapsed to zero, so the networks did not converge to a single shared weight configuration. They clustered, rather than converged, into task-conditioned regions of parameter space.

The most dramatic demonstration came from a paired control experiment that isolated the learning objective itself. Twenty networks saw exactly the same MNIST images in exactly the same order, and within each of ten seed blocks they began from bitwise identical input-layer weights. The only difference was the target rule: one network in each pair predicted digit identity, the other predicted whether the digit was even or odd. Both tasks were learned superbly, reaching 97.25 and 98.60 percent accuracy. Yet by the end of training, the networks predicting different rules had clearly separated in both their aligned input weights and their hidden representations, with a weight contrast of 0.0445 and a representational contrast of 0.1366. Even the especially stringent same-seed cross-task pairs, which started from identical weights, remained clearly separated. Changing only the prediction rule, while holding the sensory evidence fixed, redirected the emerging internal structure. The dataset effect could not be attributed to the image distribution alone.

The weight-space result also had a counterpart in hidden activity. Using linear centered kernel alignment, which compares activation spaces without requiring any one-to-one neuron match, the researchers found that hidden representations showed the same broad organization: at the final checkpoint, representational dissimilarity was 0.2544 within datasets and 0.5758 between datasets. Across all pairs, the weight measure and the representational measure were strongly correlated, with a Pearson coefficient of 0.9269. But when the within- and between-dataset groups were examined separately, the correlation among between-dataset pairs fell to 0.1071, with a confidence interval that included zero. The two measures agreed strongly about which pairs belonged to the same broad group while preserving different fine-grained information within groups. That distinction matters: it shows the robust finding is the group separation itself, whereas statements about exactly how far apart two networks are remain measure-dependent.

Perhaps the most eye-catching result concerned timing. In a densely sampled experiment that recorded checkpoints after individual optimizer updates, mean test accuracy across the datasets rocketed from 0.1043 at initialization to 0.4146 after only two SGD updates, with MNIST alone reaching 0.5105. At that same moment, the relative displacement of the weights from their initialization was a mere 0.0815 in the input layer and 0.0588 in the output layer, corresponding to root-mean-square changes per parameter of roughly one to four thousandths. Substantial functional learning had occurred through a tiny, coordinated nudge. And that nudge was already task-specific: representational separation between datasets was detectable after two updates, and canonical weight-space separation after four. The data explicitly ruled out a calibration-then-consolidation sequence in which networks first acquire most of their performance and only later settle into dataset-specific structure. Functional and structural differentiation developed together, on overlapping timescales.

The alignment comparison added an honest methodological footnote. Git Re-Basin produced the strongest neuron correspondence by every criterion, including matched-neuron activation correlations that it never directly optimized: pooled activation correspondence rose from 0.0502 in the arbitrary numbering to 0.1314 after the Hungarian procedure and 0.2734 after Git Re-Basin. The two methods agreed on only 16.5 percent of hidden neurons on average, showing they found substantially different correspondences. Still, the independently developed Hungarian procedure recovered genuine functional structure, improving every one of the 135 within-dataset pairs relative to the arbitrary original numbering. The dataset-dependent ordering survived the change of alignment method, which strengthens the substantive conclusion while limiting any claim for the homegrown procedure.

The authors frame their findings with two hypotheses and careful boundaries. The target-rich landscape hypothesis proposes that hidden-neuron permutations and ReLU rescalings generate many equivalent realizations of useful solutions, making task-compatible solution regions locally accessible through small coordinated updates, which would explain why tiny early nudges suffice for rapid learning. The results also make an attractor perspective scientifically plausible: independent trajectories repeatedly entered task-conditioned regions despite different starting points, and changing the target rule redirected that organization. But the researchers are explicit that endpoint clustering alone does not demonstrate attractor dynamics, which would require showing that perturbed networks return to the same equivalence class and mapping the basin of attraction. The study also does not show that networks trained on the same task implement an identical algorithm, and its scope is limited to small, shallow networks on low-resolution grayscale classification tasks under a single optimizer configuration.

What the study does establish is that shared inputs and objectives leave a reproducible signature in both parameters and representations, restricting the internal solutions available to a learning system without forcing them into one weight configuration. The authors draw an analogy to convergent evolution in biology, where different histories can produce similar forms under related constraints, while stressing that gradient-based training is not an evolutionary process and that the analogy motivates questions rather than answers them. The decisive next experiments, they propose, are perturbation and restart tests in symmetry-adjusted spaces: perturbing trained networks and retraining under the same objective, restarting from different nearby states, and branching trajectories while changing only the target rule. Such tests would turn the attractor image into a falsifiable dynamical account of how learning constraints shape internal organization, connecting the careful comparison of artificial networks to a broader question about how strongly shared computational constraints restrict the solutions a learning system can find.

Subject of Research: Permutation-aligned comparison of independently trained neural networks revealing task-dependent clustering in weight space

Article Title: Task-dependent clustering in permutation-aligned neural network weights

Article References: Krauss, P., Schilling, A., Maier, A., Kinfe, T., Stübner, H., Römmelt, N., & Metzner, C. (2026). Task-dependent clustering in permutation-aligned neural network weights. Frontiers in Cognition, 5, Article 1960468. https://doi.org/10.3389/fcogn.2026.1960468

Image Credits: AI Generated

DOI: 10.3389/fcogn.2026.1960468

Keywords: neural networks, machine learning, weight space, permutation alignment, training dynamics, representational similarity, computational neuroscience, task-dependent clustering, Git Re-Basin, stochastic gradient descent, attractor hypothesis, MNIST