The Hidden Ingredient Behind Deep Clustering: How Prior Knowledge Drives Machines That Sort Data Without Labels

One of the quiet revolutions in modern machine learning is the ability of neural networks to sort raw data into meaningful groups without ever being told what those groups are. This task, known as deep clustering, powers everything from anomaly detection in sensor networks to person re-identification in surveillance systems and community detection in ecological networks. Yet according to a comprehensive survey published in the open-access journal Vicinagearth by researchers at Sichuan University, the field has been telling itself an incomplete story. While most reviews attribute progress to cleverer network architectures, training strategies, or loss functions, the authors argue that the true engine of deep clustering is something far more fundamental: prior knowledge, the assumptions a method smuggles in about what the data should look like.

The core problem is deceptively simple. In supervised learning, a model learns from labeled examples, so the supervision signal is handed to it directly. In clustering, no labels exist. The network must simultaneously learn discriminative features and assign instances to clusters, with each task supposed to help the other. Without any ground truth, the algorithm needs some source of guidance to construct its own supervision. That guidance, the survey contends, is always a prior: an assumption, whether explicit or implicit, that constrains the space of acceptable solutions. From the earliest deep clustering methods to today’s state-of-the-art systems, the history of the field is really the history of priors evolving.

The authors organize the landscape into six categories of prior knowledge. The first and oldest is the structure prior, inherited directly from classical clustering algorithms such as K-means, DBSCAN, spectral clustering, and agglomerative clustering. K-means assumes instances form spherical structures around centroids; DBSCAN assumes clusters are contiguous high-density regions; spectral methods assume data lie on a locally linear manifold whose neighborhood relations should be preserved. Early deep clustering methods such as the Deep Embedding Network, SpectralNet, and PARTY simply transplanted these mature assumptions into neural network objectives, computing graph Laplacians or enforcing self-representation properties in the latent space learned by autoencoders. These methods were interpretable and theoretically grounded, and they already outperformed classic K-means on raw features thanks to the neural network’s superior feature extraction.

The second category, the distribution prior, assumes that instances from different semantic classes follow distinct probability distributions. This assumption gave rise to generative deep clustering, built on variational autoencoders and generative adversarial networks. The landmark method VaDE fit a Gaussian mixture model in the latent space, sampling a cluster distribution, then a latent vector conditioned on that cluster, and finally reconstructing the input image, with all components jointly optimized by maximizing a variational evidence lower bound. ClusterGAN later replaced explicit Gaussian components with an adversarially learned latent space, introducing a discrete one-hot variable alongside the continuous noise vector to capture cluster identity, and penalizing deviations between encoded and sampled variables to keep clusters distinct. The survey notes that Gaussian components proved redundant and could blur discriminability, which motivated this shift toward implicit distribution learning.

The third and arguably most transformative prior is augmentation invariance, the idea that different transformed views of the same image, such as crops, color distortions, and rotations, preserve its semantic content. Rather than mining structure already present in the data, researchers began constructing new supervisory signals by augmentation. Mutual-information-based methods like IMSAT and IIC maximize the information shared between the cluster assignments of an image and its augmented counterpart, while regularized information maximization pushes assignments to be both unambiguous, by minimizing conditional entropy, and balanced across clusters, by maximizing marginal entropy. Contrastive methods take a related route, pulling positive pairs together in the representation space while pushing negative pairs apart, with theoretical work showing that instance-level contrastive learning is equivalent to maximizing mutual information. Methods such as PICA, Contrastive Clustering, and DRC extended this logic to the cluster level, treating entire cluster assignment vectors as instances to be contrasted, and Twin Contrastive Learning fused instance and cluster representations into a unified embedding.

The fourth prior, neighborhood consistency, exploits a striking empirical observation: features learned through self-supervised pretext tasks map semantically similar instances to nearby points in latent space. SCAN capitalized on this by training a cluster head to make consistent predictions for each instance and its k-nearest neighbors, with an entropy term preventing collapse onto a single cluster. NNM and GCC went further, folding neighborhood information directly into contrastive objectives. GCC in particular builds a normalized symmetric graph Laplacian from a k-nearest-neighbor graph and uses it to reweight the contrastive loss, attracting neighbors rather than only augmented views of the same image. This elegantly mitigates the false-negative problem, the situation where two instances of the same class are wrongly treated as negatives and pushed apart, a known failure mode of vanilla contrastive learning.

The fifth category, pseudo-labeling, rests on a self-referential assumption: predictions in which the model is highly confident are probably correct, and can therefore serve as surrogate labels. DEC, a pioneering method, computed soft assignments with a Student’s t-distribution over distances to learnable centroids, then sharpened those assignments by squaring the probabilities and training the network to match the sharpened targets via KL divergence. DeepCluster iterated between K-means on learned features and supervised training on the resulting pseudo-labels, though its performance was limited by weak initial representations. ProPos later showed that running the same expectation-maximization loop on features from a state-of-the-art self-supervised paradigm like BYOL dramatically improves results, demonstrating that pseudo-label quality is only as good as the semantics of the representation behind it. Newer methods such as TCL and SPICE refine the selection of confident samples, using cluster-wise top-K selection or prototype-based re-assignment to keep pseudo-labels balanced and accurate, and borrow semi-supervised techniques like FixMatch to squeeze more value from them.

The sixth and most recent prior breaks with the others entirely: instead of extracting knowledge from the data itself, it imports external knowledge, most notably textual semantics. SIC builds a semantic space from meaningful texts resembling category names, generates image pseudo-labels by matching image embeddings to text centers in a CLIP-pretrained space, and then trains the cluster head with cross-entropy plus neighborhood consistency. TAC retrieves a text counterpart for each image among representative nouns, improving K-means without extra training, and introduces a mutual distillation paradigm in which image and text modalities teach each other through cluster-level contrastive losses applied to each modality’s assignments and their cross-modal nearest neighbors. The survey’s benchmark experiments confirm the payoff: external-knowledge methods achieve state-of-the-art results across five widely used image benchmarks, including CIFAR-10, CIFAR-100, STL-10, ImageNet-10, and the fine-grained ImageNet-Dogs.

Stepping back, the authors identify two grand trends in how priors have evolved. The first is a shift from mining to constructing: early methods passively extracted assumptions already latent in the data, such as manifold structure or density, whereas augmentation-based methods actively manufacture supervision by transforming inputs. The second is a shift from internal to external: the field is moving from priors derived within the dataset toward knowledge imported from open-world resources, including pretrained vision-language models. The performance gains from different priors also turn out to be largely independent and composable, which explains why methods combining augmentation invariance, neighborhood consistency, and pseudo-labeling, such as ProPos, outperform approaches relying on any single prior.

The survey also maps the road ahead. Fine-grained clustering, such as distinguishing biological subspecies that differ only in subtle markings, defeats coarse priors because color and shape augmentations may erase exactly the features that matter. Non-parametric clustering, where the number of clusters is unknown, remains computationally expensive, though DeepDPM’s Dirichlet Process mixture framework with Metropolis-Hastings-guided split-and-merge operations offers a promising template. Fair clustering must confront biases in sensitive attributes like gender and race that can distort partitions in high-stakes domains such as healthcare and employment, with recent information-theoretic metrics beginning to quantify both quality and fairness. Multi-view clustering, which fuses complementary and consistent information from different sensors or modalities, continues to expand. The authors close with a provocative suggestion: as large pre-trained models like ChatGPT and GPT-4V mature, the next generation of clustering systems may draw supervision from external knowledge sources far richer than anything the data alone can provide, completing the field’s journey from mining priors to importing them.

Subject of Research: Prior knowledge in deep clustering methods for unsupervised data grouping

Article Title: A survey on deep clustering: from the prior perspective

Article References: Lu, Y., Li, H., Li, Y., Lin, Y., & Peng, X. (2024). A survey on deep clustering: from the prior perspective. Vicinagearth, 1(1), Article 4. https://doi.org/10.1007/s44336-024-00001-w

Image Credits: AI Generated

DOI: 10.1007/s44336-024-00001-w

Keywords: deep clustering, unsupervised learning, prior knowledge, contrastive learning, pseudo-labeling, neural networks, data augmentation, representation learning, generative models, external knowledge, machine learning survey, clustering benchmarks