The work, published in the journal Knowledge and Information Systems, tackles a setting the authors call a Dynamic and Unknown Domain, or DUD. It is dynamic because the schemas of incoming messages, meaning the set of attributes each message carries, evolve constantly in number, similarity, and distribution. It is unknown because there is no fixed dictionary of attributes; new fields can appear at any moment with names nobody has seen before. In such an environment, the classical assumption underlying most clustering algorithms, that data points live in a fixed-dimensional space, simply breaks down.

The goal of schema profiling is to give practitioners a summarized view of the message structures within a sliding window of the stream. That view is built by clustering similar schemas together and representing each group by its centroid, in the spirit of the k-means algorithm. The resulting profile helps analysts understand which kinds of messages dominate the traffic, and it supports downstream tasks such as formulating queries over schemaless data. But recomputing a full clustering from scratch every time the window slides is both computationally prohibitive and analytically jarring, since profiles computed independently in successive windows would be hard to compare over time.

DSC+ addresses these constraints with a two-phase design executed at every slide of the window. In the first phase, the schemas of incoming messages are pre-aggregated into compact data structures called schema features, or SFs. Rather than storing every individual schema, an SF records, for each attribute, the empirical probability that it appears within the group of schemas it summarizes. These structures are additive and subtractive, which means they can be merged, split, and trimmed as old messages expire from the window and new ones arrive, all without ever reconstructing the original messages.

To decide which schemas should be grouped together in the first place, the algorithm borrows a trick from document comparison: Min-Hash. Each schema is passed through a set of hash functions that produce a short identifier, and the mathematics of the technique guarantees that two schemas with high Jaccard similarity, meaning they share many attributes, are likely to receive the same identifier. The researchers adapted the method so that the hash indexes remain stable even as the attribute domain grows without bound, a property the original Min-Hash formulation lacks. Schemas sharing an identifier are folded into a single schema feature, producing a pane-level coreset, and these are in turn combined into a fixed-size window-level coreset that feeds the clustering stage.

The second phase is where DSC+ departs most sharply from prior work. Instead of rerunning k-means on each window, the algorithm incrementally updates the previous clustering result through a set of formalized rules that capture five distinct evolution phenomena. Sliding occurs when the schemas within a cluster gradually change, shifting the centroid. Fading-in happens when a new source starts emitting recognizably different schemas, and fading-out when an existing source goes silent. Splitting arises when one cluster grows so internally diverse that it should become two, and merging when two clusters drift so close together that they should become one.

Each rule is triggered by statistical signals rather than guesswork. The algorithm monitors the scattering of each cluster, the average distance between its members and its centroid, and the separation between pairs of clusters, the distance between their centroids. When a robust z-score, computed with the median absolute deviation, flags a cluster as an outlier in scattering, a split is executed by locally running k-means with two centers. When a pair of clusters overlaps enough and their separation becomes anomalously low, they are merged. New schema features are simply assigned to the nearest centroid, and empty clusters are deleted. If the quality of the updated profile, measured by the Simplified Silhouette, drops by more than what the worst possible sudden change in the data could plausibly explain, a full reclustering is triggered as a safety net.

The experimental evaluation, run on both synthetic datasets engineered to simulate each evolution phenomenon and on four real-world datasets, including application logs from a consulting company, sensor data from a precision agriculture project, and network traffic logs from the Suricata monitoring tool, shows the approach outperforming the closest state-of-the-art competitors. Against the baseline of rerunning the full clustering algorithm, OMRk++, on every window, DSC+ produced profiles that were actually slightly better in terms of silhouette, an effect the authors attribute to the pre-aggregation step filtering out noise from individual schemas. Compared with the streaming competitors CSCS and FEAC-S, DSC+ tracked changes in the number of clusters far more faithfully while executing faster in most configurations.

The efficiency gains stem from the fact that the expensive clustering operates on a coreset capped at a fixed maximum size, so execution time scales sublinearly with the volume of incoming data. In tests where the window length grew from ten thousand to one million schemas, DSC+ sustained increasing stream rates while a competitor’s throughput actually declined, because that competitor struggled to compress large windows into its fixed-size summary. The researchers also tuned the key parameters, including the coreset size and the number of hash functions, showing that accuracy plateaus quickly as the coreset grows, which means substantial data reduction comes at almost no cost in profile quality.

The authors are candid about limitations. The current implementation runs on a single server, and a distributed version exploiting the additivity of schema features is left for future work. The system also lacks explicit mechanisms for handling outlier schemas that resemble nothing seen before, although such cases did not arise in the real datasets tested. Still, the contribution marks a clear step forward: a clustering method that embraces a variable and unknown attribute domain, tracks the birth, death, splitting, and merging of clusters in real time, and does so within a bounded memory footprint. For the growing ranks of organizations drowning in heterogeneous IoT and log data, DSC+ offers a way to finally see, in real time, what their streams are actually made of.

Subject of Research: Real-time schema profiling of heterogeneous data streams using dynamic stream clustering with variable-k k-means

Article Title: Dynamic stream clustering for real-time schema profiling with DSC+

Article References: Dynamic stream clustering for real-time schema profiling with DSC+. (n.d.). https://doi.org/10.1007/s10115-026-02889-w

Image Credits: AI Generated

DOI: 10.1007/s10115-026-02889-w

Keywords: schema profiling, data streams, stream clustering, k-means, coresets, Min-Hash, IoT, big data, sliding window, cluster evolution, schemaless data, real-time analytics

The intellectual foundation of ParTIpy is Pareto Task Inference, or ParTI, a framework grounded in the theory of multi-objective optimality. The core idea is elegant: when evolution drives phenotypes toward Pareto optimality, any improvement in performance on one task necessarily degrades performance on another. Under this assumption, phenotypes such as organisms, proteins, or cells, described by traits that embody trade-offs, are predicted to occupy a simple geometric structure called a polytope, a generalized shape with corners, straight edges, and flat faces, such as a triangle or tetrahedron extended to any number of dimensions. Crucially, the number of vertices of this polytope equals the number of tasks the system is optimizing. Each vertex, known as an archetype, represents a phenotype perfectly specialized for one particular task.

To recover these archetypes from data, the framework relies on archetypal analysis, a dimensionality reduction technique first introduced by Cutler and Breiman in 1994. The method identifies extreme points within a dataset, the archetypes, and then models every observation as a nonnegative linear combination of them. Formally, the archetypes are defined as convex combinations of the data points, ensuring they lie within the convex hull of the observed distribution, and each data point is reconstructed as a convex combination of the archetypes. When applied to single-cell transcriptomics, this means fitting a polytope around cells in gene expression space, where each vertex corresponds to a transcriptional program optimized for a distinct biological task. Positioning a cell relative to the vertices then reveals its bias toward one or several functions.

Previous studies have demonstrated the value of this perspective, revealing trade-offs in hepatocytes, fibroblasts, and cancer cells, and even extending the approach to spatial transcriptomics and proteomics, where groups of co-localized cells cooperate on shared tasks. However, the original ParTI software, written in MATLAB, could not scale to the massive datasets generated by modern high-throughput single-cell technologies, and its dependence on a commercial license hindered broader adoption. ParTIpy addresses both limitations directly, implementing state-of-the-art initialization and optimization strategies in Python and integrating seamlessly with the scverse ecosystem of single-cell analysis tools, including scanpy and AnnData objects.

A central technical challenge in archetypal analysis is that the underlying optimization problem is non-convex, meaning solution quality is highly sensitive to how the algorithm is initialized. The team benchmarked three initialization algorithms and two optimization procedures across twenty-one single-cell transcriptome datasets, each representing a single cell type and drawn from three independent studies spanning multiple sclerosis lesions, spatial Xenium data, and lupus patient blood. They found that combining principal convex hull analysis, or PCHA, for optimization with the Archetypal++, or AA++, initialization scheme delivered the best trade-off between reconstruction accuracy and runtime. When speed is the priority, a more efficient Frank-Wolfe variant provides a practical alternative. The package also supports a convexity relaxation parameter that allows archetypes to extend beyond the observed data cloud, a crucial option when sampling near the boundaries of the true phenotypic space is sparse.

The most striking computational innovation, however, is ParTIpy’s use of coresets. A coreset is a small, cleverly weighted subset of the data such that optimizing on the subset yields results close to those obtained from the full dataset. The researchers incorporated coreset weights directly into the objective function and modified their optimization algorithms accordingly. Their benchmarks showed that the required coreset fraction shrinks as datasets grow: for datasets exceeding one hundred thousand cells, analyzing just one to ten percent of the data was sufficient on average, cutting runtime roughly fourfold. When fitting six archetypes on one million cells using a ten percent coreset, the optimization completed in forty-five to one hundred five seconds. Critically, archetype-specific gene expression profiles derived from coresets remained highly concordant with full-data results, with a median Pearson correlation above 0.9 across coreset fractions greater than one percent.

Because the correct number of archetypes is never known in advance, ParTIpy implements three complementary selection criteria. The first examines the fraction of variance explained as archetypes are added, looking for an elbow where returns diminish. The second employs an information-theoretic criterion that penalizes model complexity, favoring solutions that balance goodness of fit against the number of free parameters. The third assesses robustness through bootstrapping: archetypal analysis is repeated on multiple resampled datasets, archetype positions are aligned across replicates using Hungarian matching, and the positional variance of each archetype is computed. Low variance signals stable, well-defined archetypes, whereas high variance warns of over-parameterization. The team validated that ParTIpy’s results are highly concordant with the original MATLAB implementation across three distinct single-cell datasets.

To demonstrate the framework in action, the researchers applied ParTIpy to single-cell RNA sequencing data of hepatocytes sampled along the liver lobule’s porto-central axis. Three selection criteria converged on a four-archetype model, and pathway enrichment analysis using Reactome gene sets via the decoupler-py univariate linear model revealed distinct specializations: one archetype was associated with drug and xenobiotic metabolism, another with lipid metabolism. When the archetypal programs were projected onto spatial transcriptomics data, the metabolism archetype localized near the central vein while the lipid archetype enriched near the portal node, matching established liver zonation patterns and suggesting that spatial gradients of oxygen and metabolites drive this functional division of labor. The package further supports inference of archetype crosstalk networks by integrating archetype-specific expression profiles with curated ligand-receptor databases.

The second demonstration tackled a longstanding weakness of conventional cross-condition single-cell analysis. Standard approaches discretize cells into clusters and test for changes in cluster abundance, imposing sharp boundaries on what are often intrinsically continuous cell states. ParTIpy instead models cells as continuous mixtures of archetypal programs. Analyzing roughly 147,000 fibroblasts from sixteen non-failing and twenty-six cardiomyopathy donor hearts, the team fit a three-archetype model chosen for superior bootstrap stability. Donor-level pseudobulk profiles revealed significant shifts toward an activated fibrotic archetype in disease, marked by genes such as POSTN, FN1, and COL1A1 and by transcription factors including TWIST1 and SCX, while a quiescent, stress-responsive archetype was closer to healthy donors. Notably, even every non-failing heart contained a small fibroblast population near the fibrotic archetype, indicating that cardiomyopathy redistributes cells among existing programs rather than creating disease-specific states.

The researchers emphasize that ParTIpy’s scalability, downstream utilities, and interoperability make Pareto task inference accessible to the broader single-cell community and beyond, with tutorials covering spatial transcriptomics applications. Unlike deep archetypal analysis approaches that rely on encoder-decoder networks and amortized variational inference, which can introduce an amortization gap, ParTIpy achieves scale through coreset-based direct optimization while preserving archetypes as extremal points in the original trait space. Future integration with RNA velocity, pseudotime, and optimal transport methods could distinguish stable task specializations from transient states. The package, version 0.2.0, is freely available on PyPI and GitHub, with comprehensive documentation at partipy.readthedocs.io, positioning ParTIpy as a versatile new lens for exploring functional trade-offs in high-dimensional biological data.

Subject of Research: Scalable archetypal analysis and Pareto task inference for single-cell transcriptomic data

Article Title: ParTIpy: a scalable framework for archetypal analysis and Pareto task inference

Article References: ParTIpy: a scalable framework for archetypal analysis and Pareto task inference. (n.d.). https://doi.org/10.1038/s44320-026-00209-6

Image Credits: AI Generated

DOI: 10.1038/s44320-026-00209-6

Keywords: ParTIpy, Pareto task inference, archetypal analysis, single-cell transcriptomics, coresets, trade-offs, cell state variability, hepatocyte zonation, cardiac fibroblasts, scverse, polytope geometry, open-source software