The work sits within a research program the authors call allotaxonometry, the quantitative comparison of the relative abundances of types across systems. The word evokes titration: just as a chemist measures how much of one substance is present in a mixture, an allotaxonometric instrument measures how much of each type, be it a word, a hashtag, or a species, is present in each of two systems. When those systems are complex, their type-probability distributions are almost always heavy-tailed, meaning a handful of types command enormous probabilities while thousands of others carry vanishingly small ones. Any comparison tool must therefore be judged not only by how it treats the dominant types at the head of the distribution but also by how sensitively it registers changes among the many rare types in the tail.

Probability-turbulence divergence, abbreviated PTD, is modeled directly on the authors’ earlier construction, rank-turbulence divergence, or RTD. RTD compares two systems by ranking their types by abundance and asking how far a type’s rank shifts between the two lists. That rank-based approach has proved remarkably robust for messy real-world data, but it has an inherent limitation: rank discards information about the actual probabilities. Two types separated by a factor of a hundred in probability might sit only a few rank positions apart if the distribution is steep. PTD restores the probability information. It works directly with the type probabilities themselves, which makes it more limited in application, since it requires normalizable probability distributions that can be measured well, but correspondingly more sensitive to genuine changes in type probability.

The instrument is built around a single tunable parameter, and the authors demonstrate with careful mathematical analysis that PTD either explicitly or functionally generalizes many classical distances and measures as special cases. Among these are the L(q) norms, the workhorse distance measures of mathematics; the Sørensen-Dice coefficient, known to machine-learning practitioners as the F1 statistic; and the Hellinger distance, a staple of probability theory and statistics. This unifying structure matters practically as well as conceptually. It means a researcher who already trusts one of these familiar measures can understand it as one point on a continuous dial, and can then turn the dial to explore how the comparison changes as the instrument becomes more or less sensitive to particular parts of the distribution.

The connections run deeper still. The authors discuss the similarities between PTD and the generalized entropies of Rényi and Tsallis, families of entropy measures that interpolate between different ways of weighting rare and common events, and the diversity indices, or Hill numbers, that ecologists have long used to quantify the diversity of a community. In ecology, the choice of diversity order determines whether a measure emphasizes species richness, the sheer count of types, or dominance, the sway of the most abundant ones. PTD inherits the same tension, and its tuning parameter plays an analogous role, letting the analyst decide which slice of the abundance spectrum should drive the measured divergence between two systems.

A persistent practical obstacle in comparing heavy-tailed distributions is the problem of exclusive types: types that appear in one system but have probability zero in the other. A word that occurs in one novel but not another, or a species present in one habitat but absent from another, cannot be handled by formulas that assume both probabilities are positive. The authors confront this head-on. They build allotaxonographs, visual displays of probability turbulence, and incorporate a way to visually accommodate zero probabilities for exclusive types, so that the most dramatic kind of difference, outright presence versus absence, is rendered legibly rather than silently discarded or forced into an arbitrary numerical convention.

Perhaps the most intuitive of the paper’s demonstrations is the flipbook. Because PTD has a single parameter, the authors can sweep that parameter across its range and show, frame by frame, how the measured divergence between two systems reorganizes. At one setting, the comparison is dominated by shifts among the most common types; at another, it is dominated by the rare types; and in between, it blends contributions across all scales. The flipbook thus gives users a direct, visual answer to a question that a single scalar divergence cannot: how do two systems diverge for types that are rare, common, and at all scales in between? Rather than committing to one weighting in advance, the analyst can watch the comparison evolve and choose the setting that answers the scientific question at hand.

One finding deserves particular attention from anyone tempted to seek a universally best setting. The authors show that PTD can be tuned to a scale-equalizing view, one in which contributions from different parts of the distribution are balanced, but that this equalizing setting is non-universal: it depends on the particular systems being compared. There is no single parameter value that will balance head and tail contributions for every pair of distributions. This is a sobering but honest result, and it reframes the tuning parameter not as a nuisance but as part of the scientific modeling itself, a choice that must be made with reference to the data and the question rather than fixed once and for all.

To ground the mathematics in reality, the authors explore comparisons of example distributions drawn from three very different domains: literature, social media, and ecology. These demonstrations show the instrument in action on the kinds of heavy-tailed data that motivate it, where a few words or species or hashtags dominate and the tail stretches out over thousands of types. The examples illustrate how different tunings reveal different stories about the same pair of systems, and how the accompanying allotaxonographs make those stories visible in a way that a bare divergence value cannot.

The paper closes by turning to open problems, chief among them the optimization of the tuning of both rank- and probability-turbulence divergence. If the parameter must be chosen with the systems in view, can that choice itself be automated, guided by the data or by the scientific goal? The authors frame this as a central challenge for future work. For now, probability-turbulence divergence offers researchers a rare combination in the comparison of complex systems: mathematical rigor that subsumes familiar measures, sensitivity to the probabilities that rank-based methods discard, and a visual, tunable interface that makes the heavy tail, where so much of the action in complex systems resides, finally open to systematic inspection.

Subject of Research: A tunable divergence measure for comparing heavy-tailed type-probability distributions in complex systems

Article Title: Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed type-probability distributions

Article References: Dodds, P. S., Minot, J. R., Arnold, M. V., Alshaabi, T., Adams, J. L., Reagan, A. J., & Danforth, C. M. (2026). Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed type-probability distributions. PLOS Complex Systems, 3(7), e0000077. https://doi.org/10.1371/journal.pcsy.0000077

Image Credits: AI Generated

DOI: 10.1371/journal.pcsy.0000077

Keywords: probability-turbulence divergence, allotaxonometry, heavy-tailed distributions, rank-turbulence divergence, complex systems, Hellinger distance, Sørensen-Dice coefficient, Hill numbers, generalized entropy, ecology, social media, computational linguistics

A team led by Megan D. Schertzer and Stella H. Park of the New York Genome Center and Columbia University, together with Jiayu Su, Fairlie Reese, Gloria M. Sheynkman and senior author David A. Knowles, believes the field has been answering that question in the wrong way. In a study published in Genome Biology, the researchers propose replacing the crude filtering steps that dominate current pipelines with a single, principled number borrowed from information theory: perplexity. Their central claim is provocative. Instead of discarding rare isoforms as noise, every transcript a gene produces, no matter how scarce, should contribute proportionally to that gene’s measured diversity.

To understand why this matters, it helps to look at how isoform analysis currently works. Long-read sequencing experiments inevitably generate some technical artifacts, spurious transcript models that do not correspond to real RNA molecules. Pipelines such as SQANTI3, which the new study uses to classify transcript structures into quality categories, help researchers separate genuine isoforms from these errors. But after artifact removal, most workflows then apply an expression threshold, keeping only isoforms that exceed some minimum abundance, typically measured in transcripts per million. Those thresholds are, as the authors point out, arbitrary. An isoform sitting just below the cutoff is thrown away, even though it may be a bona fide transcript with a real biological function. The result is a systematic undercounting of diversity that varies from study to study depending on where each team happens to set its cutoff, undermining reproducibility across labs and datasets.

Perplexity offers a way out of this arbitrariness. The concept originates in Shannon entropy, the foundational measure of uncertainty in information theory. If you treat the isoforms of a gene as the possible outcomes of a random draw, weighted by their relative abundances, the Shannon entropy quantifies how uncertain you are about which isoform you will draw next. Exponentiating that entropy yields perplexity, which has an elegant and intuitive interpretation: it is the effective number of isoforms a gene produces. A gene that expresses one dominant transcript with only trace alternatives has a perplexity close to one. A gene that spreads its expression evenly across ten isoforms has a perplexity approaching ten. The metric is closely related to the Hill numbers that ecologists have long used to measure biodiversity, a theoretical connection the authors develop in their supplementary materials, drawing on the framework formalized by Tom Leinster.

The beauty of this formulation is that no isoform is ever discarded. A transcript representing one percent of a gene’s output contributes to the perplexity calculation in proportion to its abundance, exactly as its share of the expression distribution dictates. Rare isoforms nudge the effective count upward; abundant ones dominate it. The measurement is continuous, interpretable and independent of any researcher-chosen threshold. In effect, perplexity converts the messy, long-tailed distribution of isoform abundances that long-read sequencers reveal into a single number that means the same thing in every dataset, every tissue and every laboratory.

To demonstrate that the metric delivers on this promise, the team applied it to a substantial trove of public data: 124 PacBio long-read RNA sequencing datasets from the ENCODE4 project, spanning 55 human cell types. This resource, generated by the Encyclopedia of DNA Elements consortium, captures transcriptomes across a remarkable range of biological contexts, from immune cells to neuronal lineages. After processing the data through their custom pipeline, which includes artifact filtering with SQANTI3 and annotation of open reading frames, the authors computed perplexity for every gene in every sample and asked whether the resulting diversity measurements were interpretable and reproducible.

The answer, according to the study, is yes. Perplexity proved to be a stable and meaningful quantity across genes, regulatory levels and tissues. Genes whose isoform repertoires are governed by well-characterized regulatory mechanisms showed diversity profiles consistent with known biology, while the metric distinguished cell types in ways that reflect their underlying splicing and transcription programs. Because the calculation uses all detected isoforms rather than a thresholded subset, repeated measurements of the same biological condition converge far more reliably than threshold-based counts, addressing one of the most persistent complaints about long-read transcriptomics: that different pipelines, and even slightly different parameters within a single pipeline, can yield noticeably different catalogs of isoforms.

The work also extends beyond simply counting transcripts. The authors analyzed their detected isoforms at the level of open reading frames, the stretches of RNA that are actually translated into protein, and their supplementary analyses include quadrant classifications of open reading frames and tissue-specific assignments. This reflects a growing recognition in the field that isoform diversity matters not merely as a cataloging exercise but because different isoforms can carry different protein-coding potential, potentially producing distinct protein products from the same gene. A diversity metric that faithfully captures the full isoform landscape therefore has direct implications for proteomics and for interpreting the functional consequences of genetic variation.

The implications reach into medicine as well. Alternative splicing is increasingly implicated in cancer, neurological disease and genetic disorders, and dysregulated isoform production is a hallmark of many tumors. Gloria Sheynkman, whose laboratory at the University of Virginia focuses on connecting transcriptomics to proteomics, and David Knowles, whose group at Columbia and the New York Genome Center develops computational methods for genome interpretation, have both founded or advised companies in this space, underscoring the translational interest in reliable isoform quantification. A reproducible, threshold-free measure of isoform diversity could improve the statistical power of studies that compare diseased and healthy tissue, where subtle shifts in splicing regimes may be drowned out by the noise introduced by arbitrary filtering.

There is also a satisfying intellectual symmetry in the approach. Ecologists counting species in a rainforest, linguists measuring the richness of a vocabulary and now genomicists quantifying the diversity of a transcriptome are all grappling with the same mathematical structure: a distribution over many categories, where rare categories are real but hard to handle. Shannon entropy and its exponential, perplexity, provide a common language for all of these problems. By importing that language into transcriptomics, Schertzer, Park and their colleagues have turned a long-standing methodological weakness into an opportunity, giving the field a way to embrace the full, messy richness of the human transcriptome rather than trimming it to fit a threshold. As long-read sequencing continues its march toward routine clinical and research use, metrics like perplexity may well become as standard a vocabulary for RNA biology as expression levels are today.

Subject of Research: An information-theoretic metric for quantifying isoform diversity in the human transcriptome

Article Title: Perplexity as a metric for isoform diversity in the human transcriptome

Article References: Schertzer, M. D., Park, S. H., Su, J., Reese, F., Sheynkman, G. M., & Knowles, D. A. (2026). Perplexity as a metric for isoform diversity in the human transcriptome. Genome Biology. https://doi.org/10.1186/s13059-026-04260-2

Image Credits: AI Generated

DOI: 10.1186/s13059-026-04260-2

Keywords: perplexity, isoform diversity, long-read RNA sequencing, Shannon entropy, Hill numbers, transcriptome, ENCODE4, PacBio, alternative splicing, SQANTI3, open reading frames, Genome Biology