New Tunable Divergence Tool Weighs How Heavy-Tailed Systems Drift Apart

Complex systems, from the vocabulary of a novel to the species composition of a forest, tend to be dominated by a small number of abundant types while vast numbers of types hover at the margins. Comparing two such systems, or one system with itself across time, is a deceptively hard problem: the familiar statistical distances that work well for well-behaved distributions often fail to capture the dramatic swings that occur deep in the heavy tail, where rare types live. A new study published in PLOS Complex Systems introduces a candidate solution. Researchers led by Peter Sheridan Dodds, Joshua R. Minot, Michael V. Arnold, Thayer Alshaabi, Jane Lydia Adams, Andrew J. Reagan, and Christopher M. Danforth present probability-turbulence divergence, a tunable and interpretable instrument for measuring how two type-probability distributions diverge, and they show that it either explicitly or functionally generalizes a surprising range of existing measures.

The work sits within a research program the authors call allotaxonometry, the quantitative comparison of the relative abundances of types across systems. The word evokes titration: just as a chemist measures how much of one substance is present in a mixture, an allotaxonometric instrument measures how much of each type, be it a word, a hashtag, or a species, is present in each of two systems. When those systems are complex, their type-probability distributions are almost always heavy-tailed, meaning a handful of types command enormous probabilities while thousands of others carry vanishingly small ones. Any comparison tool must therefore be judged not only by how it treats the dominant types at the head of the distribution but also by how sensitively it registers changes among the many rare types in the tail.

Probability-turbulence divergence, abbreviated PTD, is modeled directly on the authors’ earlier construction, rank-turbulence divergence, or RTD. RTD compares two systems by ranking their types by abundance and asking how far a type’s rank shifts between the two lists. That rank-based approach has proved remarkably robust for messy real-world data, but it has an inherent limitation: rank discards information about the actual probabilities. Two types separated by a factor of a hundred in probability might sit only a few rank positions apart if the distribution is steep. PTD restores the probability information. It works directly with the type probabilities themselves, which makes it more limited in application, since it requires normalizable probability distributions that can be measured well, but correspondingly more sensitive to genuine changes in type probability.

The instrument is built around a single tunable parameter, and the authors demonstrate with careful mathematical analysis that PTD either explicitly or functionally generalizes many classical distances and measures as special cases. Among these are the L(q) norms, the workhorse distance measures of mathematics; the Sørensen-Dice coefficient, known to machine-learning practitioners as the F1 statistic; and the Hellinger distance, a staple of probability theory and statistics. This unifying structure matters practically as well as conceptually. It means a researcher who already trusts one of these familiar measures can understand it as one point on a continuous dial, and can then turn the dial to explore how the comparison changes as the instrument becomes more or less sensitive to particular parts of the distribution.

The connections run deeper still. The authors discuss the similarities between PTD and the generalized entropies of Rényi and Tsallis, families of entropy measures that interpolate between different ways of weighting rare and common events, and the diversity indices, or Hill numbers, that ecologists have long used to quantify the diversity of a community. In ecology, the choice of diversity order determines whether a measure emphasizes species richness, the sheer count of types, or dominance, the sway of the most abundant ones. PTD inherits the same tension, and its tuning parameter plays an analogous role, letting the analyst decide which slice of the abundance spectrum should drive the measured divergence between two systems.

A persistent practical obstacle in comparing heavy-tailed distributions is the problem of exclusive types: types that appear in one system but have probability zero in the other. A word that occurs in one novel but not another, or a species present in one habitat but absent from another, cannot be handled by formulas that assume both probabilities are positive. The authors confront this head-on. They build allotaxonographs, visual displays of probability turbulence, and incorporate a way to visually accommodate zero probabilities for exclusive types, so that the most dramatic kind of difference, outright presence versus absence, is rendered legibly rather than silently discarded or forced into an arbitrary numerical convention.

Perhaps the most intuitive of the paper’s demonstrations is the flipbook. Because PTD has a single parameter, the authors can sweep that parameter across its range and show, frame by frame, how the measured divergence between two systems reorganizes. At one setting, the comparison is dominated by shifts among the most common types; at another, it is dominated by the rare types; and in between, it blends contributions across all scales. The flipbook thus gives users a direct, visual answer to a question that a single scalar divergence cannot: how do two systems diverge for types that are rare, common, and at all scales in between? Rather than committing to one weighting in advance, the analyst can watch the comparison evolve and choose the setting that answers the scientific question at hand.

One finding deserves particular attention from anyone tempted to seek a universally best setting. The authors show that PTD can be tuned to a scale-equalizing view, one in which contributions from different parts of the distribution are balanced, but that this equalizing setting is non-universal: it depends on the particular systems being compared. There is no single parameter value that will balance head and tail contributions for every pair of distributions. This is a sobering but honest result, and it reframes the tuning parameter not as a nuisance but as part of the scientific modeling itself, a choice that must be made with reference to the data and the question rather than fixed once and for all.

To ground the mathematics in reality, the authors explore comparisons of example distributions drawn from three very different domains: literature, social media, and ecology. These demonstrations show the instrument in action on the kinds of heavy-tailed data that motivate it, where a few words or species or hashtags dominate and the tail stretches out over thousands of types. The examples illustrate how different tunings reveal different stories about the same pair of systems, and how the accompanying allotaxonographs make those stories visible in a way that a bare divergence value cannot.

The paper closes by turning to open problems, chief among them the optimization of the tuning of both rank- and probability-turbulence divergence. If the parameter must be chosen with the systems in view, can that choice itself be automated, guided by the data or by the scientific goal? The authors frame this as a central challenge for future work. For now, probability-turbulence divergence offers researchers a rare combination in the comparison of complex systems: mathematical rigor that subsumes familiar measures, sensitivity to the probabilities that rank-based methods discard, and a visual, tunable interface that makes the heavy tail, where so much of the action in complex systems resides, finally open to systematic inspection.

Subject of Research: A tunable divergence measure for comparing heavy-tailed type-probability distributions in complex systems

Article Title: Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed type-probability distributions

Article References: Dodds, P. S., Minot, J. R., Arnold, M. V., Alshaabi, T., Adams, J. L., Reagan, A. J., & Danforth, C. M. (2026). Probability-turbulence divergence: A tunable allotaxonometric instrument for comparing heavy-tailed type-probability distributions. PLOS Complex Systems, 3(7), e0000077. https://doi.org/10.1371/journal.pcsy.0000077

Image Credits: AI Generated

DOI: 10.1371/journal.pcsy.0000077

Keywords: probability-turbulence divergence, allotaxonometry, heavy-tailed distributions, rank-turbulence divergence, complex systems, Hellinger distance, Sørensen-Dice coefficient, Hill numbers, generalized entropy, ecology, social media, computational linguistics