Gaussian Mixture Model Enables Adaptive Entropy Thresholds for AI Watermarking

Researchers at the Harbin Institute of Technology have unveiled GAMark, an adaptive watermarking framework designed to embed detectable signals in text generated by large language models without compromising the quality or functionality of the output. The work, published in the journal Cybersecurity, addresses one of the most persistent challenges in AI content authentication: reliably marking machine-generated text across tasks that mix natural language, programming code, and mathematical reasoning.

As large language models now produce everything from open-domain articles to executable code and step-by-step mathematical derivations, the need to verify authenticity, attribute sources, and protect copyright has grown urgent. Text watermarking has emerged as a leading countermeasure, typically by injecting small statistical biases into a model’s token-selection process during generation. The most influential example is the KGW scheme, which pseudorandomly partitions the vocabulary into “green” and “red” token lists and nudges the model toward green tokens, leaving behind a statistical footprint that a detector can later pick up. Google’s SynthID-Text has since demonstrated that such decoding-based watermarking can work at production scale.

The trouble, the researchers explain, is that most existing schemes assume a stable generation pattern. They rely on fixed or heuristic entropy thresholds to decide where the watermark should be applied. Token entropy—a measure of how uncertain the model is at a given step—matters enormously: in high-entropy contexts, such as creative prose, the model has many plausible choices, and biasing one direction barely changes the output. In low-entropy contexts, such as the syntax of a Python function or a symbolic derivation, the “correct” next token is nearly deterministic, and any perturbation risks breaking the code or corrupting the logic. A single fixed threshold inevitably misjudges one regime or the other, producing either redundant watermark injection in sensitive regions or noisy, unreliable detection statistics. Earlier attempts to adapt—SWEET, which applies strong bias only above a static entropy threshold, and CATMARK, which updates cluster centers online at considerable computational cost—only partially solve the problem.

GAMark’s central insight is that the semantic states traversed during generation are not uniform. The team draws on the manifold hypothesis: the high-dimensional logit distributions that a language model produces across diverse tasks do not fill their ambient space but cluster on a lower-dimensional structure corresponding to distinct latent semantic states—prose generation, code writing, formal reasoning. GAMark exploits this structure through a two-phase paradigm of offline semantic modeling and frozen online inference.

In the offline phase, the researchers collect logit-level representations from the target models and reduce them using Incremental Principal Component Analysis to a 50-dimensional subspace. They then fit a Gaussian Mixture Model with ten components, each capturing one characteristic semantic mode. In the online phase, the GMM parameters remain frozen: for each token position during generation, the current logit vector is projected and assigned soft, Bayesian posterior-weighted memberships across the semantic states. The adaptive entropy threshold is then synthesized by aggregating over those memberships, allowing the threshold to shift smoothly as the model transitions between, say, natural-language commentary and a code keyword. Only when the token’s entropy exceeds this dynamically computed threshold does the watermark gate open, biasing the green-listed tokens with strength δ while leaving deterministic tokens untouched.

Detection is handled with a matching procedure. Because the model parameters and the GMM are frozen, a detector can reconstruct the same semantic states and thresholds from the raw logits of a suspect text, recover the validity mask indicating which tokens were actually eligible for watermarking, and then compute an entropy-weighted Z-score. The weight assigned to each token scales with the excess of its entropy over the threshold, raising the exponent η to either 0 or 1. The resulting statistic follows, by the Central Limit Theorem, an approximate normal distribution under the null hypothesis, yielding a one-sided hypothesis test: a Z-score above 4.0 corresponds to a false-positive probability below 3.17 × 10⁻⁵. This weighting scheme suppresses the dilution caused by the many low-entropy, never-watermarked tokens that code and math text contain.

The experimental evaluation spanned two models—Qwen2.5-7B-Instruct and LLaMA3.1-8B-Instruct—across four benchmarks chosen for their diverse entropy profiles: C4 (RealNews) for high-entropy open-domain text, HumanEval’s 164 Python problems for functionality under strict syntax, MBPP for concise low-entropy code synthesis, and MATH-500 for interleaved natural-language and symbolic reasoning. Five baselines were compared under unified settings with a green-list ratio of 0.5 and a base logit bias of 2.0: KGW, SWEET, EWD, SynthID, and CATMARK.

The results show consistent gains on both sides of the quality–detectability trade-off. On C4, GAMark achieved a perplexity of 5.96, better than KGW (7.15), SWEET (6.65), and SynthID (6.10), indicating that the adaptive thresholds leave the generated prose nearly indistinguishable from unwatermarked output. On HumanEval, the contrast was starker: CATMARK had to suppress watermark signals to hold its Pass@1 correctness at 74.4%, paying for it with an AUROC of only 65.09% and a true positive rate of 15.85%. GAMark, by contrast, reached the highest Pass@1 of 76.2% while achieving 89.30% AUROC and 81.70% TPR—more than five times the detection sensitivity of CATMARK. On MATH-500, where entropy regimes switch within a single response, GAMark recorded 64.4% Pass@1 with 97.23% AUROC on Qwen2.5-7B-Instruct, and an AUROC of 99.27% with 97.60% TPR on LLaMA3.1-8B-Instruct.

Robustness testing under aggressive semantic attacks reinforced the picture. In a back-translation attack—routing text from English to Chinese and back via the DeepSeek API—the KGW baseline’s AUC collapsed from 0.812 to 0.650, while GAMark held at 0.910. Under full paraphrasing by Qwen2.5-7B-Instruct, most baselines shifted sharply toward false negatives, but GAMark’s AUC declined only from 0.972 to 0.880, well ahead of the second-best method, EWD, at 0.820. The researchers attribute this resilience to the fact that the watermark concentrates in semantically flexible, high-entropy regions that survive rewriting, rather than in brittle structural tokens.

Sensitivity analyses pinpointed the hyperparameters that matter. With too few GMM clusters, coarse semantic modeling fails to shield low-entropy regions; performance peaked at K = 10, beyond which quality declined slightly. The threshold scaling coefficient α governs the trade-off: α = 0.5 yields near-perfect detection (99.2% TPR on HumanEval) but crashes Pass@1 to 56.5% by mistakenly watermarking deterministic tokens, while α = 1.5 restores output quality but weakens detection below 40% TPR. The default α = 1.0 balances both, delivering 76.2% Pass@1 and 81.7% TPR on HumanEval.

Efficiency results were equally notable. Because all heavy manifold modeling is shifted offline, online inference requires only lightweight matrix projections and GMM posterior estimation. GAMark generated text at 35.92 tokens per second on HumanEval, outpacing CATMARK’s 33.13 tokens per second and even slightly exceeding the static SWEET baseline, with only 3.8% overhead compared to CATMARK’s 9.4%. An ablation study confirmed the necessity of each component: removing GMM clustering dropped Pass@1 by 7.7 points on HumanEval, and removing entropy-weighted detection cut AUROC by 7.15 points, showing that semantic state modeling preserves functionality while the weighted statistic filters the noise that unwatermarked low-entropy tokens would otherwise introduce.

The team concludes that GAMark offers a unified, practical answer for watermarking heterogeneous, cross-task generation environments—precisely the settings where modern LLMs are deployed. The researchers plan to extend the framework to more challenging adversarial settings and further optimize efficiency for large-scale deployment, work that could prove consequential as regulators, platforms, and publishers increasingly demand verifiable provenance for AI-generated content.