A Hesitation Score Helps AI Models Survive Wrong Labels and Wrong Questions

Machine learning systems that juggle several tasks at once share a hidden vulnerability that has little to do with the sophistication of their architectures: the reliability of the annotations they learn from. In a multi-task classification pipeline, every training example typically arrives with two pieces of supervision, a task identity and a class label whose meaning only makes sense in the context of that task. When annotators, automated pipelines, or dataset merges assign the wrong task to an example, the attached class label effectively answers the wrong question. A sentiment label appended to a paraphrase-detection example, for instance, is not merely noisy; it is semantically meaningless. A team of researchers at Hanoi University of Science and Technology has now formalized this failure mode and proposed a remedy built on a decades-old mathematical idea, intuitionistic fuzzy sets.

The study, published in Complex & Intelligent Systems by Hong Son Nguyen, Khac Trung Nguyen, Ngoc Linh Pham, Ngoc Thang Tran, and Ngoc Chi Lê, introduces the concept of compound task-class noise. The key insight is that task errors and class-label errors should not be treated as two independent sources of corruption. Because the interpretation of a class label depends on the task it is attached to, an error in the observed task identity can change what the observed class label means, coupling the two noise processes in a way that standard label-noise-robust methods never anticipate. The problem becomes acute in what the authors call task-agnostic inference, a deployment setting in which no task name is supplied at test time and the model must decide on its own which task should interpret the class label it predicts.

Most existing defenses against noisy labels operate under a simplifying assumption: the label is wrong, but the question it answers is right. Techniques such as generalized cross-entropy loss or transition-matrix correction methods like MTL-NL model how likely a true class is to be flipped into an observed one, yet they presume the task assignment is trustworthy. The new work shows that this assumption collapses precisely where it matters most. In experiments across text and synthetic benchmarks, baselines that remain competitive when a human verifies the task at test time degrade severely when the task must be inferred blindly, with some configurations experiencing near-total collapse under the harshest noise profile the authors tested, which they term maximal distortion.

The proposed method, named IFS-Hesitant, draws on intuitionistic fuzzy sets, a generalization of classical fuzzy set theory in which each element is characterized not only by a degree of membership and a degree of non-membership but also by a hesitation margin capturing the uncertainty that remains after both are accounted for. In the authors’ architecture, a shared encoder feeds task-specific prediction heads that output a membership value and a non-membership value for each candidate task, normalized through a softmax, together with an independent hesitation score produced by its own output channel. The joint membership used for classification is recovered by multiplying the normalized membership by one minus the hesitation, a construction based on the product T-norm from fuzzy mathematics.

What makes this design more than a notational exercise is how the hesitation score is used. During training, it acts as a trust gate: the contribution of each example to the loss is weighted by the product of the hesitation-adjusted task fit and the class satisfaction under the inferred task. Examples whose task assignment looks suspicious are automatically down-weighted, so corrupted supervision exerts less pull on the shared representation. The authors implement a floor on this weight, set to 0.1 at BERT-base scale, to prevent an early-training pathology they describe as optimization collapse, in which an overconfident hesitation signal starves the model of gradient signal before it has learned anything useful. Smaller architectures, where this starvation effect is empirically inactive, default to no floor at all.

The same hesitation signal does double duty at inference time. Because the model produces a task-fit uncertainty for each candidate task, it can route an unlabeled input to the task whose head fits best, without any external task classifier. This is what enables task-agnostic deployment: the model both selects the question and answers it, while the hesitation margin flags cases where neither selection is trustworthy. The authors show that this single scalar also supports post-hoc corruption detection, allowing practitioners to identify suspicious task-class pairs in a dataset after the fact rather than only tolerating them during training.

The empirical evaluation spans a deliberately diverse set of benchmarks. A low-capacity configuration uses a two-layer multilayer perceptron with hidden dimensions of 64 and 32 on the Blitzer cross-domain sentiment benchmark, which contains roughly 2,000 product reviews from each of four domains, and on synthetic Gaussian data with three binary tasks. A high-capacity configuration attaches the IFS heads to a BERT-base encoder on two GLUE-derived suites, one combining the Microsoft Research Paraphrase Corpus, the Recognizing Textual Entailment corpus, and the Corpus of Linguistic Acceptability, and another replacing the sentence-pair entailment task with the Stanford Sentiment Treebank. A multi-class extension uses a ResNet-18 backbone on a CIFAR-100 superclass problem with three tasks of five classes each, drawn from insects, small mammals, and non-insect invertebrates. All splits and feature extraction pipelines were refit across independent random seeds to prevent data leakage, and reported metrics are means over five seeds.

The results follow a consistent pattern across five operational noise profiles, ranging from clean data through structured corruption, annotator error, and mixed degradation to maximal distortion. When the correct task is handed to the model at test time, IFS-Hesitant is competitive with the strongest baselines, confirming that the hesitation mechanism does not sacrifice accuracy in the easy setting. When the task must be inferred without external labels, the method achieves the highest blind accuracy in every evaluated task-corrupted noise setting, with the largest margins appearing exactly where task and class errors are cross-coupled. The margins are widest on the synthetic benchmark, where task-routing signals are mathematically separable from localized class distributions, and tightest on the GLUE sentence-pair suite, where structurally dense multi-task noise makes routing intrinsically harder.

Beyond the specific benchmarks, the study makes a conceptual contribution that is likely to outlast its particular architecture. By separating task fit from class evidence and expressing the gap between them as a structured hesitation quantity, the authors offer a primitive that other robust-learning methods could adopt: a single, interpretable number that says how much a training example, or a test input, should be trusted before any prediction is made. The intuitionistic fuzzy formulation gives this quantity a principled algebra, with membership, non-membership, and hesitation combining through well-defined operations rather than ad hoc heuristics. The authors report no external funding for the work and have submitted the source code for reproduction as a supplementary file, with public release planned upon acceptance.

For practitioners deploying multi-task models in the wild, where annotation pipelines are imperfect and users do not announce which task they want solved, the message is sobering and constructive at once. Label noise is bad enough on its own, but noise that changes the question a label answers is categorically worse, and the standard toolkit was never built for it. The Hanoi team’s results suggest that teaching a model to hesitate, in a mathematically explicit way, may be one of the most economical defenses available: the same signal that protects training also routes inference and exposes corrupted data, all without requiring a separate task classifier or any change to the underlying encoder.

Subject of Research: Robust multi-task binary classification under compound task-class noise using intuitionistic fuzzy sets

Article Title: Compound task-class noise in task-agnostic inference for multi-task binary classification: an intuitionistic fuzzy approach

Article References: Nguyen, H. S., Nguyen, K. T., Pham, N. L., Tran, N. T., & Lê, N. C. (2026). Compound task-class noise in task-agnostic inference for multi-task binary classification: an intuitionistic fuzzy approach. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-026-02493-z

Image Credits: AI Generated

DOI: 10.1007/s40747-026-02493-z

Keywords: compound task-class noise, multi-task learning, task-agnostic inference, intuitionistic fuzzy sets, noisy labels, uncertainty quantification, trust-weighted loss, binary classification, BERT, CIFAR-100, GLUE benchmark, machine learning