AI Learns to Distrust Its Own Illusions: CLIP Helps Models Adapt Without Source Data

Machine learning models are often trained in one setting and deployed in another, and that transition is rarely seamless. A classifier trained on studio photographs of cars may stumble when shown cars in rain, at night, or through a security camera. Domain adaptation is the branch of machine learning that tackles this problem, and its most demanding variant, source-free domain adaptation, adds a harsh constraint: once the model leaves the training environment, the original labeled data is gone, often for reasons of privacy, storage, or proprietary restriction. All the model can carry with it is what it learned. A new study published in the journal Machine Learning by Qing Tian, Yongjiang Liu, Keyang Cheng, Weihua Ou, Jianping Gou and colleagues confronts a subtle but pervasive failure mode in this setting, one the researchers describe with an evocative word borrowed from human cognition: illusions.

The core difficulty lies in how a source model, cut off from its training data, tries to make sense of an unlabeled target domain. A popular family of methods constructs a pseudo-source domain, a synthetic stand-in for the lost source data, by selecting target samples that the source model believes look source-like. Training on this pseudo-source then reduces the statistical gap between domains. The catch, as the new paper argues, is that this construction leans entirely on the source model itself, and the source model is precisely the entity most likely to be deceived. During training, it absorbed not only the true signals that define each class, such as the shape and structure of an object, but also spurious correlations linking class identity to domain-specific features like background, lighting, or environment. When asked to identify pseudo-source samples, it may confirm its own biases, mistaking samples that share superficial environmental cues with the source domain for genuinely representative ones.

The researchers identify two intertwined challenges that undermine this process. The first is confirmation bias: once the model commits to a belief about a sample, the subsequent training on that sample reinforces the belief, whether or not it was correct. This is the machine analogue of a person who reads only news that agrees with their views. The second is domain shift, the distributional mismatch between source and target that makes the source model’s judgments unreliable in the first place. Together, these effects can contaminate the pseudo-source domain with mislabeled or unrepresentative samples, and the errors compound as adaptation proceeds. The team’s answer is a method they call Staying Rational and Resisting Illusions, or SRRI, which refuses to let the source model grade its own homework.

The key innovation is the introduction of an external referee: CLIP, the Contrastive Language-Image Pre-training model developed by OpenAI researchers, which learned to align images and text by training on hundreds of millions of image-text pairs from the web. Because CLIP’s knowledge comes from a vastly broader data distribution than the narrow source domain, it does not share the source model’s spurious correlations. SRRI uses CLIP as a source of independent evidence when deciding which target samples deserve a place in the pseudo-source domain. In effect, when the source model says a sample looks familiar, SRRI asks CLIP for a second opinion before accepting the claim.

Technically, the method proceeds in several coordinated stages. First, SRRI employs knowledge distillation, a technique in which a teacher network’s outputs guide a student network, to help the source model disentangle class-discriminative causal features from domain-specific spurious features. The goal is to teach the model which aspects of an image actually cause its label, such as the geometry of an object, and which merely co-occur with it, such as the typical backdrop of the source photographs. This disentanglement weakens the illusions at their root, making the model’s own judgments less confounded before any pseudo-source construction begins.

Distillation alone, however, cannot be trusted blindly, because in some adaptation tasks the distillation process itself performs poorly, propagating errors rather than correcting them. To guard against this, the authors design a CLIP-guided dual-model validation and class balancing strategy. Every candidate pseudo-source sample must pass inspection by both the distilled source model and CLIP, and the two models’ assessments are combined to filter out unreliable examples. Class balancing ensures that the retained samples cover all categories with reasonable richness, preventing the pseudo-source domain from being dominated by easy or overrepresented classes. This dual gatekeeping is what allows the method to remain robust even when one of its components falters on a given task.

The third pillar of SRRI is a dynamic pseudo-source domain optimization mechanism. Rather than freezing the pseudo-source once it is built, the method continuously fine-tunes the task-specific prompts of CLIP during adaptation. Prompt tuning adjusts the short text descriptions that CLIP uses to interpret images, sharpening its sensitivity to the specific categories of the target task. As these prompts improve, CLIP’s judgments on hard samples become more accurate, which in turn corrects residual bias in the target model. At the same time, the pseudo-source domain is periodically reconstructed and refined, discarding samples that no longer pass validation and admitting better ones as the models evolve. The result is a self-correcting loop in which the reference data improves alongside the adapting model.

With a trustworthy pseudo-source domain in place, SRRI applies robust supervised learning to train the target model on the pseudo-source samples, while simultaneously performing distribution alignment between the pseudo-source and the true target data. This alignment ensures that the model does not overfit to artifacts of the pseudo-source construction and that its decision boundaries remain well matched to the actual deployment distribution. The combination of reliable pseudo-labels, balanced classes, and distributional consistency addresses both of the fundamental challenges the authors set out to solve: confirmation bias is curbed by external validation, and domain shift is absorbed by the alignment procedure.

Extensive experiments reported in the paper show that SRRI outperforms state-of-the-art source-free domain adaptation methods across standard benchmarks. The improvements are attributed not to any single trick but to the architecture of trust the method builds: an independent verifier, a disentangled representation, a balanced and evolving reference set, and a training objective that keeps the target model anchored to reality. All datasets used in the study are publicly available, which should make the approach straightforward for other groups to reproduce and extend. The work was supported by the National Natural Science Foundation of China and several regional research programs, and the authors report no competing financial interests beyond these funding sources.

The broader significance of this research extends beyond a single benchmark. As artificial intelligence systems are increasingly deployed in hospitals, vehicles, and surveillance networks where raw training data cannot be shared, source-free adaptation will become a standard requirement rather than a niche concern. The lesson of SRRI is that a model adapting in the wild should not rely solely on its own inherited judgments, because those judgments may encode illusions about what really defines a category. By recruiting a vision-language foundation model as an external rational check, and by continuously refining both the verifier and the verified, the researchers offer a template for building machine learning systems that stay rational under pressure, resisting the very biases they were born with. In an era when AI is often criticized for confidently repeating its mistakes, a method explicitly designed to resist its own illusions is a welcome step toward more trustworthy machine intelligence.

Subject of Research: De-biasing source-free domain adaptation using a CLIP-verified pseudo-source domain

Article Title: Stay Rational, Resist Illusions: De-biasing Source-Free Domain Adaptation with CLIP-Verified Pseudo-Source Domain

Article References: Tian, Q., Liu, Y., Cheng, K., Ou, W., & Gou, J. (2026). Stay Rational, Resist Illusions: De-biasing Source-Free Domain Adaptation with CLIP-Verified Pseudo-Source Domain. Machine Learning, 115(10), Article 222. https://doi.org/10.1007/s10994-026-07163-2

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07163-2

Keywords: source-free domain adaptation, CLIP, pseudo-source domain, knowledge distillation, confirmation bias, domain shift, spurious correlations, prompt tuning, distribution alignment, machine learning, vision-language models, transfer learning