Ears Hold the Key: Hidden Facial Attribute Bias Exposed in DeepFake Detectors

…

…

The finding that ear visibility dominates the attribute landscape of DeepFake detection errors deserves closer scrutiny, because it connects to a broader theme in machine learning research: models rarely fail uniformly across the populations they serve. When a detector is trained on data in which certain visual configurations are underrepresented, its internal decision boundary adapts to the statistics of the training distribution rather than to the abstract concept of authenticity. In the experiments described, removing samples with visible ears from training produced the steepest drop in performance, from an accuracy of 0.806 and an AUC of 0.823 down to 0.741 and 0.763 respectively, while removing samples whose ears were not visible actually nudged performance slightly upward, to 0.813 accuracy and 0.832 AUC. This asymmetry is informative. It suggests that the detector had come to rely on information concentrated in frames where the ears are visible, and that this reliance was not merely a statistical artifact of the descriptive error analysis but a genuine generalization limitation confirmed under controlled conditions.

There are plausible perceptual reasons why ear visibility might matter so much. The ears sit at the boundary between the face and the background, a region where modern face-manipulation pipelines often struggle. Identity-swap methods, including the faceswap software used to generate the manipulated portion of the dataset studied here, typically composite a synthesized face region onto an original frame, and the seams of that composite tend to appear near the facial contour. When ears are visible, the blending boundary passes through or near them, and subtle inconsistencies in texture, lighting, or geometry may become detectable. When hair, head pose, or framing hides the ears, the manipulation boundary may fall in regions that are easier to synthesize convincingly, or the informative boundary cues may simply be absent. A detector trained predominantly on ear-visible frames may therefore learn features that are partly contingent on that visibility, and when evaluated on the full test distribution, which includes many frames without visible ears, those features transfer imperfectly.

The moderate and consistent effect of hair length adds a second layer to this picture. Hair length is not itself a forensic cue in any obvious sense, but it correlates strongly with how much of the facial contour, including the ears, is occluded. Long hair can cover the ears and the jawline, effectively removing from view exactly the regions where compositing artifacts concentrate. The reported interaction between hair length and ear visibility is consistent with this interpretation: the two attributes are not independent sources of difficulty but overlapping windows onto the same underlying signal. This kind of correlated attribute structure is common in real-world face imagery, and it illustrates why single-factor analyses can be misleading. A detector that appears biased against one attribute may in fact be responding to a visually entangled companion attribute, and only a joint analysis of the kind enabled by systematic multi-attribute labeling can disentangle the two.

The semi-supervised labeling pipeline that made this analysis possible is itself methodologically noteworthy. Annotating thousands of videos by hand with attributes such as gender, hair color, hair length, ethnicity, and ear visibility would be prohibitively expensive, and manual annotation is also vulnerable to inconsistency across annotators. The approach taken here starts from a small manually labeled seed subset and propagates labels to the remaining videos automatically, using a recognition model trained on that seed. This strategy trades a modest amount of label noise for a large gain in coverage, and it reflects a growing practice in fairness and robustness research, where the bottleneck is often not the detection model itself but the metadata needed to slice its performance into interpretable subgroups. The quality of such propagated labels matters: systematic errors in the labeling model could masquerade as subgroup disparities in the detector. The consistency between the descriptive error analysis and the controlled exclusion experiments, however, provides evidence that the attribute signals were reliable enough to support causal-style conclusions rather than mere correlations.

The experimental design of the exclusion study merits attention as well. A naive way to test whether an attribute matters would be to train on subsets and evaluate on subsets matched to that attribute, but this confounds the effect of the attribute with the effect of the changed test distribution. By keeping a single unified test set fixed across all exclusion conditions, the study isolates what happens to generalization when the training data are impoverished in a specific way. The baseline model, trained on everything, achieves 0.806 accuracy; the model deprived of ear-visible training samples drops by more than six points; the model deprived of non-visible-ear samples performs slightly better than baseline. This pattern indicates that ear-visible frames are not merely abundant in the training set but disproportionately informative, and that their removal cannot be compensated by the remaining data. In effect, the experiment functions as an ablation over demographic and appearance strata of the training corpus, a technique that could be applied to any detection benchmark where attribute metadata can be derived.

The explanation offered for the confusion-matrix asymmetry also carries practical weight. Video-level DeepFake detection typically aggregates frame-level scores, declaring a video fake when at least k of its n frames exceed a fixed threshold. When the per-frame score distribution shifts for a subgroup of videos, the fixed threshold converts that shift into an asymmetric pattern of errors: some subgroups accumulate false positives while others accumulate false negatives, depending on which side of the threshold their scores cluster. This means that part of the observed disparity between REAL and FAKE classes, or between attribute subgroups, is not a property of the learned features at all but a property of the decision rule layered on top of them. Threshold calibration, adjusting the operating point per subgroup or per deployment context, is therefore identified as a concrete remediation path, alongside data curation and augmentation. This distinction between representation-level and decision-level sources of bias is often blurred in reporting, and making it explicit is one of the more actionable contributions of the work.

The choice of the C40 compression level throughout the experiments situates the study in a deliberately adversarial regime. FaceForensics++ provides three encoding settings, from uncompressed raw frames through moderate compression to the heavily compressed c40 configuration, and it is well established that compression destroys many of the high-frequency artifacts that detectors rely on. Strong compression smooths blending boundaries, quantizes away subtle texture inconsistencies, and generally pushes detectors toward coarser, more semantic cues. That attribute-conditioned disparities persist under c40 suggests that the reliance on ear-visible regions is not limited to fragile high-frequency fingerprints but extends to the mid-level visual evidence that survives compression. It also means the findings are relevant to the realistic social-media conditions under which DeepFake detection systems must actually operate, where re-encoding is the norm rather than the exception.

It is worth considering what these results imply for the field’s standard evaluation practice. Benchmarks are typically reported as single aggregate numbers, an overall accuracy or AUC, which conceal heterogeneity across subgroups. Two detectors with identical aggregate scores may differ dramatically in how their errors distribute across gender, appearance, or occlusion categories, and the aggregate metric alone cannot reveal this. The attribute-aware framework demonstrated here offers a template for richer evaluation: enrich the benchmark with automatically derived attribute labels, slice performance by attribute, and run controlled exclusion experiments to confirm which disparities reflect genuine generalization gaps. Such a protocol could become a standard complement to aggregate reporting, much as per-class breakdowns became standard in object detection. It also aligns with broader regulatory and ethical expectations that automated systems be audited for disparate performance across population segments, an expectation increasingly applied to synthetic-media detection as it is deployed in content moderation and verification pipelines.

The semi-supervised labeling approach also raises questions worth further study. Label propagation inherits the biases of its seed set: if the manually annotated subset overrepresents certain appearances, the propagated labels and any downstream subgroup analysis will reflect that skew. Ethnicity, in particular, is a socially constructed category whose automatic annotation is fraught, and the study’s use of it as one attribute among several should be read as an operational variable for slicing performance rather than a claim about any population. The value of the framework lies in exposing where a detector is fragile, not in reifying the attributes themselves. Future work could extend the attribute vocabulary to include head pose, illumination, occlusion by accessories, and image quality, all of which plausibly interact with manipulation artifacts, and could apply the same exclusion methodology across multiple forgery families beyond the faceswap-generated subset examined here, since different manipulation methods leave artifacts in different facial regions and may therefore exhibit different attribute sensitivities.

Finally, the practical recommendations that emerge, targeted data curation, attribute-specific augmentation, and threshold calibration, form a coherent agenda. Curation ensures that every visually meaningful stratum of the deployment population is represented in training. Augmentation can synthesize the underrepresented configurations, for example by artificially occluding ears or varying hair rendering, so that the detector learns invariances rather than contingencies. Calibration adjusts the decision threshold so that aggregate error rates translate into balanced error rates across subgroups. None of these steps requires architectural novelty, which is itself an important message: some of the robustness gaps in DeepFake detection may be addressable not by new networks but by more careful accounting of who and what the training data contain, and by decision rules that acknowledge the heterogeneity the features already capture. As generative manipulation methods continue to improve and the visual gap between real and synthetic content narrows, the marginal value of such data- and decision-level discipline is likely to grow, making attribute-aware analysis a durable component of the detection toolkit rather than a passing diagnostic curiosity.

Subject of Research: DeepFake detection

Article Title: Facial attribute-aware DeepFake detection through semi-supervised facial attribute labeling

Article References: Stile, V., Caldelli, R., Balderas-Díaz, S., Guerrero-Contreras, G., & Medina-Bulo, I. (2026). Facial attribute-aware DeepFake detection through semi-supervised facial attribute labeling. Frontiers in Imaging, 5, Article 1846377. https://doi.org/10.3389/fimag.2026.1846377

Image Credits: AI Generated

DOI: 10.3389/fimag.2026.1846377

Keywords: DeepFake detection, FaceForensics++, facial attributes, semi-supervised labeling, AI bias, machine learning, digital forensics, Facial, attribute-aware, DeepFake, detection, semi-supervised