When Algorithms Cannot Be Fair: The Hidden Choices That Lock Bias Into AI

When ProPublica revealed in 2016 that the COMPAS recidivism tool flagged Black defendants as high risk at roughly twice the rate of white defendants, the company behind the system replied with a technicality that turned out to be true: the scores were equally calibrated across groups. Both sides were correct, and mathematicians soon proved why. When base rates differ between groups, an imperfect predictor cannot simultaneously satisfy calibration and error-rate balance. Ever since, the algorithmic fairness literature has treated that impossibility as the central problem, arguing endlessly over which fairness metric to prioritise in which context. A new paper in AI & Society by philosopher Alfredo Di Giorgio of the University of Salento argues that this entire debate starts too late. Before any fairness metric can be applied, a system’s designers have already decided which categories the machine recognises, which constructs become variables, which context is discarded, and which objective is optimised. Those upstream representational choices, the paper contends, can discriminate in their own right, and the tools now exist to say when that discrimination is fixable and when it genuinely is not.

The paper’s starting point is what Di Giorgio calls the ontological fracture: the gap between the lived, contextual fabric of social reality and the finite vocabulary of variables and categories into which any computational system must translate it. Encoding race as a discrete feature, or substituting the six-point Fitzpatrick skin-type scale in computer vision evaluations, does not dissolve the problem; it merely relocates the decision about which distinctions count as relevant. Two epistemic consequences follow. The first is a resolution limit, which the paper terms epistemic grain: like the grain of photographic film, a model’s feature space determines the finest distinctions it can register, and phenomena below that grain are unavailable in principle to any analysis conducted in the system’s own terms. The second is more insidious. Once categories are fixed, whatever exceeds them registers not as evidence of an inadequate category scheme but as residual error, noise to be reduced with more data. The paper calls this systematic misreading computational fog: the distortions introduced by representational choices are not representable within the representation that produced them. Gender classifiers trained on a binary scheme illustrate the structure, since a person whose identity does not fit the binary confronts an inadequacy of the scheme itself, which the system can only record as noise.

What makes the computational case distinctive, the paper argues, is not abstraction as such, which all modelling shares, but the timing of precisification. Drawing on Friedrich Waismann’s notion of open texture and H.L.A. Hart’s account of legal adjudication, Di Giorgio observes that social concepts such as race, merit, risk and need are never delimited in all possible directions; human decision-makers sharpen them case by case, in view of particulars, and can revise the sharpening when the next case differs. A computational system cannot do this. It fixes the precisification of every open-textured concept once, ex ante, at design time, and applies it uniformly at scale, with no interpreter positioned to notice that the present case exceeds the scheme and re-open it. This also explains why the familiar reassurance that a human remains in the loop underdelivers: an overseer reviewing outputs under the EU AI Act’s Article 14 can check decisions, but cannot re-open the precisification itself, because the information that would motivate re-opening has typically been discarded below the system’s epistemic grain.

The paper’s central contribution is a criterion for deciding when representational discrimination is remediable, borrowed from an unexpected source: the less-discriminatory alternative construct of United States disparate-impact doctrine. In litigation, a challenger who shows a facially neutral policy produces disparate impact can prevail by exhibiting an alternative policy serving the same legitimate purpose with a smaller disparity. Di Giorgio shifts that comparative logic one level up, from models to operationalisations. A disparity is contingent, relative to a legitimate purpose, a measure of disparity, and a class of admissible designs, when some alternative operationalisation within that class serves the purpose while reducing the disparity. It is constitutive when no such alternative exists. Nothing is constitutive in the abstract; every classification is elliptical for the purpose, the disparity measure, and the design space against which it is made. Debates that sound metaphysical, whether a bias is inherent to computation, thereby dissolve into three tractable questions: what the purpose is, what the relevant disparity is, and what the admissible design space actually contains.

The criterion’s logical form carries real teeth. Contingency is verified by exhibition: produce one admissible alternative and the classification is settled. Constitutivity, by contrast, is a negative existential claim, that no alternative exists, and can never be conclusively verified over a design space that is not surveyable. It is established only by provisional exhaustion of the known alternatives and remains permanently open to refutation by a single new exhibit. This asymmetry generates a burden structure: the critic must show contingency by exhibit, while the deployer claiming a disparity is unavoidable owes documented exhaustion of the alternatives considered and the grounds for excluding each. Neither burden is discharged by assertion. The result converts an unfalsifiable rhetorical endpoint into a procedure that specifies in advance what would show any constitutivity claim to be wrong.

Three case studies test the criterion adversarially, and the first delivers a striking reversal. The ImageNet benchmark inherited, from the WordNet lexical taxonomy beneath it, derogatory categories under the person synset, so photographs of real people were assigned to demeaning classes because the conceptual vocabulary admitted them. The bias sat exactly where the paper’s framework looks, in the category scheme itself, upstream of any training, which would ordinarily invite the label constitutive. Yet the criterion returns the opposite verdict: an admissible alternative, filtering and rebalancing the person subtree while preserving the dataset’s benchmarking purpose, was not merely available in principle but actually carried out by researchers in 2020. Upstream, the paper concludes, does not mean constitutive, and the slide from upstream to irremediable is the single most productive source of misclassification in the field.

The second case, tokenisation in large language models, resists a single verdict and forces a decomposition. Byte pair encoding builds vocabularies from corpus statistics, so well-represented languages receive compact tokenisations while under-represented ones fragment into many small units; research shows the same content can require up to fifteen times more tokens in some languages than others, compounding into higher costs, shorter outputs and less context for speakers already economically constrained. The contingent component, attributable to the construction method and allocation policy, has demonstrable remedies. The residual component is subtler: even character-level and byte-level encoding shows more than a fourfold difference in length for some language pairs, reflecting the differing information densities of scripts and the uneven byte costs of UTF-8. Within the class of systems that segment all languages with one shared inventory and price by segment count, every known alternative relocates the disparity rather than eliminating it. The paper classifies that residual layer as constitutive only relative to that class and to current knowledge, hedged exactly as the criterion demands, and refutable by any scheme achieving cross-linguistically uniform encoding costs.

The third pair of cases shows the criterion operating where layers are entangled. Amazon’s abandoned CV-screening tool penalised indicators of being a woman, and neutralising explicit indicators failed against subtler proxies; under the criterion, that documented disparity was contingent along at least one dimension, since outcome-based targets and structured competency assessments exist within the class of automated screening, while the failure of particular remedies counts as evidence about the depth of the disparity, never as proof of inevitability. The clinical case supplies the empirical anchor: Obermeyer and colleagues’ landmark 2019 study found that a widely deployed algorithm operationalising health need as health cost systematically underserved Black patients, because unequal access depresses spending conditional on illness. Re-operationalising the target through measures of health reduced the disparity by 84 percent, a remedy invisible at the model level. The paper even rereads the famous impossibility theorems through this lens: they are a constitutivity result, but one proved relative to a class of designs that holds fixed a population partition and base rates which are themselves representational commitments, not primitives of nature.

The regulatory payoff is a proposal the paper calls ontological auditing: an upstream evaluation of representational choices during design, documented in an Ontological Impact Statement. For each socially consequential construct, the statement would record what the variable purports to represent, which dimensions it excludes, what proxy relationships link it to protected characteristics, which alternative operationalisations were considered and why each was excluded, and, for any disparity the deployer deems unremediable, the parameters and exhaustion supporting that claim. Filed before deployment, such a statement in the clinical case would have exposed the cost-as-need choice while it was still cheap to revise. The paper locates the practice within the EU AI Act’s risk management and data governance provisions, noting that current obligations discipline data quality and outputs but never ask the question disparate-impact doctrine has long asked: was a less-discriminatory way of achieving the same end available? The proposal is explicitly untested, and its administrative cost unknown.

What the framework ultimately buys is a change of register. Only one of the disparities examined is classified as constitutive, and that classification is relative to a class and hedged to current knowledge, because the design space expands with technique and unhedged inevitability claims are exactly what the criterion forbids. Where a representational choice is genuinely forced, the paper asks that the fact be established, documented and weighed openly rather than discovered after deployment. Where it is not forced, and the evidence suggests that is the more common case, the criterion poses the simpler question that the vocabulary of inevitability has tended to obscure: what else could have been built, and why was it not?

Subject of Research: A criterion for distinguishing contingent from constitutive algorithmic discrimination arising from upstream representational choices in AI systems

Article Title: When algorithms cannot be fair: constitutive and contingent discrimination and the limits of less-discriminatory alternatives

Article References: Di Giorgio, A. (2026). When algorithms cannot be fair: constitutive and contingent discrimination and the limits of less-discriminatory alternatives. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03384-0

Image Credits: AI Generated

DOI: 10.1007/s00146-026-03384-0

Keywords: algorithmic fairness, algorithmic discrimination, less-discriminatory alternative, open texture, ontological auditing, EU AI Act, ImageNet, tokenisation, clinical risk scoring, disparate impact, computational opacity, AI & Society