AI Chatbots Break Human Rules in Life-or-Death Kidney Allocation Choices

When a single kidney becomes available and two patients need it, the choice is one of the starkest moral dilemmas in medicine. Age, dependents, health status, lifestyle habits — every attribute seems to demand consideration, yet no formula tells a clinician how to weigh one life against another. Economists have long treated this as a problem of mechanism design, most famously Nobel Laureate Alvin Roth, whose work on kidney exchange reshaped how transplant systems allocate scarce organs. But a new study led by researchers at Penn State asks a different and increasingly urgent question: what happens when the decision is handed to an artificial intelligence model? The answer, according to research presented at the 2026 Association for Computing Machinery Conference on Fairness, Accountability and Transparency (FAccT) and published in the conference proceedings, is that large language models do not make the same moral choices humans do — and the differences are not subtle.

The research team, led by Hadi Hosseini, associate professor of informatics and intelligent systems and associate professor of economics at Penn State, set out to map where the moral judgments of leading AI chatbots align with human values and where they diverge. Rather than building a new ethical framework from scratch, the researchers took advantage of an unusual resource: existing datasets from published academic studies in which hundreds of real participants, with no specified medical training, had already faced the kidney allocation dilemma and recorded their choices. That gave the team a human baseline against which the judgments of several AI chatbots could be measured, using the very same scenarios that human participants had encountered in the earlier studies.

The experimental design was deliberately simple and revealing. Two hypothetical patients were described side by side, each characterized by attributes such as age, the number of dependents they supported, their health status and their drinking habits. Both patients needed the same kidney, and only one organ was available. The chatbots were then asked to decide who should receive it. As Hosseini explained, the comparisons were run in several different ways: sometimes a single trait was isolated at a time to see how the models valued it on its own; sometimes several traits were mixed together to observe how the AI weighed competing factors against one another; and sometimes a flip-a-coin option was added, giving the model a way to express indecision — a key feature of human moral judgment that the researchers wanted to test explicitly.

Two findings stood out. The first concerns how AI models weigh the attributes of the patients. Human participants, the earlier studies showed, tend to balance multiple considerations simultaneously, accepting that a patient’s age might matter in one direction while their health status matters in another, and that dependents and lifestyle factors complicate the picture further. The AI chatbots did something quite different. They tended to fixate on a single factor — drinking habits was one example cited by the researchers — and let that one attribute dominate the decision, rather than integrating the full set of considerations the way people do. In effect, the models oversimplified decisions that humans experience as irreducibly complex, collapsing a many-dimensional moral trade-off into a one-dimensional judgment.

The second finding may be even more consequential. When humans confront a dilemma with no clearly correct answer, they frequently express indecision. Hosseini suggests this may be because people are reluctant to accept full agency over a decision that determines who lives and who dies — declining to choose is, in a sense, a refusal to own the outcome. The AI models almost never behaved this way. Even when the researchers directly offered the option to flip a coin, the chatbots overwhelmingly committed to a confident, deterministic answer instead. Where humans recognize ambiguity, the models projected certainty that the underlying problem simply does not support. That gap, the researchers argue, is meaningful precisely because real moral dilemmas — in medicine, in law, in the allocation of any scarce resource — often lack a single correct answer, and an advisor that cannot represent that uncertainty misrepresents the nature of the problem itself.

The stakes of this mismatch are not hypothetical. Hosseini is blunt about what is at risk: moral decisions in settings like organ allocation directly determine who lives and who dies, so getting AI’s role in them right is not optional. At the same time, he is careful to note that the study is not an argument for handing such decisions to machines. The researchers do not intend to encourage the use of AI as a substitute for professional judgment in medical decision-making or other high-stakes contexts. The point, rather, is that individuals, organizations and firms are already relying on AI to make decisions or to receive recommendations, and that reliance makes it essential to understand how these systems actually behave when the questions turn moral. A chatbot consulted informally by a clinician, a policymaker or a member of the public does not announce that its moral calculus differs from human consensus; it simply delivers an answer with confidence.

That confidence problem connects to a broader theme in the study of large language models. These systems are trained to produce fluent, decisive outputs, and their interface design rewards a single authoritative response over an expression of doubt. The Penn State findings suggest that this design tendency carries over into moral reasoning, where decisiveness is not always a virtue. Human moral judgment, by contrast, is built to tolerate ambiguity: people weigh competing values, acknowledge trade-offs, and sometimes refuse to choose at all. When an AI model collapses that process into a confident pick, it does not merely differ in its conclusion — it differs in its understanding of what kind of question is being asked. A dilemma is being treated as a calculation, and the calculation hides the dilemma.

The divergence in attribute weighting raises its own set of concerns. If a model fixates on drinking habits when allocating a kidney, it effectively imposes a moral hierarchy that human participants in the underlying studies did not adopt, penalizing one attribute out of proportion to how people balance it against dependents, age or health status. Because the researchers used the same scenarios given to humans in earlier academic work, the mismatch cannot be attributed to different framings or different questions; the models and the people were answering the same prompts. The divergence lies in the models themselves — in how they encode, prioritize and combine the moral information presented to them. That is exactly the kind of misalignment that matters for decision support, because a recommendation that silently overweights one factor can steer a human decision-maker without that person ever realizing the machine’s priorities differ from their own.

John Dickerson, chief executive officer at Mozilla.ai, who collaborated on the work, framed the problem in terms of how societies handle scarcity. When something scarce is allocated — a kidney, a job, access to some other resource — there is not always a single objectively correct answer, he observed. Humans recognize that ambiguity and codify it through open debate into the allocative process. AI models often do not. The observation points to a structural difference between human institutions and machine outputs: the legitimacy of a contested allocation decision often comes not from the answer itself but from the process of deliberation that produced it, a process in which disagreement, hesitation and debate are features rather than bugs. A model that outputs a single confident verdict bypasses that process entirely, offering the appearance of objectivity where only contested judgment exists.

Hosseini argues that closing this gap requires continued research and governance involving policymakers, regulators and other stakeholders. Asking whether AI can make moral decisions, and whether its judgments are aligned with human values, he notes, are more than philosophical musings — they sit at the core of today’s AI discourse, and the ethical stakes of AI’s role in life-altering decisions demand deep reflection. The study, published in the proceedings of the FAccT 2026 conference held in June, was partially funded by the U.S. National Science Foundation under grant numbers 2144413 and 2107173. Two students from Penn State’s College of Information Sciences and Technology contributed to the work: Samarth Khanna, a doctoral student in informatics, and Leona Pierce, a fourth-year undergraduate and Schreyer Scholar triple majoring in data sciences, mathematics and statistics. As AI systems move deeper into domains where every recommendation touches a human life, the study’s message is hard to ignore: before trusting a machine’s moral answer, it matters enormously whether that machine weighs the world the way we do — and whether it knows, as we do, when there is no right answer at all.

Subject of Research: Alignment of large language model moral judgment with human values in kidney allocation decisions

Article Title: Who gets the kidney? AI chatbots do not make the same moral choices humans do

Article References: Who gets the kidney? AI chatbots do not make the same moral choices humans do. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: artificial intelligence, large language models, AI alignment, medical ethics, organ allocation, kidney transplant, moral decision-making, FAccT 2026, decision support, human-AI values, indecision, resource scarcity