In rapidly aging societies, a growing dilemma emerges: people are living longer lives, yet family networks are shrinking. This demographic shift raises a daunting question—what happens when individuals near the end of life and are no longer able to make their own medical decisions, but lack any legally designated next of kin or trusted proxies to represent their wishes? The result is often crucial life-or-death choices falling to strangers.
Advance care planning (ACP) was developed to bridge this gap by allowing people to document their healthcare preferences in advance. However, in practice, ACP often remains a static, one-time formality, filed away and seldom updated or accessed when needed. Addressing this challenge, researchers at the Singapore University of Technology and Design (SUTD) have created an innovative AI-driven prototype named ACPAgent. This tool is designed to act as an autonomous ACP proxy, trained through interactive workshops with real users.
ACPAgent aims to tackle a profound question: when critical decisions hinge on deeply subjective personal values rather than objective facts, what role could artificial intelligence play? Over the course of four workshops involving 15 participants, users input their priorities—ranging from meaningful activities and healthcare goals to what they are willing to sacrifice for additional time. They then engaged with progressively complex medical scenarios, from treatable infections to terminal illnesses compounded by familial conflict. Participants made initial decisions independently, then reviewed the agent’s recommendations, adjusting their preferences or challenging the AI as desired.
The results revealed that participants agreed with the AI’s suggestions in nearly 87% of cases, indicating the agent effectively captured individual values. Yet, the most revealing insights surfaced in moments of discord, where users contested the AI’s grasp of nuanced considerations like family guilt, financial affordability, and realistic recovery odds—elements the prototype struggled to model accurately.
Assistant Professor Kenny Choo, lead researcher, cautions against interpreting high agreement rates as straightforward success. Such conformity could stem not only from alignment between human and machine reasoning but also from psychological phenomena like automation bias or passive acquiescence to a seemingly agreeable AI authority. Intriguingly, in about one in eight cases, the AI successfully persuaded users to revise their initial choices—offering what one participant described as “words to describe what I’m feeling.” This linguistic framing presents a double-edged sword: while facilitating expression, it risks subtly reshaping or even supplanting personal values through AI-influenced language.
The research underscores the critical need for transparency in AI decision-making, active user control, and careful logging of interactions to maintain clear boundaries of authorship and autonomy. The team proposes a spectrum of AI roles—from simple elicitors of values to full proxies—with the most promising position being that of an advocate who amplifies an individual’s wishes without replacing their human judgement.
Given the sensitive, high-stakes context, legal frameworks and integration with national ACP documents and medical records would be essential before real-world deployment. Such steps would ensure the AI’s recommendations are not only trusted but also actionable when moments of care arise. Despite these hurdles, the prototype elicited genuine interest from participants, many seeing potential in the AI as a catalyst for difficult conversations with aging family members.
Ultimately, ACPAgent serves as a pioneering probe into autonomy, trust, and the evolving interplay between people and machine intelligence in deeply personal healthcare decisions. Rather than fixating on technical capabilities, this work illuminates what users actually want—and must carefully guard—when designing AI that may someday speak for us at life’s most critical junctures.
Subject of Research: Artificial intelligence in advance care planning and healthcare decision-making
Article Title: Words to describe what I’m feeling: Exploring the potential of AI agents for high subjectivity decisions in advance care planning
News Publication Date: 2026
Web References: https://dl.acm.org/doi/10.1145/3772318.3791335
Keywords
Artificial intelligence, advance care planning, healthcare, autonomy, human-machine interaction, decision support systems
