Juvenile idiopathic arthritis is the most common chronic inflammatory rheumatic condition in children and carries a substantial burden on quality of life. Choosing a therapy is difficult because guideline evidence is limited and the trade-offs between medications can be significant. In a multimethod feasibility study conducted according to the TRIPOD-LLM checklist, researchers evaluated the decision aid, which was built on OpenAI’s o3 model, across ten de-identified clinical use cases.

Three pediatric rheumatologists scored the tool’s outputs on safety, factual consistency, treatment appropriateness, and how well it acknowledged patient preferences. They judged 97 percent of the outputs acceptable for safety and for preference acknowledgment. Factual consistency fared worse, with 60 percent of outputs rated acceptable, and 70 percent were rated acceptable for treatment appropriateness.

Agreement between the evaluators themselves varied by criterion. Inter-rater agreement among the rheumatologists was almost perfect for safety and preference acknowledgment, moderate for treatment appropriateness, and poor for factual consistency. An exploratory assessment using a second model, OpenAI’s GPT-5, as an automated judge (LLM-as-a-Judge) left these agreement levels the same or improved them, suggesting the approach may help scale evaluation of AI-generated clinical content.

One finding stood out: agreement on actual treatment recommendations, measured across all human evaluators and real-world data, was low, at less than 30 percent. The authors suggest this variability may reflect the inherent heterogeneity of acceptable treatment pathways in juvenile idiopathic arthritis, where several different drugs can be reasonable choices, as well as potential limitations in the decision aid’s decision-making logic.

The patient-facing side of the study used a mixed-methods evaluation with 34 participants to characterize the tool’s perceived value. Five themes emerged from the analysis. Patients said the tool complements rheumatologist expertise rather than replacing it, prepares them for appointments through reflection and actionable information, and builds confidence for self-advocacy.

Two further themes were more cautious. The tool’s value depended on the patient-physician relationship, indicating it may work better in some clinical dynamics than others. Patients also saw potential for broader use in self-management of juvenile idiopathic arthritis beyond the initial treatment decision.

The researchers describe the results as a proof of concept for using AI to support collaborative decision-making in pediatric rheumatology rather than as evidence the tool is ready for clinical use. Because this was a feasibility study with a small number of use cases and an exploratory patient component, the findings do not demonstrate that the model can reliably support treatment decisions in practice.

What the study does establish is a measurement framework: structured scoring of safety, factual consistency, appropriateness, and preference acknowledgment, checked against both expert rheumatologists and an automated LLM judge, alongside systematic assessment of patient perspectives. Future studies, the authors conclude, are needed to determine whether large language models can reliably support clinical practice in juvenile idiopathic arthritis and similar conditions where evidence is limited and patient values matter deeply.

Subject of Research: Feasibility evaluation of an LLM-based decision aid for collaborative treatment decision-making in juvenile idiopathic arthritis

Article Title: An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation

Article References: Barbaric, A., Benseler, S. M., Lalloo, C., Liu, X. C., Mariakakis, A., Pham, Q., Vastert, S. J., Yeung, R. S. M., Cafazzo, J. A., & Swart, J. F. (2026). An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation. PLOS Digital Health, 5(10), e0001655. https://doi.org/10.1371/journal.pdig.0001655

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001655

Keywords: juvenile idiopathic arthritis, large language models, shared decision-making, pediatric rheumatology, clinical decision support, artificial intelligence, LLM-as-a-Judge, feasibility study, TRIPOD-LLM, patient perspectives, treatment appropriateness, self-advocacy