Flinders University experts are warning that artificial intelligence (AI) must be carefully evaluated and governed before it is adopted widely in healthcare, saying rapid advances do not automatically translate into safe use for patients.

In an expert commentary titled ‘AI can reason like a physician; what comes next? published in Science, Flinders researchers caution that while new AI systems show impressive capabilities, strong results in controlled studies do not mean they are ready for routine use in hospitals or clinics.

The authors say there is an urgent need to understand how emerging AI tools can be safely integrated into everyday clinical practice, with patient outcomes remaining the central focus.

Despite these warnings, the researchers acknowledge that recent advances in AI create genuine opportunities to support doctors, particularly in busy and high-pressure care settings.

The commentary reviews new research showing that advanced reasoning-based AI systems can work through diagnostic scenarios step by step and, in some cases, closely match or even exceed the diagnostic performance of experienced doctors.

Erik Cornelisse, a PhD candidate at Flinders University and co-author of the commentary, says this shift marks a move from simple question answering tools towards algorithms capable of seemingly human-like clinical reasoning on text-based tasks.

However, the Flinders team stresses that real world medical care involves far more than text-based reasoning or test performance.

They say clinical practice depends on physical examination, listening to patients, understanding medical and social context, and taking responsibility for outcomes, elements that current AI systems cannot safely provide on their own.

“Health care decisions are complex, high stakes, and deeply human, and accuracy alone, particularly on just text-based cases, does not make a system safe for patients,” says Mr Cornelisse from the College of Medicine and
Public Health.

Senior author Associate Professor Ash Hopkins, an NHMRC Investigator and leader of Flinders’ Clinical Cancer Epidemiology Lab, says modern healthcare relies on judgement, accountability, and ethical oversight.

“AI systems have demonstrated that they can reason through clinical problems with similar performance to doctors, notably on the same scenarios used to train clinicians themselves. This presents genuine opportunities to support clinicians in the future,” says Associate Professor Hopkins.

“Multiple stakeholders are currently working on the frameworks for AI in terms of legal, professional, or moral responsibility for its decisions, and presently there is a critical need for deliberate and controlled integration into clinical care.”

The commentary highlights known risks linked to poorly evaluated systems, including bias, inequitable care, and unintended patient harm.

“History shows that algorithms can worsen outcomes when deployed without sufficient safeguards and can amplify problems as easily as they solve them, particularly when systems are trained on incomplete or unrepresentative data,” says Mr Cornelisse.

Looking ahead, the Flinders researchers argue that enthusiasm for medical AI must be matched by strong governance and clearer standards for evaluation.

“We do not allow doctors to practise without supervision and evaluation, and AI should be held to comparable standards,” says Mr Cornelisse.

The researchers stress that improvement in real patient outcomes, not exam scores, benchmarks, or demonstrations, must be the true measure of success.

Associate Professor Hopkins says AI holds enormous promise but must be applied responsibly.

“Patients deserve technology that improves care in the real world, not systems that only look impressive in studies,” he says.

“With careful design, strong oversight, and rigorous evaluation, AI could become a powerful tool to deliver safer, fairer, and more effective care across health systems worldwide,” concludes Associate Professor Hopkins.

The paper, ‘AI can reason like a physician; what comes next?’, by Ashley M. Hopkins and Erik Cornelisse is published in SCIENCE.  DOI 10.1126/science.aeg8766 (link live after embargo lifts)

Journal

Science

Funder

National Health and Medical Research Council

DOI

10.1126/science.aeg8766

Dr. Maria Alejandra Pinero de Plaza, who spearheaded the research, emphasizes that while AI is increasingly being adopted in healthcare frameworks, it is essential that such systems are compatible with the workflows of medical staff. In various acute settings, the availability of timely and accurate diagnoses can significantly affect patient outcomes. Therefore, ensuring that AI tools work seamlessly with doctors and nurses rather than hindering their critical tasks is of paramount importance.

The introduction of PROLIFERATE_AI, a unique evaluation framework, is a cornerstone of this research. This human-centered tool not only employs AI but also involves thorough analysis by researchers to understand how AI can be optimized for clinical environments. The emphasis is not solely on the accuracy of the AI itself; it also examines usability, adoption by healthcare professionals, and the overall impact on patient care, providing a holistic view of AI’s effectiveness in real-time medical scenarios.

The study published in the International Journal of Medical Informatics delves into the operational prowess of the RAPIDx AI tool. Designed to support emergency healthcare practitioners in diagnosing cardiac conditions swiftly, RAPIDx uses clinical inputs and biochemical data to generate reliable assessments. Given that chest pain is one of the top reasons patients visit emergency departments, the necessity for efficient diagnostic tools in these high-pressure environments cannot be overstated.

Conducted within the framework of a trial funded by the National Health and Medical Research Council (NHMRC), the analysis spanned twelve hospitals in both metropolitan and rural South Australia. During the trial, researchers actively engaged with medical and nursing staff, facilitating discussions around their experiences with RAPIDx. Such engagements can provide invaluable insights into the practical challenges and benefits associated with AI tools in healthcare settings.

Findings from the research indicate a pronounced divide in the tool’s usability among different experience levels among clinical staff. Senior medical professionals, particularly consultants and registrars, reported high levels of understanding and engagement with the RAPIDx AI tool. In contrast, less experienced healthcare providers, including residents and interns, encountered significant usability concerns, underscoring the need for further training. This reveals a critical insight: AI systems must be intuitive and accessible to all users in a medical environment, regardless of their expertise level.

Interestingly, registered nurses also expressed a robust emotional connection with the RAPIDx tool, viewing it as a means to enhance patient safety through improved accuracy in diagnoses. This finding reinforces the vital role of nurses in the patient care continuum and highlights the necessity for AI systems to support all members of the healthcare team, ensuring a collaborative approach.

A notable aspect of the PROLIFERATE_AI framework is its commitment to providing actionable insights. Dr. Pinero de Plaza asserts that the utility of AI tools should extend beyond just technical metrics. Evaluating AI based on real-world usability and the trust of clinicians is crucial to guarantee that these advanced technologies are not only forward-thinking but also functional and user-friendly in everyday practice.

The research outlines the clear directive for future AI implementations in healthcare. Efforts should be directed toward creating systems that work in concert with healthcare professionals, facilitating decision-making in high-stakes situations. AI should not act as a replacement for medical expertise but rather as an enriching tool that enhances the capabilities of doctors and nurses, ultimately leading to improved patient care and outcomes.

Moreover, the study sheds light on the necessity for tailored training programs and user interface designs that align with clinical workflows. When deploying AI technologies in emergency medicine, understanding the specific needs of varied healthcare professionals is vital for optimizing both usage rates and user satisfaction. The findings stress that while seasoned clinicians may have embraced the technology, there is a compelling need for additional support structures for less experienced users to ensure successful integrations.

The commitment to advancing AI tools in emergency medicine did not go unnoticed. Last month, the research team was recognized with a CSIRO award through the On PRIME innovation program. The grant serves as an impetus for their ongoing efforts to refine predictive modeling and implementation science, striving for continual enhancements in their approach to AI in healthcare.

In summary, Flinders University’s research represents a significant step toward developing AI solutions that truly enhance emergency cardiac care. By undertaking a comprehensive evaluation of the RAPIDx AI tool through the PROLIFERATE_AI framework, they have contributed to the body of knowledge in medical informatics. The study is a clarion call not only for the adoption of AI in healthcare but also for the thoughtful integration of such technologies, ensuring they are practical and beneficial for all involved in patient care.

These insights will undoubtedly inform future strategies for AI deployment in healthcare settings across the globe, with a focused commitment to improving user experiences and ultimately enhancing patient health outcomes.

Subject of Research: Evaluation of AI in Emergency Cardiac Care
Article Title: Human-centred AI for emergency cardiac care: Evaluating RAPIDx AI with PROLIFERATE_AI
News Publication Date: 28-Jan-2025
Web References: International Journal of Medical Informatics
References: National Health and Medical Research Council (NHMRC), grant number GNT1191914
Image Credits: Flinders University

Keywords: Artificial Intelligence, Emergency Medicine, Cardiac Care, RAPIDx, Healthcare Technology, PROLIFERATE_AI, Patient Safety, Medical Informatics