Osteopathic medical students test three AI platforms for pharmacology learning

When four first-year osteopathic medical students and a pharmacology faculty member at Baptist University College of Osteopathic Medicine decided to interrogate three leading artificial intelligence platforms with the same pharmacology questions, they expected differences. What they found was a striking gap in the quality of the evidence behind the answers. Consensus Deep, a research-focused subscription service costing 65 dollars a month, returned references that were 100 percent peer-reviewed for one query and scored highest on both credibility and verifiability. Perplexity Pro, at 17 dollars a month, drew freely from blogs, websites, and even social media posts, dragging its credibility scores down. ChatGPT Plus, at 20 dollars a month, produced readable, patient-centered summaries but often cited sources that needed independent checking. The study, published in Frontiers in Pharmacology on 27 October 2026, offers one of the first structured looks at how paid generative AI tools perform when osteopathic medical students use them to learn pharmacology.

The team designed three queries that map onto real challenges in pharmacological education. The first asked about the risks of long-term use of GLP-1 receptor agonists for weight loss or diabetes, a topic dominating current public and clinical attention. The second asked whether osteopathic manipulative treatment, or OMT, reduces or supplements opioid use in chronic pain management, a question that sits at the intersection of pharmacology and osteopathic philosophy. The third asked what prevention measures exist for adverse drug reactions in geriatric populations with polypharmacy when a whole-person approach is used. Each query was submitted to Consensus Deep, Perplexity Pro, and ChatGPT Plus, all accessed through one-month paid subscriptions during April to June 2026.

The methodology was deliberately careful. Three researchers were assigned to each query, and every prompt was entered at least twice on each platform at different times on different days. Prompt wording was refined through trial runs, and the same final wording was used by all researchers for consistency. Follow-up questions, when asked, were standardized across the team. A faculty coauthor independently ran every query on every platform and compared the outputs, although the reference scoring itself was done by the four students. Because the researchers submitted queries independently, the sets and numbers of references returned varied even for the same platform and query, a finding that itself says something about the reproducibility of generative AI outputs.

To evaluate what the platforms produced, the team built a two-dimensional scoring system. Reference credibility was scored from 1 to 3: a blog, commercial website, news article, or opinion piece earned a 1; a reputable medical source such as a StatPearls chapter or professional organization page earned a 2; and peer-reviewed original research, systematic reviews, meta-analyses, randomized controlled trials, or clinical practice guidelines earned a 3. Reference verifiability was scored on a parallel scale: a fabricated or nonexistent reference earned a 1, an existing but hard-to-trace reference earned a 2, and a fully traceable reference with clear authors, dates, and claims earned a 3. Discrepancies between raters triggered reevaluation by the whole team.

Inter-rater reliability was assessed with a one-way random-effects intraclass correlation coefficient, chosen because the rotating pool of raters did not require the same three people to score every output. The results were reassuring for the scoring system itself. The three-rater average showed good reliability for credibility, ICC(1,k) = 0.87 with a 95 percent confidence interval of 0.62 to 0.97, and excellent reliability for verifiability, ICC(1,k) = 0.93 with a 95 percent confidence interval of 0.80 to 0.98. Single-rater reliability was moderate for credibility at 0.70 and good for verifiability at 0.83. Both omnibus F-tests were statistically significant at p less than .001, indicating that differences between outputs far exceeded differences between raters.

The first query revealed the sharpest contrasts. For GLP-1 receptor agonists, ChatGPT Plus returned an average of 15 references, only 49 percent of which were peer-reviewed, with credibility and verifiability averages of 2.38 and 2.83. Notably, most of its pharmacological references addressed semaglutide specifically rather than the broader drug class, omitting agents such as liraglutide or dulaglutide. Perplexity Pro returned an average of 23 references spanning scientific articles, websites, blogs, YouTube videos, and Facebook posts, with only 39 percent peer-reviewed and a credibility average of just 1.95. Consensus Deep delivered exactly 20 references for each researcher with no repeats, 100 percent peer-reviewed, and perfect verifiability scores of 3. It also stated transparently how many articles it retrieved, found eligible, and included, and annotated each citation with article type, quality ratings, and a description of its contribution.

The second query, which explicitly mentioned OMT, predictably drew osteopathic-affiliated references from all three platforms. ChatGPT Plus framed OMT as an adjunctive, nonpharmacological therapy rather than a guaranteed replacement for opioids, but its response remained brief and broad, and the authors note that no evidence-supported claims were provided. Perplexity Pro produced a concise summary emphasizing OMT as a potentially opioid-sparing adjunct, drawing on a small number of references focused mainly on low back pain and clearly distinguishing observed trends from the lack of large, high-quality trials quantifying opioid dose reduction. Consensus Deep again led on source quality, with 92 percent peer-reviewed references and an academically grounded narrative citing controlled and observational trials in which OMT was associated with decreased postoperative opioid consumption and reduced opioid use in chronic lower back pain.

The third query exposed a different weakness. On adverse drug reaction prevention in geriatric polypharmacy, all three platforms listed standard measures such as medication reconciliation, deprescribing, the Beers Criteria, STOPP/START criteria, dose adjustment, and caregiver involvement. But peer-reviewed reference percentages collapsed: 14 percent for ChatGPT Plus, 30 percent for Perplexity Pro, and 98 percent for Consensus Deep. Osteopathic affiliation was essentially absent across all outputs. The authors attribute the richer content on GLP-1 agonists, compared with the thinner coverage of geriatric polypharmacy, to the sheer volume of internet attention surrounding weight-loss drugs rather than to any true difference in platform capability, a caution about confusing topical popularity with evidentiary depth.

Beneath the numbers lies a structural problem for osteopathic education. The tenets of osteopathic philosophy, including the body as a unit of body, mind, and spirit, the capacity for self-healing, and the reciprocal relationship between structure and function, are underrepresented in the large language model training datasets because the osteopathic literature itself is a small fraction of the biomedical record. The authors cite the Association of American Medical Colleges 2026 workforce dashboard showing that DO physicians constitute 8.2 percent of total physicians. When platforms were asked about topics without explicit osteopathic framing, they defaulted to conventional pharmacologic priorities, and the authors note the risk of bias toward prescription drug interventions over combined OMT and cognitive behavioral therapy for chronic pain.

The study’s limitations are candidly acknowledged. The results capture a snapshot of rapidly evolving platforms; the same query can yield different outputs within the same week, and platform memory influenced ChatGPT Plus responses while Consensus Deep generated answers independently. Prompt engineering skill matters, and the team, though experienced users, lacked formal training in it. Cost is a barrier: the best-performing platform was also the most expensive, meaning a student’s experience may depend on subscription affordability. Perplexity Pro’s Model Council feature, which routes queries to different underlying models, and ChatGPT’s selectable modes add further variability, and the opaque reasoning pathways of all platforms complicate reproducibility. OpenEvidence, restricted to licensed clinicians and medical students, was excluded. The authors’ recommendations are practical: be specific with prompts, revise them iteratively, establish context before asking substantive questions, match the platform to the task, treat AI as a supplement rather than a sole source, and always investigate where the answers come from. In pharmacology, where errors can reach patient care, critical appraisal of AI output is not optional.

Subject of Research: Comparative evaluation of three AI platforms for pharmacology education in osteopathic medical school

Article Title: To chat or not to chat: how osteopathic students utilize artificial intelligence platforms in pharmacology education

Article References: Barr, L., Bakeer, S., Bemowski, B., Orio, S., & Zaveri, N. (2026). To chat or not to chat: how osteopathic students utilize artificial intelligence platforms in pharmacology education. Frontiers in Pharmacology, 17, 1930362. https://doi.org/10.3389/fphar.2026.1930362

Image Credits: AI Generated

DOI: 10.3389/fphar.2026.1930362

Keywords: artificial intelligence, pharmacology education, osteopathic medicine, medical students, large language models, ChatGPT, Perplexity, Consensus, reference credibility, OMT, polypharmacy, GLP-1 receptor agonists