Mental health apps often match controls, but that doesn’t mean they fail

Mental health apps have become one of the most visible promises of digital medicine: inexpensive, scalable tools that put evidence-based therapy techniques into the pockets of millions of people who might otherwise never see a clinician. Yet the field has been haunted by a recurring and discouraging finding. In randomized controlled trials, many of these apps fail to outperform their control conditions. Reviewers, funders and the press often read that result as a verdict: the app does not work. A new Comment published in Nature Mental Health argues that this verdict is frequently wrong, and that the way the field evaluates digital interventions is systematically misreading what a failed comparison against a treatment control actually means.

The article, authored by Ashley D. Kendall of the University of Illinois Chicago, Noah Robinson of XRHealth USA, Robin J. Mermelstein of the University of Illinois Chicago, and Steven D. Hollon of Vanderbilt University, makes a deceptively simple point with far-reaching consequences. When a mental health app is tested against a treatment control — a comparison condition that itself contains some active therapeutic ingredient, such as psychoeducation, mood tracking, relaxation exercises or generic support — the trial is not measuring whether the app works. It is measuring whether the app works better than another intervention that may already deliver a meaningful share of the benefit. A null result in that context is not evidence of treatment failure. It may simply be evidence that two modestly effective treatments are, on average, equally effective.

The authors ground this argument in decades of psychotherapy research, a literature that digital mental health has often ignored when importing the randomized controlled trial as its gold standard. Landmark studies in face-to-face therapy, including the widely cited 2010 JAMA analysis by Fournier and colleagues on antidepressant medication and psychotherapy, established long ago that the difference between an active treatment and a credible control tends to shrink as controls become more structurally similar to treatments. Meta-analytic work by Cuijpers and colleagues on psychotherapy for depression reached the same conclusion: the choice of control condition can swing effect sizes dramatically, and pill placebos, psychological placebos and waitlists produce systematically different estimates of how well a therapy works. In other words, the problem Kendall and her colleagues describe is not unique to apps. It is an old and well-understood feature of clinical trials design that the app field has largely failed to absorb.

The technical logic is straightforward. A treatment control in a mental health app trial is not inert. It typically involves contact with researchers, expectation of benefit, structured engagement with content derived from therapeutic principles, and regular self-monitoring — all of which are known to produce symptom improvement on their own. When the active app and the control app both reduce symptoms by a comparable amount, the trial’s statistical machinery reports no significant difference. But the absolute benefit experienced by participants in both arms may be substantial. Reporting that result as a treatment failure conflates comparative advantage with clinical efficacy, two entirely different questions. The first asks whether the app beats a competitor; the second asks whether it helps people relative to no treatment at all. Treatment controls are designed to answer the first question, yet their results are routinely interpreted as answers to the second.

This interpretive habit has real-world costs. Apps that fail to beat treatment controls may be deprioritized by funders, rejected by regulators, passed over by health systems and dismissed by clinicians, even when the trials show that users improved meaningfully. Meanwhile, the enormous unmet need in mental health care — driven by workforce shortages, cost barriers and geographic inequity — means that even modestly effective scalable tools could deliver enormous public health value. The Comment argues that holding apps to a standard of superiority over other active interventions is a standard that many established, widely reimbursed psychotherapies would themselves struggle to meet, particularly against well-constructed psychological placebos. Chambless and Hollon’s influential 1998 framework for establishing empirically supported treatments recognized this problem nearly thirty years ago, emphasizing that efficacy must be demonstrated against appropriate comparison conditions and that equivalence to an established treatment can itself constitute evidence of effectiveness.

The authors also point to a related distortion: expectation effects and common factors. Any intervention that engages a person in a structured effort to feel better recruits hope, motivation, self-efficacy and the therapeutic alliance in diluted form. These ingredients are not noise to be controlled away; they are part of why psychological treatments work. A treatment control deliberately packages many of them. When both arms of a trial contain them, the measured difference between arms reflects only the incremental contribution of the app’s specific content over and above a foundation of generic benefit. For interventions whose mechanisms overlap heavily with those of their controls — as is often the case for apps built on behavioral activation, cognitive restructuring or mindfulness — small incremental effects are close to a statistical inevitability, not a signal of clinical worthlessness.

What, then, should the field do instead? The Comment calls for a diversification of trial designs rather than an abandonment of rigor. Single-case experimental designs, in which each participant serves as their own control across staggered phases of exposure to the intervention, can isolate within-person change with far fewer participants and are well suited to digital platforms that can randomize in real time. Sequential multiple assignment randomized trials, described in recent methodological reviews by Collins and colleagues, allow researchers to evaluate adaptive interventions in which the app’s content changes in response to a user’s progress. Hybrid effectiveness-implementation designs, formalized by Curran and colleagues, evaluate both whether an intervention works and how it functions in real-world care settings, providing information that traditional efficacy trials systematically omit. Dismantling and additive designs can determine which components of an app contribute to benefit, rather than asking only whether the whole package beats a competitor.

The authors also emphasize the value of alternative control strategies borrowed from other areas of medicine. Work by Mulla, Guyatt and colleagues on the interpretation of trials against active comparators shows that non-inferiority and equivalence designs, properly powered and pre-specified, can establish that a new treatment preserves most of the benefit of an existing one while offering advantages in cost, access or tolerability. For mental health apps, whose defining advantages are scalability and low marginal cost, demonstrating equivalence to clinician-delivered care in a non-inferiority framework may be a far more informative and honest question than chasing small superiority effects against enriched controls. Furukawa and colleagues’ work on minimally important differences provides the statistical scaffolding: what matters clinically is not whether a p-value crosses a threshold, but whether the difference between arms — or the improvement within an arm — exceeds the smallest change patients would consider meaningful.

The timing of this intervention matters. The digital mental health sector has matured rapidly, with recent syntheses by Torous and colleagues in World Psychiatry and by Linardon and colleagues mapping a crowded landscape of apps, growing regulatory attention and persistent uncertainty about which products deserve clinical endorsement. A 2022 analysis by Goldberg and colleagues found that while app-based interventions show pooled benefits for depression and anxiety, effects measured against active controls were markedly smaller than those measured against waitlists — precisely the pattern the new Comment predicts. As health systems begin to prescribe apps and payers consider reimbursement, getting the evaluation framework right is no longer an academic quibble. Misreading null superiority results could throttle the pipeline of accessible care just when demand for it has never been higher.

None of this means the authors are asking for a lower bar. They are asking for a more precise one. An app that helps users improve but does no better than a thoughtful control deserves scrutiny of its mechanisms, its engagement design and its incremental value — not a blanket verdict of failure. Conversely, the framework they propose would still condemn apps that produce no meaningful improvement at all, and non-inferiority and single-case designs are arguably more demanding than the status quo, requiring explicit specification of what counts as a clinically meaningful change. The core message is statistical as much as clinical: a null difference between two active treatments is an uninformative result about efficacy, and decades of psychotherapy research already knew it. Digital mental health, the Comment suggests, will only fulfill its promise once its trials start asking the questions its technology can actually answer.

Subject of Research: Evaluation methodology for mental health app clinical trials

Article Title: Failure to outperform treatment controls does not imply treatment failure for mental health apps

Article References: Failure to outperform treatment controls does not imply treatment failure for mental health apps. (n.d.). https://doi.org/10.1038/s44220-026-00715-4

Image Credits: AI Generated

DOI: 10.1038/s44220-026-00715-4

Keywords: mental health apps, digital mental health, treatment controls, randomized controlled trials, clinical trial design, psychotherapy research, non-inferiority trials, effect sizes, Nature Mental Health, outcomes research, implementation science, depression and anxiety