A qualitative investigation into how health researchers interact with instructional computational materials reveals that practical and software-level obstacles often prevent users from grasping core analytical principles. Published in PharmacoEconomics – Open, the study evaluated an R tutorial designed for trial-based economic evaluations in health technology assessment. Although methodological tutorials are widely published by academic specialists to improve analytical quality, transparency, and statistical rigor, applied researchers frequently encounter severe roadblocks long before methodological reasoning can even begin. By examining real-time engagement and post-hoc reflections from practicing researchers, the research team demonstrated that technical barriers surrounding software configuration, file navigation, syntax errors, and code adaptability heavily dictate whether statistical guidelines can be successfully implemented in applied workflows.
Economic evaluations and budget impact analyses are standard components of health technology assessments, providing healthcare decision-makers worldwide with vital evidence to guide efficient resource allocation across pharmaceuticals, medical devices, and public health interventions. To reliably inform policy and avoid inefficient expenditures, the methodological quality of these analyses must be rigorous and robust. However, contemporary literature demonstrates that published evaluations often suffer from critical methodological shortcomings. Trial-based analyses regularly overlook skewed cost data, baseline imbalances in patient characteristics across treatment arms, and missing observations, while model-based assessments frequently lack transparency in data inputs, uncertainty characterization, and structural validation. These widespread deficiencies typically arise from stringent project timelines, resource limits, and an absence of accessible, practical guidance tailored to clinical investigators and applied health economists.
To assess how applied researchers navigate instructional resources designed to resolve these statistical deficits, an investigative team led by Romain Jonathan Collet, Johanna Maria van Dongen, and colleagues conducted a multi-stage qualitative study using an R tutorial published by their group in 2023 as an illustrative case study. This underlying tutorial provides explanatory narrative text, executable R code, and a hypothetical trial dataset to address complex challenges such as baseline covariate adjustments, correlated costs and effects, missing values, and skewed distributions. The qualitative investigation engaged twenty researchers across early-, mid-, and late-career stages from Vrije Universiteit Amsterdam and Amsterdam UMC in the Netherlands, alongside Mahidol University in Thailand, representing the authentic end-users of health economic methods rather than methodologists or software developers.
The study design combined two methodological components: an adapted three-step test interview protocol with seventeen researchers who had no prior exposure to the specific tutorial, and semi-structured qualitative interviews with three researchers who had previously implemented the tutorial in their own applied empirical research projects. The three-step test interview procedure involved direct, real-time observation where participants verbalized their thought processes while working through the tutorial files using a think-aloud technique. This was immediately followed by a focused interview using non-leading probes to explore specific troubleshooting behaviors and actions, culminating in a semi-structured reflective dialogue regarding overall clarity, usability, and design enhancements. Interviews were conducted either in person or remotely via Microsoft Teams using screen sharing, with verbatim transcripts systematically analyzed via reflexive thematic analysis to actively construct coherent descriptive themes grounded in participant experiences.
The participant cohort comprised twenty researchers, including seventeen based in the Netherlands and three in Thailand, with ages ranging from 24 to 57 years and a median age of 31. Eighteen participants identified as female and two as male. Among the seventeen participants who completed the three-step test interviews, twelve held master’s degrees as their highest qualification, and the majority reported being beginners or completely unfamiliar with R software (82 percent) and economic evaluations (88 percent), despite all having participated in or preparing to conduct trial-based economic evaluations. The three-step interview sessions ranged from 42 to 111 minutes with a median duration of 61 minutes, while the complementary interviews with prior tutorial users lasted between 14 and 27 minutes with a median duration of 18 minutes.
Through rigorous reflexive thematic analysis, the investigators identified three primary overarching themes: getting in and getting it to run, understanding and making sense of what tutorials teach, and adapting and applying tutorials in practice. The first theme illuminated substantial practical and technical friction encountered prior to any analytical work. Novice users found the preliminary setup of R and RStudio highly intimidating and time-consuming, requiring ad-hoc web searches to complete installation. Many participants struggled to extract compressed repository archives, locate the correct project files, or understand basic environment workflows, frequently opening isolated scripts instead of project directories and triggering path errors. Long code execution runtimes generated severe confusion for beginners who could not verify whether scripts were actively executing, crashed, or completed, whereas experienced programmers recognized execution duration as a normal consequence of computational complexity.
Interpreting console error messages and system warnings presented another persistent challenge within the initial operational phase. When unexpected script errors emerged, novice researchers routinely continued running downstream code chunks without resolving underlying issues or felt paralyzed by obscure terminology. To overcome installation and execution obstacles, participants frequently relied on external support channels, including informal consultations with departmental statisticians, previous academic course materials, and generative artificial intelligence tools such as ChatGPT to diagnose syntax errors. Furthermore, structural alignment between the narrative manuscript and computational files proved vital: participants valued numbered code chunks that closely mirrored the published paper’s progression, noting that typographical errors or discrepancies between printed text and online repositories severely degraded user confidence.
The second major theme captured cognitive challenges in comprehending methodological principles and interpreting analytical outputs. While experienced practitioners found the specialized terminology appropriate, beginners struggled with domain-specific jargon, abbreviations such as Quality-Adjusted Life Years, and advanced concepts like utility scores, multiple imputation, regression modeling, and bootstrapping. Participants widely expressed a preference for explicit, transparent base code over packaged, wrapper-style functions, viewing the latter as opaque black boxes that obscured underlying operational mechanics. Participants also reported difficulty locating generated tables and statistical graphics within the RStudio interface. Intriguingly, many researchers relied on error-free code execution as a proxy for methodological accuracy, assuming that if a script executed without warnings, the underlying statistical procedures were correctly implemented, regardless of their own capacity to verify intermediate calculations.
The third theme centered on the practical adaptation of tutorial workflows to real-world datasets, an endeavor participants characterized as vastly more demanding than stepping through pre-packaged hypothetical examples. Modifying sample code required fundamental programming fluency, substantial analytical time, and independent verification capabilities that novice researchers rarely possessed. Consequently, participants expressed deep doubts regarding the validity of modified outputs, with some expressing a preference for restructuring their empirical datasets to fit the rigid tutorial format rather than altering the tutorial scripts. Facilitating factors for real-world translation included cross-checking derived results against established software like STATA, obtaining hands-on mentoring from experienced colleagues, and consulting generative artificial intelligence tools to adapt code syntax to novel data structures.
While generative artificial intelligence emerged as a frequent coping mechanism for troubleshooting syntax and explaining code chunks, the researchers highlighted critical operational risks associated with overreliance on automated chatbots. Inexperienced investigators often lack the domain-specific background required to formulate precise prompts, critically appraise machine-generated code, or detect subtle methodological flaws, underscoring that artificial intelligence cannot replace formal methodological training or expert supervision. Methodological tutorials remain highly valuable, efficient educational instruments when accompanied by clear architectural design, plain language, and transparent code, but their successful deployment fundamentally depends on addressing user capability, environment access, and ongoing institutional guidance.
The study authors acknowledged key methodological strengths and limitations within their research design. Strengths included the combination of real-time observational testing and practice-based retrospective interviews, alongside investigator triangulation and participant checking. However, focusing solely on a single R-based tutorial evaluated primarily within academic centers across the Netherlands and Thailand may limit direct transferability to alternative computing languages such as Python, STATA, SAS, or SPSS, as well as lower-resource non-academic environments. Furthermore, potential participant self-selection and the dual role of two investigators as tutorial co-authors were transparently noted. The researchers concluded that structured design guidance, including formalized development checklists, should be established to ensure future methodological tutorials remain accessible, transparent, and realistically adaptable for applied researchers worldwide.
Subject of Research: Applied researchers' practical usability and engagement with an R tutorial for trial-based economic evaluation
Article Title: How Applied Researchers Engage with Methodological Tutorials: A Three-step Test Interview Study of an R Tutorial for Trial-Based Economic Evaluation
Article References: Collet, R. J., Leek, R., de Wit, G. A., Ben, Â. J., Bosmans, J. E., & van Dongen, J. M. (2026). How Applied Researchers Engage with Methodological Tutorials: A Three-step Test Interview Study of an R Tutorial for Trial-Based Economic Evaluation. PharmacoEconomics – Open, 10(6), 1039-1050. https://doi.org/10.1007/s41669-026-00677-4
Image Credits: AI Generated
DOI: 10.1007/s41669-026-00677-4
Keywords: health technology assessment, economic evaluation, R programming, methodological tutorials, three-step test interview, qualitative research, reflexive thematic analysis, trial-based economic evaluation, reproducible research, statistical methods, RStudio, health economics, implementation barriers, code adaptation, generative AI in research, software usability, data analysis workflows
Cite Scienmag News
Ophelia Keating. (November 21, 2026). Study Finds Technical Hurdles Limit Applied Researchers Using Methodological R Tutorials. Scienmag. https://scienmag.com/study-finds-technical-hurdles-limit-applied-researchers-using-methodological-r-tutorials/
Ophelia Keating. "Study Finds Technical Hurdles Limit Applied Researchers Using Methodological R Tutorials." Scienmag, 21 November 2026, https://scienmag.com/study-finds-technical-hurdles-limit-applied-researchers-using-methodological-r-tutorials/. Accessed 21 November 2026.
Ophelia Keating. "Study Finds Technical Hurdles Limit Applied Researchers Using Methodological R Tutorials." Scienmag. November 21, 2026. https://scienmag.com/study-finds-technical-hurdles-limit-applied-researchers-using-methodological-r-tutorials/








