Validated LLM thematic-analysis tutorial shows high human-model agreement in Reddit posts

A structured, human-in-the-loop workflow for large language model-assisted thematic analysis achieved high agreement between human and LLM coding under controlled conditions, according to a tutorial in Behavior Research Methods. The demonstration analyzed 2,952 publicly available Reddit posts and reported a Cohen’s kappa of 0.91. This was a methodological demonstration rather than a general assessment of language-model performance: the agreement was observed within the study’s defined workflow, data and validation conditions.

The tutorial organizes the work into three phases. First, researchers manually analyze a subset of the data to establish an initial codebook and assess which patterns are relevant. Second, the same subset is analyzed with an LLM, allowing researchers to refine prompts iteratively and evaluate the alignment between human and model outputs. Third, the validated approach is applied to the full corpus. This sequence places the model’s proposed interpretations against a human-developed benchmark before they are extended across the larger dataset, while keeping prompt engineering as an ongoing part of the analytic process rather than treating instructions as a one-time technical step.

For the full-corpus analysis, the workflow uses a modular four-step pipeline: extracting key ideas, assigning thematic codes at the chunk level, consolidating codes into themes, and clustering themes. Structured prompt design and JSON-formatted outputs are central to this design. They support traceability between the extracted material, codes, themes and source data, allowing a researcher to follow how an interpretation is represented across the pipeline. The tutorial also identifies hallucination and overgeneralization as failure modes and incorporates iterative human verification through random sampling of outputs. Sampling supplies repeated checks on model-generated results during the scaled analysis, rather than relying on a single end-stage review.

The reported Cohen’s kappa of 0.91 summarizes strong agreement for the controlled comparison conducted in the demonstration; it does not establish that the same performance will occur with other models, prompts, corpora or research questions. The tutorial’s stated contribution is therefore a practical framework for integrating LLMs into large-scale qualitative analysis while retaining explicit validation, structured outputs and human review. Its broader reproducibility support includes open availability of prompts, code and reproducibility materials through an Open Science Framework repository, allowing researchers to inspect the documented workflow and assess the approach within their own studies.

Subject of Research: Human-in-the-loop LLM workflows for reproducible large-scale thematic analysis

Article Title: Human-in-the-loop LLM-assisted thematic analysis: A validated, reproducible tutorial for large-scale qualitative research

Article References: Jafarian, M. (2026). Human-in-the-loop LLM-assisted thematic analysis: A validated, reproducible tutorial for large-scale qualitative research. Behavior Research Methods, 58(11), Article 329. https://doi.org/10.3758/s13428-026-03207-6

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03207-6

Keywords: large language models, thematic analysis, qualitative research, human-in-the-loop AI, research reproducibility, prompt engineering, Reddit posts, Cohen’s kappa, open science, Human-in-the-loop, LLM-assisted, thematic