Large language model prompting strategies show no stable advantage for predicting SDS screen-positive status in university students

A comparative benchmark found that large language model prompting strategies were competitive with logistic regression and XGBoost at some operating points, but did not demonstrate a stable or general advantage for predicting questionnaire-defined SDS screen-positive status among university students. The study used de-identified records from 6,395 undergraduates at Huangshan University in China, and its findings apply to screening-threshold prediction rather than diagnosis of depressive disorder.

Researchers defined the outcome as a standard score of at least 53 on the 20-item Zung Self-Rating Depression Scale, or SDS. The predictors were sex, age, scores from the Self-Rating Anxiety Scale, Cognitive Fusion Questionnaire and Acceptance and Action Questionnaire-II, and eight non-depression SCL-90 symptom dimensions plus an additional-items score. These inputs comprised 126 non-SDS questionnaire items, so the method did not reduce questionnaire burden relative to directly administering the SDS.

The grade cohorts were assigned distinct analytical roles: 1,951 first-year students formed the Development cohort, 2,186 second-year students formed the Test cohort, and 2,258 third-year students formed the Holdout cohort. Development was divided into training and validation subsets for fitting the traditional models, selecting XGBoost calibration and choosing supplementary thresholds. Test labels were not used for fitting, calibration or threshold selection, while Holdout evaluated two Test-selected language-model conditions and both structured-data comparators. Because all cohorts came from one university and survey period, Holdout constituted within-institution grade-cohort validation rather than external or temporal validation.

The study compared logistic regression and isotonic-calibrated XGBoost with 12 conditions formed by combining GPT, Claude and Gemini models with four prompt families. These included zero-shot prompts, a rule-informed prompt and two example-augmented conditions containing two or four balanced demonstrations selected from Development-training records. The model calls were made through OpenRouter using fixed model identifiers, prompts and decoding settings, and outputs were required to contain a label, a positive probability and evidence. Hard-label performance was evaluated at 0.5 and at thresholds selected only in Development-validation, alongside discrimination, calibration, structured-output validity, cohort shift, construct overlap, subgroup performance and repeated-call stability.

At the fixed 0.5 threshold on Test, GPT E1, Gemini E2 and XGBoost produced macro-F1 values of 0.732, 0.730 and 0.674, respectively. The confidence intervals for pairwise macro-F1 differences among the three leading language-model conditions included zero, and their differences were only 0.001 to 0.003. Rankings changed when thresholds were selected in Development-validation: Gemini E2 ranked first in 99.9% of paired bootstrap replicates, while its Test macro-F1 remained 0.730 and GPT E1’s fell to 0.702. XGBoost’s Test macro-F1 was 0.660 at its selected threshold of 0.174, with sensitivity rising from 0.316 at 0.5 to 0.871 and specificity falling from 0.979 to 0.622.

Probability quality did not mirror the fixed-threshold ranking. XGBoost had the lowest Brier scores on Test and Holdout, 0.129 and 0.147, and the lowest expected calibration error across the tested binning specifications. Its AUROC was 0.846 on Test and 0.835 on Holdout, compared with 0.818 and 0.808 for GPT E1 and 0.834 and 0.820 for Gemini E2. Calibration intercepts indicated systematic miscalibration for logistic regression and both language-model finalists, so their reported probabilities were not consistently calibrated estimates of risk. In Holdout at 0.5, GPT E1 achieved sensitivity of 0.655, specificity of 0.821 and macro-F1 of 0.729; Gemini E2 achieved 0.723, 0.796 and 0.739, respectively.

Repeated calls exposed a substantial difference in output stability. Across the complete Holdout cohort and 10 calls at temperature 0.1, GPT E1 changed the thresholded prediction for 532 of 2,258 students, an inconsistency rate of 23.6% with a 95% confidence interval of 21.9% to 25.4%. Gemini E2 was inconsistent for 34 cases, or 1.5%, with a 95% confidence interval of 1.1% to 2.1%. GPT E1 instability was concentrated in cases with median probabilities close to 0.5: all 47 cases within 0.05 of the cutoff were inconsistent, compared with 51 of 1,497 cases at least 0.20 from it. In a representative 400-case subset, reducing the temperature to 0 did not clearly remove GPT E1 variability.

The questionnaire predictors also shared substantial measured content. In the combined sample, SDS and SAS scores had a Spearman correlation of 0.60 with a 95% confidence interval of 0.58 to 0.61. Removing SAS reduced the AUROC of logistic regression and XGBoost by approximately 0.025 to 0.029 across Test and Holdout, while removing the non-depression SCL-90 group reduced XGBoost AUROC by 0.019 on Test and 0.018 on Holdout. Models based only on sex and age had AUROCs of 0.531 and 0.538 for logistic regression and 0.498 and 0.521 for XGBoost on Test and Holdout, respectively, compared with 0.844 and 0.832 for full logistic-regression models. These associations describe predictive overlap and are not evidence of causal effects or an independent marker of depression.

Per-1,000 projections further illustrated the operating-point trade-offs. At 0.5 in Holdout, Gemini E2 produced 202 true positives, 147 false positives and 78 missed positives, while XGBoost produced 74 true positives, 17 false positives and 205 missed positives. Lowering XGBoost to its Development-derived threshold increased projected true positives to 245 and false positives to 303, while reducing missed positives to 34. Exploratory sex-stratified results varied by model and metric and did not constitute a comprehensive fairness assessment. Overall, the study establishes an auditable offline benchmark rather than clinical utility: it did not validate diagnoses, external transportability, fairness or reduced questionnaire burden, and it identified repeated-call stability, calibration and threshold choice as separate evaluation dimensions.

Subject of Research: Comparative evaluation of large language model prompting strategies for predicting SDS screen-positive status from structured questionnaire data in university students

Article Title: SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies

Article References: Yang, X., Cao, Y., Wang, H., Li, H., Jiang, X., Xu, H., Yao, T., Zhang, X., Chen, Y., Li, J., Zhou, B., Xu, L., Chen, Z., & Zhu, X. (2026). SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies. Frontiers in Psychiatry, 17, Article 1942164. https://doi.org/10.3389/fpsyt.2026.1942164

Image Credits: AI Generated

DOI: 10.3389/fpsyt.2026.1942164

Keywords: large language models, mental health screening, structured questionnaires, SDS threshold prediction, prompting strategies, calibration, repeated-output stability, XGBoost, university students, screen-positive, prediction, structured