Students Rate AI Article Recommender Below Usability Benchmark in Pilot Study

Undergraduate students evaluating an experimental academic literature recommendation tool rated its usability below standard benchmarks, according to a pilot study published in Scientific Reports. Conducted by researchers Ahmad Hassan Afridi, Sumaira Johar, Naveed Ahmad, Tariqullah Jan, and Ruhul Amin Khalil, the single-institution evaluation examined ScholarMind, an interface prototype in which a large language model generates academic article recommendations directly for users. When participants evaluated their overall user experience, the system achieved a mean score of 56.6 on the System Usability Scale, trailing well behind the widely recognized industry usability benchmark of 68. Furthermore, when the participating undergraduates were asked to state their comparative preference between the new prototype and the established Google Scholar search platform, their selections were evenly split.

The research focused on how readers perceive interactive controls, addressing serendipity as an interface-design question rather than purely as an algorithmic property, which has been the primary focus of prior recommender system studies. ScholarMind incorporated three specific experimental features into its user interface: interactive sliders intended to adjust accuracy and serendipity, cognitive-level classifications assigned to academic articles using Bloom’s taxonomy labels, and an explanation panel displaying model-generated weights presented as SHAP values. However, an inspection of the underlying source code revealed an important implementation constraint: rather than recalculating or retrieving new recommendations dynamically, the sliders functioned solely to reorder a fixed set of previously generated articles.

To assess reactions to these components, the authors recruited forty-one undergraduate students at a single academic institution in Pakistan. Every participant in the trial completed a mandatory evaluation workflow that required them to interact with all three experimental interface features. Following the completion of this task, the students completed a detailed self-report battery combining established instruments with custom questionnaires. The measurement protocol included the Recommender systems Quality questionnaire (ResQue), the System Usability Scale (SUS), the short version of the User Experience Questionnaire (UEQ-S), and three separate, unvalidated scales created to evaluate each of the three individual interface features.

Across the unvalidated feature assessments, mean participant ratings fell on the agreement side of the scale midpoint, ranging between 3.62 and 3.80 on five-point rating scales. Despite these ostensibly positive initial ratings, the researchers detected evidence of acquiescent responding among the students, as reverse-worded questionnaire items correlated negatively with their corresponding scales. The study also evaluated exploratory associations between specific feature impressions and broader attitudes. Perceptions of the explanation panel demonstrated a moderate positive correlation with perceived trust (rₛ = 0.47, 95% confidence interval [0.18, 0.68]). Concurrently, student perceptions of the LLM-assigned Bloom’s taxonomy labels were positively correlated with perceived usefulness (rₛ = 0.51, 95% confidence interval [0.23, 0.71]).

The authors emphasized critical constraints regarding how these results should be interpreted, noting that the empirical findings describe subjective perceptions of an interface prototype rather than the objective performance or information retrieval quality of the recommender system itself. Because the trial was conducted as an exploratory pilot involving forty-one undergraduates at a single institution, the findings provide preliminary insight into how learners navigate explanations and control mechanisms in generative literature tools without demonstrating broader algorithmic efficacy.

Subject of Research: Undergraduate perceptions of interface features in an LLM-based academic article recommender

Article Title: User perceptions of serendipity-control, cognitive-level labelling, and explanation features in an LLM-based academic article recommender: a single-institution pilot study

Article References: Afridi, A. H., Johar, S., Ahmad, N., Jan, T., & Khalil, R. A. (2026). User perceptions of serendipity-control, cognitive-level labelling, and explanation features in an LLM-based academic article recommender: a single-institution pilot study. Scientific Reports. https://doi.org/10.1038/s41598-026-78243-5

Image Credits: AI Generated

DOI: 10.1038/s41598-026-78243-5

Keywords: ScholarMind, academic article recommender, large language models, serendipity control, Bloom's taxonomy labels, explanation panel, SHAP values, System Usability Scale, ResQue, UEQ-S, interface design, acquiescent responding, perceived trust, perceived usefulness, Google Scholar, pilot study, undergraduate perceptions