New Scale Measures User Trust in Large Language Models

Researchers at Bournemouth University have developed and validated a new psychometric instrument designed to measure how users perceive the trustworthiness of the large language models (LLMs) they use most frequently. Published in Discover Artificial Intelligence, the study introduces the Perceived Trustworthiness of LLMs scale, abbreviated as PT-LLM-8. This self-report tool consists of eight items that evaluate specific qualities of an AI system, such as truthfulness, safety, and privacy. The scale distinguishes between a user’s willingness to rely on a system and their judgment of the system’s inherent attributes. By focusing on perceived trustworthiness rather than trust itself, the instrument provides a nuanced metric for assessing human-AI interaction in everyday contexts.

The development of the PT-LLM-8 was grounded in the TrustLLM framework, a comprehensive model that synthesizes evidence from over 600 studies to define eight dimensions of LLM trustworthiness. These dimensions include truthfulness, safety, fairness, robustness, privacy, machine ethics, transparency, and accountability. However, the researchers noted that the machine ethics dimension, which refers to the internal moral reasoning of an AI, is not directly observable by end users. To address this, the study replaced machine ethics with compliance with laws and regulations. This adjustment reflects the idea that users often judge an AI’s ethical standing based on its adherence to legal and institutional safeguards rather than its internal design assumptions.

The study recruited 966 participants from the United Kingdom via the online platform Prolific. After applying eligibility screening and data quality checks, a final sample of 752 participants was retained for analysis. The participants were required to be at least 18 years old, reside in the UK, and identify as British. They also had to meet specific criteria regarding their engagement with their primary LLM, defined as the model they used most often. The final sample had a mean age of 28.58 years, with 50.3% identifying as male and 48.8% as female. This demographic profile reflects a relatively young and technologically engaged population, a limitation the authors acknowledge when discussing the generalizability of their findings.

Each of the eight items on the scale was rated on an 11-point scale ranging from 0 to 10. This format was chosen to capture greater variability in perceptions than shorter Likert-type scales, allowing respondents to express more nuanced judgments. The items were phrased to evaluate the outcomes of interaction with the LLM, such as receiving reliable information or having data protected, rather than the internal technical processes of the model. For example, one item asked participants to rate their trust that their primary LLM would provide factually accurate and reliable information. Another focused on the model’s ability to handle personal data securely. This user-centered approach ensures that the scale measures perceptions formed through direct experience with the AI system.

To validate the scale, the researchers conducted exploratory and confirmatory factor analyses on the collected data. The exploratory analysis, performed on a subsample of 376 participants, supported a one-factor structure, indicating that the eight items collectively measure a single latent construct of perceived trustworthiness. The confirmatory analysis, conducted on an independent subsample, confirmed this structure with good model fit indices. The results showed that the scale has high internal consistency, with a Cronbach’s alpha of 0.90 and a composite reliability of 0.91. These values suggest that the items are strongly related and consistently measure the same underlying concept.

Measurement invariance tests were also conducted to determine if the scale functions similarly across different groups. The results supported configural, metric, scalar, and strict invariance across gender. This means that the factor structure, loadings, intercepts, and residuals were equivalent for male and female participants. Consequently, differences in scores between men and women can be interpreted as genuine differences in perceived trustworthiness rather than measurement artifacts. This property is crucial for the scale’s utility in research and applied settings, as it allows for valid group comparisons and monitoring of trust dynamics across diverse populations.

The study also examined the associations between PT-LLM-8 scores and various user characteristics. A regression analysis revealed that self-efficacy, agreeableness, and conscientiousness were positively associated with perceived trustworthiness. This suggests that individuals who are confident in their abilities, cooperative, and dependable tend to view LLMs as more trustworthy. In contrast, extraversion was negatively associated with the scale scores, indicating that more outgoing individuals may apply different evaluative standards to AI systems. The model explained 10.4% of the variance in trustworthiness scores, highlighting that while personality traits play a role, they are not the sole determinants of how users perceive AI trustworthiness.

Contextual factors also influenced perceived trustworthiness. Participants who used their primary LLM in both stationary and mobile settings reported higher scores than those who used it only in stationary environments. Similarly, users who interacted with the LLM through both text and voice reported higher perceived trustworthiness than those who used text alone. Device type, however, was not significantly associated with the scores. These findings suggest that the breadth and modality of interaction, rather than the specific hardware used, are more closely linked to how users evaluate the trustworthiness of their AI tools.

The researchers emphasize that the PT-LLM-8 is designed to capture user perceptions rather than to diagnose the objective technical properties of LLMs. While system-level frameworks like TrustLLM focus on intrinsic properties such as safety and robustness, the PT-LLM-8 measures how these properties are perceived by users in practice. This distinction is important because a system may be technically trustworthy without earning user trust, or it may elicit overconfidence despite limited trustworthiness. By providing a reliable measure of perceived trustworthiness, the scale allows researchers and developers to examine whether technical safeguards are effectively translated into user confidence.

Despite its strengths, the study has several limitations. The validation was based on a sample of British adults, which may limit the generalizability of the findings to other cultural or demographic groups. The participants were also relatively engaged and technologically literate, which might not reflect the experiences of casual or skeptical users. Furthermore, the scale measures perceptions of a single primary LLM, so it does not capture variations in trustworthiness across different models or task contexts. Future research should extend the validation of the PT-LLM-8 to diverse populations and examine its relationship with other constructs, such as general trust and reliance on AI systems.

The development of the PT-LLM-8 represents a significant methodological contribution to the field of human-AI interaction. As large language models become increasingly integrated into personal and professional life, understanding how users perceive their trustworthiness is essential for designing effective and responsible AI systems. The scale provides a practical tool for researchers, practitioners, and developers to monitor shifts in user trust and evaluate the impact of design interventions. By offering a validated instrument that captures the multifaceted nature of perceived trustworthiness, the study facilitates more systematic investigations into the social and psychological dynamics of human-AI relationships.

Beyond personality characteristics, the authors evaluated how practical engagement and user capability relate to perceived trustworthiness through a separate regression analysis. This model accounted for 24.3 percent of the variance in scale scores. Perceived helpfulness in personal life emerged as the strongest single predictor, followed by self-reported competence in operating the tool and helpfulness in professional work tasks. Notably, the reported frequency of model use showed only a positive but non-significant relationship with trustworthiness ratings once helpfulness and competence were controlled. This contrast indicates that perceived trustworthiness does not automatically accumulate through sheer repetition or frequent exposure; rather, it is shaped primarily by meaningful, problem-solving interactions where the system successfully assists with concrete personal or workplace objectives.

Individual item scores also revealed clear variations in how respondents judged distinct aspects of model behavior. Across the eight items, individual means ranged from 5.91 to 7.95 on the eleven-point scale, with standard deviations spanning 1.71 to 2.68. Participants assigned the lowest average rating to the item assessing whether the model handled personal data securely, reflecting persistent public caution regarding privacy protections. Conversely, the item measuring performance across a wide variety of topics received the highest average score. This broad topical versatility explained the largest share of variance and correlated strongly with factual accuracy, safety, and explanatory clarity, suggesting that users view generalist competence as a primary indicator of dependable performance rather than expecting narrow, domain-specific specialization.

From an applied perspective, the researchers outline specific guidelines for administering and interpreting the scale in research and organizational settings. Although exploratory factor analysis demonstrated a dominant first factor accounting for 64.3 percent of the variance, justifying the use of an aggregate composite score, relying entirely on a single summary figure carries operational limitations. A single score can mask divergent movements across separate dimensions, such as an improvement in perceived safety occurring alongside growing doubts about data privacy or accountability. To address this risk, the study recommends a dual analytical strategy: employing the total summed score as an overarching index of user sentiment while simultaneously monitoring item-level diagnostics. This granular tracking allows developers, educators, and governance bodies to detect localized deficiencies, measure targeted interventions, and identify which specific functional attributes govern public acceptance over time.

Subject of Research: Development and validation of a scale assessing perceived trustworthiness in large language models

Article Title: Development and validation of a scale assessing perceived trustworthiness in large language models

Article References: Yankouskaya, A., Barajeeh, B., Babiker, A., AlShakhsi, S., Ma, Y. T., Ho, C. S. M., & Ali, R. (2026). Development and validation of a scale assessing perceived trustworthiness in large language models. Discover Artificial Intelligence, 6(1), Article 1855. https://doi.org/10.1007/s44163-026-02494-4

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02494-4

Keywords: Development, validation, scale, assessing, perceived, trustworthiness, large, language, models