Finnish Work Performance Test Passes Its First Big Statistical Stress Check

When someone on long-term sick leave is trying to return to work, the stakes of a single assessment can be enormous. Benefits, rehabilitation plans, and even the course of a person’s career can hinge on how accurately a clinician judges their ability to perform a job. Yet measuring work ability is notoriously difficult: it is a complex, multifaceted phenomenon, and the tools used to capture it are not always as reliable as the decisions built upon them. A new study published in the Scandinavian Journal of Occupational Therapy now offers the first rigorous statistical examination of the Finnish version of the Assessment of Work Performance, a structured observation tool designed to show how well a person actually works, not just how well they describe working.

The Assessment of Work Performance, or AWP, was originally developed in Sweden and has now been translated into ten languages, with Finnish being the most recent addition. Unlike interview-based instruments, the AWP relies on direct observation. An occupational therapist watches a client carry out a real or simulated work task and rates fourteen distinct observable skills on a four-point scale, from incompetent to competent performance. These skills fall into three domains: motor skills such as mobility, coordination, and strength; process skills such as time management, planning, and adaptation; and communication and interaction skills such as social contact and information exchange. Crucially, the instrument is not tied to any particular diagnosis. Because it measures the interaction between person and environment, it can be applied to clients with mental health conditions, musculoskeletal problems, or any other work-related challenge, in genuine workplaces or constructed assessment settings.

In the new study, a team led by Jennie Nyman of Metropolia University of Applied Sciences, together with colleagues at Linköping University in Sweden, put the Finnish translation, the AWP-FI, through a demanding statistical technique known as Rasch analysis. Seventeen occupational therapists across Finland, all trained in a one-day session and two follow-up meetings, collected ninety-four assessments as part of their routine clinical work. The clients ranged from 18 to 60 years of age, and a majority of those with reported diagnoses had mental or behavioural disorders. Most assessments were conducted through direct observation, and the overwhelming majority, 84 percent, took place in simulated settings, with tasks such as book binding, household chores, and arranging and indexing magazines standing in for real jobs.

Rasch analysis is a form of modern psychometrics that converts ordinal rating-scale data into interval-level measurements. The method estimates two quantities simultaneously: the ability of each person and the difficulty of each item, placing both on a common scale. If the data behave as the model expects, the instrument can be said to measure a single coherent underlying trait, in this case work performance, in a way that is fair and comparable across individuals. The researchers structured their evaluation around three questions drawn from established psychometric guidance: whether the scale’s difficulty range matched the abilities of the sample, whether a reliable measurement scale had been constructed, and whether the individuals in the sample were being measured reliably and meaningfully.

The headline result was encouraging. The AWP-FI showed an overall fit to the Rasch model, with a non-significant chi-square test indicating that observed responses aligned well with model expectations. Individual items performed within acceptable statistical limits, and, importantly, no item behaved differently for different groups. Tests for differential item functioning, which detect whether an item is biased, for example by being harder for women than for men at the same underlying ability level, came back clean across gender, across direct versus participatory observation, and across simulated versus genuine work tasks. This stability suggests the Finnish instrument can be used confidently with men and women alike, in real workplaces or assessment laboratories, and whether the therapist watches from the side or joins the work activity as a participant.

The analysis was not without complications. Five of the fourteen items showed disordered thresholds, meaning that adjacent response categories were not being used in a consistent, ordered way. In practice, the middle rating of the four-point scale was rarely the most probable response for those items, likely because the observed clients clustered at similar performance levels and the full range of the scale was seldom used. The researchers collapsed the affected categories and reran the analysis, a standard corrective procedure, but they caution that any permanent change to the rating scale should wait for confirmation in larger and more varied samples.

Two deeper issues also emerged. First, targeting was suboptimal: the mean person location of 1.6 sat well above the item mean of zero, and the item difficulty range, spanning roughly from minus 1.3 to 0.9 on the logit scale, was narrow. In plain terms, the clients assessed were generally more capable than the hardest items could measure, leaving the scale without items at the demanding end of the work-performance continuum. The authors attribute this largely to the tasks chosen: most assessments used simulated, non-work-specific activities with relatively low demands. Their clinical recommendation is that assessors think carefully about task selection, since the nature and demands of the chosen task directly shape what the assessment can reveal. Adding harder items could widen the measure, but the authors urge caution, because new items must remain faithful to the instrument’s theoretical foundation in the Model of Human Occupation, and expanding item counts carries its own psychometric risks.

Second, the analysis flagged local dependency among items and indications of multidimensionality. Twenty-four of ninety-one residual correlations exceeded the statistical cut-off, and these correlations clustered neatly within the three AWP domains, suggesting that items within a domain share information beyond what the single underlying trait of work performance can explain. When the researchers compared person estimates derived from different domains, between 11 and 17 percent of respondents fell outside the range expected for a truly unidimensional scale, well above the 5 percent benchmark. Notably, this finding diverges from an earlier evaluation of the Swedish original, which found no multidimensionality. The authors suggest the clinical impact may be minimal, since the domain structure reflects the deliberate theoretical architecture of the Model of Human Occupation, but they recommend that future studies with larger samples apply multidimensional or testlet-based Rasch approaches to confirm the pattern.

On the reliability front, the news was solid. The person-separation index reached 0.83, comfortably above the 0.8 benchmark, indicating the scale can distinguish roughly three clinically meaningful groups of clients. Only four participants, about 4 percent, showed erratic response patterns outside acceptable fit limits, meaning nearly everyone was measured in a statistically coherent way. Taken together, the results provide an initial validation of the AWP-FI and support its use as a valid and reliable observational tool for assessing work performance in Finland, where the Finnish Institute for Health and Welfare has explicitly called for more reliable and valid vocational assessment practices. Used alongside the interview-based Worker Role Interview, whose Finnish version passed a similar evaluation in 2023, the AWP-FI can contribute to the kind of holistic, multi-source assessment that sickness insurance and vocational rehabilitation systems increasingly demand.

The study’s limitations are candidly acknowledged. The sample of ninety-four assessments is modest, the DIF analysis suffered from very unequal group sizes, with simulated tasks dominating at 84 percent, which raises the risk that genuine item bias could have gone undetected, and the narrow spread of client performance likely constrained several of the analyses. The authors therefore frame this as a first step rather than a final verdict, recommending further testing with larger and more diverse samples to resolve the local dependency and targeting issues. For clinicians, the practical message is immediate and actionable: the AWP-FI works, but its usefulness depends on choosing work tasks demanding enough to reveal the full range of a client’s capability, and on remembering that the fourteen observed skills are organised into three theoretically distinct domains that should inform, not obscure, the interpretation of results.

Subject of Research: Psychometric validation of the Finnish version of the Assessment of Work Performance using Rasch analysis

Article Title: Psychometric evaluation of the Finnish version of the Assessment of Work Performance (AWP-FI)

Article References: Nyman, J., Sandqvist, J., Pihlava, J., Ekbladh, E., & Yngve, M. (2025). Psychometric evaluation of the Finnish version of the Assessment of Work Performance (AWP-FI). Scandinavian Journal of Occupational Therapy, 32(1), Article 2555182. https://doi.org/10.1080/11038128.2025.2555182

Image Credits: AI Generated

DOI: 10.1080/11038128.2025.2555182

Keywords: psychometrics, Rasch analysis, occupational therapy, work ability, vocational rehabilitation, AWP-FI, construct validity, differential item functioning, Model of Human Occupation, Finland, assessment tools, work performance