Lesotho’s School Test Reports Are Credible but Contextually Thin, Study Finds

A new qualitative study has scrutinised the national reports that Lesotho produced for three successive rounds of the Southern and Eastern Africa Consortium for Monitoring Educational Quality, and its verdict is deliberately two-sided. The reports, published after the SACMEQ II assessment of 2000, SACMEQ III of 2007 and SACMEQ IV of 2013, come across as methodologically credible and genuinely useful for tracking broad trends in Grade 6 reading and mathematics. Yet their value for national, context-sensitive policymaking is limited by uneven contextual explanation, incomplete discussion of who was and was not included in the sampled population, restricted dissemination, and limited transparency about some reporting limitations. Crucially, the researchers say, none of this challenges the technical validity of SACMEQ scores themselves; rather, it shows that technically defensible results can still require substantial national interpretation before they adequately support decisions in ministries and classrooms.

The study, published in SN Social Sciences by Musa Adekunle Ayanwale of the University of Pretoria and Maipato Charlotte Khoase of the National University of Lesotho, addresses a question that has received surprisingly little attention in the large literature on international assessments. Researchers have extensively examined the technical quality, comparability and policy use of cross-national testing programmes, but far less work has asked whether a country’s own national report clearly states its evidential boundaries, represents the sampled population faithfully, interprets results against national conditions, and actually reaches the stakeholders expected to act on them. To fill that gap, the authors developed a provisional Authenticity Evaluation Framework built around four reporting functions: methodological transparency, fidelity of representation, contextual interpretation, and policy accessibility and usefulness. The framework is explicitly presented as an analytical heuristic rather than a validated measurement model.

A central conceptual move in the paper is the separation of technical validity from what the authors call report authenticity. Validity, in the psychometric tradition associated with the Standards for Educational and Psychological Testing and Michael Kane’s argumentation framework, concerns whether accumulated evidence supports intended interpretations and uses of scores. Authenticity in assessment literature has traditionally referred to the realism or meaningfulness of assessment tasks, not to reports. The authors extend the term to documents, defining report authenticity as the fidelity, transparency, contextual adequacy and practical interpretability with which a report represents the educational reality within its stated scope. This distinction matters because a testing programme can generate sound scores while a national report communicates them with varying degrees of clarity about scope, uncertainty and implication. The two functions are complementary, and neither substitutes for the other.

The division of labour within the consortium underpins the study’s framing. Regional coordination structures, historically run with the UNESCO International Institute for Educational Planning, established common instruments, standardised administration, central data validation and cross-national reporting. National ministries, research coordinators and country teams were responsible for interpreting the evidence and producing national reports for domestic planning. Consequently, the authors direct claims about omitted context, accessibility or policy translation at Lesotho’s national reporting arrangements rather than at the regional coordination centre. The unit of analysis is the trio of Lesotho’s national reports, not the consortium’s international outputs, and the study does not extend to the consortium’s fifth cycle, now conducted under the name SEACMEQ.

Methodologically, the researchers adopted an interpretivist qualitative case study combining documentary analysis with semi-structured interviews involving four purposively selected senior stakeholders drawn from Lesotho’s Ministry of Education and Training and affiliated institutions. The informants, aged between 48 and 60, had direct experience in curriculum development, educational planning, assessment and quality assurance. Analysis followed inductive content analysis principles, supported by ATLAS.ti 24 for coding and theme development, with the four framework dimensions serving as sensitising concepts rather than predetermined codes. Trustworthiness was addressed through methodological triangulation, an audit trail, reflexive notes and repeated checking of themes against transcripts and reports. The authors are candid that four participants provide depth rather than representativeness, and that interview testimony cannot independently verify the implementation quality of every technical procedure described in the documents.

The documentary evidence confirmed a clear descriptive trend: mean literacy performance rose from 451 in SACMEQ II to 511 in SACMEQ IV, while numeracy climbed from 447 to 514, moving Lesotho from below the SACMEQ reference mean of 500 to slightly above it. Stakeholders interpreted these gains cautiously, distinguishing statistical improvement from the attainment of nationally desired standards. Interviewees accepted that the assessments were sufficiently aligned with the Grade 6 curriculum to support regional comparison, and they praised the data management chain, citing double data entry, national cleaning, central validation at the Coordination Centre and continuity in questionnaire design, with new modules added while core questions were retained. One informant noted that the consortium does not fundamentally change its questionnaires between cycles, which participants saw as supporting comparability, although the authors caution that perceived continuity does not by itself establish measurement invariance or full cross-cycle equivalence.

The limitations clustered around boundaries and context. Participants worried that the reports did not adequately represent learners with disabilities, with one informant observing that children with visual impairment in particular appeared to be excluded, in tension with the policy of inclusion. The authors are careful to note that a precisely defined sampling frame is not inherently a weakness; probability sampling permits defensible inference to a specified population even when that population is narrower than all children of the relevant age. The reporting problem arises when exclusions, non-response and the limits of generalisation are not made visible, or when conclusions drift beyond the eligible population. Participants also reported that reasons for non-response went unstated, that methodological limitations received only brief discussion, and that wider influences on achievement, including parental involvement, community support and household socio-economic conditions, received comparatively limited attention. A perceived divergence between SACMEQ results and Lesotho’s separate national assessment was treated strictly as a prompt for contextual inquiry, since the two programmes differ in grade, instruments, sampling and metric, with SACMEQ reporting Item Response Theory scale scores and the national programme reporting percentages.

On policy usefulness, the picture was again mixed. Stakeholders valued SACMEQ evidence for monitoring, benchmarking against the consortium’s member states, curriculum review, teacher development and resource allocation discussions, and the documentary record confirmed these intended functions. Yet dissemination emerged as the strongest constraint. One informant remarked that results typically stayed with government officials and a few participating academics, whereas some other countries make publication of the results a national agenda item. Participants also pointed to resistance to pedagogical change among some teachers and to systemic constraints, including shortages of qualified teachers, limited learning resources, inadequate facilities, financial pressure and weak parental participation. The authors stress that these conditions are not deficiencies of SACMEQ reporting; rather, they explain why credible evidence does not automatically translate into educational improvement, and they distinguish report credibility from implementation capacity.

The recommendations that follow from the evidence are practical and largely fall within national reporting and knowledge mobilisation. The authors suggest that Lesotho’s ministry and national research team make the target population, exclusions, non-response, uncertainty and limits of inference prominent in future reports; separate descriptive associations from causal explanations; explain how findings relate to national curriculum and system conditions; provide disaggregated analyses only where sample design and precision support them; and accompany technical reports with concise, audience-specific briefs for policymakers, curriculum specialists, district officers, school leaders and teachers. Participants additionally called for closer links between assessment evidence and in-service teacher training, and for reports to combine quantitative results with richer qualitative explanation. The authors emphasise that these strategies have not been evaluated prospectively, so their reach and effectiveness should be tested.

The study’s broader significance lies in its bounded, carefully qualified contribution to how the education world thinks about assessment reporting. It does not validate a new scale, estimate the psychometric quality of SACMEQ instruments, or generalise stakeholder perceptions to all report users, and it explicitly excludes SEACMEQ V and international reports from its corpus. Its theoretical claim is modest but pointed: validation of assessment-based inferences and evaluation of how national reports communicate those inferences are distinct tasks, and the second deserves systematic attention. The Authenticity Evaluation Framework offers a provisional vocabulary for that second task, pending expert content review, independent coding studies and multi-country testing. For Lesotho, the message is one of complementary responsibility: regional structures supply comparable, technically sound evidence, while national teams must supply the contextual interpretation, transparent boundaries and active dissemination that turn numbers into decisions. Technically defensible scores, the study concludes, are a foundation rather than a finished product.

Subject of Research: Authenticity and policy usefulness of Lesotho's national SACMEQ literacy and numeracy assessment reports

Article Title: Authenticity of Lesotho’s SACMEQ II-IV literacy and numeracy reports

Article References: Authenticity of Lesotho’s SACMEQ II-IV literacy and numeracy reports. (n.d.). https://doi.org/10.1007/s43545-026-01841-7

Image Credits: AI Generated

DOI: 10.1007/s43545-026-01841-7

Keywords: SACMEQ, Lesotho, report authenticity, large-scale assessment, literacy, numeracy, educational reporting, psychometric validity, qualitative case study, education policy, SEACMEQ, dissemination