Development and validation of an assessment tool for public health emergency management program

Researchers in Ethiopia have developed and validated what they describe as the first standardized, psychometrically tested instrument for assessing how well public health emergency management programs are being implemented, according to a study published in Global Health Research and Policy. The 45-item tool, built through a year-long mixed-methods process involving literature reviews, expert workshops, translation, and statistical validation, achieved high content validity, a robust fifteen-factor structure, and excellent internal consistency. Its developers say the instrument fills a longstanding gap in emergency preparedness evaluation, particularly for low-resource countries that have lacked rigorous, context-appropriate means of measuring their readiness for epidemics, disasters, and other health crises. Until now, they note, many governments have had to rely on ad hoc checklists or donor-driven reviews that vary widely in scope and rigor, making it difficult to track progress over time or compare performance across regions and institutions.

The need for such a tool is grounded in a sobering global record. Between 1994 and 2015, weather-related disasters killed more than 1.5 million people and affected over 4 billion worldwide, a burden that has grown as climate change intensifies floods, droughts, and storms. The COVID-19 pandemic, which had produced more than 777 million cases and 7 million deaths by early 2022, exposed deep vulnerabilities in health systems, from fragile supply chains to overstretched workforces and slow coordination between levels of government. So did the 2013–2016 West African Ebola outbreak, which caused over 28,000 cases and 11,000 deaths and demonstrated how quickly an epidemic can overwhelm under-resourced health services and spread across borders when surveillance and response structures are weak. Low-income nations shoulder a disproportionate share of this burden, bearing up to 91 percent of weather-related disaster mortality, a figure that reflects not only geography but also differences in infrastructure, housing quality, early warning systems, and the capacity of health services to absorb shocks. In Ethiopia, home to roughly 130 million people, threats ranging from malaria and emerging infectious diseases to floods, droughts, and conflict-related disruptions make emergency management a persistent national priority, yet the country’s preparedness efforts have been constrained by workforce shortages, limited funding, weak governance, and poor infrastructure. These constraints mirror those reported across much of sub-Saharan Africa, where public health emergency management is a relatively young professional discipline and formal training pipelines remain limited.

Existing approaches to evaluating emergency preparedness, the researchers argue, have serious shortcomings. Many assessment tools focus narrowly on specific hazards or single sectors rather than offering an integrated view of a program’s overall effectiveness. Others emphasize implementation processes over programmatic outcomes, or lack empirical grounding in their development and validation, producing inconsistent and fragmented evaluations. Without standardized metrics, findings from one assessment cannot reliably be compared with those from another, and managers have little objective basis for deciding where to direct scarce funds. Local assessments in Ethiopia have revealed deficiencies in workforce, infrastructure, resources, and intersectoral collaboration, while studies of vector control in South Asia and the Middle East and global disease surveillance efforts have documented poor monitoring, fragmentation between sectors, and resource constraints. The International Health Regulations monitoring framework and the Joint External Evaluation process provide country-level benchmarks, but the study team argued that a granular, program-level instrument was still missing. To overcome these problems, the team set out to build a standardized instrument capable of capturing readiness, response capabilities, and overall effectiveness across a range of emergency scenarios.

The study, conducted from July 2023 to June 2024, was guided by Donabedian’s Structure-Process-Outcome framework, a classic health services model that separates evaluation into the resources and infrastructure available (structure), the activities carried out (process), and the results achieved (outcome). Developed originally to evaluate the quality of medical care, the framework has been widely applied across health systems research because it offers a logical chain of accountability: adequate structures enable sound processes, which in turn produce better outcomes. Applying this model to emergency management, the team designed the tool to measure structural elements such as resources and workforce capacity, process elements such as multi-sectoral coordination and resource allocation, and outcome-oriented concepts such as community resilience and response effectiveness. Although the study did not directly measure health outcomes, the framework allows the researchers to link improvements in structures and processes to anticipated gains in public health resilience, creating a theory-driven rationale for how better inputs and practices should translate into saved lives and reduced disruption.

Instrument development unfolded in deliberate stages. A team of eight subject matter experts conducted an extensive literature review to identify existing frameworks, core preparedness domains, and validated assessment tools, defining pertinent domains and items for the new instrument. A focused desk review in the town of Adama consolidated this evidence, appraised conceptual coherence, and distilled the material into a preliminary framework. A two-day consultative workshop in Hawassa then brought together twelve experts from every region of Ethiopia, including officials from the Ethiopian Public Health Institute and the Ministry of Health, who refined the draft items through Q-sort procedures, focus groups, and iterative consensus-building. Q-sorting, a technique in which participants sort statements into ranked categories, helped the group surface disagreements about item wording and relevance in a structured way, while the focus groups allowed frontline perspectives to shape the draft. The process deliberately embedded Ethiopian perspectives at every step, ensuring the tool’s cultural relevance and local applicability rather than simply adapting a questionnaire designed abroad. Item generation ultimately produced 60 items: 15 addressing overall implementation status, 15 evaluating individual-level capacities and attitudes, and 30 assessing organizational characteristics.

Because Ethiopia’s emergency management workforce operates largely in Amharic, the questionnaire underwent rigorous translation. Four independent translators produced forward translations from English to Amharic, and two additional language experts translated each version back into English. Back-translation is a standard safeguard in cross-cultural instrument development because it exposes ambiguities: if the backward version drifts from the original meaning, the forward translation likely does too. Comparisons between the back-translations and the original found no discrepancies, and no modifications were required, establishing linguistic equivalence before formal testing began.

Validity assessment proceeded on several fronts. For face validity, 30 Ethiopian public health emergency management professionals with at least a master’s degree and five years of experience rated each item for clarity and comprehensibility on a 4-point scale; items achieving a Face Validity Index of at least 0.80 were retained. For content validity, eight qualified experts independently rated each item’s relevance on a 4-point Likert scale, with scores of 1–2 coded as irrelevant and 3–4 as relevant. Following recommendations from seminal instrument-development literature, items with an Item-level Content Validity Index below 0.83 were eliminated; the threshold reflects a statistical adjustment for small expert panels, which naturally limits how closely ratings can agree. Eight of the original 60 items fell below this threshold and were removed, and recomputation confirmed that the remaining 52 items achieved scale-level content validity scores exceeding 0.83.

Construct validity was tested through exploratory factor analysis using survey responses from 260 public health emergency management professionals recruited across Ethiopia’s national, regional, zonal, and district levels, with support from regional health bureaus. This multi-level recruitment was important because emergency management responsibilities differ sharply between a ministry office in the capital and a district health post, and a valid instrument must function across that spectrum. The sample achieved a 100 percent response rate with no missing data, an unusual and valuable result for survey research. Participants were predominantly male (73.9 percent), relatively young (45.4 percent aged 20 to 30), and split between bachelor’s (53.85 percent) and master’s degrees (46.15 percent); half worked in health facilities, and 50.4 percent were based in Addis Ababa. The data proved highly suitable for factor analysis: the Kaiser–Meyer–Olkin measure of sampling adequacy rose from 0.831 in the first analysis to 0.840 after refinement, Bartlett’s Test of Sphericity was highly significant, and item communalities ranged from 0.526 to 0.808. Values of this kind indicate that the items share enough common variance for meaningful underlying dimensions to be extracted.

The factor analyses led to further item pruning. An initial extraction identified 16 factors explaining 65.31 percent of variance, but two items on implementation status failed to load meaningfully on any dimension and were dropped. A second analysis flagged five additional poor-fitting items concerning work factors, resource availability, and information sharing. After their removal, the final structure settled at 15 dimensions comprising 45 items, together explaining 74.8 percent of the variance, comfortably above the 60 percent threshold the authors cite as evidence of good construct validity. The retained dimensions spanned implementation status, individual capacity building, individual reinforcement and engagement, individual attitudes, and a suite of community-level factors including information sharing, integration, planning, delegation, resource availability, leadership, monitoring and evaluation, coordination, infrastructure, and the legal framework. This blend of individual and organizational dimensions is notable: many existing tools evaluate either personal competencies or institutional systems, but rarely both in a single instrument. Internal consistency was strong, with an overall Cronbach’s alpha of 0.863 and all subscales exceeding 0.70, the conventional benchmark indicating that items within each subscale reliably measure the same concept.

The study’s authors are candid about its limitations. Validation occurred in a single country, so generalizability elsewhere requires further testing; factor structures and item meanings can shift when a questionnaire crosses cultural and linguistic boundaries. The youthful age profile of many pilot participants raises questions about the depth of professional experience represented, and the team acknowledges potential biases arising from stakeholder input and the need for cultural adaptation in other contexts. They also reflect on methodological lessons, noting that rigid adherence to numerical cutoffs during content review may have overlooked nuanced qualitative perspectives on borderline items; more transparent documentation of expert debates and divergent rationales, they suggest, would strengthen future validation efforts.

Even so, the implications extend well beyond Ethiopia. The tool’s emphasis on multi-sectoral coordination, resource allocation, funding transparency, workforce capacity, and provision of essential supplies addresses weaknesses commonly identified in developing countries, and its structured approach can be adapted through local translation and contextualization for other low- and middle-income settings, including regions contending with climate change, urbanization, or conflict. The researchers recommend operationalizing the instrument within national preparedness plans, using it for ongoing monitoring and benchmarking to guide investments, and testing its responsiveness over time in longitudinal and implementation studies. Repeated application could reveal whether training programs, funding increases, or governance reforms actually move the needle on preparedness, an evidence link that has been difficult to establish with existing tools. By standardizing the measurement of capacities and gaps, they argue, the tool can help policymakers prioritize resources, strengthen resilience, and mount more coordinated responses when the next crisis arrives.