New validated observation tool spots great digital teaching in real classrooms

Researchers in Spain have developed and rigorously validated a direct classroom observation guideline designed to identify good educational practices that incorporate digital technologies, and the new instrument works across every level of education, from early childhood classrooms to university lecture halls. The study, published in the Journal of New Approaches in Educational Research, describes a systematic design process grounded in Activity Theory and refined through expert consensus and real-world pilot testing, offering schools, administrators and researchers a practical, evidence-based way to recognize teaching that genuinely improves learning with technology rather than merely using it.

The motivation for the work stems from a persistent gap in education systems worldwide. Governments and educational administrations increasingly call for the identification and dissemination of good practices with digital technologies in order to improve teaching quality, and international repositories such as the European School Education Platform and the UNESCO ICT in Education Repository collect and share such experiences. Yet defining what actually counts as a good practice has remained surprisingly elusive. Previous literature reviews have highlighted criteria such as effectiveness, transformative effect, sustainability, legitimacy and replicability, while regional agencies in Spain have emphasized features including curriculum integration, institutional support, improved outcomes, interdisciplinarity, inclusion and the involvement of the educational community. Without clear, observable criteria, judgments about quality risk being subjective and inconsistent.

To anchor the instrument in a coherent theoretical framework, the research team, based at the Department of Pedagogy of Universitat Rovira i Virgili, turned to Activity Theory, a cultural-historical framework originally developed by Vygotsky, Leontiev and Engestrom. Activity Theory conceives educational practice as an intentional, object-oriented action involving students and teachers working toward a common goal, mediated by physical and sociocultural tools. The theory models an activity system through its core components, including the subject, tools, community, rules, division of labor and object, and it emphasizes the dynamic, dialectical interactions among these elements. This holistic view aligns closely with the structure of a learning situation and allows observers to detect tensions, contradictions and disruptions within the classroom activity system, which the researchers see as essential to understanding how digital technologies are actually integrated into teaching and learning.

The study adopted a Design-Based Research approach, a methodology structured around iterative cycles of analysis, design, development and evaluation. The first cycle began with a systematic literature review following the PRISMA 2020 model, searching databases including Web of Science, Scopus, ERIC, SciELO and Dialnet for publications from 2013 to 2023 describing educational practices with digital technologies at any educational level. From an initial pool of 718 publications, 27 studies were selected for full content analysis using ATLAS.ti software. This analysis identified 26 variables linked to the seven components of Activity Theory, each operationally defined with its theoretical meaning, the type of evidence required for observation and its intended use within the guideline, forming the initial prototype of the instrument.

Validation proceeded through the Delphi method, a structured technique for aggregating expert judgment over successive rounds. The team formed a coordinating group and recruited a panel of experts across four professional profiles: teachers from early childhood, primary and secondary education; university professors; researchers; and professionals involved in educational management. Thirty-two experts were invited and 26 completed the competence questionnaire, which calculated an Expert Competence Index combining self-assessed knowledge and the quality of argumentation sources. Twenty-three experts scored at or above the 0.8 threshold denoting high competence and continued into the Delphi rounds, a panel size consistent with established guidance that Delphi studies typically require between 15 and 35 participants.

The three Delphi rounds progressively sharpened the instrument. In the first round, 23 experts ranked the variables derived from each Activity Theory component, and the researchers computed a Relative Importance Index to convert the rankings into interpretable percentages, selecting the two highest-priority variables per component to yield 14 variables and 14 indicators. In the second round, 20 experts rated the clarity and relevance of the proposed indicators, with most indicators exceeding 85 percent positive validation and five achieving unanimous acceptance; all indicators met the Content Validity Index threshold of at least 78 percent, the criterion the team used to classify indicators as excellent. In the third round, nine experts assessed their satisfaction with the revised indicators, and mean scores rose while standard deviations fell, with two indicators reaching a perfect average of 4.00 with zero variability, confirming that the iterative refinements had addressed the panel’s concerns.

With the validated prototype in hand, the team moved to pilot testing in authentic educational settings, an essential step for establishing external validity and feasibility. The pilot involved eight educational practices distributed across four stages, early childhood, primary, secondary and higher education, with two practices per stage, all hosted by different public institutions that had previously collaborated with the university. Sessions lasted one hour in the school stages and two hours at university, with class sizes ranging from 28 students in early childhood settings to as many as 80 university students. Researchers observed each practice simultaneously using individual copies of the guideline and then jointly interviewed the observed teachers with a complementary 13-question semi-structured interview guide, allowing triangulation of the observational data.

Reliability analyses during the pilot demonstrated that different observers could apply the instrument consistently. Weighted Cohen’s Kappa values ranged from 0.64 to 1.00 across the 14 indicators, with 10 of the 14 indicators exceeding 0.80, indicating substantial to almost perfect agreement according to commonly accepted interpretive benchmarks. Intraclass Correlation Coefficients, calculated under the two-way random-effects absolute-agreement model, ranged from 0.67 to 1.00 for single measures, while the global ICC reached 0.89 for single measures and 0.94 for average measures. These figures confirm that the guideline is robust, internally coherent and replicable across observers and contexts, a critical property for any instrument intended to support fair evaluation of teaching quality.

The pilot also generated concrete improvements to the instrument’s design and usability. The researchers added a new section capturing contextual information such as the observed teacher’s name, type of session, school type and dimensions, and technological devices used, reoriented the layout from horizontal to vertical for easier field use, reorganized indicators so that all indicators for each component appeared on the same page, expanded the open-observation field from three to seven lines, and moved illustrative examples directly into the guideline itself. One indicator was reworded to replace a conjunction with a disjunctive form, broadening its scope to capture situations in which community resources or external agents are identified but not necessarily used. Post-pilot analysis showed that 58 percent of the modifications involved new sections, 32 percent design adjustments and 10 percent content changes.

Beyond its immediate technical achievement, the instrument carries significant practical potential. The authors note that teacher training has traditionally emphasized technical mastery of digital tools rather than their pedagogical integration, and the observation guideline is well positioned to serve as a formative assessment tool, a self-reflection resource in mentoring programs, or an evidence-based component in departmental evaluations of technology-enhanced teaching. Its flexible design allows adaptation to different disciplines, educational levels and modalities, including hybrid and virtual environments. The authors caution that further research is needed, including criterial and convergent validation, sensitivity testing to detect changes in teaching practice over time, assessment of transferability across educational systems, and development of an English-language version, but the validated guideline already represents a substantial step toward making the evaluation of good digital teaching practices systematic, transparent and grounded in direct evidence from real classrooms.

Subject of Research: Design and validation of an Activity Theory-based observation instrument for identifying good educational practices with digital technologies through direct classroom observation

Article Title: Good educational practices with digital technologies: validation of an instrument for direct classroom observations

Article References: Good educational practices with digital technologies: validation of an instrument for direct classroom observations. (n.d.). https://doi.org/10.1007/s44322-026-00065-0

Image Credits: AI Generated

DOI: 10.1007/s44322-026-00065-0

Keywords: digital technologies, educational practices, classroom observation, Activity Theory, Delphi method, instrument validation, teacher digital competence, Design-Based Research, educational research, inter-rater reliability, content validity, teaching and learning