PaperlayerView exact revision on arXiv

Source-linked AI summary

I code or AI code: A comparative evaluation of AI-rated scores in classroom observations

Y. Fong, J. Xiang, T. Y. D. Chan, K. Lee, E. Y. H. Lau

arXiv:2609.18274v1cs.CLcs.AI

TL;DR

Classroom observation supports education-quality assessment but depends on substantial resources and trained observers. This study compared transcript-based AI scoring with human ratings using CLASS. Alignment was greater for Emotional Support and Quality of Feedback, while other domains diverged; the authors position AI as a complement to human judgment.

  • Problem

    Classroom observation is resource-intensive and depends on trained observers, while empirical work on AI for standardized classroom observation remains limited.

    Source: §1 · p. 3, §1.2 · p. 4

  • Method

    The study compared Perplexity GPT-5.0 scores from classroom observation transcripts with human ratings across CLASS, using 87 observations from 38 classrooms.

    Source: §1.4 · p. 6, §2.2 · p. 7, §2.3.3 · p. 9

  • Results

    AI alignment with human judgment was acceptable only for Quality of Feedback and the Emotional Support domain, with divergence for interactions involving nonverbal, procedural, and contextual cues.

    Source: §5 · p. 19

  • Takeaways & Limitations

    AI-assisted observation is a limited complement to human judgment, with outputs framed as evidence for teacher reflection rather than evaluation.

    Source: §5 · p. 19, §6 · p. 20

    The study examined one LLM using transcripts and was limited to Hong Kong early-childhood classrooms serving children aged 3–4, constraining generalizability.

    Source: §4.2 · p. 18, §5 · p. 19

Abstract

from arXiv · show

Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet they remain resource-intensive and dependent on trained observers. This study evaluated the feasibility of using a LLM (GPT-5 model) to score teacher-child interactions in early childhood classrooms, benchmarked against human raters. The study analyzed 87 video-recorded observations from 38 classrooms across 30 kindergartens in Hong Kong. Using observation transcripts, the AI model was configured to apply the full Classroom Assessment Scoring System (CLASS) framework. AI-rated scores were then compared with human ratings by examining correlations and differences in mean scores of the CLASS domains and dimensions. The results showed greater convergence between AI and raters for the Emotional Support domain and, in particular, the Quality of Feedback dimension, which captures how teachers use feedback to extend children's learning. Greater divergence emerged for interactions that were more procedural or context-dependent, particularly within the Classroom Organization and Instructional Support domains. These findings suggest that transcript-based AI scoring may capture some of the relative variation in teacher-child interactions but cannot yet reproduce calibrated human judgements consistently across the full CLASS framework. AI-assisted observation may therefore be more appropriate as a preliminary screening tool rather than as a replacement for trained observers, providing teachers with evidence for reflection rather than high-stakes evaluation. Future research should examine whether domain-specific training and incorporation of contextual and visual information can improve alignment between AI and human rated scores.

1. Introduction

Classroom observation supports evaluation and teacher development but requires substantial time, training, and coordination. This study tests whether AI scores align with human ratings across CLASS domains and dimensions.

1.1 Value of Classroom Observations

  • Classroom observations can give educators evidence for reflection and professional growth, while standardized instruments such as CLASS assess teacher-child interaction quality.CLASS is grounded in developmental theories of process quality and has been used in professional development and improvement initiatives.Source: §1.1 · pp. 3–4

1.2 Challenges in Resource Intensity for Classroom Observations

  • Time, workload, funding, scheduling, observer fatigue, and intensive training and calibration make sustained classroom observation difficult to prioritize.Video recording may offer flexibility, but it does not eliminate fatigue, bias, or the need for rater calibration.Source: §1.2 · pp. 4–5

1.3 Growing prominence of AI as a resource

  • AI research in education has largely focused on student-centered assessment, leaving comparatively little empirical work on standardized classroom observation.Earlier work includes activity labeling from audio and analysis of lecture exchanges; a pilot using multiple LLM agents reported an 18-fold efficiency gain but used binary coding across two subscales.Source: §1.3 · pp. 5–6

1.4 The Current Study

  • The study asks whether AI-rated CLASS scores align with human ratings, addressing the resource demands of trained-observer classroom assessment.The analysis examines convergence and divergence across domains and dimensions.Source: §1 · p. 3, §1.4 · p. 6
  • The study uses CLASS to compare AI and human scores, hypothesizing closer alignment for instructional language exchanges but uncertainty where transcripts only partly represent interactions.The authors identify Emotional Support and Classroom Organization as domains whose interactions may be only partially represented in transcripts.Source: §1.4 · p. 6

2. Method

The study selected classroom videos from Hong Kong kindergartens using audio-visual quality and coding-feasibility criteria. The final analysis included 87 observations from 38 K1 classrooms.

2.1 Procedures

  • Trained research assistants filmed classroom activities under a standardized protocol after written consent was obtained from teachers, children, and parents.Incidental footage of participants without consent was removed, and private activities such as washroom breaks were not filmed.Source: §2.1 · p. 6

2.2 Sample

  • The final sample comprised 87 observations of 15–20 minutes each from 38 K1 teachers, totaling about 26.5 hours across 30 kindergartens.Videos were screened for quality and relevance; recess and unstructured free play were excluded, and single-educator classrooms were prioritized.Source: §2.2 · p. 7

Teacher demographics

The CLASS Pre-K framework rates classroom quality through three domains and their constituent dimensions. Teacher demographic information was available for only part of the sample.

Teacher demographics

  • Demographic records were missing for 20 of 38 teachers; among 18 with age data, mean age was 32.17 years (SD 7.42), while mean teaching experience was 10.89 years (SD 7.99).The table reports 18 female teachers and education attainment for 18 teachers.Source: Teacher demographics · p. 7

2.3.1 The Classroom Assessment Scoring System Pre-K

  • CLASS Pre-K assesses Emotional Climate, Classroom Organization, and Instructional Support through ten dimensions scored on a seven-point quality scale.Domain scores are aggregated from dimension scores.Source: §2.3.1 · pp. 7–8

CLASS Scale Overview

Individual observation-cycle scores were compared between AI and trained human raters. Human scoring involved calibration and overlapping ratings, while the AI was trained and calibrated using transcripts and consolidated human scores.

CLASS Scale Overview

  • Individual observation cycles were compared, although CLASS typically averages four to seven observations per classroom for a classroom-level estimate.The study used individual cycles to examine similarities and differences between raters and AI-rated coding.Source: CLASS Scale Overview · pp. 8–9

2.3.2 Human raters

  • Eight trained human raters coded observations; their weighted Cohen’s kappa values ranged from .672 to .861, and overlapping scores were averaged for analysis.After two rounds of calibration, the remaining 77 observations were each assigned to two raters.Source: §2.3.2 · p. 9

2.3.3 AI Rater

  • After training and calibration, the AI independently scored the remaining 71 classroom observation transcripts across CLASS dimensions.The model was Perplexity GPT-5.0, and all coding used transcripts because of difficulty interpreting video content at the time.Source: §2.3.3 · pp. 9–10

Training and calibration procedure for Humans and AI

The analysis reversed Negative Climate scores and winsorized dimension outliers before aggregating domain scores. It compared human- and AI-rated scores using descriptive, correlation, reliability, and mean-difference analyses.

2.4 Data Analysis

  • Negative Climate scores were reversed, and outliers in 0.423% of rater and 0.704% of AI dimension data were winsorized at ±3 SD before domain aggregation.Source: §2.4 · p. 12
  • The analyses used correlations, inter-rater reliability, and independent t-tests to compare human- and AI-rated scores across CLASS domains and dimensions.Source: §2.4 · p. 12
  • Descriptive statistics were computed first, and all quantitative analyses used IBM SPSS Statistics Version 30.Source: §2.4 · p. 12

3. Results

Descriptive comparisons showed different rater–AI scoring patterns across the CLASS domains, with larger differences in selected dimensions. Table 2 presents scores across all domains and dimensions.

3.1 Preliminary Results

  • Raters scored higher than AI in Emotional Support, especially for Positive Climate and Negative Climate; Table 2 presents scores across domains and dimensions.Source: §3.1 · p. 12, Table 2 · p. 13
  • In Classroom Organization, raters scored slightly lower than AI, with generally similar dimension scores and the largest difference in Behavior Management.Source: §3.1 · p. 12
  • In Instructional Support, raters scored lower than AI at domain and dimension levels, with the largest difference in Concept Development.Source: §3.1 · p. 12
  • Only raters’ Negative Climate scores had a non-normal distribution, showing extreme kurtosis.Source: §3.1 · p. 12

Descriptive statistics of raters (n = 71) and AI (n = 71)

Figure 3 shows that raters’ reverse-scored Negative Climate ratings clustered at the scale ceiling, while AI ratings were more varied.

Descriptive statistics of raters (n = 71) and AI (n = 71)

Distribution of Negative Climate in Raters’ and AI-rated scores

The scoring-alignment analysis covered three CLASS domains and ten dimensions, using Pearson correlations for domains and Spearman correlations for dimensions.

Distribution of Negative Climate in Raters’ and AI-rated scores

3.2 Scoring alignment between raters and AI

Correlations between raters’ and AI-rated scores

AI–human agreement varied across CLASS constructs: Emotional Support and Quality of Feedback aligned more than Classroom Organization and other Instructional Support dimensions. The findings support only conditional use of transcript-based scoring.

Correlations between raters’ and AI-rated scores

  • AI–rater correlations were significant for Emotional Support and Quality of Feedback, but not for Classroom Organization or the other Instructional Support dimensions.Three of four Emotional Support dimensions had significant associations; weighted kappa was 0.681 for consolidated raters versus AI.Source: Correlations between raters’ and… · p. 15
  • Raters scored Emotional Support higher and Instructional Support lower than AI, while Classroom Organization showed no significant mean difference.Table 4 compares raters’ and AI-rated scores at the domain and dimension levels.Source: Correlations between raters’ and… · p. 15, Table 4 · p. 16

4.1 Where AI aligns with raters: verbal and structured

  • Alignment was strongest for explicit, structured language; Quality of Feedback showed significant correlations without significant mean differences.The authors suggest its structured, exchange-based indicators may suit LLM pattern recognition.Source: Correlations between raters’ and… · p. 17

4.2 Where AI diverged: nonverbal, procedural, and contextual constructs

  • Classroom Organization showed minimal alignment, while Concept Development and Language Modeling had low correlations and higher AI scores than raters.The authors link these divergences to rapid, context-sensitive, nonverbal management cues and the pedagogical demands of Instructional Support.Source: §4.1 · p. 17, §4.2 · p. 18

5. Limitations and Future Directions

The findings support limited complementary use of AI, while the study’s model, data, and observation scope constrain generalizability. The authors call for broader models, modalities, contexts, and observation cycles.

5. Limitations and Future Directions

  • The study used one transcript-based LLM, Hong Kong classrooms serving children aged 3–4, one teacher per video, and single observations under CLASS.More than 80% of the full video sample was excluded because of transcription constraints; the authors note these factors may limit generalizability.Source: §5 · p. 19
  • Future work should compare multiple models, incorporate multimodal data, extend to other age groups and contexts, and use more observation cycles and frameworks.The authors also recommend scalable pipelines and analyses with greater sample size and power.Source: §5 · p. 19
  • AI approximated acceptable reliability only for Quality of Feedback and Emotional Support, with divergence for constructs requiring nonverbal, procedural, or contextual cues.The authors conclude that AI-assisted observation is complementary to human judgement, not an independent observation system.Source: §5 · p. 19

6. Conclusions and Implications

The authors position AI as a complement to human observation, especially as a reference or screening tool when intensive observation capacity is constrained. They recommend reflection-focused use and further validation.

6. Conclusions and Implications

  • Schools with limited human resources may use AI as a reference or screening tool, particularly when intensive observation capacity is constrained.The authors advise framing AI outputs as evidence for reflection rather than evaluation.Source: §6 · p. 20
  • AI-assisted observation should complement, not replace, human raters, with teachers trained on the model’s capabilities and limits.The proposed role is to inform reflection on feedback and prompting practices.Source: §6 · p. 20
  • The authors call for comparisons across AI models, multimodal data, diverse contexts, workload evaluation, and targeted training on classroom discourse.They identify collaboration between education researchers and AI developers as important for responsible and effective use.Source: §6 · p. 20

Related papers

Loading 2609.18274v1…