PaperlayerView exact revision on arXiv

Source-linked AI summary

EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration

Jia-Kai Dong, Yi-Cheng Lin, Hung-yi Lee

arXiv:2607.18529v2cs.HCcs.AI

TL;DR

Teaching videos need scalable evaluation that captures multimodal pedagogical quality for particular learners, not a universal quality score. EduPanel uses specialized agents to provide learner-conditioned judgments, improving expert evaluation while experts still detect unreliable outputs.

  • Problem

    Automatic teaching-video judges have limited support for multimodal evidence and evaluation relative to the intended learner rather than a universal quality standard.

  • Method

    EduPanel is a multimodal, rubric-grounded, learner-conditioned judge that decomposes evaluation across three specialized agents with inspectable rationales.

  • Results

    Experts improved scoring with EduPanel’s commentary while remaining able to distinguish unreliable outputs, achieving AUC = 0.77 in a planted-error study.

  • Takeaways & Limitations

    EduPanel is best used as decision support for experts, who can benefit from its feedback without accepting its judgments wholesale.

  • Takeaways & Limitations

    Agreement with the adjudicated ground truth is not an independent validation because one researcher constructed the reference standard using AI commentary.

Abstract

from arXiv · show

Teaching videos are becoming a major medium for education, creating a growing need for scalable evaluation of their pedagogical quality. Existing automatic judges do not fully address this setting because teaching quality depends on multimodal evidence and should be evaluated with respect to the intended learner rather than as a universal property. We present EduPanel, a rubric-grounded, learner-conditioned LLM judge that decomposes evaluation across specialized agents to produce interpretable assessments for different aspects of teaching quality. Across expert studies, architecture ablations, and learner-persona analyses, EduPanel achieves reliability comparable to a median human expert. In expert evaluation, its feedback improves scoring accuracy (MAE 0.87 to 0.73), while experts remain able to detect unreliable outputs (AUC = 0.77) instead of accepting them blindly. These results suggest that EduPanel can serve as effective assistants for educational evaluation rather than replacements for human experts.

1 Introduction

EduPanel is a multimodal, rubric-grounded, learner-conditioned three-agent judge designed to evaluate teaching videos for specified learners. The paper examines its reliability, design factors, and usefulness as a critically verifiable assistant to experts, while limiting conclusions to the proposed system and its calibration.

  • Motivation: Teaching quality is learner-dependent: the same instructional material may help advanced students but overwhelm beginners lacking prerequisites.Effective teaching depends on prior knowledge and expertise, which learner-independent judges cannot capture.
  • System: EduPanel evaluates teaching videos relative to a specified learner instead of assigning a single learner-independent quality score.Learner profiles serve as evaluation conditions, allowing the same video to be assessed differently for different intended audiences.
  • System: Three specialized agents decompose evaluation into reconstructing and fact-checking instruction, assessing course-specification alignment, and judging appropriateness for the target learner.The decomposition mirrors the distinct reasoning required to evaluate teaching videos and provides inspectable intermediate evidence.
  • Evaluation: The study evaluates reliability, design choices, persona sensitivity, input and architecture ablations, and practical utility in expert workflows.Its research questions cover evaluation quality, design validation, and critically verifiable assistance, including complementary error detection.
  • Expert workflow: AUC = 0.77: experts distinguished unreliable from reliable outputs in a planted-error study, while EduPanel commentary improved expert scoring and contributed complementary observations.These results support using the judge as an assistant whose commentary experts critically verify rather than accept wholesale.
  • Scope: The experiments use a deliberately small, curated benchmark of 12 videos, 10 evaluation dimensions, four subject domains, and one deployed backbone.Accordingly, the conclusions concern the proposed system and its calibration rather than population-level estimates.

2 Related Work

Prior work extends LLM judging to multimodal inputs and role-differentiated agents, while separate literatures address teaching-content generation, learner-conditioned assessment, and human–AI reliance. EduPanel combines these threads around full-length pedagogical evaluation and its use within human rating workflows.

  • LLM and multimodal judges: LLM judges achieve high agreement with human preferences but exhibit known biases and are typically evaluated on curated, text-only benchmarks.This motivates extensions beyond text-only judging and closer analysis of reliability.
  • LLM and multimodal judges: Multimodal judges incorporate visual and audio-visual signals, while role-differentiated agents structure evaluative reasoning; no prior judge evaluates full-length teaching videos pedagogically.EduPanel builds on both threads while changing the evaluation target and purpose.
  • Teaching-content generation: Teaching-video generation pipelines use metrics such as VLM-as-a-judge aesthetic scores and knowledge-recovery quizzes, which assess information conveyance rather than whether a video teaches well.This distinguishes generation-time quality metrics from pedagogical assessment.
  • Personalized, learner-conditioned evaluation: Instructional effectiveness depends on learners’ prior knowledge and expertise, yet most automated assessment does not explicitly model learner characteristics.LLMs can role-play learners with specified backgrounds, and this capability has been used to train teachers against synthetic students.
  • Human–AI collaboration and critical use: Human–AI teams benefit when errors differ, but people can over-rely on automation; prior reliance research largely concerns assistants, classifiers, or LLM outputs rather than LLM evaluators.Pairing learner-conditioned evaluation with human-team analysis in rating workflows is described as previously unoccupied.

3 Method

EduPanel evaluates teaching videos with a learner-conditioned, rubric-grounded three-agent pipeline, using course requirements and student personas as evaluation context. The method combines multimodal agent specialization with a two-round expert protocol that tests both scoring agreement and resistance to unreliable AI feedback.

  • Evaluation Set: The evaluation set contains 12 independently created teaching videos across four subject areas and four instructional topics, each paired with a course requirement and target student persona.The subjects are physics, biology, mathematics, and computer science, with three videos per topic.
  • Rubric Design: The initial 24-metric rubric was reduced to 10 evaluated dimensions after pilot testing found ceiling effects that limited discrimination among high-quality videos.Only A1, C4, and D2 require visual information according to the rubric’s declared modality needs.
  • Judge Architecture: EduPanel scores each rubric dimension from 1 to 5 across three independent passes, averaging outputs from three agents that share the gemini-3-flash backbone.The pipeline receives the video, course requirement, and student persona as evaluation inputs.
  • Judge Architecture: Agent 1 maps video content and flags factual or visual issues, Agent 2 text-scores objective dimensions from that report, and Agent 3 simulates the target learner using the video and persona.Dimension assignment follows the rubric’s evaluator field: A1 and A2 go to Agent 2, while remaining dimensions go to Agent 3.
  • Human Evaluation: Twelve experts rated all 12 videos blindly and then with AI assistance, verifying scores before optionally revising them while being told to verify rather than copy.Some Round 2 AI scores were planted false positives or false negatives, differing from the original by at least two points and limited to at most three cells per video.
  • Evaluation Metrics: Primary evaluation uses MAE, Pearson correlation (r), and Krippendorff’s α, with within-1 rate as a secondary metric and ROC AUC for planted-error detection.Behavioral response to unreliable feedback is evaluated separately from error detection.

4 Results

EduPanel matches typical human-rater consistency, while its modality-dependent bias varies by backbone and reflects video use rather than a universal LLM-judge property. Its learner conditioning, agent decomposition, and expert-assistance workflow improve interpretability, adaptability, scoring accuracy, and error detection, although experts rarely revise ratings.

  • Reliability and bias: MAE 0.85 matches the median blind expert’s 0.87 against AI-free human consensus, while adjudicated-ground-truth agreement reaches MAE = 0.55 and within-1 = 92%.AI-free human consensus is treated as the primary reference because adjudication incorporated AI commentary.
  • Reliability and bias: +0.67 versus +0.24 signed bias distinguishes visually grounded from transcript-only dimensions on Gemini, while GPT shows no net visual bias, indicating backbone-dependent leniency.The modality gradient differs little between objective and subjective agents, and removing video flips visual bias from +0.67 to −0.45.
  • Ablations: 0.55→1.07 transcript-only MAE after video removal, alongside within-1 92%→72%, shows that video is indispensable for evaluation performance.Without video, scores flatten toward the scale center and visual dimensions default to a neutral score.
  • Persona conditioning: F-family MAE 0.42→0.90 after neutral-persona replacement contrasts with A-family MAE 0.67→0.72, showing selective effects on learner-adaptability dimensions.All three F dimensions worsen, including F2: 0.75→1.17, while objective-content differences remain comparable to run-to-run variation.
  • Persona conditioning: ∆= +1.43 for vocabulary and ∆= +1.47 for prerequisite knowledge when switching from middle-school to university persona exceeds human shifts of +0.57 for both dimensions.Judge and human score changes correlate at 0.37 overall, strongest for vocabulary (r = 0.41), while pacing changes modestly (∆= +0.36).
  • Expert assistance and trust calibration: 0.87 to 0.73 MAE after expert assistance improves accuracy by 0.14, while planted-error detection reaches ROC AUC = 0.77 and revision rates remain low.Eight of 12 experts improved, but only 17% of ratings on AI-correct cells and 21% on incorrect cells were revised.

5 Discussion

EduPanel evaluates pedagogical fitness for specified learners rather than a universal instructional-quality notion, with learner conditioning changing evaluations. Its findings support a multimodal, three-agent decision-support workflow that assists experts while preserving human verification.

  • Beyond multimodal judging: EduPanel evaluates pedagogical fitness for a specified learner rather than estimating a single universal notion of instructional quality.Changing the learner changes the evaluation, while removing the learner profile selectively degrades learner-adaptability dimensions.
  • Experimental insights: EduPanel performs well on transcript-grounded dimensions but is more lenient on visually grounded ones, a pattern specific to Gemini rather than LLM judges generally.The pattern is absent on the GPT backbone.
  • Experimental insights: The three-agent architecture improves calibration more than aggregate accuracy by preserving score dispersion and interpretable intermediate evidence.A monolithic judge achieves comparable MAE but compresses the score distribution.
  • Implications for deployment: EduPanel should assist experts rather than replace them, because experts improved accuracy and consistency while still identifying many incorrect AI outputs.Experts mainly discerned problems during verification rather than score revision, making disagreement an informative signal for additional review.

6 Limitations

EduPanel’s conclusions are bounded by limitations in ground-truth independence, causal identification, benchmark scale, learner-persona coverage, and architectural generalizability. These constraints motivate broader benchmarks, stronger protocols, controlled assistance studies, and expanded learner modeling.

  • Ground-truth construction: Ground truth relies on one researcher’s adjudication incorporating AI commentary, so agreement with it is not independent validation; AI-free consensus and five alternative rules provide robustness checks.This dependence limits how independently the judge is validated.
  • Causal interpretation of expert assistance: The within-subject blind-then-assisted design lacks a no-AI re-rating control, preventing clean separation of accuracy gains from practice, familiarization, or repeated exposure.A controlled condition is needed to establish a causal effect of AI assistance.
  • Scale and statistical power: The benchmark contains 12 videos and 10 rubric dimensions, making per-dimension findings descriptive trends and limiting conclusions to this curated benchmark.The study’s findings should therefore be interpreted within the scope of the present study.
  • Scope of learner conditioning: Learner conditioning uses only two learner profiles and a crowd reference, with moderate human-aligned co-variation for vocabulary but weak alignment for prerequisite awareness and especially pacing.The judge adjusted scores in expected directions for vocabulary and prerequisite dimensions, but broader generalization remains uncertain.
  • Scope of the architectural findings: The three-agent architecture improves transparency, controllability, and persona-dependent evaluation but yields limited aggregate-accuracy gains over a monolithic judge on this benchmark.The architecture’s benefits should not be interpreted as broad accuracy superiority.
  • Scope of the architectural findings: Modality-dependent leniency may be specific to the deployed Gemini backbone rather than a general property of LLM judges, motivating broader validation and learner modeling.Future work should address larger benchmarks, stronger ground-truth protocols, controlled AI-assistance studies, and broader learner modeling.

7 Future Work

Future work should test EduPanel’s causal impact with stronger experimental and ground-truth protocols, assess its generality across benchmarks, models, and learner personas, and extend learner-conditioned evaluation beyond instructional videos.

  • Causal validation: Future studies should establish the causal effect of AI assistance through controlled experiments and stronger ground-truth protocols, including AI-blind adjudication.These designs are intended to clarify whether AI assistance itself improves evaluation outcomes.
  • Generality: Larger and more diverse benchmarks, additional LLM backbones, and broader learner-persona coverage should clarify the generality of EduPanel’s findings.The proposed evaluations vary datasets, model backbones, and intended learners.
  • Scope expansion: Future work should extend learner-conditioned evaluation beyond instructional videos to other settings.The passage identifies this as a further direction without specifying the additional settings.

8 Conclusion

EduPanel is a multimodal, rubric-grounded judge that evaluates teaching videos relative to their intended learners. It achieves human-level consistency on some dimensions but has limitations, particularly for visual reasoning, while supporting expert decision-making as a learner-conditioned tool.

  • EduPanel evaluates teaching videos with multimodal evidence and rubric grounding, conditioning judgments on the intended learner rather than a universal quality standard.This learner-relative design is central to the proposed judge.
  • Across a curated benchmark, EduPanel achieves human-level consistency on some evaluation dimensions.
  • EduPanel shows clear limitations on evaluation dimensions requiring visual reasoning.
  • The workflow study suggests learner-conditioned judges can serve as effective decision-support tools for experts.The supplied passage ends before specifying the full workflow finding.

A Materials and rubric details

EduPanel uses a 24-metric, learner-conditioned rubric routed across three specialized agents, with persona-specific requirements making pedagogical fitness scorable for each video. A pilot excluded saturated metrics that could not support human–AI agreement analysis.

  • Dimension-selection pilot: 4.90 ± 0.30 for A3, 4.80 ± 0.40 for A6, 4.80 ± 0.40 for C2, and 4.60 ± 0.49 for C3 showed ceiling effects among strong videos.Because these metrics made strong videos indistinguishable, they could not support human–AI agreement analysis.
  • Rubric and routing: The rubric contains 24 metrics across six dimensions, uses 1–5 anchors, and assigns each metric to a scoring agent.Agents 1 and 3 access the full video, while Agent 2 is text-only and consumes Agent 1’s report.
  • Learner conditioning: Persona conditioning uses video-specific course requirements and Has/Lacks inventories to make dimension F scorable relative to a target learner.For the meiosis example, the Lacks list includes chromosome structure vocabulary and mitosis versus meiosis.
  • Rubric and routing: A1–A2 are scored by the objective agent, while the remaining eight evaluated dimensions are scored by the persona-simulator agent.This routing is declared in the rubric’s evaluator field.

B Ground truth and planted errors

The study compares five ground-truth construction rules based on 12 blind expert scores per cell, with curated adjudication as the primary ground truth. It also evaluates 26 valid planted-error cells spanning false positives and false negatives.

  • Ground-truth construction rules: Five ground-truth rules are evaluated over 12 blind expert scores per cell in the sensitivity analysis.The rules are curated, mean, median, mode, and trimmed mean.
  • Ground-truth construction rules: Curated ground truth uses researcher adjudication that consolidates blind scores and discussion while consulting AI commentary.It is the study’s primary ground-truth construction rule.
  • Ground-truth construction rules: Mean, median, and mode aggregate blind scores without AI, while trimmed mean removes one highest and one lowest score before averaging.Mode ties are resolved toward the value nearest the median; trimmed mean is intended to be robust to outlier raters.
  • Planted errors: 26 valid planted cells comprise 9 FP and 17 FN errors, with at most 3 per video and gaps of ≥2 points from the AI’s real evidenced score.The validity screen is described in Section 3.3, with full per-item details provided in the supplement.

C Judge configuration and reproducibility

EduPanel’s three agents use a fixed Gemini-based video-scoring setup with native MP4 ingestion, repeated passes, and deterministic decoding. Reproducibility is supported by releasing per-pass outputs and associated evaluation materials.

  • Model and input configuration: All three agents use gemini-3-flash with native MP4 ingestion, while the GPT cross-backbone variant samples approximately 48 evenly spaced frames.The GPT variant is used specifically for cross-backbone comparison.
  • Scoring procedure: Each video is scored over three independent passes and the scores are averaged, with run-to-run stability reported in Table 10.Repeated scoring provides a basis for assessing run-to-run stability.
  • Reproducibility: All scores use a fixed snapshot, temperature 0, and decode seed 42, and the authors release per-pass outputs, rubrics, prompts, requirements, personas, and anonymized rater data.The fixed window uses snapshot gemini-3-flash-preview.

D Robustness re-analyses

Three zero-cost re-analyses show that EduPanel’s headline findings are robust to ordinal treatment, leave-one-video-out resampling, and retained-dimension variance concerns, while the deployed Gemini judge remains highly stable across runs. The retained-dimension check bounds, but does not eliminate, possible selection bias.

  • Judge run-to-run reliability: The deployed Gemini judge is highly stable across scoring passes, whereas the GPT backbone is noisier, especially on subjective or visual dimensions F2, C4, and A1.The table reports GPT backbone run-to-run stability only; its accuracy and bias versus Gemini are reported elsewhere.
  • Rank-based and ordinal treatments: Ordinal analyses preserve every conclusion: judge–GT discrimination is ρ = 0.82 versus r = 0.84, while assistance reliability rises from 0.36→0.47 or 0.38→0.50.Both Krippendorff α results remain below the conventional 0.667 threshold.
  • Leave-one-video-out: Leave-one-video-out results show no headline metric depends on one video: judge MAE remains [0.51, 0.60], assistance ∆MAE [+0.13, +0.16], and detection AUC [0.76, 0.78].The full-sample values are 0.55, +0.14, and 0.77, respectively; the assistance ∆MAE is positive throughout.
  • Retained-dimension variance: All 10 retained alignment dimensions also show substantial human blind-score variance, with per-dimension SD 1.07–1.44, rather than being discriminated only by the judge.Because experts never scored the 14 excluded dimensions, the selection cannot be recomputed from human variance; this check bounds but does not eliminate selection concerns.
Loading 2607.18529v2…