PaperlayerView exact revision on arXiv

Source-linked AI summary

VIRA-Bench: Diagnosing Pedagogical Contingency Understanding in Online Tutoring Videos

Weihang Wang, Hansu Gu, Guanming Liu, Peng Zhang, Ning Gu, Tun Lu

arXiv:2610.29428v1cs.HC

TL;DR

Existing benchmarks rarely test whether a model links a visible tutoring intervention to the learner response and adjusted follow-up. VIRA-Bench evaluates that short-horizon contingency in one-to-one tutoring, finding distinct layer-specific evidence dependencies and a substantial gap from human performance.

  • Problem

    Video and educational-dialogue benchmarks rarely test the linked relation between a visible teacher intervention, its elicited learner response, and a subsequent adjusted follow-up.

    Source: §1 · p. 2

  • Method

    VIRA-Bench organizes audited, expert-verified tutoring-video questions into three theory-grounded diagnostic layers for observable grounding, contingency reasoning, and support-response modeling.

    Source: §1 · pp. 2–3, §3.2 · p. 4, §3.4.2 · p. 5

  • Results

    The best of 11 models reaches 61.3% against a 91.0% human reference, and video removal most degrades OEG while event-order disruption most degrades ICR.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results identify distinct behavioral dependencies in observable grounding and order-sensitive contingency reasoning, without attributing them to an internal architectural mechanism.

    Source: §5 · p. 9

    The benchmark covers 15–60-second excerpts from one-to-one online English tutoring, so findings may not transfer directly to multi-party, cross-domain, or longer-horizon interactions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Understanding a one-to-one tutoring video requires more than recognizing its events. A model must connect a visible teacher intervention to the learner response it elicits and to the follow-up adjusted to that response. Existing video and educational-dialogue benchmarks typically emphasize event recognition, temporal reasoning, social-state inference, or transcript-based interaction, but rarely diagnose this action-response-adjustment relation. We introduce VIRA-Bench, a focused diagnostic benchmark for short-horizon pedagogical contingency in one-to-one online tutoring. VIRA-Bench contains 1,005 tutoring clips and 1,322 multiple-choice questions with researcher evidence audit and expert verification, organized into Observable Evidence Grounding (OEG), Interaction Contingency Reasoning (ICR), and Support-Response Modeling (SRM). Across 11 models, the best reaches 61.3%, well below the 91.0% human reference on a stratified subset, and ICR generally has the lowest layer accuracy among leading models. Under controlled input manipulations, removing video most strongly degrades OEG, whereas disrupting temporal order most strongly degrades ICR. Free-response, Stem+Options, and synchronized-shuffle controls show that response format, static question cues, and video-ASR mismatch do not explain these layer-specific patterns. The results identify distinct behavioral evidence dependencies without claiming an internal architectural mechanism. VIRA-Bench provides a focused evaluation resource for multimodal dialogue grounding, pedagogical discourse, and temporal reasoning in online tutoring.

1 Introduction

VIRA-Bench targets a gap between recognizing tutoring events and recovering how teacher interventions, learner responses, and follow-ups relate. It evaluates that capability in short-horizon one-to-one tutoring and finds distinct evidence dependencies across its diagnostic layers.

1 Introduction

  • Existing video benchmarks emphasize event recognition, temporal relations, or holistic social-state inference, while educational-dialogue resources typically rely on transcripts.The introduction frames the missing test as linking a visible intervention to its elicited response and then to a subsequent adjustment.Source: §1 · p. 2
  • Tutoring offers a short-horizon action-response-adjustment chain with visual and temporal evidence, including teacher operations on a shared screen and learner replies.The motivating account grounds this setting in the zone of proximal development, formative assessment, and contingent scaffolding.Source: §1 · p. 2
  • VIRA-Bench evaluates visually grounded pedagogical contingency through three layers: observable evidence grounding, interaction contingency reasoning, and support-response modeling.The benchmark contains 1,005 tutoring clips and 1,322 questions, constructed with researcher evidence audits and expert verification.Source: §1 · pp. 2–3
  • Across 11 models, the best reaches 61.3%, versus a 91.0% human reference, while removing video most degrades OEG and disrupting event order most degrades ICR.ICR generally has the lowest layer accuracy among leading models.Source: §1 · pp. 2–3

2 Related Work

Related work spans video understanding and educational interaction, but the paper identifies a gap between those lines: testing response-dependent pedagogical contingency from recorded visual evidence.

Video benchmarks

  • Video benchmarks cover short- and long-form understanding and diagnostic tests of temporal evidence or static shortcuts, while causal video QA emphasizes physical or event causality.The supplied passage also cites video benchmark and diagnostic-suite examples.Source: §2 · p. 3

Educational dialogue and scaffolding theory

  • Educational NLP resources model uptake, classroom dialogue, and tutoring criteria, while video-based classroom observation and feedback tools provide adjacent forms of recorded-interaction analysis [Whitehill and LoCasale-Crouch, 2024].The passage notes that ClassMind uses pre-segmented lessons.Source: §2 · p. 3
  • VIRA-Bench translates IRF exchange structure and contingent scaffolding theory into a benchmark for interaction understanding from recorded evidence.The related-work passage presents this as complementary to user simulators for multi-turn evaluation.Source: §2 · p. 3

3 The VIRA-Bench Benchmark

VIRA-Bench evaluates short-horizon teacher intervention, learner response, and follow-up in one-to-one tutoring clips. Its diagnostic layers distinguish observable grounding, contingency reasoning, and support-response modeling, supported by a multi-stage, audited construction pipeline.

3.1 Task Formulation

  • VIRA-Bench poses multiple-choice questions about short-horizon pedagogical contingencies in 15–60-second tutoring clips, focusing on a single exchange rather than longitudinal scaffolding.The exchange is defined as a teacher intervention eliciting a learner response, followed by a response-based teacher follow-up.Source: §3.1 · pp. 3–4

3.2 Diagnostic Framework

  • The three diagnostic layers separate observable evidence grounding, directed intervention-response relations, and modeling whether support matches learner state.The framework is grounded in the IRF exchange structure and contingent teaching cycle, and layers differ by cognitive operation rather than a simple difficulty order.Source: §3.2 · p. 4

3.3 Clip Extraction from Tutoring Sessions

  • Clip extraction uses ASR segmentation and visual-content-event detection to center 15–60-second windows on teacher interventions, then filters candidates for clip quality and recoverable evidence.From 2,760 event anchors, successive screening and auditing stages yield 1,005 clips with at least one valid MCQ.Source: §3.3 · pp. 4–5

3.4 Multi-Stage Question Construction

  • Construction separates audited structured descriptions from candidate-question generation and uses disjoint model sets for construction and held-out validation.GPT-4o produces descriptions, GPT-5 generates questions, and held-out validators handle shortcut filtering and diagnostic calibration.Source: §3.4 · p. 5

3.4.1 Structured Evidence Description

  • Structured evidence descriptions cover screen events, student behavior, teacher speech acts, temporal dynamics, and cognitive demand, and researchers audit them against source videos before question generation.The audit corrects factual errors and removes clips whose key events cannot be reliably recovered.Source: §3.4.1 · p. 5

3.4.2 Question Construction and Filtering

  • Question filtering matches option length, specificity, and tone, screens for video-agnostic shortcuts and evidence support, then applies expert review before retaining 1,322 MCQs.The expert audit accepted 340 items without change, retained 16 after minor revision, and removed 4 from its 360-question sample.Source: §3.4.2 · pp. 5–6

3.5 Dataset Overview

  • The final benchmark contains 1,322 questions across 1,005 clips, with 497 OEG, 529 ICR, and 296 SRM questions.Clips average 32.4 seconds and come from 72 learners and 22 teachers; each question has four options and one gold answer.Source: §3.5 · p. 6, Figure 3 · p. 6

4 Experiments

The experiments compare 11 models with a stratified human reference, then use input and order manipulations to probe which evidence supports each diagnostic layer. Models remain far below the human reference, with video removal selectively affecting OEG and temporal-order disruption selectively affecting ICR.

4.1 Experimental Setup

  • The evaluation covers 1,322 questions from 1,005 clips with a stratified 200-question human reference and tests zero-shot Video+ASR inputs alongside evidence perturbations.The ASR-only condition removes video; visual shuffle disrupts event order, while synchronized video–ASR shuffle preserves local cross-modal alignment.Source: §4.1 · pp. 6–7

4.2 Human–Model Gap and Layer Profiles

  • 61.3% is the best Video+ASR model accuracy, 29.7 points below the 91.0% human reference on a stratified subset.The five leading models range from 55.9% to 61.3% with overlapping confidence intervals; all top-tier models follow SRM > OEG > ICR.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Response-Format and Shortcut Controls

  • 29.2–29.7% under STEM+OPTIONS, versus 55.9–59.7% under full Video+ASR, shows static question cues alone do not reproduce full-input performance.Free response lowers overall accuracy by 12.1 and 11.7 points, but both tested models retain the OEG-over-ICR ordering.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 Visual Evidence for OEG

  • 12.7–22.0 points is the OEG reduction across five models when video is removed, with every OEG decline significant after Holm correction.Action Identification and Content Recognition drop more than Behavior Extraction, indicating transcripts partially support behavior extraction but cannot replace visual grounding.Source: §4.4 · p. 8

4.5 Temporal-Order Evidence for ICR

  • 11.9–19.1 points is the ICR reduction across five models when visual event order is shuffled, and all five declines survive Holm correction.The largest temporal-order effects concentrate in Sequence Recognition; synchronized video–ASR shuffling also preserves ICR-selective degradation while retaining local alignment.Source: §4.5 · p. 8, Figure 5 · p. 8, Table 3 · p. 9

4.6 Diagnostic Implications

  • Paired interventions suggest OEG development targets event-level speech, teacher-action, and screen-change alignment, while ICR targets preserving actor, target, and order across the exchange.The paper recommends reporting full-input scores with perturbation deltas to distinguish grounding, order-sensitive reasoning, and pedagogical judgment.Source: §4.6 · p. 9

5 Conclusion

VIRA-Bench decomposes short-horizon pedagogical contingency understanding into three theory-anchored layers for one-to-one online tutoring. Experiments show a substantial human–model gap and distinct evidence dependencies for observable grounding and contingency reasoning.

5 Conclusion

  • VIRA-Bench decomposes short-horizon pedagogical contingency understanding into three theory-anchored layers within one-to-one online tutoring.The results identify distinct behavioral evidence dependencies, not an internal architectural mechanism.Source: §5 · p. 9
  • Across 11 models, the best reaches 61.3% against a 91.0% human reference on a stratified subset, with ICR generally lowest among leading models.Removing video most strongly degrades OEG, whereas disrupting event order most strongly degrades ICR, including under aligned video–ASR shuffling.Source: §5 · p. 9
  • Within its one-to-one tutoring scope, VIRA-Bench separates progress in observable grounding from progress in order-sensitive contingency reasoning.Source: §5 · p. 9

6 Limitations

The benchmark evaluates short excerpts from one-to-one online English tutoring, and its findings may not transfer directly to other interaction settings.

6 Limitations

  • Findings from 15–60-second one-to-one online English tutoring excerpts may not transfer directly to multi-party, cross-domain, or longer-horizon interactions.Source: §6 · p. 9
  • LLM-assisted construction receives multiple-model and expert checks, but occasional annotation artifacts may remain.Source: §6 · p. 9
  • The multiple-choice video-plus-ASR protocol targets reproducible multimodal understanding, not open-ended generation, action, or multi-turn adaptation.Source: §6 · p. 9

Ethical Considerations

The source recordings were institutionally approved and shared under protections that limit access and use.

Ethical Considerations

  • Sessions were collected with guardian consent and institutional ethics approval; released clips are de-identified, face-blurred, and restricted to non-commercial academic research.Re-identification is prohibited, and the benchmark is intended for model evaluation rather than high-stakes assessment of individual teachers or students.Source: Ethical Considerations · pp. 9–10

A Clip Extraction Details

VIRA-Bench clips are extracted around visible screen events and interaction windows, then screened and filtered. Evidence descriptions and MCQs are generated and checked through staged validation, yielding the final benchmark.

A Clip Extraction Details

  • Clip extraction combines ASR-guided instructional intervals, screen-change events, and response-aware windows, followed by quality screening and evidence audit.Windows extend 3 seconds before and 12 seconds after merged visual events, can expand to nearby ASR turn boundaries, and are limited to 15–60 seconds; 1,005 clips remain with at least one valid MCQ.Source: A Clip Extraction Details · pp. 13–14

B Prompting and Item Construction Details

  • GPT-4o drafts evidence descriptions, GPT-5 generates candidate questions, and held-out GLM-5V and Claude-Haiku-4.5 validate them; construction models are excluded from the leaderboard.The pipeline uses a construction-evaluation separation protocol.Source: B Prompting and Item Construction… · p. 14

B.1 Structured Evidence Description

  • GPT-4o receives each clip and ASR transcript and produces timestamped structured evidence, marking uncertain observations rather than inferring hidden mental states.The record includes screen events, student behaviors, and teacher speech.Source: §B.1 · p. 14

B.2 MCQ Generation

  • GPT-5 generates one four-option MCQ per target subtask from audited evidence, with constraints against shortcuts and answers inferable from text alone.Subtask instructions specify the cognitive operation, evidence fields, and distractor strategy.Source: §B.2 · pp. 14–15

B.3 Generate-Judge-Regenerate Loop

  • Each item can undergo up to three generate-judge-regenerate iterations, combining programmatic checks, option-only shortcut detection, and evidence-supported difficulty filtering.Items failing after three iterations are discarded.Source: §B.3 · p. 15

B.4 Item Filtering Statistics

  • 1,322 MCQs remain after expert adjudication of items passing the multi-stage filtering pipeline.The benchmark’s filtering stages are summarized in Table 5.Source: §B.4 · p. 15

C Expert Verification

Three domain experts audited a stratified set of benchmark items, checking clarity, answer correctness, evidence sufficiency, and subtask alignment. The audit found high acceptance and bounded the estimated residual defect rate.

C Expert Verification

  • Three experts reviewed 300 stratified random and 60 high-risk items across four quality dimensions; Fleiss’ κ for binary acceptability was 0.76.Evidence sufficiency had the lowest per-dimension agreement (κ = 0.68), particularly for Support-Response Modeling items; Interaction Contingency Reasoning had slightly more revisions due to temporal-evidence judgments.Source: C Expert Verification · pp. 15–16
  • 340 of 360 audited items were accepted without change, 16 after minor revision, and 4 removed, for 94.4% direct and 98.9% final acceptance.The removals reflected insufficient visual evidence, an ambiguous gold answer, or a subtask mismatch.Source: C Expert Verification · p. 16
  • The 360-item expert audit bounds expected residual defective items in the 1,322-item benchmark to fewer than 37.The reported Wilson confidence interval is [0.4%, 2.8%].Source: C Expert Verification · p. 16

D Benchmark Statistics

The benchmark draws on short clips from English-medium online tutoring, with learners from varied first-language backgrounds and teacher code-switching. Its instructional-context diversity is limited.

D Benchmark Statistics

  • Across 1,005 clips, duration averages 32.4 seconds, and each clip contains one to three MCQs, with a mean of 1.32.Seventy-two percent of clips have one question, 24% have two, and 4% have three.Source: D Benchmark Statistics · p. 16
  • The source sessions are English-medium tutoring, while learners’ first-language backgrounds and teacher code-switching provide the primary variation.Learners are primarily Mandarin L1, with some English L1 students; teacher speech is estimated at 75% English and 25% other languages by word count.Source: D Benchmark Statistics · p. 16
  • Instructional-context diversity is limited because the sessions share an English-medium format; variation is mainly in learner L1 background and teacher code-switching.The paper identifies this as a scope limitation.Source: D Benchmark Statistics · pp. 16–17

E Evaluation Protocol

The evaluation uses video and transcript inputs with controlled evidence perturbations, standardized scoring, and paired significance tests. A stratified expert evaluation provides a 91.0% majority-vote human reference.

E Evaluation Protocol

  • Under the full condition, models receive native-resolution video sampled at 1 frame per second and ASR as a separate text input.The clips are 720p and 15–60 seconds long; models use provider-default generation parameters in a zero-shot setting.Source: E Evaluation Protocol · p. 17
  • Accuracy is the proportion of items matching the gold answer, reported overall, by diagnostic layer, and across eight subtasks.Option order is randomized per model–item pair, and responses are parsed from JSON or fallback option matching; the random baseline is 25%.Source: E Evaluation Protocol · p. 17
  • 47.6% of shared errors between two Qwen3-VL variants selected the same distractor content after independent option shuffling.A permutation test gave p < 0.001, consistent with shared preference for plausible distractors rather than fixed answer positions.Source: E Evaluation Protocol · p. 18
  • Paired item-level accuracy comparisons use two-sided McNemar tests with Holm correction across comparisons.Adjusted p-values are assessed at α = 0.05, 0.01, and 0.001.Source: E Evaluation Protocol · p. 18

F Human Reference Details

  • Three applied-linguistics experts independently answered the 200 questions after viewing clips with ASR transcripts, and majority voting yielded 91.0% accuracy.Pairwise agreement on correct versus incorrect decisions exceeded 90% across all expert pairs.Source: F Human Reference Details · p. 18
  • 91.0% majority-vote accuracy was achieved by three experts on the 200-question stratified human reference set.The set sampled 25 questions from each of eight subtasks; individual expert totals averaged 87.0%.Source: F Human Reference Details · p. 18

G Paired Significance Analysis of Evidence Dependency

Paired perturbations show distinct evidence dependencies across diagnostic layers, while the benchmark’s design distinguishes interactional causality from physical event causality. The surrounding sections describe transcript de-identification and the benchmark’s intended scope and use.

G Paired Significance Analysis of Evidence Dependency

  • 12.7–21.9 percentage-point OEG drops followed video removal, while 11.9–19.1-point ICR drops followed visual shuffling; SRM changes were mostly nonsignificant.The OEG declines and ICR declines remained significant after correction; SRM showed larger model-specific effects under particular perturbations.Source: G Paired Significance Analysis of… · p. 18, Table 10 · p. 20
  • Visual shuffling permutes coherent event segments, disrupting intervention–response–follow-up order while preserving local visual evidence.OEG stayed largely stable for most models, whereas ICR declined; Sequence Recognition was expected to show the strongest subtask degradation.Source: G Paired Significance Analysis of… · pp. 18–19

I Positioning Against Causal Video QA

  • Unlike physical-causality video QA, VIRA-Bench tests whether one participant’s action elicits another’s response and shapes subsequent behavior.Its diagnostic layers are grounded in pedagogical theory, and controlled evidence perturbations support construct-validity analysis beyond aggregate accuracy.Source: I Positioning Against Causal Video QA · p. 19

J Transcript De-identification

  • Transcript de-identification replaces detected names with role labels, preserving who addresses whom while removing personal names.The process detected 1,340 person mentions across 1,005 released transcripts; no subtask asks for identity or uses names as answer-bearing evidence.Source: J Transcript De-identification · p. 19

K Datasheet

  • VIRA-Bench comprises 1,322 multiple-choice questions across 1,005 real tutoring clips from English-medium instruction with Chinese L1 students.Sessions were recorded from 2018 to 2021, with clips selected for instructional richness and interaction density.Source: K Datasheet · pp. 19–20
  • The benchmark is intended for multimodal-model evaluation, not teacher evaluation, student assessment, high-stakes decisions, or training.Using it as training data would compromise its diagnostic validity.Source: K Datasheet · p. 20

Related papers

Loading 2610.29428v1…