PaperlayerView exact revision on arXiv

Source-linked AI summary

Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Xusen Hei, Jiali Chen, Jinyu Yang, Yi Cai

arXiv:2611.02928v1cs.MM

TL;DR

Experiment videos let learners observe procedures but not perform them, and converting them into virtual experiments requires fidelity to the apparatus and state changes as well as operational preconditions. Lab2Sim provides a benchmark, and EduVideo2Code uses hierarchical procedure extraction, state-machine-guided generation, and execution-based verification and repair. EduVideo2Code improves all four evaluation dimensions on both tested backbones, with apparatus-fidelity gains of up to 24.15 points.

  • Problem

    Experiment videos let learners observe rather than conduct experiments, while interactive conversion must reproduce apparatus and state transitions and enforce operation preconditions.

    Source: §1 · p. 1

  • Method

    Lab2Sim pairs 218 experiment videos with grounded specifications and 11,881 rubric items; EduVideo2Code extracts procedures hierarchically, uses state machines to guide code generation, and verifies and repairs pages through browser execution.

    Source: §1 · pp. 1–2

  • Results

    EduVideo2Code improves all four evaluation dimensions on both tested backbones, with gains of up to 24.15 points in apparatus fidelity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that usable virtual experiments require code generation together with explicit procedural modeling and execution-based verification.

    Source: §1 · p. 2

    Generated pages were assessed only with automatic metrics, their effects on learners remain unstudied, and educators should review them before classroom use.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.

1 Introduction

The paper addresses the challenge of turning experiment videos into virtual experiments that are both faithful to demonstrations and operable by learners. It introduces Lab2Sim and EduVideo2Code, and reports that the framework improves evaluation results despite a gap between runnable pages and apparatus fidelity.

1 Introduction

  • Existing approaches either reconstruct interfaces without inferring experimental physical states and procedural constraints or produce textual scientific-video outputs rather than interactive environments.The paper frames the remaining challenge as making video-derived virtual experiments both faithful to demonstrations and operable by learners.Source: §1 · p. 1
  • EduVideo2Code improves all four evaluation dimensions on both tested backbones, with gains of up to 24.15 points in apparatus fidelity.Across six multimodal models, more than 93% of directly generated pages are runnable, but no model reaches 60% apparatus fidelity.Source: §1 · p. 2
  • Lab2Sim pairs 218 real-world experiment videos with evidence-grounded specifications and 11,881 executable rubric items for evaluating fidelity and operability.EduVideo2Code extracts procedures hierarchically, uses an explicit state machine to guide code generation, and verifies and repairs pages through browser execution.Source: §1 · pp. 1–2

2 Related Work

Prior work reconstructs and evaluates interfaces, while scientific-video research focuses on understanding experiments and producing textual outputs. EduVideo2Code targets interactive virtual experiments that respond to learner operations.

2.1 Front-End Code Generation

  • Interface-generation research progressed from reconstructing static layouts to evaluating interactive behavior and recovering states, actions, and transitions from recordings.The cited work covers screenshots, sketches, and design specifications as well as execution-based benchmarks and interface recordings.Source: §2.1 · p. 2

2.2 Scientific Experiment Understanding

  • Prior physical-demonstration methods target settings without scientific procedural constraints; EduVideo2Code instead infers an experiment state machine and checks page states against the video.Scientific-video methods produce labels, answers, or text, but not an environment in which the experiment can be performed; EduVideo2Code generates one whose states respond to learner operations.Source: §2.1 · p. 2, §2.2 · pp. 2–3

3 Lab2Sim

Lab2Sim evaluates browser-based virtual experiments against video-grounded specifications and rubric items. Its construction screens experiment videos, annotates evidence-linked specifications, and converts them into rubrics across fidelity and operability dimensions.

3.1 Task Definition

  • Each Lab2Sim instance supplies a source video and instruction, with a gold specification and derived rubric used to score the generated page against experiment content and behavior.A page must run in the browser before its rubric items are scored; Runnable Rate is reported separately.Source: §3.1 · p. 3, §3.4 · p. 4

3.2 Video Collection

  • The authors screened more than 500 public instructional and demonstration videos from Bilibili and YouTube, retaining 218 whose procedure, apparatus, and observable state transitions were identifiable.They excluded duplicate recordings, videos below 720p, overly complex or occluded experiments, and footage without the procedure.Source: §3.2 · p. 3

3.3 Evidence-Grounded Annotation

  • Gold specifications record apparatus, initial conditions, ordered steps, preconditions, and state transitions, with each annotated element linked to source frames or a time interval.Human reviewers verify model-drafted specifications against the video; fixed templates derive traceable rubric items, including domain-labeled violation probes.Source: §3.3 · pp. 3–4

3.4 Rubric Evaluation Dimensions

  • The rubric separates fidelity—apparatus and state-transition matching—from operability—step execution and precondition enforcement.Scores target video-visible elements and violation probes; items based on apparatus inference or domain knowledge are retained but excluded from the main scores.Source: §3.4 · p. 4

3.5 Benchmark Statistics

  • Lab2Sim contains 218 instances across chemistry, physics, and biology, averaging 7.1 steps and 23.2 annotated state transitions per instance.The benchmark contains 11,881 rubric items, averaging 54.5 per instance.Source: §3.5 · p. 4

4 EduVideo2Code

EduVideo2Code turns a video-grounded procedure into an interactive page through procedure extraction, state-machine-driven generation, and browser-based verification and repair. The stages use evidence and runtime behavior to guide targeted corrections.

4.1 Overview

  • The compile–verify–repair framework extracts steps and state transitions, compiles them into an experiment state machine for page generation, then replays and checks the page in a browser.Detected errors can trigger targeted repair of the procedure, state machine, or apparatus, followed by repeated verification.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 Hierarchical Procedure Extraction

  • Hierarchical procedure extraction uses a coarse full-video pass to identify phases, apparatus, and step order, then inspects ambiguous segments with denser sampling and high-resolution frames.Each output step is linked to supporting video frames, and the resulting procedure is both the input to generation and the reference for verification.Source: §4.2 · p. 5

4.3 State-Machine-Driven Generation

  • State-machine-driven generation separates experimental logic from layout and interaction code by compiling the procedure into states, operations, and operation preconditions.Operations become learner gestures and states become visual appearances, so learners perform the experiment rather than replaying a fixed animation.Source: §4.3 · p. 5

4.4 Execution-Based Verification

  • The independent browser verifier compares apparatus and spatial relations, tests invalid operations, and replays the demonstrated procedure against video evidence.A score below threshold prompts targeted repair: apparatus errors trigger asset search, while operation or state-transition errors prompt procedure reinspection or state-machine revision.Source: §4.4 · pp. 5–6

5 Experiments

Across Lab2Sim’s 218 instances, evaluation separates page executability from experimental fidelity and operability. EduVideo2Code improves all four rubric dimensions on both tested backbones, while ablations and case studies expose stage dependencies and distinct reconstruction failures.

5 Experiments

  • The experiments compare EduVideo2Code with six multimodal baselines and test the framework with GLM-5.3-Flash and GPT-5.6-Sol as backbones.All systems are evaluated on the 218 Lab2Sim instances using four rubric dimensions and diagnostic metrics.Source: §5 · p. 6, §5.1 · p. 6

5.1 Experimental Setup

  • All eight systems exceed 93% Runnable Rate, yet AF ranges from 20.62 to 61.32 and PE from 20.11 to 41.40; loading and responding does not establish experimental fidelity.Evaluation runs generated pages in a sandboxed browser, replays annotated steps, and separately scores rubric items against attached video evidence.Source: §5.2 · p. 6, §5.1 · p. 6

5.2 Main Results

  • Both EduVideo2Code variants improve AF, PE, PCC, and STF over direct generation with the same backbone; AF rises by 20.88 points for GLM-5.3-Flash and 24.15 for GPT-5.6-Sol.Both variants also reduce OFR and reach a 100% Runnable Rate, but PCC gains differ sharply: 22.00 points for GLM-5.3-Flash versus 1.19 for GPT-5.6-Sol.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Ablation Study

  • Removing any of the three stages worsens every metric; HPE removal causes the largest AF, PE, and PCC drops, while SMG removal causes the largest STF drop.The ablation analysis suggests effective precondition enforcement needs preconditions recovered by HPE and checked in the browser by EBV; the stages depend on one another.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Case Study

  • Case studies show failures ranging from blank rendering and wrong experiments or apparatus to operations that do not produce the demonstrated state changes.EduVideo2Code reproduces demonstrated operations and states in the examples, but represents condensation with text rather than a corresponding visual change.Source: Figure 4 · p. 8, §5.4 · p. 8

6 Conclusion

The paper introduces Lab2Sim for evaluating interactive virtual experiments reconstructed from real-world experiment videos, and proposes EduVideo2Code to generate and refine them. Results show a persistent gap between runnable pages and faithful reconstructions, which the framework narrows through explicit state machines and browser-based verification.

6 Conclusion

  • Lab2Sim provides video-grounded specifications and executable rubric items as concrete targets for reconstructing interactive virtual experiments.Browser execution connects these targets to generated-environment behavior.Source: §6 · p. 8
  • EduVideo2Code is a compile-verify-repair framework that uses explicit state machines and execution-based verification to narrow the gap between runnable pages and faithful experiments.The framework is presented as a way to translate physical demonstrations into faithful, executable environments.Source: §6 · pp. 8–9
  • Evaluations across six models reveal a persistent gap between producing runnable pages and faithfully reproducing the demonstrated experiment.The paper describes its work as advancing the study of translating physical demonstrations into faithful, executable environments.Source: §6 · pp. 8–9

Limitations

The benchmark evaluates pages against what source videos show, within a bounded set of videos and disciplines, and relies partly on model judging. Its annotation guidelines specify how evidence, steps, states, and preconditions are recorded.

Limitations

  • Lab2Sim does not evaluate scientifically expected behavior that the source video does not demonstrate; its 218 videos cover three disciplines and do not represent all languages, regions, or laboratory settings.Model-judge agreement with human judgments was measured on 50 sampled cases, and generated pages have not been evaluated for learner effects or classroom readiness.Source: Limitations · p. 9

A Gold Annotation Guidelines

  • Evidence collection uses sampled frames and targeted resampling until apparatus is visible in cited evidence or confirmed absent; unsampled frames are not evidence of absence.Near-duplicate filtering can be bypassed for edited still-shot videos when frames are confirmed to show a real state change.Source: A Gold Annotation Guidelines · pp. 10–11
  • Annotation proceeds from global experiment phases to local steps, recording each manipulation or observation, its timing, apparatus, state transitions, evidence labels, and cited frames.Packaging such as title cards is excluded, and timestamps must be monotonic and within the video duration.Source: A Gold Annotation Guidelines · p. 11
  • States encode one attribute update for a named object, while each precondition records an operation and state overrides that make the condition fail.Facts are labeled video_visible, apparatus_inferred, or domain according to their evidential basis.Source: A Gold Annotation Guidelines · p. 11

B Rubric Instantiation Templates

The rubric is instantiated from reviewed gold-specification fields, with additional probes testing whether pages block invalid operations. Its 11,881 items cover fidelity and operability, while inferred-apparatus and domain items are excluded from the main scores.

B Rubric Instantiation Templates

  • Each rubric item records its question, gold answer, check specification, evidence, source field, and inherited evidence label.Checks may use model judgment, a script, or both; evidence is attached according to item type and may fall back to source-field frames when a mosaic is unavailable.Source: B Rubric Instantiation Templates · p. 12
  • Violation probes test skipped steps, swapped adjacent steps, fivefold quantities, or substitutions to another same-kind container, and pass when the page blocks the operation with textual feedback.These domain-labeled probes count toward PCC alongside constraint items.Source: B Rubric Instantiation Templates · p. 12
  • 11,881 rubric items comprise 15.54% identity, 13.03% execution, 42.57% state, and 28.86% PCC items, including constraint items and violation probes.The PCC share includes 8.10% constraint items and 20.76% violation probes; the PE and STF shares correspond to 7.1 steps and 23.2 state transitions per instance.Source: B Rubric Instantiation Templates · p. 12
  • Gold-specification fields map through fixed templates to identity, execution, constraint, and state items, while separate violation probes count toward PCC.The templates map fields to candidate rubric items; each field yields one item, with duplicates removed during human review.Source: B Rubric Instantiation Templates · pp. 11–12, Table 5 · p. 13
  • Apparatus-inferred and domain-labeled items remain in the released rubric but are excluded from the main scores.Main scores include video-visible elements and violation probes.Source: B Rubric Instantiation Templates · pp. 12–13

C Quality Control

Quality control combines schema checks, frame-grounded human review, and separate verification of apparatus localization. Reviewers also filter candidate items, while judge agreement is measured on a sampled subset.

C Quality Control

  • Every gold specification is schema-validated before evidence derivation, including checks on frame references, entities, boxes, labels, step timing, and required metadata.The validator also rejects pseudo-steps such as title cards or pure equipment displays.Source: C Quality Control · p. 13
  • Two reviewers check each gold specification against cited frames, with a third deciding disagreements; about 20% of drafted elements are corrected or removed.Review covers apparatus, steps, state transitions, preconditions, and their evidence labels.Source: C Quality Control · p. 13
  • Apparatus boxes are independently verified; failed boxes are retried up to three rounds before the entity is downgraded to apparatus_inferred.The verification rejects empty, incorrect, truncated, or not-visible boxes; no unverified box enters the gold specification.Source: C Quality Control · p. 13
  • The assisting model annotates each instance three times, keeping majority readings for most fields but inspecting full-resolution frames when directly observable readings disagree..Source: C Quality Control · p. 13
  • Each candidate item is checked by two reviewers, with a third deciding disagreements; about five percent of candidate items are removed.Items are kept by default, and removals are assigned a reason such as ambiguity, insufficient evidence, or duplication.Source: C Quality Control · pp. 13–14
  • On 2,983 kept items from 50 sampled cases, judge-human agreement reached Cohen’s κ of 0.79; scaling calibration to the full benchmark remains future work.Source: C Quality Control · p. 14

D Evaluation Protocol Details

The evaluation runs every system on all 218 instances, using uniformly sampled video frames as the only visual input. Pages are checked for runnability before rubric scoring, then an agent replays steps and scores are aggregated across items.

D Evaluation Protocol Details

  • All systems are evaluated on the full 218-instance set, with no instance excluded.Source: D Evaluation Protocol Details · p. 14
  • Each baseline receives 32 uniformly sampled frames per video as its only visual input.Source: D Evaluation Protocol Details · p. 14
  • A page is runnable if it renders qualifying content within 15 seconds without a fatal runtime error; rubric items are scored only for runnable pages.Non-runnable pages lower Runnable Rate but their rubric items are marked unscorable and omitted from the four rubric dimensions.Source: D Evaluation Protocol Details · p. 14
  • The evaluation agent executes annotated steps in order in one page session, using screenshots and extracted interactive elements to choose actions, with six turns per step.Unexecuted steps receive up to two retries, each with a fresh six-turn budget, for at most 18 turns per step.Source: D Evaluation Protocol Details · pp. 14–15
  • Items receive scores of 1, 0.5, or 0, or are marked unscorable; unscorable items are excluded from their dimension’s denominator.Apparatus fidelity uses a judge’s match, mismatch, or uncertain verdict; procedure execution checks whether the core operation is accepted and produces a change, phenomenon, or success message.Source: D Evaluation Protocol Details · p. 15
  • State items combine script assertions and visual probes when page state is exposed; only a score of 1 counts as a pass in aggregation.If state is not exposed or the gold value lacks a page-vocabulary counterpart, scoring uses the probe alone; dimensions aggregate pass fractions over pooled scored items.Source: D Evaluation Protocol Details · p. 15

E Generation, Agent, and Judge Prompts

The appendix provides the generation, agent, and judge prompts used in the experiments. The generation instruction is shared across instances, while evaluation prompts guide the agent and specify JSON verdicts for the judge.

E.1 Generation Instruction

  • The generation instruction contains no experiment-specific information, so the video is the only source of experimental content.Source: §E.1 · p. 15
  • The instruction was written in Chinese and English with equivalent content, and the appendix reproduces the English version.Source: §E.1 · p. 15
  • The instruction refers to a video file, supplied in the experiments as 32 uniformly sampled frames.Source: §E.1 · p. 15
  • Figure 5 presents the generation instruction shared by all baselines and instances.Source: §E.1 · p. 15, Figure 5 · p. 16

E.2 Evaluation Agent

  • At each turn, the evaluation agent receives Figure 6’s prompt and the current screenshot, with interactive elements supplied in the prompt’s {elements} field.It executes one annotated step at a time and returns a single action per turn.Source: §E.2 · p. 17

E.3 Judge Prompts

  • For each item, the judge receives a dimension-specific Figure 7 prompt requesting a JSON verdict that is parsed into the item score.Source: §E.3 · p. 17

F Additional Case Studies

Additional Lab2Sim cases show both successful reconstructions with missed interactions and distinct failure modes, including procedure drift, unresponsive operations, and stalled initialization.

Successful reconstructions

  • Successful pages reproduce key experimental effects and block an invalid reagent addition, yet still miss individual interactions or visible state changes.The acetaldehyde page reproduces the color change and blocks CuSO4 before NaOH, but lacks shaking; the polarization page misses the initial analyzer movement, and the transpiration page’s airflow step has no visible effect.Source: F Additional Case Studies · p. 17

Failure cases

  • Procedure drift makes the deformation page require an unshown hand-pull step, causing it to reject weight-hanging steps demonstrated in the video.Removing the weight still works as demonstrated.Source: F Additional Case Studies · p. 17
  • In the rusting case, operations leave the test tubes unchanged and rust is never rendered; the page scores PE 0 and STF 14.3.It reproduces part of the apparatus but none of the state transitions.Source: F Additional Case Studies · p. 17
  • The manometer page loads its frame and step panel but remains loading, with zero steps and an empty workbench, despite being counted as runnable.Figures 8 and 9 present source-video frames above generated-page frames; the cases include procedure-execution verdicts and rubric scores.Source: F Additional Case Studies · p. 17, Figure 9 · p. 21

Related papers

Loading 2611.02928v1…