PaperlayerView exact revision on arXiv

Source-linked AI summary

LabWorld: Learning Interactive 4D Worlds for Scientific Experimentation

Suozhi Huang, Kelsey Fu, Binxu Li, Luna Lyu, Kaize Wu, Wenbo Lu, David Smerkous, Jiahan Li, Siqi Zhu, Zhizhou Sha, Jiacheng Guo, Zhixuan Liang, Shilong Liu, Le Cong, Mengdi Wang

arXiv:2610.36592v1cs.CV

TL;DR

Laboratory simulators often rely on curated assets and predefined procedures, while ordinary monocular videos leave equipment, layout, and interactions implicit. LabWorld reconstructs those videos as editable 4D worlds with event-constrained motion; it achieves the highest reported mean scores for pipetting and vortexing, while event conditioning improves modeled physical-constraint satisfaction in a five-video ablation.

  • Problem

    Existing laboratory simulators rely on manually constructed assets and predefined procedures, while monocular laboratory video leaves equipment, layouts, and interactions implicit.

    Source: §1 · p. 1

  • Method

    LabWorld reconstructs an editable 4D laboratory world from monocular video, combining reusable assets and metric scene state with time-varying objects and event-conditioned motion constraints.

    Source: §1 · pp. 1–3

  • Results

    LabWorld achieves the highest mean Mask IoU (0.501) for pipetting and Tube-Length–Normalized PCK-AUC (0.638) for vortexing; event conditioning raises modeled constraints satisfied from 5/35 to 35/35 on five videos.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Recorded human experiments can provide a source for explicit, editable representations in which scene geometry, object motion, and procedure events can be inspected and modified.

    Source: §6 · p. 10

    Severe hand occlusion can increase geometry and motion uncertainty, and event constraints depend on accurate event extraction and scene geometry estimation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Autonomous scientific agents must not only reason about experiments but also interact with the physical environments in which they are executed. Existing laboratory simulators typically rely on manually constructed assets and predefined procedures, while laboratory videos capture real equipment, layouts, and interactions but leave this information implicit in pixels. We introduce LabWorld, an agent-driven Real2Sim framework for reconstructing interactive 4D laboratory worlds from monocular RGB video. LabWorld recovers reusable 3D assets, a metric scene, time-varying object states, and procedure events. Multi-frame observations provide complementary evidence for asset and scene reconstruction, while motion reconstruction combines visual and geometric observations with event constraints encoding the spatial and temporal requirements of laboratory interactions. We evaluate LabWorld on real pipetting and vortexing videos collected in our laboratory against general-purpose agentic and dedicated reconstruction baselines. For pipetting, LabWorld achieves a mean Mask IoU of 0.501, exceeding the strongest baseline by 0.033. For vortexing, it achieves a mean Tube-Length-Normalized PCK-AUC of 0.638, exceeding the strongest baseline by 0.035. On five pipetting videos, event-conditioned reconstruction increases the fraction of modeled physical constraints satisfied from 14.3% (5/35) to 100% (35/35). The resulting worlds expose geometry, motion, articulation, and events as explicit, editable state for replay, inspection, and future embodied scientific systems.

1. Introduction

Laboratory videos contain equipment, layout, and interactions implicitly, while existing approaches leave gaps in recovering reusable assets, observed motion, and procedure constraints together. LabWorld reconstructs editable 4D worlds from monocular video, and its evaluations show strong benchmark results while illustrating that image agreement alone can diverge from modeled physical validity.

1. Introduction

  • Existing systems do not jointly recover reusable laboratory assets, observed object motion, and procedure-specific interaction constraints from ordinary monocular video.Laboratory simulators rely on curated assets and predefined procedures, while many Real2Sim systems assume asset libraries, depth sensing, or active capture.Source: §1 · p. 1
  • 0.501 mean Mask IoU for pipetting and 0.638 mean Tube-Length–Normalized PCK-AUC for vortexing were the highest across evaluated baselines.On a five-video ablation subset, removing event-conditioned reconstruction raised pipette Mask IoU from 0.508 to 0.572 while reducing modeled physical requirements satisfied from 35/35 to 5/35.Source: §1 · p. 2
  • LabWorld converts monocular laboratory video into an editable 4D world with reusable assets, a metric scene, time-varying object states, and procedure events.The resulting representation exposes geometry, camera state, trajectories, articulation, and events as independently inspectable variables.Source: §1 · pp. 2–3
  • Event-conditioned reconstruction constrains motion using the spatial and temporal requirements of procedure events, beyond visual agreement alone.A pipette–beaker interaction requires approach, entry, dwell, and withdrawal through the opening rather than merely appearing near the beaker in projection.Source: §1 · pp. 2–3
  • The evaluation uses real laboratory pipetting and vortexing videos and examines visual refinement, geometric initialization, temporal regularization, and event-conditioned reconstruction.The planning and editing panels in Figure 1 are conceptual illustrations, not evaluated downstream results.Source: Figure 1 · p. 2, §1 · p. 3

2. Related Work

Related work spans laboratory simulation, Real2Sim reconstruction, and perception and representation components. LabWorld addresses the complementary task of recovering explicit assets, scene state, motion, and interaction structure from observed laboratory procedures.

2.1 Laboratory Simulators and Digital Twins

  • Laboratory simulators and digital twins provide scientific environments but primarily build them from curated assets, protocols, or engineered workstations rather than observed human-procedure states.LABWORLD addresses a complementary representation problem to systems for closed-loop scientific reasoning, robotic execution, and XR guidance.Source: §2.1 · p. 3

2.2 Real2Sim and Motion Reconstruction

  • Many Real2Sim and motion-reconstruction systems require active capture, RGB-D, robot state, or controlled interaction, while closer video methods focus on scene reconstruction or manipulated-object motion.LABWORLD reconstructs object motion within an explicit laboratory world and uses procedure events to constrain ambiguous trajectories.Source: §2.2 · p. 4

2.3 Assets, Geometry, and Constraint Representations

  • View synthesis, geometry, tracking, segmentation, asset generation, articulation, and relational constraints provide useful components but do not alone define reusable objects and functional interaction requirements.LABWORLD integrates reusable assets and metric geometry with source-video events that constrain rigid and articulated motion.Source: §2.3 · p. 4

3. Method

LabWorld recovers an explicit laboratory world from monocular RGB video through object and event identification, asset and scene reconstruction, and constrained motion reconstruction. The resulting representation keeps geometry, motion, articulation, and event timing editable and independently inspectable.

3. Method

  • LabWorld represents a laboratory world using reusable assets, a metric static scene, camera parameters, time-varying rigid poses and articulated states, and a structured event list.The pipeline identifies objects and procedure events, constructs assets and a metric scene, then reconstructs motion under visual, geometric, temporal, event, and physical constraints.Source: §3 · p. 4

3.1 Object Identification and Event List Generation

  • A vision-language model identifies task-relevant entities and functional parts, while video segmentation tracks object identities and an agent infers structured procedure events.Events specify interaction types and time windows; for example, a dispense event links a pipette tip to a beaker opening over the interaction interval.Source: §3.1 · pp. 4–5

3.2 Laboratory Assets and Scene Reconstruction

  • LabWorld selects complementary object views to generate or reuse assets, then combines estimated depth, point maps, camera parameters, support surfaces, and known dimensions to assemble a metric scene.Assets include canonical frames, physical scales, collision geometry, and functional landmarks; articulated equipment is represented as links and joints.Source: §3.2 · p. 5

3.3 Motion Reconstruction with Event Constraints

  • Motion reconstruction combines rendered-observation agreement, metric geometry, temporal smoothness, event-specific spatial and temporal relations, and physical constraints.Event relations can require ordered approach, entry, dwell, and withdrawal phases; physical terms enforce dimensions, articulation limits, support, and non-penetration. The output is independently editable and replayable.Source: §3.3 · p. 6

4. Evaluation Protocol

The evaluation measures image-space agreement and, for pipetting, satisfaction of modeled physical interaction requirements. It fixes camera alignment, references, scoring, and ablation conditions to compare reconstruction methods and components.

4.2 Baseline Comparison Protocol

  • Camera alignment is frozen before motion scoring, references are excluded only when unavailable or unreliable, and video-level metrics are averaged equally.Reconstructions retain native geometry, camera, scale, and trajectory, with no post-hoc alignment or score-driven output selection; baselines receive no LABWORLD trajectories, event constraints, or evaluation reference points.Source: §4.1 · p. 6, §4.2 · p. 7

4.1 Image-Space Reconstruction Evaluation

  • Pipetting Mask IoU compares rendered silhouettes with visible SAM3 masks, while vortexing uses Tube-Length–Normalized PCK-AUC because hand occlusion makes silhouette IoU unsuitable.The vortexing metric measures 2D full-tube center localization, not shape, orientation, 3D pose, or physical interaction; its references are source-assisted and provisional.Source: §4.1 · pp. 6–7

4.3 Physical Constraint-Satisfaction Diagnostics

  • Pipetting diagnostics test seven fixed geometric and temporal requirements at 24 Hz across five videos, yielding 35 checks per condition.The checks overlap with event-conditioned constraints and measure consistency with correctly executed demonstrations, not independent recovery of hidden events or liquid transfer.Source: §4.3 · p. 7

4.4 Ablation Protocol

  • Ablations use the same observations, masks, camera, asset geometry, and evaluation, changing event planning, temporal smoothing, depth initialization, or visual refinement.The event ablation retains generic physical constraints; trajectory acceleration is additionally reported for the temporal ablation.Source: §4.4 · pp. 7–8

5. Results

LABWORLD has the highest reported mean reconstruction accuracy on both pipetting and vortexing, although the task metrics measure different properties. Reduced visibility coincides with lower pipetting agreement on average, while event conditioning preserves modeled interaction constraints despite lower image-space overlap.

5.1 Reconstruction Accuracy

  • LABWORLD achieves the highest mean Mask IoU for pipetting (0.501) and highest Tube-Length–Normalized PCK-AUC for vortexing (0.638), ahead of every listed baseline.Pipetting: Do as I Do 0.468, GPT-6 Astra 0.156, FoundationPose 0.026; vortexing: Do as I Do 0.603, GPT-6 Astra 0.479, FoundationPose 0.232.Source: §5.1 · pp. 8–9, Table 2 · p. 9

5.2 Reconstruction Under Reduced Visibility

  • Higher-visibility pipetting observations have mean Mask IoU 0.545 (n = 503), versus 0.349 for lower-visibility observations (n = 144).Across 647 observations, the occlusion proxy has Pearson r = −0.347 but Spearman ρ = −0.070; the proxy and IoU both depend on the predicted silhouette.Source: Figure 4 · p. 9, §5.2 · p. 9

5.3 Ablation Study

  • Removing event constraints raises Mask IoU from 0.508 to 0.572 but drops physical constraint satisfaction from 35/35 to 5/35 checks.The component summary attributes image-agreement gains to visual refinement and depth initialization, smoother motion to temporal regularization, and preserved 3D interaction structure to event conditioning.Source: §5.3 · p. 10, Table 3 · p. 10

6. Conclusion

LABWORLD turns human laboratory demonstrations into explicit, editable 4D worlds, with event constraints shaping motion reconstruction. Evaluation and component ablations provide empirical support, while the approach depends on accurate event extraction and scene geometry.

6. Conclusion

  • LABWORLD converts recorded laboratory scenes and procedures into explicit, editable worlds where geometry, object motion, and events can be inspected and modified.The paper presents these worlds as a way to construct environments for physical-intelligence research and a foundation for future robot learning and experimental planning.Source: §6 · p. 10
  • Event-conditioned reconstruction incorporates spatial and temporal requirements of experimental interactions into motion reconstruction alongside image agreement.The conclusion says evaluation on pipetting and vortexing videos, together with component ablations, provides empirical support for the framework and the role of event constraints.Source: §6 · p. 10
  • Severe hand occlusion can increase geometry and motion uncertainty, and event constraints depend on accurate event extraction and scene geometry estimation.Source: §6 · p. 10

A. Agent Workflow and Reproducibility Scope

The workflow automates preparation and reconstruction, exporting an editable world with separate scene, object-state, and event information. Its motion reconstruction uses event-specific geometric requirements and landmark-based trajectory reasoning.

A. Agent Workflow and Reproducibility Scope

  • The current reproduction package starts from cached observations, reusable assets, and saved configurations, so it does not independently establish fully automatic reconstruction from raw video.End-to-end evaluation must separately record input provenance, agent actions, and human intervention.Source: A. Agent Workflow and Reproducibility… · p. 15

B. Executable World Representation

  • The exported world separates reusable assets, a static metric scene, camera state, rigid-object poses, articulated-joint states, and a scientific event list for independent inspection and editing.A downstream agent can change an initial object state or action while retaining the remainder of the reconstructed laboratory.Source: B. Executable World Representation · p. 15

C. Event Vocabulary and Geometric Constraints

  • Event constraints encode functional spatial relations, and temporally tapered confidence softens their influence around event boundaries without guaranteeing trajectory continuity.The dispensing proximity term is not a complete event constraint or completion test; it must be combined with phase-specific depth, timing, and non-penetration conditions.Source: C. Event Vocabulary and Geometric… · pp. 15–16, Table 4 · p. 15

C.1 Pipetting Event Definitions

  • Pipette–beaker evaluation checks ordered entry, dwell, and withdrawal through the opening, while rack or tip-box region access alone does not establish exact contact or insertion.The configured shallow insertion depth is 8 mm and is an explicit reconstruction constraint, not a measurement of the hidden tip; the event evaluates pipette motion, not liquid transfer.Source: §C.1 · p. 16

D. Trajectory Reconstruction Details

  • For elongated tools, semantic endpoints or landmarks and a canonical length constrain lifted 3D endpoints, while roll requires additional asymmetric or non-collinear evidence.When that evidence is occluded, roll is propagated temporally; the final optimizer couples translation and rotation corrections across time and applies event and physical terms after image-space alignment.Source: D. Trajectory Reconstruction Details · pp. 16–17

E. Evaluation and Validity Masks

Evaluation uses sequence-level camera alignment and validity flags to report image-space metrics only on observable, trustworthy frames. The system also has limitations in asset fidelity, monocular geometry, physical modeling, and domain scope.

E. Evaluation and Validity Masks

  • A single camera alignment estimated from static-scene evidence is frozen across the sequence, and per-frame validity is determined without access to reconstruction error.Frames are not excluded because IoU is zero or a predicted trajectory is poor; the report includes both valid-frame metrics and validity coverage.Source: E. Evaluation and Validity Masks · p. 17
  • An oracle global translation is retained only as a diagnostic upper bound for camera error and never modifies the reconstructed trajectory.The same diagnostic translation is used for mask and box metrics.Source: E. Evaluation and Validity Masks · p. 17

F. Current Limitations

  • Generated assets may miss fine geometry, transparency, materials, or exact dimensions; monocular reconstruction remains uncertain for thin or reflective tools, especially under severe hand occlusion.Contact and non-penetration constraints do not replace full system identification of mass, friction, compliance, and fluid dynamics, and transfer beyond tabletop laboratory procedures remains open.Source: F. Current Limitations · p. 17

G. Baseline Evaluation Details

Baseline comparisons use frozen reconstructions and matched source timestamps, while physical-constraint checks are recomputed from saved scenes. Those checks measure consistency with known properties of the demonstrations, not exact metric recovery.

G. Baseline Evaluation Details

  • Baselines produce one frozen reconstruction per episode, are compared at matched source timestamps, and are averaged equally across videos without silently dropping missing or incompatible outputs.Do as I Do receives the source video and midpoint reference frame for pipetting; FoundationPose uses independently predicted depth and camera inputs, an inferred-scale mesh, and an initial mask.Source: G. Baseline Evaluation Details · pp. 17–18

H. Physical Constraint-Satisfaction Definitions

  • Seven pipetting requirements are recomputed from the final saved 3D scene at 24 Hz, with fixed event windows, geometry, and tolerances across ablation conditions.The checks cover approach, entry, dwell, withdrawal, collision avoidance, final table support, and floor clearance; five videos yield 35 checks per condition.Source: H. Physical Constraint-Satisfaction… · p. 18
  • Because the laboratory demonstrations record correctly executed procedures, satisfying the checks indicates consistency with known source-motion properties, not exact metric recovery or independently verified event correctness.Collision and support diagnostics overlap with requirements enforced during event-conditioned reconstruction.Source: H. Physical Constraint-Satisfaction… · p. 18

I. Ablation Implementation Details

The ablations isolate event conditioning, temporal regularization, depth initialization, and iterative visual refinement while holding other evaluation conditions fixed. A motion-roughness diagnostic measures saved-scene pipette trajectories, not source-motion ground truth.

I. Ablation Implementation Details

  • The five-video subset was prespecified, and ablations retained the same observations, masks, calibration, asset geometry, alignment, and evaluation procedure unless a component was removed.Optimization weights, constraint tolerances, and event windows were also shared across videos, with no per-video tuning.Source: §I · pp. 18–19
  • Event removal bypassed event-conditioned contact and orientation projections, prescribed terminal rest, and dense cup-interaction planning, while generic floor nonpenetration and reviewed lid animation remained active.This functionally removed event conditioning rather than merely setting its loss weight to zero.Source: §I · p. 19
  • The temporal ablation removed trajectory smoothing, temporal penalties, neighboring-pose repairs, and continuity restrictions, while retaining prepared observations, per-frame priors, interpolation, and event planning.The depth ablation replaced depth-derived heights with a fixed 0.20 m above-table prior for every observation and video.Source: §I · p. 19
  • The visual-refinement ablation retained masks, mask-derived initialization, depth-derived heights, smoothing, and event constraints, but removed iterative mask and anchor optimization, endpoint refinement, and mask-guided candidate selection.Trajectory roughness was measured as mean second finite differences of saved-scene pipette-center positions, scaled by the squared 24 Hz sampling rate; it is not source-motion ground truth.Source: §I · p. 19

J. Extended Ablation Study

The extended ablation separates physical interaction validity, image-space alignment, and trajectory coherence across five pipetting episodes. Event conditioning preserves modeled interaction constraints even when its removal improves Mask IoU.

J. Extended Ablation Study

  • 35/35 physical constraints were satisfied with event conditioning versus 5/35 without it, although removing it raised Mask IoU from 0.508 to 0.572.The full pipeline retained the approach–entry–dwell–withdrawal structure, showing that higher image-space agreement alone did not ensure physically meaningful trajectories.Source: Table 5 · p. 19, J. Extended Ablation Study · pp. 19–20
  • Removing temporal regularization raised Mask IoU from 0.508 to 0.514 and preserved 35/35 physical constraints, but increased mean trajectory acceleration from 5.410 to 6.070 m/s2 (+12.2%).The result contrasts slightly better frame-wise alignment with rougher motion.Source: J. Extended Ablation Study · p. 20
  • Without depth-derived initialization, Mask IoU fell from 0.508 to 0.487 while physical constraint satisfaction remained 35/35.Depth provided spatial initialization for image-space alignment, while event constraints preserved interaction geometry.Source: J. Extended Ablation Study · p. 20
  • Removing iterative visual refinement reduced Mask IoU from 0.508 to 0.223 while preserving all 35/35 physical constraints.Geometric and event constraints maintained interaction structure, but visual refinement was needed for precise image-space localization.Source: J. Extended Ablation Study · p. 20
Loading 2610.36592v1…