PaperlayerView exact revision on arXiv

Source-linked AI summary

Underwater C3-JEPA: An Object-Centric Cross-View World Model for ROV Salvage

Yuncong Yang, Jinlong Li, Yulong Xue, Feng Wu, Chunwen Zhang, Lei Qiao, Xuyang Wang

arXiv:2609.30214v1cs.ROcs.AI

TL;DR

Near-field underwater salvage must track partially visible objects across cameras and account for delayed vehicle response without contact sensing. C3-JEPA predicts object-centric latent states from multi-view video and control, combining weak binding, cross-view fusion, and geometric regularization. It reports stronger task-object representation and predictive results, including on real underwater video.

  • Problem

    Near-field salvage leaves targets partially visible across cameras, while few real-system trials, unknown actuation delays, and limited contact sensing constrain learning and perception.

    Source: §I · p. 1

  • Method

    C3-JEPA binds target and gripper tokens across views, retains free context slots, and predicts future latent states conditioned on vehicle control.

    Source: §I · p. 1, A. Architecture overview · p. 3

  • Results

    Group Dice is approximately 0.9 versus approximately 0.1 for unguided slots; persistence terminal motion error is cut by a third on the interaction diagnostic and by half at 5 s on AutoGrasp recordings.

    Source: §VII · p. 11

  • Takeaways & Limitations

    The predictive interface supports candidate-command evaluation and imagined-rollout behavior-agent training, and real underwater video shows withheld-view recovery and performance ahead of persistence.

    Source: §I · p. 2, §VI · p. 10

    The reported controller coupling is an interface demonstration, not a powered success-rate study; a closed-loop field trial remains future work.

    Source: §VI · p. 10

Abstract

from arXiv · show

We present Underwater C$^{3}$-JEPA (cross-view, control-conditioned, context-extended), an object-centric multi-view predictive world model for near-field heavy-load underwater ROV salvage. Without contact sensors, it predicts in latent space how the task-object state evolves through contact interaction and under the hydrodynamic lag of the vehicle, from synchronized multi-view RGB observations and vehicle control signals. C$^{3}$-JEPA encodes multi-camera observations into task-object and context tokens, fuses cross-camera evidence through held-out-view attention, and directly predicts future states conditioned on control. Weak binding anchors the target and gripper at low annotation cost, while SIGReg sharpens the geometric representation. Experiments show that the learned representation transfers substantially more task-relevant information to downstream probes than a reconstruction-free latent baseline, while keeping the predictor lightweight. The resulting predictive interface supports model-predictive-control (MPC) candidate evaluation and imagined-rollout behavior-agent training. Validation on real underwater video shows the same architecture recovering a withheld camera's object state and staying ahead of persistence, so the recipe transfers beyond simulation.

I. INTRODUCTION

Near-field salvage requires a work-class ROV to envelop a large, shape-uncertain object despite occlusion, contact-dependent gripper motion, and delayed vehicle response. C3-JEPA focuses on target and gripper state across camera views and predicts its evolution under control for imagined training and command ranking.

I. INTRODUCTION

  • The platform's underactuated, coupled gripper and hydrodynamically lagged base must envelop a large, shape-uncertain object without contact sensing.The mission proceeds through acquisition, approach, alignment, enveloping contact, lift/hold, and verification.Source: §I · p. 1
  • Near-field targets may be occluded and partially visible across cameras, while mapping methods reconstruct more scene detail than the decision requires.Visual SLAM can drift or lose tracking under backscatter; structured-light scanning requires short standoff and dwell-and-scan motion.Source: §I · p. 1
  • Few real-system trials, unknown actuation delays, difficult gripper simulation, and fragile tactile instrumentation constrain alternative routes to learning this behavior.The gripper's closed kinematic loops can force tiny simulation steps or solver failure, while tactile sensing requires pressure housing and cabling.Source: §I · p. 1
  • OCPD holds target and gripper slots at full fidelity and aligns them across cameras, while free context slots retain unaligned evidence.Guided object-centric cross-view fusion uses weak anchors, with human review reserved for ambiguous frames.Source: §I · p. 1
  • One compact control-conditioned model supports imagined-rollout behavior training offline and candidate-command ranking online.At 1.33 M parameters, predicting fused tokens rather than pixels keeps the predictor lightweight while learning command-to-motion lag.Source: §I · p. 2
  • The paper asks whether onboard visual channels preserve identifiable target and gripper state across views and contact while supporting rollouts.Its stated scope is narrower than the full sensing needs of salvage, which also include sonar, navigation, and force sensing.Source: §I · p. 2

II. RELATED WORK

Prior underwater perception often reconstructs scene structure, while object-centric learning and multi-view methods offer alternatives for retaining task-relevant entities across views. C3-JEPA adapts those ideas to a moving underwater camera rig and near-field task-object state.

A. Underwater perception: scene mapping versus task-object state

  • Mapping and structured-light approaches reconstruct scene geometry first, whereas OCPD resolves and aligns task objects without committing to a metric map.Mapping remains suited to search and survey; the approaches differ in cost and what survives occlusion and contact.Source: A. Underwater perception: scene… · p. 2

B. Object-centric visual representation

  • Weak binding anchors only the target and gripper, addressing slot permutation or absorption of small task objects while leaving remaining slots for context.DINOv3-feature clustering proposes masks from a few region prompts; the design retains instance-level target and gripper roles.Source: B. Object-centric visual representation · p. 2

C. Multi-view fusion and underwater predictive models

  • C3-JEPA withholds the target camera from cross-view keys and values, then uses a view-conditioned decoder to reconstruct its features.This adapts held-out-view fusion to cameras rigidly mounted on one moving ROV while preserving view-dependent appearance.Source: C. Multi-view fusion and underwater… · pp. 2–3

D. Latent prediction and planning

World models provide states whose evolution can be queried under candidate actions, supporting planning and imagined behavior learning. This work combines object-centric state with latent prediction and a multi-view architecture that aligns task objects while retaining context.

D. Latent prediction and planning

  • World models query state evolution under candidate actions for short-horizon MPC, counterfactual evaluation, imagined policy training, or downstream estimation.Architectures include pixel-generative, latent-dynamics, and object-centric models.Source: D. Latent prediction and planning · p. 3
  • Latent planning methods predict candidate consequences, but do not by themselves guarantee preservation of target and gripper identity through occlusion and contact.Low latent prediction error also does not guarantee object semantics, observable calibration, or task-relevant information transfer.Source: D. Latent prediction and planning · p. 3
  • C3-JEPA combines weak binding, held-out-view fusion, and latent prediction so MPC and imagined-rollout evaluation can query candidate consequences.Its comparison with a LeWM-style reconstruction-free latent model uses the same multi-view recordings and control signals and identical downstream probes.Source: D. Latent prediction and planning · p. 3

A. Architecture overview

  • The predictive state combines cross-view target and gripper tokens with three unaligned context slots from a reference view.The fixed ROV frame has a binding-only slot and is excluded from fusion and the predictive state.Source: A. Architecture overview · p. 3

B. Object-centric slots and geometric regularization

The representation uses weak binding to identify task objects and regularizes latent geometry separately from assigning object semantics. Its architecture retains free context slots alongside bound task-object slots.

B. Object-centric slots and geometric regularization

  • Stage I maps synchronized multi-view RGB to bound task-object slots and free context slots; Stage II injects control and predicts state autoregressively.The architecture also presents a predictive interface.Source: Figure 2 · p. 4
  • Weak binding uses frozen DINOv3 patch evidence and target/gripper anchors to guide corresponding VideoSAUR slot masks.The supervision is auditable and inexpensive, but is not fully unsupervised segmentation.Source: B. Object-centric slots and geometric… · pp. 3–4
  • SIGReg projects token embeddings onto random directions, scores projected normality, and averages the scores to match the latent marginal to an isotropic Gaussian.It regularizes token geometry; weak binding and object-feature reconstruction supply target and gripper semantics.Source: B. Object-centric slots and geometric… · p. 4

C. Held-out-view fusion

Held-out-view cross-attention fuses visible-view tokens into an object token while withholding the target view from the attention keys and values. A view-conditioned decoder predicts the withheld view’s feature evidence, testing whether the fused token retains cross-view object information.

C. Held-out-view fusion

  • Cross-attention fuses visible-view tokens into a shared token for each bound object.The target view is excluded from the keys and values during training.Source: C. Held-out-view fusion · p. 4
  • A view-conditioned decoder predicts feature evidence for the withheld target view.This tests cross-view object evidence without requiring all camera appearances to collapse.Source: C. Held-out-view fusion · p. 4
  • Masks the model identifies as genuinely invisible are excluded from the held-out-view test.Source: C. Held-out-view fusion · p. 4

D. Control-conditioned JEPA prediction

The predictor conditions on normalized vehicle commands to forecast future object tokens from a short state history. It predicts twelve future states with a lightweight model, and the shared table configuration applies across experiments.

D. Control-conditioned JEPA prediction

  • The control vector combines normalized thruster and gripper command channels and produces AdaLN-Zero parameters for the predictor.Source: D. Control-conditioned JEPA prediction · p. 4
  • From four history states, the control-conditioned predictor forecasts twelve future states using token MSE plus cosine distance.Its 1.33 M parameters keep the predictive interface lightweight; command inputs let it learn thrust buildup, added-mass inertia, and cross-couplings within the predictor.Source: D. Control-conditioned JEPA prediction · p. 4

IV. EXPERIMENTAL SETUP AND EVALUATION

  • The visual encoder and control-conditioned predictor share the training configuration listed for all experiments.Source: §IV · p. 4

A. Dataset taxonomy and split policy

The study uses synchronized simulator recordings with disjoint train/validation partitions and reports a separate real-field deployment. Evaluation proceeds through representation, prediction, rollout, calibration, and decision/transfer evidence, with each level interpreted on its own.

A. Dataset taxonomy and split policy

  • ROS–Gazebo recordings provide synchronized cameras, vehicle state, gripper signals, poses, and event annotations; privileged state supplies training targets and evaluation references, not model inputs.The camera model uses channel attenuation values (0.3, 0.3, 0.09) and RGB noise with standard deviation 0.02.Source: A. Dataset taxonomy and split policy · pp. 4–5
  • The release includes 705 recordings across two privileged auto-grasp batches, plus 41 grasping, 30 interaction, and 14 gripper-motion recordings; partitions are recording-disjoint.Two field deployments on a submerged real target provide synchronized camera streams and navigation telemetry.Source: A. Dataset taxonomy and split policy · p. 5

B. Metrics and evidence ladder

  • Evaluation spans representation, prediction, recursive rollout, observable calibration, and decision/transfer; improvement at one level is not evidence for a later one.Source: B. Metrics and evidence ladder · pp. 5–6

V. RESULTS

Binding-guided training gives target and gripper slots clear task semantics, while held-out-view fusion improves representation measures and recovers withheld-view evidence. The control-conditioned predictor beats persistence on terminal relative-motion error in an interaction-focused diagnostic, and imagined rollouts track recorded motion across near and far distances.

A. Representation evidence

  • Group Dice rises from 0.09/0.12 to 0.90/0.88 for target/gripper slots with binding guidance, while feature reconstruction error increases from 0.0157 to 0.0186.In the equal-budget fusion ladder, held-out-view cross-attention reduces feature MSE from 0.0231 to 0.0200 (−13.4%) and raises target decoder Dice from 0.713 to 0.820 (+10.7 pp).Source: A. Representation evidence · p. 6, Table II · p. 7
  • With a withheld camera, target Dice improves from 0.2206 to 0.7604 and gripper Dice from 0.2034 to 0.8695; held-out feature MSE also falls for both objects.The evaluation uses 512 fixed random windows, not a recording-disjoint split; front-camera UUV Dice is 0.456 versus 0.755–0.800 for the other views.Source: A. Representation evidence · p. 6
  • Adding SIGReg at weight 0.03 reduces estimated 3D target-center error from 0.625 to 0.496 and increases Dice from 0.887 to 0.895.This representation sweep isolates SIGReg in the geometry renderer while holding the other factors fixed.Source: A. Representation evidence · p. 6

B. Conditional rollout evidence

  • 25.7 cm terminal relative-motion error versus 39.1 cm for persistence on a 229-recording interaction-focused diagnostic set.The reported recording-level bootstrap 95% confidence intervals are 23.6–27.9 cm for the predictor and 36.7–41.5 cm for persistence.Source: B. Conditional rollout evidence · p. 6
  • Longer-context and unroll-regularized variants do not improve overall on the deployed 1 s-context model: their terminal errors are 32.2 cm, 32.2 cm, and 28.6 cm versus 25.7 cm.The 8 s variant is the exception at the 10 s horizon on Interacting recordings.Source: B. Conditional rollout evidence · pp. 6–7
  • Replacing history with mean tokens raises future error by 536%, and using a distant window from the same recording raises it by 260%; reversing or permuting the pre-anchor frames shifts predictions by 1.6% or 0.6%.These perturbations test sensitivity to observation continuity and frame ordering.Source: B. Conditional rollout evidence · p. 7
  • Predicted UUV mask centers track recorded motion with about 0.09 mean error over 3 s at ≈3.4 m and under 0.05 normalized units over a recursive 5 s rollout at ≈0.8 m.The near-field rollout shows no endpoint divergence; cross-view fusion aggregates evidence when an object moves between cameras.Source: B. Conditional rollout evidence · p. 7

C. Cross-view consistency

Predicted tokens preserve the observed cross-view ray consistency, while control-conditioned predictions show gains over unconditioned motion and support imagined-rollout agent training. The agent result reflects critic-predicted progress, not independent task success.

C. Cross-view consistency

  • Predicted and observed tokens have closely matching ray spreads: 16–20° versus 17–22°, with predicted spread no larger in any horizon bin.Source: C. Cross-view consistency · p. 7

D. Causal consistency and behavior-agent training

  • At 15 s, direction cosine reaches about 0.42 with control versus 0.23 without, with positive control gain throughout grasp/contact.The gain grows from +0.07 at 1 s to about +0.24 at 15 s.Source: D. Causal consistency and… · pp. 7–8
  • Approach-phase control gain is slightly negative at 1 s, then +0.06 at 5 s and up to about +0.16 at 10 s.Source: D. Causal consistency and… · p. 8
  • On held-out recordings, the agent exceeds behavior cloning in critic-predicted progress by +0.383 in search and +0.209 in grasp/contact.Progress uses the same frozen critic that shapes training, so this reflects imagined-rollout-loop consistency rather than independent task success.Source: D. Causal consistency and… · p. 8

E. Interface validation for model-predictive control

The world model’s predicted endpoints rank matched-endpoint candidate controls above shuffled-control rollouts, and C3-JEPA transfers more task information than the reconstruction-free baseline. The transfer findings distinguish SIGReg alone from its use with reconstruction and binding.

E. Interface validation for model-predictive control

  • On 90 matched-endpoint groups, predicted endpoints select the true best candidate first in 53.3% versus 23.3% for shuffled controls and 25% for random choice.Mean regret falls from 0.0091 to 0.0058.Source: E. Interface validation for… · p. 8

F. Downstream transfer against a reconstruction-free world model

VI. FIELD VALIDATION ON REAL UNDERWATER VIDEO

Field recordings test whether C3-JEPA transfers to real underwater video without simulator training. The model recovers withheld-camera object state and predicts ahead of persistence, though field evidence is open-loop and reported in relative units.

VI. FIELD VALIDATION ON REAL UNDERWATER VIDEO

  • Both deployments use raw field recordings alone, with no simulator participation, in a 60 m by 20 m test pool 10 m deep.Deployment A records navigation above a steel frame; Deployment B records contact with an aluminum pipe, using a synchronized camera pair without measured intrinsics or extrinsics or metric target ground truth.Source: §VI · p. 9, Table IV · p. 9, Figure 9 · p. 10
  • Token error is per-dimension mean squared error against reference predictions, while mask quality is a floor/model/ceiling intersection-over-union triple.References include training-session mean tokens, persistence, and decoded true tokens of the predicted view.Source: §VI · p. 9
  • 76% and 65% lower token error on the two deployments, cross-camera prediction recovers withheld-camera masks at 88% and 81% of the achievable range.The study notes that the rigidly co-mounted, largely co-visible cameras test hidden-view recovery rather than a dedicated fusion benchmark.Source: §VI · p. 9, Figure 6 · p. 10, Figure 7 · p. 10
  • 30% below persistence and 89% below the mean-token floor, the field-trained predictor’s 36-step self-fed rollout reduces token error.Mask quality does not improve: 0.159 against 0.184 for persistence (CI −0.053 to +0.002).Source: §VI · p. 9, Figure 6 · p. 10
  • Field evidence is open-loop, uses relative units, and does not evaluate actuated behavior outside simulation.The reported 36-step result is a self-fed prediction rollout against recorded motion, not a closed-loop field trial.Source: §VI · pp. 9–10

VII. CONCLUSION

C3-JEPA combines object-centric cross-view fusion with a control-conditioned latent world model for underwater ROV salvage. The conclusion reports improved task-relevant representation and real-video transfer, while keeping the predictive interface compact.

VII. CONCLUSION

  • C3-JEPA combines weak binding, held-out-view fusion, free context slots, and control-conditioned latent prediction for near-field ROV operation without contact sensing.The conclusion reports withheld-camera recovery to 81–88% of achievable range and persistence gains of 23–31% at a five-second horizon.Source: §VII · pp. 10–11
  • Against a reconstruction-free latent world model trained on identical data, C3-JEPA transfers substantially more task-relevant information to identical probes.The conclusion also reports group Dice of approximately 0.9 versus approximately 0.1 for unguided slots.Source: §VII · p. 11
  • The predictive interface supports MPC candidate rollouts, candidate ranking, and imagined-rollout behavior-agent training; simulation couples it to a conventional automatic controller.Cross-view decoding compares each camera’s own-token mask with the mask from merged cross-view tokens.Source: §VII · p. 11, Figure 8 · p. 11
Loading 2609.30214v1…