PaperlayerView exact revision on arXiv

Source-linked AI summary

Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos

Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan, Ian Fuller, Lee Caesar, Jiaye Wu, Yantian Zha

arXiv:2609.38966v1cs.RO

TL;DR

Real-fish video offers a scalable way to capture morphology and swimming, but realistic, controllable fish grounded in those observations and standardized biological-fidelity evaluation remain missing. Video2SwimFish builds individual fish assets and swimming policies from video, and its benchmark finds that task success and fidelity to real-fish locomotion do not necessarily align.

  • Problem

    Simulation lacks realistic, deformable, controllable fish grounded in real-world observations and standardized tools to evaluate biological fidelity, while video-based policy learning lacks joint-level demonstrations and task rewards alone do not specify how fish should swim.

    Source: §1 · p. 2

  • Method

    Video2SwimFish reconstructs controllable, deformable fish assets from real-fish videos and uses each fish’s observed midline curvature to define an individual low-dimensional action space for learning swimming policies.

    Source: §1 · p. 2

  • Results

    Task success and biological fidelity do not necessarily align: policies that succeed at tasks can still deviate substantially from their corresponding real fish’s locomotion.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Faithfully reproducing an individual animal’s locomotion while learning a task-capable policy remains an open challenge.

    Source: §1 · p. 3, §5 · p. 9

    The simulations use approximate quasi-steady hydrodynamics rather than full fluid–structure interaction, sim-to-real transfer was not tested, and experiments covered only subsets of the 120 fish.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/.

1 INTRODUCTION

Video2SwimFish addresses the lack of realistic, controllable fish grounded in real observations and tools for evaluating biological fidelity. It reconstructs fish assets and individual swimming policies from video, then benchmarks task success alongside resemblance to each fish’s locomotion.

1 INTRODUCTION

  • Real-fish video offers a low-cost, non-invasive source of morphology and swimming observations, while real-world aquaculture training is costly, hard to scale, and can pose animal-welfare risks.The identified gap is realistic, deformable, controllable fish grounded in observations, paired with standardized simulation tools for evaluating locomotion and biological fidelity.Source: §1 · p. 2
  • Reconstructing an articulated 3D fish is difficult because existing reconstruction methods do not directly provide controllable internal articulation and perform poorly on fast, highly deformable fish.Methods that generate articulated assets from visual input mainly target rigid objects, whereas fish articulation must be inferred and coupled with a deformable mesh.Source: §1 · p. 2
  • Learning biological swimming is difficult because task rewards do not specify how to swim, video lacks joint-level states, and predefined locomotion patterns are not individualized.The paper therefore trains policies to follow swimming motion observed in the real-fish video.Source: §1 · p. 2
  • Video2SwimFish automatically converts real-fish videos into realistic, controllable, deformable simulated fish assets, each equipped with an individual swimming policy.The pipeline is presented as a way to address the reconstruction and locomotion-learning challenges and enable systematic evaluation.Source: §1 · p. 2
  • The Biological Locomotion Manifold (BLM) encodes an individual fish’s observed midline curvature as a low-dimensional action space bounded by its observed deformation range and rate.This gives reinforcement learning a control space grounded in that individual’s motion rather than a hand-designed pattern.Source: §1 · p. 2
  • The paired datasets comprise synchronized multi-view videos of 120 identified fish across six species and 120 controllable fish assets with learned policies.The video dataset has 20 individuals per species; each sequence contains exactly one identified fish, unlike the cited closest prior work, which records several fish of one species per sequence and outputs trajectories rather than individual morphology.Source: §1 · pp. 2–3
  • Task success and biological fidelity do not necessarily align: methods can succeed while deviating substantially from their corresponding real fish’s locomotion.The paper identifies faithful reproduction of individual animal locomotion as an open challenge.Source: §1 · p. 3

2 RELATED WORK

Prior work provides tools for reconstruction, locomotion learning, simulation, and embodied-agent benchmarks, but leaves a gap in deformable, controllable fish grounded in biological observations and standardized biological-fidelity evaluation.

From Video to Controllable 3D Assets

  • Video2SwimFish couples real-fish video reconstruction with articulation generation to produce deformable, controllable fish, unlike prior reconstruction and visual asset methods that lack control-ready articulation or target rigid objects.The cited methods also struggle with fast, highly deformable motion.Source: §2 · p. 3

Learning Locomotion from Videos

  • Video2SwimFish evaluates task performance against biological fidelity because prior video-imitation, reference-motion, and fish-like-robot approaches have limitations for individual real-fish motion.Video imitation can depend on aligned observation spaces and an inverse dynamics model; reference-motion methods need joint-level references, while CPG and RL parameters are shared across individuals.Source: §2 · p. 3

Simulation Assets and Embodied-Learning Benchmarks

  • Existing simulation platforms and embodied-learning benchmarks provide assets and scalable training, but FishGym’s fish are hand-designed and standardized biological-fidelity evaluation is absent.Video2SwimFish links simulated fish directly to their real-world counterparts.Source: §2 · p. 3

3 VIDEO2SWIMFISH

Video2SwimFish turns synchronized views of an individual fish into a deformable mesh, internal articulation, and a swimming policy. Its stages select a canonical frame, construct articulation with VLM feedback, and derive an action space from observed curvature.

3.1 MESH RECONSTRUCTION

  • 0.88 mean affine-aligned IoU measures each reconstructed mesh’s lateral silhouette against the segmented canonical frame.A VLM selects the canonical frame, and Meshy reconstructs a mesh scaled to the fish’s estimated body length using known tank dimensions for calibration.Source: §3.1 · pp. 4–5

3.2 ACTOR–CRITIC AUTO-CONSTRUCTION OF FISH ARTICULATION

  • A fine-tuned actor VLM proposes bone-template edits and a critic scores physical plausibility and morphology compatibility, iterating until acceptance.The actor is fine-tuned on 74 manually constructed examples; joint degrees of freedom, limits, and drive parameters use a shared physically validated preset.Source: §3.2 · p. 5

3.3 BIOLOGICAL LOCOMOTION MANIFOLD (BLM)

  • The BLM applies PCA to observed body curvature and uses four coefficients, capturing 99.92% of recorded curvature variance, as actions bounded by observed ranges and rates of change.Coefficient limits and maximum per-step changes are derived from real-fish trajectories.Source: §3.3 · p. 6

3.4 DATA COLLECTION

  • The synchronized top- and front-view corpus contains 120 individual fish across six species, while 74 manually constructed examples train the actor VLM.The examples come from twelve of fourteen controllable fish models; two models are held out for validation.Source: §3.4 · pp. 6–7

4 BENCHMARK

The benchmark evaluates trajectory following, reward-free transfer of swimming behavior from video, and fish capture by a BlueROV, measuring biological fidelity alongside task performance where applicable.

4 BENCHMARK

  • The benchmark covers trajectory reproduction, reward-free behavior transfer, and downstream robot interaction, with biological fidelity measured alongside task performance in the first two tasks.Task 1 measures completion, path distance, phase monotonicity, curvature distance, body-wave frequency, and swimming speed.Source: §4 · p. 7, §4.2 · pp. 7–8

4.1 EXPERIMENTAL SETUP

  • All methods and fish use the same actuator preset and physics step, with fluid forces from a shared quasi-steady resistive panel model.The BLM policy acts in a manifold rather than joint space; coefficient changes stay within rates observed in that fish’s video and are mapped to joint targets by a fish-specific controller.Source: §4.1 · p. 7

4.2 TASK 1: TRAJECTORY FOLLOWING

  • Task 1 follows a re-rooted, reoriented 5 s window of the fish’s own trajectory and scores progress along its path, so swimming the right shape slowly is not penalized.Episodes complete when the phase variable reaches the path end; the reward uses a look-ahead point 0.3 BL ahead, heading alignment, and phase increase.Source: §4.2 · p. 7
  • BLM+RL completes 0.89 of episodes versus 0.70 for the next-best method, with the lowest frequency and speed errors but the largest Fréchet distance.Joint RL+AMP has the most accurate path at 0.34 BL and closest curvature distribution at W1(κ) 0.78, but completes only half its episodes and has the worst tail-beat frequency.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 TASK 2: FREE SWIMMING FROM VIDEO

  • BLM+IL halves curvature distance (0.74 vs. 1.63) and cuts speed error by more than threefold (0.11 vs. 0.39), while BCO better matches frequency (0.012 vs. 0.038).The task evaluates fidelity rather than retracing a particular recording, using 20 real initial states and 5 s roll-outs without reward.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 CASE STUDY: BLUEROV CAPTURE

  • The BlueROV2 capture case trains a PPO-controlled robot with a 6-DoF manipulator and rigid basket to capture a fish swimming under its own Task-2 BLM+IL policy.Capture requires the fish to be inside the basket with its opening facing up; the case uses a channel catfish and a lake sturgeon.Source: §4.4 · p. 9, Table 3 · p. 9

5 LIMITATIONS AND DISCUSSIONS

Video2SwimFish produces controllable simulated fish from individual-fish videos, and its benchmark highlights the unresolved challenge of combining task capability with fidelity to the individual animal.

5 LIMITATIONS AND DISCUSSIONS

  • Learning a policy that is both task-capable and faithful to the individual animal remains open, and the datasets and metrics are offered as a basis for community progress.The benchmark finds that task success and locomotion fidelity come apart, and BLM does not dominate the baselines.Source: §5 · p. 9
  • The simulations use approximate quasi-steady hydrodynamics rather than full fluid–structure interaction, and the work does not test sim-to-real transfer.The stated reasons for not testing transfer include limited test-site access, robot waterproofing and heat-dissipation issues, and animal-welfare concerns.Source: §5 · p. 9
  • Experiments cover only subsets of the 120 fish: twelve in Task 1, six in Task 2, and two in the Case Study, due to limited computation.Source: §5 · p. 9

A DATASET DETAILS

The dataset pairs synchronized views of 120 fish across six species with a reproducible path from video to canonicalized 3D meshes.

A DATASET DETAILS

  • The source dataset records one fish per video in synchronized top and front views at 32 fps, covering six species and 20 individuals per species.Behavior signals are resampled to 30 Hz; body length is estimated from valid silhouettes, with species-median substitution flagged when individual data are insufficient.Source: A DATASET DETAILS · p. 16
  • Table 4 summarizes asset properties including bone-chain coverage and alignment, defined as the fraction of body length spanned and mean bone-axis angle to the local centerline.Source: Table 4 · p. 16

B RECONSTRUCTION PIPELINE

B.1 CANONICAL FRAME AND CROP

  • A VLM selects a fully visible, lateral front-view frame, then background subtraction isolates the fish in a crop.Source: §B.1 · p. 16

B.2 IMAGE-TO-3D AND MESH CRITIC

  • The cropped image is sent to an image-to-3D service; a VLM critic checks four rendered views, followed by deterministic component cleanup and one possible resubmission.The critic rejects meshes that are not a fish, are truncated, contain multiple fish, or show the tank rather than the animal.Source: §B.2 · p. 16

B.3 CANONICALIZATION

  • The mesh is oriented with its first principal axis along +x, scaled to measured body length, and recentered at the bounding-box origin.Source: §B.3 · p. 16

B.4 SKELETON CONSTRUCTION: VLM ACTOR–CRITIC WITH GEOMETRIC REGULARIZATION

The skeleton pipeline combines VLM-proposed articulation with geometric regularization, then exports and checks deformable fish assets and extracts observed swimming behavior.

B.4 SKELETON CONSTRUCTION: VLM ACTOR–CRITIC WITH GEOMETRIC REGULARIZATION

  • A VLM actor proposes bone edits and a critic scores the skeleton, while geometric verification and structural completion constrain bones to the body and fill or extend the chain.The loop runs for at most six iterations; accepted skeletons receive deterministic completion, including gap filling and extension toward the skull and caudal-fin base, capped at 22 bones.Source: §B.4 · pp. 16–17

B.5 EXPORT AND STAGE-1 VERIFICATION

  • Regularized exported chains contain 12–19 bones, with a median of 15, and 114 of 120 fish receive a final critic score of 5.87 of 120 assets pass the strict containment test; regularized chains span 64–83% of body length.Source: §B.4 · p. 17, §B.5 · p. 17

B.6 HYDRODYNAMICS

  • A panel model applies quadratic normal drag and skin friction to FEM skin slices, with impulse and speed caps to prevent explicit-integration runaway.The model uses normal-drag coefficient 1.0, skin-friction coefficient 0.01, and a body-speed cap of 1.5 BL/s.Source: §B.6 · p. 17

B.7 BEHAVIOR EXTRACTION

  • Top-view videos are segmented into 20-station midline curvature sequences; per-fish four-mode PCA bases are fitted, with species-pooled bases used for fish with fewer than 100 clean frames.The same frames provide re-rooted 5 s reference trajectories for trajectory following and biological-fidelity metrics.Source: §B.7 · p. 17

C MESH FIDELITY: SILHOUETTE IOU AGAINST THE CANONICAL FRAME

The mesh-fidelity evaluation compares generated fish silhouettes with their canonical input frames, using aligned IoU and examining segmentation-related disagreement. The section also specifies the simulation and control action spaces used for fish models.

C MESH FIDELITY: SILHOUETTE IOU AGAINST THE CANONICAL FRAME

  • 0.88 mean affine IoU on the clean subset, reaching 0.93 for white bass #008 and 0.91 for lake sturgeon #002.The subset comprises the two most cleanly segmented fish per species; removing fins changes mean IoU by less than 0.01.Source: Table 5 · p. 18, C MESH FIDELITY: SILHOUETTE IOU… · p. 18
  • Silhouettes are compared after head-matched alignment and a 6-DOF affine refinement, with no non-rigid warping.Photo masks use U2-Net segmentation and thresholding; alignments are rigid or affine in 2-D.Source: C MESH FIDELITY: SILHOUETTE IOU… · pp. 17–18
  • Photo segmentation can lower the measured IoU because it drops translucent fins or retains tank-floor pixels, making reported IoU a lower bound on reconstruction accuracy.The overlays also show residual differences from fin poses and slight body bends that a global 2-D transform cannot remove.Source: C MESH FIDELITY: SILHOUETTE IOU… · p. 18, Figure 4 · p. 19

D SIMULATION AND CONTROL

  • The control setup provides joint-target, BLM-coefficient, and CPG action spaces for fish simulated with FEM in Isaac Sim and Isaac Lab.BLM coefficients decode through a per-fish calibration matrix; the CPG policy outputs amplitude, frequency, and phase-offset changes.Source: D SIMULATION AND CONTROL · p. 20

E.1 TASK 1: TRAJECTORY FOLLOWING

The evaluation covers trajectory following, reward-free swimming from video, and a BlueROV2 capture case study. It specifies each task’s setup, metrics, or conditions for evaluation.

E.1 TASK 1: TRAJECTORY FOLLOWING

  • Trajectory-following episodes use each fish’s own 5 s position-and-heading path, re-rooted at reset, and track completion, path distance, curvature, frequency, speed, and phase progression.Baselines include joint RL, Joint RL+AMP, CPG+RL, BCO+RL, and BLM+RL.Source: §E.1 · p. 20

E.2 TASK 2: FREE SWIMMING FROM VIDEO

  • Free-swimming evaluation tests whether policies transfer video behavior without reward, using real initial states and 5 s roll-outs without a target.The policy observes top-view quantities and its BLM coefficients; evaluation includes path distance, speed, heading stability, curvature, and frequency metrics.Source: §E.2 · p. 20

E.3 CASE STUDY: ROV CAPTURE

  • The BlueROV2 capture study trains a robot with a 6-DoF manipulator and basket to capture fish driven by their Task-2 BLM+IL policies.Capture policies are trained for channel catfish 002 and lake sturgeon 016; success requires the fish inside the basket with its opening facing up.Source: §E.3 · p. 21

E.4 TRAINING PROTOCOL AND STOPPING RULES

Runs stop after sustained task success or a plateau, while runs with repeated high blow-up rates are retried. Fish are selected from assets passing Stage-1, with one numerically unstable sturgeon replaced by a larger conspecific.

E.4 TRAINING PROTOCOL AND STOPPING RULES

  • Runs stop when completion or capture success stays above its threshold for three epochs, or when performance plateaus after at least 40 epochs.The thresholds are 0.90 for completion and 0.85 for capture success; a plateau means no improvement above 0.02 over 20 epochs.Source: §E.4 · p. 21
  • A run exceeding a 0.30 blow-up rate for three epochs after warm-up is marked failed and retried once with a different seed.The protocol also restarts a process that produces no epoch for 40 minutes from its latest checkpoint.Source: §E.4 · p. 21
  • Task 1, Task 2, and case-study experiments use two, one, and two fish per species respectively, selected from assets with the most clean video frames.One sturgeon was replaced because its FEM body was numerically unstable at the shared physics step.Source: §E.4 · p. 21

F COMPUTE

The appendix reports compute costs, describes manually constructed skeleton templates and their training examples, and specifies the quasi-steady hydrodynamics approximation used in simulation.

F COMPUTE

  • A trajectory-following run takes about 10 minutes per PPO epoch when two runs share a GPU, while the workstation sustains about 0.4 epochs per minute.Mesh generation costs about 30 Meshy credits per fish, and the actor–critic loop takes 13–40 minutes per fish on one GPU.Source: F COMPUTE · p. 21

G MANUALLY CONSTRUCTED SKELETON TEMPLATES

  • Fourteen hand-skeletonized fish models provide templates and actor-training examples, with bones placed along the midline and no per-bone joint parameters.The library spans 14 body plans; the fine-tuning set has 74 examples from 12 models, with roach and trevally held out for validation.Source: G MANUALLY CONSTRUCTED SKELETON… · pp. 21–22

G.1 HYDRODYNAMICS

  • A resistive panel model replaces full fluid simulation with quadratic drag on deformable-body panels, summing forces and moments into a wrench per bone.The quasi-steady model neglects added mass, vortex shedding, and wake history, so absolute swimming speeds are specific to this simulator.Source: §G.1 · p. 22
Loading 2609.38966v1…