PaperlayerView exact revision on arXiv

Source-linked AI summary

All You Need Is Low Fidelity: Zero-Shot Sim-to-Real of Learned Robotic Fish Control

Liam Maloney, Simon Ramchandani, Mike Y. Michelis, Ronan Hinchet, Robert K. Katzschmann

arXiv:2609.36993v1cs.RO

TL;DR

Soft underactuated fish need controllers that adapt their gait, but high-fidelity fluid simulation is costly and low-cost models omit unsteady effects. The paper trains a controller in a stateless simulator using a band-limited trajectory generator, then deploys it unchanged on hardware for several closed-loop tasks. The results show transfer within the regime covered by the fluid identification, while the simulator remains unreliable outside that bounded regime.

  • Problem

    The paper asks whether a controller learned in a low-cost simulator with stateless, quasi-steady fluid can transfer to a physical fish and generalize beyond one gait.

    Source: §I · p. 1

  • Method

    The policy controls a band-limited trajectory generator in a stateless fluid simulator, keeping its motions within the regime covered by the fluid identification.

    Source: §I · p. 1, B. Trajectory Generator · p. 5, §VIII · p. 8

  • Results

    One policy deployed unchanged on a real soft fish reached static targets, rejected disturbances, generalized beyond its training distribution, and tracked a moving target.

    Source: §I · p. 1, §VIII · p. 8

  • Takeaways & Limitations

    A low-cost simulator can support transfer when it is accurate over the bounded motions the controller can produce.

    Source: §VII · p. 7, §VIII · p. 8

    The simulator is unreliable where omitted unsteady effects, including vortex shedding and added-mass transients, matter, and the experiments do not establish whether a tighter fluid model would transfer.

    Source: §VII · p. 7

Abstract

from arXiv · show

Complex tasks for underwater robots remain limited by the capabilities of their controllers. Learning a better one for a soft, underactuated robotic fish trades simulator cost against fidelity. We show that an intentionally low-fidelity simulator is enough: a stateless, quasi-steady fluid model with no wake and no added-mass history suffices to learn a \emph{general}, closed-loop controller that transfers to hardware without tuning. Our platform is a soft, single-motor, tendon-driven fish whose policy observes only what the hardware can measure. A staged pipeline grounds the simulator in two independent identifications, fixing the tail dynamics and a stateless fluid model; the policy then acts through a band-limited rhythmic trajectory generator rather than commanding the tail directly. Deployed unchanged in an outdoor pool, a single policy performs closed-loop target reaching, disturbance rejection, and out-of-distribution target acquisition and tracking. The transfer rests on the constraint rather than the fidelity: the generator cannot leave the band over which the fluid was identified. This raises the question of how much of the physics can reside in the controller rather than in the simulator.

I. INTRODUCTION

Soft underactuation makes whole-body control difficult, while learning requires simulation at a scale that high-fidelity fluid models make costly. The paper asks whether a low-fidelity model can support a general controller that transfers unchanged to real fish hardware.

I. INTRODUCTION

  • Soft robotic fish have largely been limited to straight-line cruising or hardcoded gaits, motivating learned whole-body control for underactuated swimmers.The introduction contrasts this gap with versatile whole-body controllers learned for rigid quadrupeds and humanoids.Source: §I · p. 1
  • Soft-fish control must adapt the swimming stroke itself because the underactuated tail shape emerges from coupled motor, body, tendon, and fluid dynamics.Feedback over hand-designed gaits can steer the stroke but cannot change it, while reinforcement learning needs millions of interactions and therefore requires simulation.Source: §I · p. 1
  • High-fidelity CFD resolves swimming physics but is costly per step and difficult to parallelize, whereas fast stateless models make learning feasible but coarsely approximate unsteady fluid.The unresolved question is whether policies learned in these low-cost simulators can transfer to physical fish and generalize beyond a single gait.Source: §I · p. 1
  • A low-fidelity simulator supports zero-shot transfer when the policy is constrained to motions within the regime identified for its fluid model.The policy controls a band-limited trajectory generator rather than commanding the tail directly, and one policy performs target reaching, disturbance rejection, and out-of-distribution target acquisition and tracking.Source: §I · p. 1, Figure 1 · p. 1, Figure 2 · p. 2

II. RELATED WORK

Prior work spans high-fidelity simulation, low-cost fluid models, and limited real-fish control; this paper brings whole-gait authority and hardware-measurable observations to zero-shot deployment. Its platform and simulation twin use a single tendon-driven tail and a stateless, quasi-steady fluid model.

II. RELATED WORK

  • Prior work leaves open a combination of general closed-loop deployment on a soft underactuated fish, whole-gait policy authority, hardware-measurable observations, and a stateless fluid simulator.CFD-trained policies transfer to agile maneuvers at substantial simulation cost; low-cost deployment on an underactuated eel is limited to straight-line swimming, while prior fish tracking uses a steering offset or privileged tail-state access in simulation [9].Source: §II · p. 2

A. Hardware Fish

  • The soft fish uses one motor and antagonistic tendons routed along a compliant tail, with policy observations from a motor encoder, IMU, and AprilTag.A sealed, oil-filled head houses onboard compute, while the motor in the flooded body rotates the tendon pulley.Source: A. Hardware Fish · p. 2

B. Simulation Fish

  • The MuJoCo twin uses a rigid head and five-hinge tail, with CAD-derived inertial properties and a stateless ellipsoid fluid model.Each ellipsoid has five fluid coefficients, with three independently identified sets extending the model to fifteen coefficients; forces depend only on instantaneous body velocity.Source: B. Simulation Fish · pp. 2–3

IV. SYSTEM IDENTIFICATION

The first identification isolates the tail’s passive stiffness and damping in air, then maps a motor-shaft measurement to per-hinge stiffness using tendon geometry. This fixes tail dynamics before the fluid fit.

A. Tail Dynamics

  • The tail is modeled as a damped oscillator, and an in-air sinusoidal frequency sweep separates its effective stiffness, inertia, and damping.The real component of the torque-to-angle response is affine in frequency squared; the imaginary component yields damping after separating amplitude-dependent Coulomb friction from viscous damping.Source: A. Tail Dynamics · p. 3
  • The identified effective stiffness is approximately 0.98 N · m/rad, compared with 0.26 N · m/rad for the idealized model at the same shaft.The effective damping is approximately 0.011 N · m · s/rad.Source: A. Tail Dynamics · p. 3
  • The measured motor-shaft stiffness is converted to per-hinge stiffness through tendon moment arms and payout geometry.The five hinges share modulus and thickness but taper in height, so geometry sets their relative stiffnesses and the measured effective stiffness sets the scale; the resulting values run from 2.99 to 1.32 N · m/rad from body to fin.Source: A. Tail Dynamics · p. 3

B. Fluid Model

The fluid fit assigns separate coefficients to the head and body, tail, and caudal fin, with the stateless model compensating for unsteady fin thrust through inflated coefficients. Acceleration weighting and held-out runs assess the fitted twin.

B. Fluid Model

  • The model uses 15 coefficients across head/body, tail, and caudal-fin zones, allowing high body angular drag without locking the oscillating fin.With one shared set, body-stabilizing angular drag would also inhibit fin motion needed for thrust.Source: B. Fluid Model · p. 3
  • The fin’s angular-drag and Kutta-lift coefficients are an order of magnitude larger than the head’s, compensating for unsteady thrust absent from the stateless model.A separate identification on different data produces similar inflation, which the authors attribute to the model rather than fitting error.Source: B. Fluid Model · p. 3, Table I · p. 4
  • Start-of-run position errors receive four times the weight of end-of-run errors to avoid a fitted twin that initially swims backward.The model fixes added mass from ellipsoid geometry, so remaining transient effects are absorbed into fitted coefficients.Source: B. Fluid Model · pp. 3–4
  • Held-out runs test whether the weighting identified the fluid or merely the transient; Figure 5 compares measured and identified-twin paths for one fitted and one validation run at the same condition and on the same axes.Circles mark each path’s position at t = 8 s.Source: B. Fluid Model · p. 4, Figure 5 · p. 4

C. Motor

The post-deployment motor identification tests how the real position-controlled actuator differs from the idealized servo used for policy training. The identified model matches measured shaft motion more closely on held-out commands.

C. Motor

  • The deployed policy was trained against an effectively ideal servo, unlike the physical motor, which is weak, heavy, speed-limited, and delayed.Table II contrasts the idealized and identified actuator parameters; joint damping caps peak shaft speed near the measured 5 rad/s.Source: C. Motor · p. 4, Table II · p. 4
  • The actuator model was fitted in the time domain by replaying recorded commands and fitting measured encoder angles, then validated on held-out commands.The authors use this approach because saturation and a back-EMF speed cap make the motor’s frequency response amplitude-dependent.Source: C. Motor · p. 4
  • 1.0° RMSE for the identified servo versus 3.6° for the idealized model on measured encoder motion.Figure 6 shows commanded setpoint, hardware, the training model, and the post-deployment identified model over a representative stroke.Source: Figure 6 · p. 4, C. Motor · p. 4

V. CONTROL

The stateless fluid model cannot distinguish large, slow strokes from small, fast flutter when they produce the same thrust. The control problem is to prevent policies from exploiting gaits outside the validated regime.

V. CONTROL

  • The stateless model computes thrust from instantaneous velocity alone, so a large, slow stroke and small, fast flutter can produce the same thrust.The real fish instead produces thrust through unsteady, vortex-driven motion with a visible, large-amplitude stroke.Source: A. The Control Problem · p. 4
  • Direct motor-angle control can exploit the model’s small-amplitude, high-frequency flutter, which the authors expect not to transfer to the real fish.Bounding motor velocity or low-pass filtering the setpoint does not remove the exploit: the policy can lower amplitude and raise frequency.Source: A. The Control Problem · p. 4
  • The authors frame the remedy as constraining how the policy acts, rather than improving the plant model.The concern is that stateless-model-rewarded gaits lie outside the regime validated on the real fish.Source: A. The Control Problem · pp. 4–5

B. Trajectory Generator

The trajectory generator constrains policy actions to smooth rhythmic strokes with a frequency ceiling set by the fluid model’s identified band. Reinforcement learning uses hardware-measurable observations and a staged curriculum.

B. Trajectory Generator

  • The trajectory generator emits a smooth rhythmic setpoint with a hard frequency ceiling, while preserving policy control over gait amplitude, offset, and turning.Capping frequency makes a given speed require larger amplitude, steering away from the small, fast flutter the stateless model rewards.Source: B. Trajectory Generator · p. 5
  • The ceiling matches the band used to fit the fluid coefficients, so the action space cannot leave the regime where the stateless model was validated.The generator’s six parameters relax toward policy-set targets with a 0.1 s time constant, adapting within a stroke without jumping between steps.Source: B. Trajectory Generator · p. 5

C. Reinforcement Learning

  • Soft actor-critic trains on 24 hardware-measurable observations, including motor and yaw signals, trajectory-generator phase, commands, and recent camera frames.The six-dimensional action sets trajectory-generator parameters; the reward favors proximity, lightly penalizes large actions, and gives a terminal arrival bonus.Source: C. Reinforcement Learning · p. 5
  • A ten-stage curriculum adds observation noise and delay, physics randomization, then wider target arcs; training reached ±120°, making targets beyond that band out-of-distribution.The deployed policy completed the schedule through the ±120° stage.Source: C. Reinforcement Learning · p. 5

VI. SIM-TO-REAL TRANSFER

The policy was deployed unchanged on the physical fish and closed the loop on target reaching, disturbance rejection, and out-of-distribution acquisition and tracking. Its clearest boundary was tight turning in the lateral-rear sector and figure-eight crossings.

A. Experimental Setup

  • An overhead camera tracked AprilTags on the fish and target at roughly 60 fps; the policy received recent poses, IMU, and encoder states at 50 Hz.A laptop sent the three most recent poses to the Raspberry Pi 5, which ran the policy and commanded the motor.Source: A. Experimental Setup · p. 5

B. Deployment

  • From 1.6 m, the fish converged to within 4.5 cm; it reached the 5 cm training radius on 11/22 static and 5/8 perturbed targets.Because targets were hand-placed, these are descriptive session counts, not an unbiased reachability rate.Source: B. Deployment · p. 5

C. Beyond the Training Distribution

  • It reached targets at +158° and −175° and at 2.8 m, and followed a hand-moved figure-eight target in both attempts.The targets exceeded the training range; the two rearward approaches closed to 2.1 cm and 1.6 cm, and the 2.8 m target to 2.3 cm.Source: C. Beyond the Training Distribution · p. 5, Figure 7 · p. 6
  • With no external disturbances in training and changed trajectory-generator scaling constants, it reached 7/8 perturbed targets, recovered from imposed starts and prods, and swam through stirred water.The deployed amplitude and turn-bias scalings were 2.7× and 5.3× smaller, and its frequency ceiling was 1.5× higher than in training.Source: C. Beyond the Training Distribution · p. 6
  • The lateral-rear 90°–120° band had 0/4 successes, and only one of ten figure-eight attempts produced clean crossing loops, both limited by turning radius.The text attributes these failures to the tight turn required within the pool, rather than to tracking failure.Source: C. Beyond the Training Distribution · p. 6, Figure 8 · p. 6

D. Why the Transfer Works

The trajectory generator keeps simulated tail-velocity power within the identified frequency band more reliably than unconstrained direct action. Hardware measurements show a shortfall, while a setpoint filter performs unevenly when the gait reverses.

D. Why the Transfer Works

  • Tail velocity, rather than commanded setpoint or tail angle, is the diagnostic because the fluid model uses velocity to calculate thrust.The setpoint contains 50 Hz controller stepping that the motor removes before the fluid sees the motion.Source: D. Why the Transfer Works · p. 6
  • 62% of deployed hardware velocity power is below 4 Hz; rescaling for the 1.5× frequency-ceiling error recovers 77%, with the remainder unexplained.Asynchronous encoder sampling leaves residual broadband content, making the 62% figure a lower bound.Source: D. Why the Transfer Works · p. 7, Figure 9 · p. 7
  • A first-order setpoint filter keeps 93% of velocity power in band during forward swimming but only 68% when the gait reverses.Unlike setpoint attenuation, the trajectory generator bounds frequency by construction in both directions.Source: D. Why the Transfer Works · p. 7
  • The policy comparison is not a controlled experiment because the policies used different fluid fits and motor models.The velocity-frequency signature is comparable across them, but the training differences qualify the comparison.Source: D. Why the Transfer Works · p. 7
  • Trajectory-generator runs place 97% of velocity power below 4 Hz, versus 44% for unconstrained direct action; open-loop identification runs place 92% below 4 Hz.Source: D. Why the Transfer Works · pp. 6–7, Figure 9 · p. 7

VII. DISCUSSION

The discussion frames sim-to-real transfer as a bounded-model problem: unconstrained learning can exploit omitted physics, while a band-limited trajectory generator keeps actions within identified conditions. It also connects this constraint to system identification and explains out-of-distribution target behavior through error-based feedback.

VII. DISCUSSION

  • Fig. 10 compares commanded setpoints with measured motor angles for TG, DA, and DA-FOF; direct action requests a rapid oscillation the motor does not reproduce.This exposes a hardware-side mismatch alongside the simulator's modeling limits.Source: Figure 10 · p. 7
  • A stateless fluid cannot distinguish a large, slow stroke from a small, fast flutter, and an unconstrained policy converges on the flutter outside the coefficient-fit regime.The authors suggest this may generalize beyond this robot, but say their experiments do not prove it.Source: §VII · p. 7
  • The simulator is reliable only in a bounded moderate-amplitude, moderate-frequency regime; the band-limited trajectory generator keeps gaits there despite omitted unsteady vortex shedding and added-mass transients.The authors did not test an unconstrained policy on the fish, so they cannot determine whether a tighter fluid model would transfer it.Source: §VII · p. 7
  • Fixing inertial properties from CAD removed simulated backward swimming; otherwise, the fluid fit could absorb structural error and produce coefficients predicting the wrong thrust sign.The discussion presents constraining parameter absorption during identification as analogous to constraining policy actions during control.Source: §VII · p. 7
  • Beyond-training targets remain errors rather than new input types: the policy observes target offset and learns an error-driven servo, while the generator bounds its gait to the identified family.Further, behind, or moving targets create larger or time-varying errors, not a map over the training distribution.Source: §VII · pp. 7–8

VIII. CONCLUSION AND FUTURE WORK

A single unchanged policy trained in a cheap stateless simulator achieved several forms of closed-loop control on a real fish. The conclusion proposes ways to refine the approach and asks how much physics can reside in the controller rather than the simulator.

VIII. CONCLUSION AND FUTURE WORK

  • A single unchanged RL policy trained in a cheap stateless simulator gave a real underactuated fish zero-shot closed-loop reaching, disturbance rejection, out-of-distribution acquisition, and moving-target tracking.It was deployed in an outdoor pool.Source: §VIII · p. 8
  • Two independent identifications grounded the simulator, and its band-limited trajectory generator kept learned gaits within the regime those identifications covered.An unconstrained policy instead exploited small-amplitude, high-frequency flutter rewarded by the stateless fluid.Source: §VIII · p. 8
  • The authors propose directly identifying the simulator's validity boundary and replacing the fixed harmonic-oscillator constraint with a less blunt one to potentially recover agility.Whether this trade holds beyond this fish remains an open question.Source: §VIII · p. 8
  • They propose sampled-grid targets for an unbiased reach map and randomized external forces during training to build disturbance rejection.The observed disturbance rejection was not designed for.Source: §VIII · p. 8
Loading 2609.36993v1…