PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to school in the presence of hydrodynamic interactions

Mattia Gazzola, Andrew A. Tchieu, Dmitry Alexeev, Alexia de Brauer, Petros Koumoutsakos

arXiv:1509.04605v1physics.flu-dyn

TL;DR

The role of hydrodynamics in fish schooling remains a largely unexplored question, especially because self-propelled swimmers must adapt to changing fluid loads. This paper models swimmers as interacting dipoles and uses reinforcement learning and evolutionary optimization to study formation control and swimming effort. Adaptive policies maintain prescribed formations, and optimization identifies an hourglass school associated with reduced collective effort.

  • Problem

    How self-propelled swimmers respond to one another’s wakes, and how flow-mediated interactions relate to schooling, remain largely unanswered.

    Source: §I · p. 2

  • Method

    The study models self-propelled swimmers as finite-width dipoles and uses reinforcement learning to learn formation-maintaining actions, then evolutionary optimization to identify low-effort school arrangements.

    Source: §I · p. 2, §II · p. 3

  • Results

    Adaptive policies maintain prescribed formations where fixed rules do not robustly do so, and optimization identifies an hourglass arrangement associated with a 10-fold reduction in collective effort.

    Source: §I · p. 2, D. Optimal shape of a school of dipole… · p. 12, §IV · p. 15

  • Takeaways & Limitations

    The findings suggest that adaptive responses to flow-mediated interactions matter for schooling, while tight packing can aid collective effort but eventually makes formation maintenance harder and collisions more likely.

    Source: §IV · p. 15

Abstract

from arXiv · show

Schooling, an archetype of collective behavior, emerges from the interactions of fish responding to visual and other informative cues mediated by their aqueous environment. In this context, a fundamental and largely unexplored question concerns the role of hydrodynamics. Here, we investigate schooling by modeling swimmers as vortex dipoles whose interactions are governed by the Biot-Savart law. When we enhance these dipoles with behavioral rules from classical agent based models we find that they do not lead robustly to schooling due to flow mediated interactions. In turn, we present dipole swimmers equipped with adaptive decision-making that learn, through a reinforcement learning algorithm, to adjust their gaits in response to non-linearly varying hydrodynamic loads. The dipoles maintain their relative position within a formation by adapting their strength and school in a variety of prescribed geometrical arrangements. Furthermore, we identify schooling patterns that minimize the individual and the collective swimming effort, through an evolutionary optimization. The present work suggests that the adaptive response of individual swimmers to flow-mediated interactions is critical in fish schooling.

I. INTRODUCTION

Schooling models need to account for the feedback between swimmers’ behavior and the flow they create. This work combines a self-propelled dipole model with learning to study formations under hydrodynamic interactions.

I. INTRODUCTION

  • Classical agent-based models omit the flow environment, although gait changes perturb the flow and affect neighbors; self-propelled fish must adapt to nonlinear hydrodynamic loads.Unlike towed swimmers, self-propelled swimmers must adjust their gait to maintain a desired direction under varying loads.Source: §I · p. 2
  • Fixed local rules do not robustly sustain finite schools under flow coupling, whereas reinforcement-learning agents maintain prescribed arrangements and evolutionary optimization identifies low-effort schools.The model examines hundreds of dipole swimmers and their internal structure and global school shape.Source: §I · p. 2

II. LEARNING OPTIMAL BEHAVIOR IN A FLUID-MEDIATED ENVIRONMENT

  • Reinforcement learning uses trial and error to map sensed states to actions that maximize reward and maintain swimmers’ relative positions in a prescribed formation.The model evolves swimmers in an inviscid fluid, distinguishing hydrodynamic effects from ad hoc local interaction rules.Source: §II · p. 3, A. A finite width dipole model for… · p. 4

A. A finite width dipole model for hydrodynamically interacting swimmers

  • The low-order finite-width dipole model represents swimmers as self-propelled dipoles in inviscid flow, capturing leading-order coupling for swimmers spaced more than one body length apart.It abstracts from morphology, inertia, three-dimensionality, and viscous effects, supporting qualitative inquiries rather than detailed swimmer-level resolution.Source: A. A finite width dipole model for… · p. 3

B. Swimming gaits and maneuvering through circulation change

Each dipole swimmer has five discrete gaits, implemented through changes in vortex circulation. This limited action set reduces the computational cost of learning a policy.

B. Swimming gaits and maneuvering through circulation change

  • The five actions are nominal-speed travel, slower or faster straight travel, and left or right turns at a specified radius.Turns occur at nominal speed; the available actions are mapped to integers 1 through 5.Source: B. Swimming gaits and maneuvering… · p. 4, Figure 3 · p. 5
  • Swimmers implement these gaits by adjusting vortex circulation, which changes the flow field and influences every swimmer in the system.The additional circulation changes are fixed by the swimming parameters for speed and turning radius.Source: B. Swimming gaits and maneuvering… · p. 5
  • The five-action repertoire simplifies natural swimming, but reduces the computational cost of identifying an optimal policy through reinforcement learning.Natural swimmers have many kinematic degrees of freedom and can fine-tune gaits in response to environmental cues.Source: B. Swimming gaits and maneuvering… · p. 5

C. The Aoki-Couzin behavioral model with hydrodynamics interactions

The Aoki-Couzin model specifies collision avoidance, alignment, and attraction rules. Here, those desired directions are translated into circulation changes before swimmers evolve with hydrodynamic coupling.

C. The Aoki-Couzin behavioral model with hydrodynamics interactions

  • The Aoki-Couzin model uses collision avoidance, alignment with a larger neighborhood, and attraction to a still larger neighborhood to set each agent’s desired direction.Changing interaction-zone sizes produces qualitatively different behaviors, including dynamically parallel and highly parallel schooling.Source: C. The Aoki-Couzin behavioral model… · p. 5
  • To apply these rules to dipoles, each agent determines its desired direction and adjusts its vortex circulations accordingly.The conversion assumes no other swimmers are present when computing the circulations needed to match the desired bearing.Source: C. The Aoki-Couzin behavioral model… · p. 6
  • After the circulations are set, the swimmers’ system dynamics are evolved with hydrodynamic coupling.The model examines the resulting behavior under hydrodynamic interactions.Source: C. The Aoki-Couzin behavioral model… · p. 6

D. Reverse engineering of dynamic interaction rules via reinforcement learning

The section uses reinforcement learning to infer interaction policies for dipole swimmers tasked with maintaining prescribed formations, rather than specifying local rules in advance. The policy combines a target-following reward, sensory states, available gaits, and trial-and-error learning.

D. Reverse engineering of dynamic interaction rules via reinforcement learning

  • Swimmers learn to maintain assigned target positions through reinforcement learning, which identifies interaction policies rather than relying on handcrafted local rules.The framework defines the goal through a reward, sensed state, available actions, and trial-and-error learning.Source: D. Reverse engineering of dynamic… · pp. 6–7
  • One-step Q-learning updates action values from experience, with an ε-greedy choice balancing the best-known action and exploration.All swimmers share a policy and update it using their personal experience.Source: D. Reverse engineering of dynamic… · p. 6
  • A shared policy reduces convergence time linearly with the number of agents and yields a hundreds-fold computational-cost reduction, though individual policies may allow finer tuning.The shared approach has every agent update the same action-value function.Source: D. Reverse engineering of dynamic… · p. 6
  • The reward favors following an assigned target while penalizing unnecessary maneuvers and rewarding reduced effort through slowing down.The reward weights target distance at 0.9 and action effort at 0.1.Source: D. Reverse engineering of dynamic… · p. 7
  • Each swimmer’s state bins target distance and orientation into 900 possibilities, while five available gaits form the action set.The continuous motion remains in two-dimensional space; only the sensed quantities are discretized.Source: D. Reverse engineering of dynamic… · p. 7

E. Optimization of schooling patterns via evolution strategies

The paper couples reinforcement learning with CMA-ES to search school geometries according to collective swimming effectiveness. For each candidate geometry, swimmers first learn to swim coherently, then the school’s effort-based cost is evaluated and returned to the optimizer.

E. Optimization of schooling patterns via evolution strategies

  • CMA-ES evaluates each candidate school only after reinforcement learning trains its swimmers, then uses the resulting fitness to select and evolve geometries.The process repeats across generations until an optimal solution is reached.Source: E. Optimization of schooling patterns… · p. 7, Figure 4 · p. 8
  • The optimizer samples school-configuration vectors from a multivariate Gaussian and adapts its covariance using configurations with successful cost values.Each evaluation follows a learning period and a separate evaluation period.Source: E. Optimization of schooling patterns… · p. 7
  • The school-level objective minimizes collective swimming effort, measured through additional circulation expenditure relative to nominal cruise swimming.The paper uses f = 0 as the reference for cruise swimming by isolated dipoles.Source: E. Optimization of schooling patterns… · pp. 7–8

A. Learning process for an individual dipole swimmer

A single dipole learns to follow a target that alternates between straight runs and fixed-radius turns. After an initial transient, its reward improves and its learned policy follows the pseudo-random path indefinitely.

A. Learning process for an individual dipole swimmer

  • The target alternates straight runs with random turns of fixed radius, and the swimmer chooses among turning, accelerating, slowing, or continuing straight.The swimmer senses its position, bearing, and target position, but is not given predefined local action rules.Source: A. Learning process for an individual… · p. 8
  • After an initial transient, the swimmer’s reward rises as it learns to track the target, ultimately following its pseudo-random path indefinitely.The intermediate policy follows for longer before failing; the non-adaptive trajectory fails at its first turn.Source: A. Learning process for an individual… · p. 8, Figure 5 · p. 9
  • Figure 5a plots reward normalized by its maximum, while panels 5b–d show non-adaptive, intermediate, and well-adaptive trajectories against the target.The caption identifies the agent and target paths and includes streamlines as a reference.Source: Figure 5 · p. 9

B. Classical agent based models versus learning agents in the presence of hydrodynamics

Reinforcement learning lets dipole swimmers maintain prescribed formations by adjusting their gaits to hydrodynamic loads, whereas fixed-gait and classical agent-based rules do not robustly preserve schooling. With hydrodynamics, collision counts rise sharply under both tested Aoki-Couzin rules, although the study does not exhaust their parameter space.

B. Classical agent based models versus learning agents in the presence of hydrodynamics

  • Adaptive gait changes learned through reinforcement learning let swimmers maintain prescribed formations despite varying hydrodynamic loads.Fixed-gait swimmers instead strain apart and collide in square and random arrangements.Source: B. Classical agent based models versus… · p. 9
  • Unlike infinite periodic dipole lattices passively stabilized by a stationary flow, finite schools require adaptive gaits to respond to varying loads.The authors frame their approach as complementary to the infinite-school analysis [29].Source: B. Classical agent based models versus… · pp. 9–10
  • With hydrodynamics, collisions increase by 40% for the ‘dynamic parallel group’ model and 700% for the ‘highly parallel group’ model.Figure 7 compares collision counts over time; collisions occur when agents are less than 2ℓ apart.Source: B. Classical agent based models versus… · pp. 10–11, Figure 7 · p. 11
  • The study does not explore the full parameter space, so it cannot rule out parameter combinations that sustain structured schooling.The authors report that they could not identify such a set by handcrafting or direct optimization, but explicitly state this is not a mathematical proof.Source: B. Classical agent based models versus… · p. 11

C. Optimal internal structure of a school of dipole swimmers

Optimization identifies an hourglass school with substantially lower collective effort, while internal-structure optimization favors striated, closely packed patterns rather than equidistant hexagons. The hourglass geometry combines boundary alignment and elongation to exploit flow-mediated interactions.

C. Optimal internal structure of a school of dipole swimmers

  • Internal structure is optimized by minimizing the sum of swimmers’ gait-adjustment efforts for the 50 agents nearest the center of a roughly 200-agent school.Diamond, rectangular, and equidistant-hexagon arrangements are compared while edge effects are excluded.Source: C. Optimal internal structure of a… · pp. 11–12
  • Diamond and rectangular optimizations form striated patterns, while equidistant hexagons provide no collective benefit.The model assumes swimmers remain more than one characteristic length ℓ apart; the reported minimum spacing is 2.55ℓ, and tighter packing is suboptimal.Source: C. Optimal internal structure of a… · p. 12, Figure 8 · p. 13

D. Optimal shape of a school of dipole swimmers

  • Stochastic optimization minimizes whole-school effort while jointly varying the shape and area of a 100-swimmer formation.Effort f = 0 corresponds to cruise swimming by isolated swimmers.Source: D. Optimal shape of a school of dipole… · p. 12
  • The optimized hourglass school contracts area ∼80%, reduces collective effort tenfold, and outperforms Aoki-Couzin models by 190% in swimming effort.The authors report that circulation strength decreases for all swimmers in the optimized arrangement.Source: D. Optimal shape of a school of dipole… · pp. 12–13
  • Hourglass swimmers align in striated patterns and synchronize a stronger global dipolar field, maintaining forward speed with smaller circulation strength.Swimmers in the neck can draft from those ahead while being pushed by those behind.Source: D. Optimal shape of a school of dipole… · p. 12
  • The hourglass combines boundary alignment with elongation, reducing counterflow from lateral neighbors; square and diamond arrangements are detrimental to effort.For both prescribed arrangements, the reported whole-school fitness is f = 0.004.Source: D. Optimal shape of a school of dipole… · p. 13

IV. DISCUSSION

Adaptive policies maintain schooling under hydrodynamic interactions, and evolutionary optimization identifies formations with lower collective effort. The optimized schools are elongated and striated, while extending the approach to viscous flows remains future work.

IV. DISCUSSION

  • Classical handcrafted interaction rules are not robust with hydrodynamics, whereas reinforcement learning identifies adaptive policies that help swimmers maintain formations.The swimmers adjust their actions in response to the dynamic environment to maximize a numerical reward.Source: §IV · pp. 13, 15
  • The optimized solution has fbest = −0.067, versus f = 0.004 for both square and diamond formations.The best configuration uses striated swimmer patterns; its minimum separation is 1.35ℓ, above the model’s assumed one-characteristic-length separation.Source: Figure 10 · p. 14, Figure 9 · p. 14
  • Hourglass-shaped schools elongated along the swimming direction minimize collective effort, while internal striated arrangements support drafting and pushing.The striated pattern is reported as optimal independently of the school’s overall shape.Source: §IV · p. 15
  • Tight packing can reduce collective effort by enhancing the global dipolar field, but excessive packing makes formation-keeping harder and collisions more likely.Source: §IV · p. 15
  • Extending the learning and optimization techniques to two- and three-dimensional viscous flows at intermediate Reynolds numbers is identified as future work.Source: §IV · p. 15

V. APPENDIX

The appendix describes adaptive schooling error comparisons, numerical time-step and collision handling, and a CMA-ES/RL procedure for searching school configurations.

1. Time integration and handling of collisions

  • The solver uses forward Euler with a distance-dependent time step, and colliding dipoles are marked dead and assigned zero circulation.The time step is bounded by dtmax when dmin ≥ 2ℓ and by dtmin = 5e−4 when dmin ≤ ℓ/2.Source: §1 · p. 15

2. Error analysis of non-adaptive and adaptive schools of dipole swimmers

  • Adaptive simulations maintain schooling patterns more robustly than non-adaptive simulations under hydrodynamic interactions.Figure 11 compares the summed errors between agents and their assigned target points for diamond, square, and random formations.Source: §2 · p. 15, Figure 11 · p. 16

B. Optimal schooling formations

  • CMA-ES searches school geometry while reinforcement learning trains agents to follow target points for each proposed configuration.Each evaluation includes 10,000 training time units followed by a 100-unit evaluation; the population size is p = 100.Source: B. Optimal schooling formations · pp. 15–16

1. CMA-ES settings and parameterization for the optimization of school shape

The school’s external shape is defined by a mirrored cubic-spline profile whose area is controlled through average swimmer spacing. CMA-ES varies shape parameters and, separately, search-space parameters for internal lattice geometries.

1. CMA-ES settings and parameterization for the optimization of school shape

  • Swimmers are placed maximally distant from one another and the boundary inside a school shaped by a mirrored cubic-spline profile.The profile is shown as a solid line with its symmetric complement dashed; red dots mark control points.Source: §1 · p. 16, Figure 12 · p. 17
  • The school shape varies with four parameters: average swimmer spacing, two radius-scaling constants, and a spline angle, searched over specified ranges.The four-dimensional search space is × [0.1, 2] × [0.1, 2] × [π/6, 5π/6].Source: §1 · p. 16

2. CMA-ES settings for the optimization of internal lattice structure of a school

  • For internal lattice optimization, the diamond, rectangular, and hexagonal search spaces vary over the specified ranges for h, b, and β.The diamond and rectangular spaces use three parameters each; the hexagonal space uses two.Source: §2 · pp. 16–17

Related papers

Loading 1509.04605v1…