PaperlayerView exact revision on arXiv

Source-linked AI summary

Approximately Equivariant Networks for Imperfectly Symmetric Dynamics

Rui Wang, Robin Walters, Rose Yu

arXiv:2201.11969v4cs.LG

TL;DR

Real-world dynamical data often departs from the strict symmetries assumed by equivariant networks, despite symmetry in governing physical laws. This paper defines approximate symmetry and builds networks that relax equivariance constraints; these models outperform baselines with no symmetry bias, overly strict symmetry, and state-of-the-art approximate equivariance in the reported smoke and jet-flow experiments.

  • Problem

    Equivariant networks assume perfect symmetry, although real-world dynamics can break symmetries through varying forces, boundary conditions, or missing values.

    Source: §1 · p. 1

  • Method

    The paper defines approximate symmetry and introduces approximately equivariant networks that relax equivariance constraints while retaining symmetry bias.

    Source: §1 · p. 2

  • Results

    The models outperform baselines with no symmetry bias, overly strict symmetry, and state-of-the-art approximate equivariance on simulated smoke and experimental jet-flow data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments on smoke-plume simulations and real-world jet-flow data support approximate equivariance as a balance between inductive bias and model flexibility.

    Source: §6 · p. 9

    Relaxed group convolution is limited to discrete or compact groups and is inefficient when the group order is large.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Incorporating symmetry as an inductive bias into neural network architecture has led to improvements in generalization, data efficiency, and physical consistency in dynamics modeling. Methods such as CNNs or equivariant neural networks use weight tying to enforce symmetries such as shift invariance or rotational equivariance. However, despite the fact that physical laws obey many symmetries, real-world dynamical data rarely conforms to strict mathematical symmetry either due to noisy or incomplete data or to symmetry breaking features in the underlying dynamical system. We explore approximately equivariant networks which are biased towards preserving symmetry but are not strictly constrained to do so. By relaxing equivariance constraints, we find that our models can outperform both baselines with no symmetry bias and baselines with overly strict symmetry in both simulated turbulence domains and real-world multi-stream jet flow.

1. Introduction

Real-world dynamics often break the strict symmetries assumed by equivariant networks, motivating models that retain symmetry bias while allowing deviations. The paper defines approximate symmetry and introduces approximately equivariant networks, which outperform several types of baselines on simulated smoke and experimental jet-flow data.

1. Introduction

  • Real-world data are rarely perfectly symmetric, and forces, boundary conditions, or missing values can break symmetries in turbulence modeling.The paper argues that these mismatches hinder the potential applications of strictly equivariant networks.Source: §1 · p. 1
  • Prior work relaxed symmetry constraints, but approximate symmetry lacked a formal definition and high-dimensional dynamics symmetries remained unexplored.The introduction also notes that existing research focused on translation symmetry.Source: §1 · p. 2
  • The paper formally characterizes approximate equivariance as an interpolation between no inductive bias and a strong equivariance bias.This formalization motivates a model class that relaxes rather than removes symmetry constraints.Source: §1 · p. 2
  • Approximately equivariant networks model imperfectly symmetric dynamics by relaxing equivariance constraints.The proposed schemes generalize weight relaxation for group convolution and introduce input-dependent kernel-basis dependencies for steerable CNNs.Source: §1 · p. 2
  • The proposed models outperform baselines with no symmetry bias, overly strict symmetry, and state-of-the-art approximate equivariance on simulated smoke and experimental jet-flow data.The paper applies the networks to fluid-flow forecasting and studies rotation, scaling, and Euclidean symmetries.Source: §1 · p. 2

2. Mathematical Preliminaries

The preliminaries define equivariance and invariance, describe common architectural mechanisms for strict equivariance, and introduce approximate equivariance for dynamics that are not perfectly symmetric.

2.1. Equivariant Functions and Neural Networks

  • Equivariance is defined through group actions on inputs and outputs, while invariance is the special case with a trivial output representation.The section describes group representations acting on the input and output spaces.Source: §2.1 · p. 2
  • Strictly equivariant networks remain equivariant when their layers are equivariant, and commonly use group convolution or steerable CNNs as trainable linear layers.The section identifies weight sharing and weight tying as the two strategies used in these layers.Source: §2.1 · p. 2
  • Group convolution achieves equivariance by sharing kernel weights across pairs indexed by the same relative group element.The kernel weight depends on g^-1h.Source: §2.1 · p. 2
  • Steerable convolution imposes constraints on a matrix-valued kernel, whose solutions can be expressed as a trainable combination of equivariant kernel-basis elements.The kernel basis follows from the linear constraint on the kernel.Source: §2.1 · pp. 2–3

2.2. Approximate Equivariance

  • Imperfect metal thickness or composition can break the rotational symmetry of heat diffusion, leaving the resulting dynamics approximately symmetric.The figure contrasts uniform diffusion, which gives perfect symmetry, with slightly varying diffusion, which gives approximate symmetry.Source: §2.2 · p. 3, Figure 2 · p. 3
  • The definition makes strict equivariance the zero-error case of approximate equivariance.Approximate equivariance is defined for a function under group actions on its input and output.Source: §2.2 · p. 3

3. Approximately Equivariant Networks

The section develops approximately equivariant networks that relax weight-sharing and weight-tying constraints, retaining a symmetry bias while allowing imperfect symmetry. It presents relaxed group and steerable convolutions, regularization, alternative constructions, and an analysis of equivariance error.

3. Approximately Equivariant Networks

  • The framework relaxes strict symmetry constraints in group convolution and steerable CNNs to model dynamics with approximate symmetries, including rotation, scaling, and Euclidean symmetries.It generalizes a weight-relaxation approach to these symmetry groups.Source: §3 · p. 3

3.1. Relaxed Group Convolution

  • Relaxed group convolution mixes multiple kernels with coefficients varying by group element, introducing symmetry-breaking dependence on the specific pair (g, h).The number of kernels controls flexibility: smaller L imposes stronger symmetry, while larger L allows more; L = 3 performed best in most experiments.Source: §3.1 · p. 3

3.2. Relaxed Steerable Convolution

  • Relaxed steerable convolution makes weights depend on kernel position, and rotation-group weights can depend only on offset angle.Weights are initialized equally across positions and penalized for differences; input-position-dependent weights are too large to use directly, so the method proposes low-rank factorization.Source: §3.2 · p. 4

3.3. Soft Equivariance Regularization

  • Soft equivariance regularization adds loss terms on symmetry-independent weights to encourage equivariance and prevent over-relaxation.The hyperparameter α places an equivariance prior; it does not directly set the model’s equivariance level.Source: §3.3 · p. 4

3.4. Other Alternatives for Approximate Symmetry

  • Alternative constructions include Lift Expansion, constrained locally connected networks, and models that begin with non-equivariant layers before equivariant layers.The locally connected network retains convolution-like locality but uses unshared spatial weights and soft equivariance constraints.Source: §3.4 · pp. 4–5

3.5. Equivariant Error Analysis

  • Equivariance error measures a function’s departure from equivariance, and a function is ϵ-approximately equivariant when that error is below ϵ.The definition measures the worst-case discrepancy between transforming the input before applying the function and transforming its output.Source: §3.5 · p. 5
  • A strictly equivariant approximator cannot perfectly learn an approximately equivariant function, while an approximately equivariant model class can converge to the target’s equivariance error as model error falls.The convergence result bounds the difference in equivariance errors by 2c + ϵ when model error is at most c.Source: §3.5 · p. 5

4. Related Work

Related work covers equivariant models for structured data and physical dynamics, alongside approaches that relax strict symmetry constraints. The paper situates its approach among physics-guided methods for learning dynamical systems.

4.1. Equivariance and Invariance

  • Convolutional, recurrent, graph, capsule, and equivariant networks use symmetry in deep learning, with equivariant models applied extensively to image data.The related work also notes a growing body of deep-learning approaches to dynamical systems.Source: §4.1 · p. 5, §4.2 · p. 5
  • Symmetry has a close connection to physics through Noether’s correspondence between symmetry groups and conserved quantities, motivating equivariant models for dynamical systems.Prior work applies rotational equivariance to trajectory prediction and studies scaling, rotation, and uniform motion in physical dynamics.Source: §4.1 · p. 5
  • Approximate equivariance addresses overly restrictive symmetry assumptions; prior approaches relax weight sharing or combine equivariant and non-equivariant layers, while this method uses efficient convolutional layers.The cited MLP approach is described as unsuitable for large images or high-dimensional physical dynamics because of its weight count.Source: §4.1 · p. 5

4.2. Learning Dynamical Systems

  • Physics-guided learning incorporates physical-system inductive biases, including models built around RANS-LES coupling or Hamiltonian and Lagrangian mechanics.This work encodes approximate symmetries as inductive biases for dynamics prediction without over-constraining representation power.Source: §4.2 · p. 6

5. Experiments

Across smoke simulations and experimental jet flow, approximately equivariant models generally outperform fully equivariant and other approximate-equivariance baselines. Their performance varies with the symmetry in the data, and all tested symmetry-bias levels perform similarly on ocean dynamics after fine-tuning.

5. Experiments

  • Baselines span non-equivariant, fully equivariant, and approximate-equivariance models; evaluations test future-time and cross-domain generalization using 20-step RMSE averaged over three runs.EMLP is excluded because it cannot handle large image data.Source: §5 · p. 6

5.1. Experiments on Synthetic Smoke Plumes

  • On synthetic smoke, RGroup significantly outperforms baselines on both test sets for approximate translation, while RSteer has the lowest RMSE for rotation and scaling.The table reports prediction RMSE across three approximate-symmetry datasets, with Future and Domain denoting temporal and spatial-domain test splits.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2. Learning Different Levels of Equivariance

  • Across 10 smoke datasets with varied rotational equivariance, RSteerR learns the data’s equivariance levels more accurately than baselines, while E2CNN remains perfectly equivariant even when the data is not.The datasets vary buoyancy differences across inflow positions to control data equivariance error.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3. Experiments on Experimental Jet Flow Data

  • On jet flow, relaxed-weight-sharing models achieve lower RMSE than fully equivariant and approximately equivariant baselines; RSteerTR outperforms RGroup and RSteer.Figure 5 also shows the relaxed steerable CNNs have the lowest prediction errors; on ocean dynamics, all models perform similarly after fine-tuning.Source: Table 2 · p. 8, §5.3 · p. 8, Figure 5 · p. 8
  • On ocean dynamics, all models have very close prediction performance after fine-tuning, unlike the smoke and jet experiments where approximate equivariance better matched the ground truth than strict or absent symmetry bias.The authors hypothesize that strict rotational symmetry provides little advantage over a baseline CNN for ocean currents, so soft approximate equivariance also provides no advantage.Source: §5.3 · p. 9

6. Discussion

The paper proposes approximately equivariant networks that relax symmetry constraints to better fit real-world dynamics, and reports that they outperform many baselines on smoke-plume simulations and real-world jet flow.

6. Discussion

  • The proposed approximately equivariant networks relax symmetry constraints to better fit real-world scenarios.The approach relaxes weight-sharing and weight-tying schemes in group and steerable convolutions.Source: §6 · p. 9
  • The methods aim to balance inductive bias and model flexibility by relaxing weight-sharing and weight-tying schemes.The relaxations are applied in group convolution and steerable convolution.Source: §6 · p. 9
  • On smoke-plume simulations and real-world jet-flow data, the proposed networks outperform many baselines with no symmetry bias or overly strict symmetry constraints.Source: §6 · p. 9
  • Future work includes applying relaxed weight sharing to graph neural networks and analyzing universal approximation and generalization.Source: §6 · p. 9

A.1. Hyperparameter Tuning

The experiments use grid search to tune training and architecture settings, including method-specific parameters, while keeping model parameter counts below 107 for fairness.

A.1. Hyperparameter Tuning

  • Grid search tunes learning rate, batch size, hidden dimension, layer count, and prediction-error training steps, as listed in Table 3.Source: §A.1 · p. 13, Table 3 · p. 13
  • The search also tunes filter-bank count for group convolutions and the weight-constraint coefficient α for relaxed weight sharing.Source: §A.1 · p. 13
  • Input length is fixed at 10, and each model has fewer than 107 trainable parameters for a fair comparison.Source: §A.1 · p. 13

A.2. Additional Experiments on Real-world Ocean Dynamics

The ocean-current experiments use NEMO reanalysis data and compare equivariant, approximately equivariant, and proposed relaxed models; Table 4 is identified as reporting prediction RMSE.

A.3. Additional Experiments on Real-world Ocean Dynamics

  • Table 4 presents prediction RMSE on the ocean-current data.Source: Table 4 · p. 13
  • The dataset comprises 36 64×64 Pacific Ocean subregions from 2021, with test sets differing from training in time range and spatial domain.Source: §A.2 · p. 13
  • The prediction comparison includes three equivariant models, three best approximately equivariant baselines, relaxed steerable CNNs, and relaxed group convolutions.Source: §A.2 · p. 13

A.3. Equivariance Error Analysis

The appendix analyzes how approximate-equivariance error relates to approximation error, including a lower bound for strictly equivariant approximators and a bound on error-estimate deviation.

A.3. Equivariance Error Analysis

  • For an approximately equivariant function and a strictly equivariant approximator, at some input the approximation error is at least half the function’s equivariance error.The proof considers the input x and its transformed counterpart gx; one of those two approximation errors must exceed half the equivariance error.Source: §A.3 · p. 13
  • If f has equivariance error ϵ and an approximately equivariant model approximates f within c uniformly, their equivariance-error estimates differ by at most 2c + ϵ.Source: §A.3 · p. 13
  • The second bound assumes a G-invariant norm and an approximately equivariant model class with varying equivariance error.Source: §A.3 · p. 13

A.4. Additional Figures

The additional figures locate axial measurement fields of view for multistream jets and show test-set prediction RMSEs across regularization settings for a scaling RSteer model.

A.4. Additional Figures

  • Figure 6 locates streamwise-plane fields of view at the jet centerline, with three vertical stations at each axial location.The white boxes mark fields of view, and pink lines illustrate the vertical stations.Source: Figure 6 · p. 14
  • Figure 7 presents test-set prediction RMSEs for a scaling RSteer model trained with different regularization parameter α.Source: Figure 7 · p. 14

Related papers

Loading 2201.11969v4…