Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

0. NOTATION

• Signals on a domain are denoted x(u) ∈ X(Ω, C), with x mapping each domain element u to C.Source: §0 · p. 7
• Functions on signals are written f(x) ∈ F(X(Ω)), with f : X(Ω) → Y.Source: §0 · p. 7
• Groups and their elements are denoted G and g, respectively.Source: §0 · p. 7
• For discrete signals, xu ∈ C^s represents the vector at domain element u, while xuj ∈ C denotes its jth component.Source: §0 · p. 7
• F(X) denotes a function on discrete-domain signals that returns another discrete-domain signal represented as a matrix.Source: §0 · p. 7
• A mesh is represented as T = (V, E, F), consisting of nodes V, edges E, and faces F.Source: §0 · p. 7
• The symbols ℓ(γ) and ℓuv denote a curve’s length and the discrete metric on edge (u, v), respectively.Source: §0 · p. 7

Deep learning foundations

• Deep learning’s practical success is attributed to representation learning and local gradient-descent methods, typically implemented through backpropagation.These principles support feasible solutions to difficult tasks such as computer vision, Go, and protein folding at appropriate computational scale.Source: §1 · p. 8

Motivation

• Geometric deep learning exposes physical-world regularities and low-dimensional structure to address the curse of dimensionality across applications.Rather than learning generic high-dimensional functions, it exploits pre-defined regularities through unified geometric principles.Source: §1 · p. 8

Geometric regularity

• Known data symmetries, including translation and rotation, motivate architectures built around geometric structures such as image grids, time sequences, and molecular coordinates.The text presents these models as instances of a shared principle of geometric regularity.Source: §1 · p. 8

Unification

• The geometric-unification perspective provides a common mathematical framework for CNNs, RNNs, GNNs, and Transformers.It is framed in the spirit of Felix Klein’s Erlangen Program.Source: §1 · p. 8
• It also offers a constructive procedure for incorporating prior physical knowledge into neural architectures and designing future architectures.This is the second stated purpose of the geometric blueprint.Source: §1 · p. 8

Scope and limitations

• The work focuses on representation-learning architectures and data symmetries, not detailed reviews of self-supervised, generative, reinforcement-learning, optimization, or regularization pipelines.It studies selected architectures in depth rather than attempting to summarize the entire Geometric Deep Learning literature.Source: §1 · p. 9

2 Learning in High Dimensions

• High-dimensional supervised learning estimates an unknown label function from i.i.d. samples in X = R^d, with modern neural networks often operating in the interpolating regime.The learned function matches all observed labels, while performance is evaluated on new samples from the data distribution.Source: §2 · p. 9

2.1 Inductive Bias via Function Regularity

• Effective learning therefore requires inductive bias through function-class construction or regularisation, including explicit complexity penalties and implicit minimum-norm solutions from optimisation.For neural networks, weight decay and path-norm regularisation constrain network complexity, while gradient descent on under-determined least squares selects interpolating solutions with minimal L2 norm.Source: §2 · p. 10, §2.1 · pp. 10–11

2.2 The Curse of Dimensionality

• For 1-Lipschitz functions, ensuring approximation error ϵ requires N = Θ(ϵ^-d) samples, so sample complexity grows exponentially with dimension.Sobolev regularity yields a minimax rate of order ϵ^-d/s, improving the picture substantially only when s ∝ d, which is unrealistic in practice.Source: §2.2 · pp. 12–13
• Sparsity-promoting fully connected networks can break the curse under low-dimensional projection assumptions, but complex long-range correlations motivate geometric priors based on spatial structure.The projection assumption is described as unrealistic for many applications including vision, speech, physics, and chemistry.Source: §2.2 · p. 12

3 Geometric Priors

• Geometric priors exploit domain structure through symmetry and scale separation, as illustrated by CNN weight sharing and pooling.Signals are modeled as functions on a domain Ω, with scale separation preserving important characteristics when transferring signals to coarser domains.Source: §3 · pp. 13, 15

3.1 Symmetries, Representations, and Invariance

• Symmetries act on domains and signals through group actions and representations, constraining functions to improve learning efficiency.A representation assigns invertible matrices ρ(g) satisfying ρ(gh) = ρ(g)ρ(h), while group actions on signals induce corresponding transformations of feature spaces.Source: §3.1 · pp. 17, 19
• Invariant functions ignore group transformations, whereas equivariant functions transform outputs compatibly; CNNs combine equivariant convolutions with invariant global pooling.Object classification is shift-invariant, convolutional layers are shift-equivariant, and segmentation requires the output mask to follow input shifts.Source: §3.1 · pp. 19–21

3.2 Isomorphisms and Automorphisms

• The relevant symmetry group depends on the domain’s structure, progressing from bijections on sets to homeomorphisms, diffeomorphisms, and isometries.A symmetry is an invertible structure-preserving map; additional structures include topology, differentiability, distance, and orientation.Source: §3.2 · pp. 21–22
• Symmetries are automorphisms preserving an object’s structure, while isomorphisms establish structural equivalence between distinct objects; adding preserved structure selects progressively smaller subgroups.For example, E(2) is a subgroup of Diff(2), and SE(2) is a subgroup of E(2).Source: §3.2 · pp. 22–23

3.3 Deformation Stability

• Deformation stability applies both to signals on fixed domains and to changing domains such as graphs and manifolds, using costs or distances based on structure-preserving alignments.Graph edit distance vanishes for isomorphic graphs, while Gromov-Hausdorff distance vanishes for isometric Riemannian manifolds.Source: §3.3 · pp. 24–26
• Deformation stability replaces exact invariance with a complexity-weighted notion that measures how far transformations lie from a symmetry subgroup, while excluding unconstrained large deformations.Small deformations do not form a group because composition violates closure, whereas using all of Diff(Ω) can allow transformations that materially change semantic content.Source: §3.3 · p. 24

3.4 Scale Separation

• Fourier analysis exposes global signal and domain properties and extends beyond Euclidean spaces through intrinsic differential operators, but its global support makes it unstable under arbitrarily small deformations.The Laplacian provides a route to Fourier transforms on graphs, groups, and manifolds, while approximate translations can destabilize Fourier representations regardless of deformation size.Source: §3.4 · pp. 27–29
• Multiscale wavelet representations localize information in space and frequency, yielding approximately deformation-equivariant features and motivating hierarchical coarsening from local interactions toward coarse scales.A locally stable function approximately factors as f ≈ f_j ◦ P_j, with coarsening applicable to grids, graphs, and manifolds.Source: §3.4 · pp. 28–30
• FMM groups nearby sources and treats each group as a single source for long-ranged n-body force calculations.Source: §3.4 · p. 30

3.5 The Blueprint of Geometric Deep Learning

• Linear invariants are too restrictive, whereas equivariant maps composed with nonlinearities and group averaging can yield expressive invariant functions and universal approximators.For translation-invariant images, linear averaging would retain only the average RGB color; localized equivariants address the tension between global invariance and deformation stability.Source: §3.5 · pp. 31–32
• Composing local equivariant layers expands receptive fields while preserving stability, and interleaved coarsening further enlarges them, supporting a common framework across geometric domains.Methods differ in their domains, symmetry groups, and implementations, while the blueprint encompasses many existing deep learning architectures.Source: §3.5 · pp. 32, 34
• The blueprint combines local equivariant maps, global invariant maps, and coarsening operators to construct rich function spaces with prescribed invariance and stability.These blocks are combined into representations for signals on domains equipped with a symmetry group.Source: §3.5 · pp. 31–33

4 Geometric Domains: the 5 Gs

• The 5G framework unifies grids, groups and homogeneous spaces, graphs, geodesics and manifold metrics, and gauges as local frames for tangent or feature spaces.Groups represent global symmetry transformations, geodesics represent metric structures on manifolds, and gauges represent local reference frames.Source: §4 · p. 34, Figure 9 · p. 35

4.1 Graphs and Sets

• Graphs require permutation-invariant graph outputs and permutation-equivariant node outputs, with local shared functions over neighbourhood features providing the geometric deep learning construction.For graphs, adjacency and node features must be permuted together; when the local function is injective, the construction is equivalent to one Weisfeiler-Lehman refinement step.Source: §4.1 · pp. 36–38

4.2 Grids and Euclidean spaces

• Grids are graphs with fixed node ordering, so their stronger geometric prior is translation invariance, yielding shared-weight local operators represented by circulant matrices and discrete convolutions.Unlike sets and general graphs, fixed neighbour ordering lets the local function distinguish left and right neighbours without permutation invariance.Source: §4.2 · p. 40
• Convolution emerges from translational symmetry because circulant matrices commute with shifts, while their shared eigenbasis is the discrete Fourier basis, producing the discrete Fourier transform.A matrix is circulant iff it commutes with the shift, and the Fourier transform diagonalises all circulant convolution operators.Source: §4.2 · pp. 41–42
• The grid Fourier-convolution blueprint extends to graphs through eigenvectors of adjacency or Laplacian matrices, but graph Fourier bases depend on graph structure and lose directional organization.Consequently, Fourier transforms across different graphs cannot be directly compared, and graph filters are isotropic because graph frequencies are organized only by magnitude.Source: §4.2 · pp. 42–43
• Translation-equivariant linear operators are convolutions, with the Fourier basis jointly diagonalizing shift operators and enabling frequency-domain representations.On grids, Fourier filtering can be computed with an FFT in O(n log n) complexity, while the convolution kernel is obtained by inverse Fourier transformation.Source: §4.2 · pp. 42, 44, §4.3 · p. 44

4.3 Groups and Homogeneous spaces

• Group convolution generalizes Euclidean convolution to symmetry groups and is G-equivariant, while group Fourier transforms project signals onto irreducible-representation matrix elements.The construction assumes a suitable invariant measure, and its outputs are functions on the group G rather than necessarily on the original domain Ω.Source: §4.3 · pp. 44, 45, 47
• For spherical signals, SO(3) acts on S2, so spherical convolution produces a function on the three-dimensional group SO(3), requiring subsequent layers to operate on group-defined signals.This differs from translations, whose convolution outputs remain on the same domain and can be composed directly.Source: §4.3 · pp. 46–47
• Convolution is practical only on small or low-dimensional groups, whereas permutation groups and higher-dimensional affine groups are computationally infeasible to integrate over directly.Equivariant maps between low-dimensional domains can nevertheless be represented as generalized convolutions even when the acting group is large.Source: §4.3 · p. 47

4.4 Geodesics and Manifolds

• General manifolds lack global symmetries, so their geometric treatment relies on metric-preserving transformations and local reference-frame changes rather than straightforward filter sliding.Manifold models compactly represent 3D object surfaces, ignore empty space and irrelevant interiors, and support invariance to nearly-isometric non-rigid deformations.Source: §4.4 · pp. 48–49
• A Riemannian metric equips each tangent space with smoothly varying inner products that measure vector lengths, angles, and local volume independently of coordinates.Tangent vectors are coordinate-free entities, while coordinates and metric matrices depend on the chosen local basis.Source: §4.4 · pp. 49–50
• Intrinsic functions on Riemannian manifolds are invariant to isometries and remain stable under approximate isometric deformations.A manifold equipped with a metric has intrinsic properties, and expressing functions through intrinsic quantities guarantees invariance to metric-preserving transformations.Source: §4.4 · p. 50
• Smooth Riemannian manifolds can be embedded in sufficiently high-dimensional Euclidean spaces, but different isometric embeddings may represent the same intrinsic geometry.The Nash (1956) Embedding Theorem establishes existence of such embeddings, while embedding uniqueness is not guaranteed.Source: §4.4 · pp. 50–51
• The intrinsic gradient replaces Euclidean displacement by tangent-vector derivatives and identifies the direction of steepest increase for scalar fields.Differentials act as linear functionals on tangent vectors, while the Riemannian metric represents them through the intrinsic gradient.Source: §4.4 · pp. 51–52
• Geodesics are metric-dependent shortest curves that enable parallel transport, exponential maps, geodesic distances, and locally defined convolution-like filters in tangent spaces.Parallel transport preserves vector lengths and angles while connecting tangent spaces; the exponential map locally flattens the manifold into a tangent space, and geodesic distance is the shortest-path length.Source: §4.4 · pp. 52–54
• Intrinsic manifold symmetries are distance-preserving isometries, whose invariant structures include geodesic distances and the Laplacian.On connected manifolds, metric and Riemannian isometries coincide; self-isometries form the isometry group, and the Laplacian is intrinsic and isometry-invariant.Source: §4.4 · pp. 55–56
• The Laplacian eigenbasis provides manifold Fourier analysis: its orthogonal eigenfunctions represent smooth signals, and it is optimal for truncated smooth-signal approximation.For compact manifolds, the spectrum is discrete, eigenfunctions are orthogonal, and Aflalo et al. (2015) showed that no other basis achieves a better approximation error.Source: §4.4 · p. 57
• Spectral convolution is intrinsically isometry-invariant but isotropic, while direct eigendecomposition is expensive and geometrically unstable under small near-isometric perturbations.Higher-frequency Laplacian eigenfunctions can change dramatically under small domain perturbations; polynomial spectral transfer functions avoid spectral decomposition and connect spectral and spatial views.Source: §4.4 · pp. 58–59
• Spatial manifold convolution can be isometry-invariant through intrinsic exponential-map constructions, but it requires local filters, path-dependent transport, and coordinates in a chosen tangent-space gauge.Manifolds generally lack a global group structure, the exponential map is local, and abstract tangent-space signals must be represented relative to local bases.Source: §4.4 · pp. 59–60
• Consistent global gauges may not exist or may be arbitrary, causing singularities and orientation-dependent, nonsmooth feature maps that motivate gauge-independent constructions.The sphere S2 is non-parallelisable, and even practically constructed frames can vary wildly near singularities or lack a principled correspondence between directions at different points.Source: §4.4 · p. 60

4.5 Gauges and Bundles

• Gauge transformations change coordinate representations without changing geometric objects, and bundle structure groups specify the allowable frame transformations and their induced feature transformations.A gauge is a frame for a fibre; for tangent bundles transformations act through GL(s), while restricted frames use groups such as O(s) or SO(s). The RGB example interprets independent channel permutations as gauge symmetry.Source: §4.5 · pp. 61–62
• Gauge-equivariant maps remain invariant under gauge changes while transforming input and output features through potentially different representations of G.Gauge transformations act independently on feature vectors at each point, unlike node permutations in graphs.Source: §4.5 · p. 63
• Gauge symmetry alone permits position-dependent filters with no spatial weight sharing, while unconstrained vector-field kernels face an overly strong equivariance constraint.For vector fields, independently selectable gauges at source and target points force restrictive matrix-kernel conditions.Source: §4.5 · pp. 63–64
• Connections resolve this constraint by parallel-transporting vectors to a common tangent space, yielding gauge-equivariant convolutions with locally supported filters.The construction assumes a unique connecting geodesic, so locality is required; equivariance additionally requires the filter to commute with the structure-group representation.Source: §4.5 · p. 64

4.6 Geometric graphs and Meshes

• Meshes extend graphs with oriented triangular faces and manifold conditions, while their edge lengths induce a metric whose intrinsic properties are preserved by isometries.A manifold mesh has disk-like 1-hop neighborhoods, and intrinsic quantities depend only on edge lengths.Source: §4.6 · p. 65
• The cotangent mesh Laplacian is local, symmetric, positive semidefinite, and converges to the continuous manifold Laplacian under suitable refinement; expressing weights intrinsically gives isometry invariance.Its eigenvalues analogize frequencies, eigenvectors are orthogonal, and deformations preserving the mesh metric leave the Laplacian unchanged.Source: §4.6 · p. 67
• Fourier-based mesh convolutions are unstable across perturbed or different domains, whereas polynomial and rational Laplacian filters offer efficient alternatives with deformation stability but different locality properties.Polynomial filters are localized but resolution-dependent; rational filters are resolution-independent, exponentially decaying, and can be computed approximately with linear complexity.Source: §4.6 · pp. 68–69
• Viewing meshes through intrinsic operators such as the Laplacian enables operator-based learning, but different discretisations of one surface may vary in connectivity and node count.An intrinsic operator can encode the mesh structure up to isometric embedding, while cross-mesh correspondence cannot generally be represented by node permutation alone.Source: §4.6 · p. 70
• Functional maps replace point correspondences with linear operators between functions on two domains, represented as matrices and illustrated against pointwise maps in Figure 13.In the spectral domain, the map acts on truncated Laplacian Fourier coefficients as x′ = Φ′ĈΦᵀx.Source: §4.6 · p. 70, Figure 13 · p. 71
• Area-preserving functional maps are orthogonal, so their inverse is the transpose, C⁻¹ = Cᵀ.Orthogonality places C in O(n) and provides the condition used for invertibility.Source: §4.6 · pp. 70–71
• Functional maps induce remeshing-invariant or equivariant mesh functions by mapping signals across domains, applying an operator, and mapping them back.Permutation invariance and equivariance are recovered as a special case because permutation matrices are orthogonal; reordering nodes is a trivial remeshing.Source: §4.6 · p. 71
• Wang et al. (2019a) showed that remeshing-invariant functions of Q depend only on its spectrum, explaining robust spectral constructions and permitting Dirac, Steklov, or learnable operators beyond Laplacians.For Q = VΛVᵀ, invariant and equivariant functions take the forms f(Q) = f(Λ) and F(Q) = VF(Λ).Source: §4.6 · p. 71

• The section first derives CNNs, group-equivariant CNNs, and GNNs as direct implementations of the Geometric Deep Learning prescriptions.Source: §5 · p. 72
• For unknown graph structures such as unordered sets, GNN variants recover Deep Sets and Transformer architectures.Source: §5 · p. 72
• Geometric graph and mesh constructions yield equivariant message passing networks and intrinsic mesh CNNs, including Geodesic CNNs, MoNet, and gauge-equivariant mesh CNNs.Source: §5 · p. 72
• Temporal-grid analysis derives translation-equivariant RNNs and connects stability to time warping with gated RNNs such as LSTM and GRU models.The section presents time-warping stability as desirable for handling long-range dependencies, with class invariance yielding gated RNNs.Source: §5 · p. 72
• Rather than exhaustively cataloguing architectures, the exposition aims to help classify future developments through invariances and symmetries.Source: §5 · p. 72

Convolutional operators

• CNNs implement local translation-equivariant operators as convolutions, with compact filters expressed through basis generators and extended to tensor-valued multi-channel features.For scalar images, the domain is a discrete grid; with multiple channels, filters form tensors mapping M input channels to N output channels.Source: §5.1 · pp. 73–74

Efficient multiscale computation

• A vanilla CNN layer composes an equivariant linear convolution, grid coarsening, and pointwise nonlinearity, with global pooling producing translation-invariant representations.ReLU rectifies features and supports higher-order interactions across scales; pooling commonly uses anti-aliased low-pass filtering followed by downsampling or nonlinear max-pooling.Source: §5.1 · pp. 74–75

Deep and Residual Networks

• Empirical evidence favors deeper, thinner CNNs, while ResNets reparameterize layers as perturbations of previous features and enable substantially deeper training.The residual formulation corresponds to a Forward Euler discretization of an underlying ODE, yielding a more favorable optimization geometry than directly learning feature positions.Source: §5.1 · pp. 75, 77

Normalisation

• Normalization replaces activations using first- and second-order moment information, and batch normalization popularized this principle, although a general theory of its benefits remains missing.Variants include layer, weight, instance, and group normalization, while proposed explanations based on better-conditioned optimization landscapes are not yet comprehensive.Source: §5.1 · p. 77

Data augmentation

• Data augmentation adds transformed training images to encode semantic-preserving changes such as lighting, color, rotations, and dilations that CNNs do not explicitly model.Although widely successful in vision and representation learning, augmentation is provably sub-optimal in sample complexity compared with architectures having richer invariance groups.Source: §5.1 · pp. 77–78

Group convolution

• Group convolution moves a filter across a homogeneous space using a transitive group action, generalizing Euclidean translation-based convolution to transformations such as rotations and translations.The filter can be moved to any position because the group action is transitive.Source: §5.2 · p. 78

Discrete group convolution

• Discrete group convolutions model 3D volumetric images with translations and 90-degree grid rotations, while DNA sequences additionally use reverse-complement symmetry combined with translations.For DNA, the reverse-complement operation forms the two-element group Z2, whose nonidentity transformation reverses the sequence and complements each letter.Source: §5.2 · pp. 78–79

Transform+Convolve approach

• Group convolution is implemented by transforming a canonical filter for each non-translation symmetry and applying standard translational convolutions to the resulting copies.For discrete rotations and DNA symmetries, transformations simply permute filter coefficients, enabling efficient indexing with pre-computed indices.Source: §5.2 · p. 80
• The outputs can be stored as Euclidean feature maps with orientation channels, and group equivariance acts by transforming feature maps and permuting those channels.This representation retains standard CNN-like structure, allowing design patterns such as residual networks to be used with group convolutions.Source: §5.2 · pp. 80–81

Spherical CNNs in the Fourier domain

• Spherical CNNs implement convolutions spectrally using spherical harmonics on S2 and Wigner D-functions on SO(3), where Fourier-domain multiplication and FFT-like algorithms provide efficient computation.The convolution on S2 produces functions on SO(3), so multi-layer spherical CNNs require Fourier transforms on both domains.Source: §5.2 · p. 81

5.3 Graph Neural Networks

• GNNs are permutation-equivariant functions F(X, A) built by applying shared permutation-invariant neighbourhood functions φ to node features and graph structure.The local operation is variously called diffusion, propagation, or message passing, and forms a GNN layer.Source: §5.3 · p. 82
• Across the three flavours, learnable transformations ψ and φ surround permutation-invariant aggregation L, typically implemented by sum, mean, or maximum.L can also be constructed using recurrent neural networks.Source: §5.3 · p. 82
• Convolutional GNNs use fixed neighbourhood weights, attentional GNNs use feature-dependent scalar weights, and message-passing GNNs compute arbitrary vector-valued edge messages.These correspond to progressively richer interaction modelling across graph neighbourhoods.Source: Figure 17 · p. 82, §5.3 · p. 83
• The flavours form a containment hierarchy: convolution ⊆ attention ⊆ message-passing, because increasingly general edge interactions subsume the preceding cases.Attentional layers recover convolution through feature-independent coefficients, while both can be represented as restricted message functions.Source: §5.3 · pp. 83–84
• Message passing is harder to train and more memory-intensive, whereas convolution often suits homophilous graphs and attention provides a more scalable middle ground.The three-flavour summary excludes higher-dimensional Weisfeiler–Lehman and explicitly Fourier-transform-based spectral GNNs.Source: §5.3 · p. 84

Empty edge set

• Deep Sets apply a shared transformation independently to each node, forming a permutation-equivariant model and a special case of a convolutional GNN with A = I; Zaheer et al. (2017) established universal-approximation properties.For point clouds, this architecture is commonly called PointNets (Qi et al., 2017).Source: §5.4 · pp. 84–85

Complete edge set

• Complete-graph GNN aggregation is equivalent in expressiveness to the empty-edge case because the invariant summed input is identical for every node and can be ignored.Thus, modeling all possible links with ordinary convolutional aggregation does not recover useful relational distinctions without a more expressive mechanism.Source: §5.4 · p. 85

Transformers

• Self-attention makes complete-graph aggregation expressive by learning normalized coefficients a_uv = a(x_u, x_v), effectively inferring a soft adjacency matrix; Transformers are attentional GNNs.This perspective is attributed to Joshi (2020).Source: §5.4 · pp. 85–86
• Transformers preserve sequence order by augmenting node features with positional encodings, while graph settings can use Laplacian eigenvectors, as in the Graph Transformer of Dwivedi and Bresson (2020).Transformer positional encodings correspond to eigenvectors of a circular-grid graph Laplacian through the discrete Fourier transform.Source: §5.4 · p. 86

Inferred edge set

• Latent graph inference learns an adjacency matrix between I and 11⊤ to capture useful structure, but must balance discrete structure learning with downstream objectives, making optimization highly challenging.It can improve on A = I and avoid the memory and large-neighborhood costs of A = 11⊤, while potentially supporting hypotheses about causal relations.Source: §5.4 · pp. 86–87

• Equivariant layers separately transform node features and spatial coordinates, requiring coordinates to follow E(3) transformations while preserving the appropriate feature transformation.For g ∈ E(3), coordinates transform as ρ(g)x = Rx + b, with R modeling rotations and reflections and b translations.Source: §5.5 · p. 87
• Satorras et al. (2021) use equivariant message passing whose spatial dependence is through pairwise distances, yielding efficient invariance to rotations, reflections, and translations.Distances remain unchanged under E(3), and the layer is a particular message-passing GNN flavour.Source: §5.5 · p. 88
• The approach treats node features as transformation-invariant scalars, limiting representations of directional or higher-order spatial information despite a velocity variant that handles rotating 3D vectors.General E(3)-equivariant networks may instead need features encoding vectors or tensors of arbitrary dimensionality.Source: §5.5 · p. 88
• Broader equivariant methods fall into irreducible and regular representations, extending beyond the simplified scalar-feature layer to richer geometric feature transformations.Irreducible methods solve equivariance constraints using Wigner D-matrices, Clebsch-Gordan matrices, and spherical harmonics, whereas regular methods store latent-feature copies for all group elements.Source: §5.5 · pp. 88–89
• Irreducible-representation models include Tensor Field Networks, 3D Steerable CNNs, and SE(3)-Transformers, while LieConv and LieTransformer pursue broader Lie-group equivariance.Earlier related architectures include Dynamic Graph CNN, SchNet, and DimeNet for point clouds and quantum chemistry.Source: §5.5 · p. 89

Geodesic patches

• Geodesic CNNs construct local mesh patches from geodesic polar coordinates and discretise them through weighted neighbouring nodes that act as soft pixels.The patch is defined using the exponential map and local polar frame; patch radius must remain below the injectivity radius.Source: §5.6 · pp. 90–92

Gauge handling

• Gauge ambiguity makes geodesic patch orientation arbitrary at each node, motivating direction-independent filters, fixed gauges, or aggregation over all filter rotations.Isotropic filters preserve gauge invariance but discard directional information, whereas fixed gauges can be non-intrinsic or singular; angular max pooling retains anisotropic filters while pooling their rotated responses.Source: §5.6 · pp. 90–91

Patch operators

• Patch operators aggregate multi-hop neighbours with radial-angular weighting functions, using predefined weights in Geodesic CNN and Anisotropic CNN and learnable weights in MoNet.The weighting functions form soft pixels, while learnable filter coefficients combine their aggregated responses channel-wise.Source: §5.6 · pp. 91–92, Figure 18 · p. 92

Gauge-equivariant filters

• Gauge-equivariant mesh CNNs let features transform under arbitrary representations of SO(2) or O(2), rather than restricting them to gauge-invariant trivial representations.Examples include tangent vectors under the standard representation and rotated filter responses under cyclic regular representations.Source: §5.6 · p. 92
• Gauge-equivariant message passing parallel-transports neighbour features before applying anisotropic kernels, with learned self and neighbour matrices constrained to intertwine input and output representations.The transporter encodes parallel transport between node gauges, while the neighbour kernel depends on each neighbour’s angular coordinate and treats directions differently.Source: §5.6 · pp. 92–93

Sequential inputs and domains

• RNNs process arbitrarily long sequences on a fixed domain, such as video frames, fMRI meshes, or traffic graphs, while dynamic domains require separate treatment.The section focuses on Ω(t) = Ω, although dynamic graphs arise when domains change over time.Source: §5.7 · pp. 93–94

Recurrent aggregation

• A symmetry-respecting encoder maps each input X(t) to a latent vector z(t), which an RNN recurrently aggregates into h(t) for online or sequence-level prediction.For video, a translation-invariant CNN produces a k-dimensional frame representation, and the final summary h(T) can classify the whole sequence.Source: §5.7 · pp. 94, 96

Geometric interpretation

• RNNs are geometric architectures because they implement an unusual symmetry over the temporal input vectors z(t), rather than merely serving as sequence summarizers.The recurrent update is shared across time and can be unrolled for backpropagation through time.Source: §5.7 · pp. 94, 96

Translation equivariance

• Standard RNNs are not generally translation-equivariant under sequence shifts because shifting inputs changes the boundary condition imposed by the initial state h(0).A one-step shift would require h(0) = R(z(1), h(0)), which does not generally hold.Source: §5.7 · p. 96
• Shift equivariance is recovered by left-padding and choosing h(0) as a fixed point satisfying h(0) = R(0, h(0)); contraction of γ(h) = R(0, h) guarantees a unique fixed point.Iterating the contraction until convergence is equivalent to padding the sequence with sufficiently many zero-vectors.Source: §5.7 · p. 97

Training dynamics

• Recurrent depth and parameter sharing cause vanishing or exploding gradients, motivating more sophisticated layers such as LSTMs to preserve learnability over long sequences.Sigmoidal activations can make gradients vanish, whereas ReLUs may cause exploding gradients because updates accumulate across every input step.Source: §5.7 · p. 98, Figure 20 · p. 99

Long Short-Term Memory networks

• LSTMs reduce vanishing-gradient effects by retaining a learned proportion of the previous cell state instead of fully overwriting it.A memory cell stores c(t), while learned gates control how information enters, is retained in, and exits the cell.Source: §5.8 · pp. 99–100
• Three sigmoid gates control LSTM updates: the input gate admits candidate features, the forget gate retains prior state, and the output gate produces h(t).The gates are vector-valued in [0, 1]^m and are applied through element-wise multiplication to decode the new cell state and summary vector.Source: §5.8 · p. 100

Time warping invariance of gated RNNs

• Gated RNNs are tightly coupled to time-warping invariance because their gates can learn the derivative of the time-warping transformation.The derived invariant update has the gated-RNN form, whereas SimpleRNNs fully overwrite h(t) and lack the required second term.Source: §5.8 · pp. 101–104
• The time-warping construction is limited to non-destructive warpings whose derivative is not too large, because otherwise intermediate signal changes may be unsampled.The discrete approximation requires dτ(t)/dt ≲ 1; time dilations remain allowed within the framework.Source: §5.8 · p. 103
• Chrono initialisation samples gating biases to produce forgetting times across a target dependency range and has been shown to improve long-range dependency modelling.The recommendation is bΓ ∼−log(U(Tl, Th)−1), where U is the uniform real distribution.Source: §5.8 · p. 104

Sequence-to-sequence learning with RNNs

• Seq2seq RNNs pass the encoder’s final summary vector h(T) to an autoregressive decoder, creating a fixed-capacity bottleneck for representing entire input sequences.Recurrent attention addresses this pressure by letting each output step query and weight representations from all input time steps.Source: §5.8 · pp. 104–105, Figure 21 · p. 105

Chemistry and Drug Design

• Graph neural networks support virtual drug screening, exemplified by discovering Halicin as a potent antibiotic, including against antibiotic-resistant bacterial strains.The model predicted whether candidate molecules inhibited growth in Escherichia coli.Source: §6 · p. 107

Drug Repositioning

• Drug repositioning models biological interactions as graphs, enabling side-effect prediction for drug combinations and investigation of therapies for COVID-19.Zitnik et al. (2018) formulated polypharmacy as edge prediction in a drug-drug interaction graph, while Gysi et al. (2020) studied COVID-19 applications.Source: §6 · pp. 107–108

Protein biology

• Protein applications span sequence-to-structure-to-function prediction, with contact-graph GNNs improving function prediction and MaSIF modeling 3D surfaces for interaction and binder design.MaSIF used mesh convolutions on local geodesic patches and addressed interface prediction, ligand classification, docking, and de novo protein design.Source: §6 · pp. 108–109

Recommender Systems and Social Networks

• Graph representation learning powers large-scale recommendation and social-network analysis, including Pinterest’s PinSage, production recommenders, and Fabula AI’s misinformation detection.PinSage scaled to graphs with millions of nodes and billions of edges, while Fabula modeled information spread and identified echo-chamber clustering.Source: §6 · pp. 109–110

Traffic forecasting

• GNNs model road networks for ETA prediction, with DeepMind’s predictor deployed in Google Maps and Baidu Maps serving travel-time predictions through the spatio-temporal ConSTGAT model.Road intersections are represented as nodes and road segments as edges with features such as length and historical speeds.Source: §6 · p. 110

Object recognition

• CNNs drove major advances in grid-based visual recognition, with AlexNet sweeping ILSVRC 2012, later CNN architectures surpassing human-level ImageNet performance, and translation-equivariant models supporting detection.The section also identifies R-CNN models and SegNet as influential approaches for detection and semantic segmentation.Source: §6 · p. 111

Game playing

• CNN-backed reinforcement learning enabled AlphaGo to defeat Lee Sedol, while successive systems removed human or game-specific biases and retained CNNs as their input-representation backbone.AlphaGo Zero learned through self-play, AlphaZero extended the approach to Chess and Shogi, and MuZero learned game rules on the fly.Source: §6 · p. 112

Text and speech synthesis

• Text and speech synthesis apply geometric deep learning to one-dimensional grids for tasks including text-to-speech, completion, and translation, increasingly using CNNs and Transformers instead of RNNs.Earlier systems commonly used recurrent architectures such as seq2seq and recurrent attention.Source: §6 · p. 113

Sequential architectures

• Dilated convolutions expand receptive fields exponentially with equivalent parameters, rival RNNs in translation and protein interaction, and reduce computation through parallelism.They underpin WaveNet’s human-like raw-waveform speech and piano generation, while WaveRNN enabled industrial-scale and on-device deployment.Source: §6 · p. 113
• Transformers surpassed recurrent and convolutional architectures in machine translation, while BERT and GPT-3 extended attention-based learning to broad language applications and coherent few-shot text generation.GPT-3 scaled language modeling to 175 billion parameters trained on web-scale text.Source: §6 · p. 114

Healthcare Applications

• Healthcare applications use grid, surface, brain-network, patient-network, and EHR representations for diagnosis, prediction, and characterization of anatomical and clinical structure.Examples include improved pulmonary-nodule detection with roto-translation convolutions, joint geometric-functional brain analysis, graph-based neurological diagnosis, and promising EHR graph learning.Source: §6 · pp. 114–115

Particle physics and astrophysics

• In particle physics, graph-based models represent unordered particles and their interactions naturally, supporting jet classification and architectures incorporating physical symmetries, mechanics, symbolic reasoning, and improved generalisation.These methods also offer greater interpretability for domain experts.Source: §6 · p. 116
• Geometric neural networks improved astrophysical-neutrino detection in IceCube by separating source events from background, while equivariant CNNs support radio-galaxy and cosmic-microwave-background analysis.The relevant signals are naturally represented on irregular detector geometries or spheres.Source: §6 · p. 117

Virtual and Augmented Reality

• In virtual and augmented reality, mesh networks address deformable 3D shape correspondence and synthesis, including mobile, real-time reconstruction of articulated body avatars from video.A hybrid CNN encoder and geometric decoder enabled realistic articulated-hand avatars faster than real time on a mobile phone.Source: §6 · pp. 117–118

Symmetry in Mathematics and Physics

• Group theory became the modern mathematical language of symmetry through Galois, Lie, and Klein, while Cartan later integrated Riemannian geometry into Klein’s framework.Noether linked differentiable symmetries of physical actions to conservation laws, and gauge invariance helped unify electromagnetism and the weak and strong forces in the Standard Model (Noether, 1918; Maxwell, 1865; Yang and Mills, 1954).Source: §7 · pp. 118–119

Early Use of Symmetry in Machine Learning

• Early machine-learning symmetry work developed equivariant feature detectors, exposed single-layer perceptron limits, and led through Neocognitron shift invariance to convolutional neural networks.Representation-theoretic treatments of invariant and equivariant networks appeared with Wood and Shawe-Taylor (1996), followed by later work including Makadia et al. (2007), Esteves et al. (2020), and Cohen and Welling (2016).Source: §7 · p. 120

Graph Neural Networks

• The first GNNs proposed by Gori et al. (2005) and Scarselli et al. (2008) used recurrent mechanisms, while Li et al. (2015) introduced GGNNs with gating and backpropagation through time.Source: §7 · p. 121

Computational chemistry

• Computational chemistry independently drove graph learning for molecular property prediction, with molecular graph networks anticipating contemporary edge-conditioned weights and global pooling.This line progressed from ChemNet and early molecular graph work to fingerprinting, quantum-chemical prediction, and successful virtual screening of antibiotic drugs.Source: §7 · p. 121

Node embeddings

• Graph representation learning also developed through random-walk node embeddings, supervised extensions, and later contrastive objectives, with untrained GNNs sometimes matching DeepWalk when node features are available.Representative methods include DeepWalk (Perozzi et al., 2014), node2vec (Grover and Leskovec, 2016), LINE (Tang et al., 2015), Planetoid (Yang et al., 2016), DGI, GRACE, BERT-like objectives, and BGRL.Source: §7 · pp. 121–122

Probabilistic graphical models and the Weisfeiler-Lehman formalism

• Analyses by Morris et al. (2019) and Xu et al. (2018) conclude that GNNs in the outlined three flavours cannot exceed WL-test power, while GIN is maximally expressive.For real-valued node features, injective aggregation requires multiple aggregators, specifically equal to the receiver node’s degree; moments are one example.Source: §7 · p. 124

Signal processing and harmonic analysis

• Harmonic-analysis foundations for CNNs include M-theory, steerable pyramids, and scattering transforms, which connect symmetry handling, multiscale wavelets, deformation stability, and depth.Scattering transforms replace trainable filters with multiscale wavelet decompositions and provide a framework for understanding CNNs.Source: §7 · p. 126

Signal Processing on Graph and Meshes

• Graph signal processing generalized Fourier analysis through graph adjacency and Laplacian eigenvectors, enabling spectral GNNs that helped reignite machine learning on graphs.Graph convolutional networks using spectral filters by Defferrard et al. (2016) and Kipf and Welling (2016a) became among the field’s most cited models.Source: §7 · p. 126

Computer Graphics and Geometry Processing

• Geometry-processing research established intrinsic metrics, symmetries, geodesic CNNs, MoNet, and GATs, while point-cloud models introduced dynamic, task-dependent latent graph construction.DGCNN updates nearest-neighbour graphs between layers; GAT generalizes MoNet’s attention mechanism by incorporating node features.Source: §7 · p. 127

Algorithmic reasoning

• Algorithmic reasoning designs neural architectures to preserve algorithmic invariants, spanning neural computers, primitive-algorithm models, algorithmically aligned GNNs, and simulation-oriented systems.General-purpose neural computers are often harder to optimize and are typically outperformed by simpler relational reasoners, while GNN alignment has extended to iterative and linearithmic algorithms, data structures, persistent memory, and implicit planning.Source: §7 · pp. 128–129
• Extrapolation remains difficult because algorithmic invariants must hold beyond the training distribution; proposed remedies constrain GNN modules toward linear targets or use environment-invariant graph representations.Xu et al. (2020b) give a geometric argument for rectifier-based extrapolating GNNs, while Bevilacqua et al. (2021) frame extrapolation through causal reasoning.Source: §7 · pp. 129–130

Geometric Deep Learning

• Geometric Deep Learning was named in 2015, popularized in 2017, and framed through “4G” before adding Geodesics for manifold metric invariants and intrinsic symmetries.The authors regard the field’s emergence as at least partially fulfilled, while noting that “gauges” is technically a stretch because the underlying structures are manifolds and bundles.Source: §7 · p. 130