Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks

1 Main

• The framework addresses black-box neural networks, scaling costs, data inefficiency, architecture trial-and-error, and loss of plasticity during sequential training [1] [2] [5].It motivates principled analysis of internal structure rather than relying predominantly on empirical scaling and ever-larger models.Source: §1 · p. 2
• Local fibrations, opfibrations, and coverings generalize rigid global symmetries used in geometric deep learning while capturing diverse neural-network structures [9] [12] [13] [18] [23].These symmetries arise in computational-graph input and output trees and were originally introduced through category theory and graph theory [23].Source: §1 · p. 2
• Covering symmetries emerge through synchronized SGD updates, enabling model compression and state-of-the-art continual learning after carefully breaking those symmetries [5].The framework is validated in MLPs, CNNs, recurrent networks including LSTMs, and Transformers under supervised and reinforcement learning.Source: §1 · p. 2

An hierarchy of symmetries in computational graphs

• Fibration classes group nodes with isomorphic input trees, allowing compression into a base graph that preserves forward computation and lifting back without changing functional output.Balanced coloring identifies the fiber partition by iteratively grouping nodes receiving the same set of input colors.Source: An hierarchy of symmetries in… · p. 4
• The symmetry hierarchy progresses from fibrations and opfibrations to coverings and automorphisms, with increasing strictness producing more color classes, fewer equivalent nodes, and less compression.Automorphisms are the most restrictive global symmetry and were never observed in the trained networks described here.Source: An hierarchy of symmetries in… · p. 4

Emergence of symmetries during learning

• During SGD, fibers, opfibers, and covers emerge across MLPs, CNNs, RNNs, and Transformers, with formation propagating forward for fibers and backward for opfibers.In an LSTM trained on Beam Rider, approximate covers initially grow but later break because reinforcement-learning exploration incentives favor symmetry breaking.Source: §2.1 · p. 5, Computational graphs and synchronization · p. 5, Covers and the emergence of symmetry · p. 7

Fibrations synchronize activity, opfibrations synchronize error

• Fibers synchronize activity and flavored opfibers synchronize backpropagated errors for any input, making corresponding weight updates identical under gradient descent.Approximate synchronization is bounded by εfib for quasi-fibers, while quasi-opfibers use an analogous threshold εop.Source: Fibrations synchronize activity,… · pp. 5–6, Synchronized learning · p. 7

Covers and the emergence of symmetry

• Covering partitions are preserved under gradient descent, can merge but not split, and therefore form stable attractors of stochastic gradient descent.The covering theorem identifies the weights of covers as an invariant set under GD, yielding stable attractors under stochastic GD for sufficiently large learning rates.Source: Covers and the emergence of symmetry · p. 7

Synchronization captures regularity in the data

• Class-conditional synchronization clusters coarsen fiber partitions, indicating that fibers capture hierarchical regularity shared across training data and increasing across network layers.The compression rule preserves exact forward computation and loss for exact symmetries, while quasi-symmetry preservation is approximate and bounded by precision and a data-dependent coefficient.Source: Compression rule preserves activity… · p. 8, Synchronization captures regularity in… · p. 9

Fibration and covering compression versus performance

• 17% of the original MLP size is reached with no change in loss or accuracy, while covering compression outperforms fibration compression and conventional pruning.For the Transformer, fibration compression reaches 70% of its original size at εLoss = 0 and 18% at εLoss = 0.45 without affecting translation performance.Source: Fibration and covering compression… · p. 10, Figure 3 · p. 11

Fibration symmetry breaking for continual learning

• 95% accuracy by task 5,000, versus 90% previously, is achieved by FSB, which outperforms continual backpropagation after task 1,500.FSB first compresses covers, then sprouts randomly initialized nodes to restore degrees of freedom without changing the learned base.Source: Figure 3 · p. 11, Fibration symmetry breaking for… · p. 12

• Stochastic gradient descent stabilizes covering symmetries, producing increasingly coarse structures that explain node collapse, synchronized activity, and simpler models despite over-parameterization [24].The framework attributes this behavior to a hierarchical loss landscape and noise-induced stabilization.Source: §3 · p. 12
• 17-18% model size is achieved by merging functionally redundant nodes, preserving the model’s exact function while complementing quantization and low-rank decomposition.The framework interprets weight-tying and symmetry-breaking techniques as graph-symmetry operations, including dropout, residual connections, and redundancy-reduction methods.Source: §3 · p. 13
• 50% reduction in the performance gap on sequential ImageNet benchmarks versus continuous backpropagation supports Fibration Symmetry Breaking as a plausible lifelong-learning foundation.FSB compresses emergent covers to preserve past knowledge and injects randomized naive nodes to restore optimization degrees of freedom without disrupting the learned base.Source: §3 · p. 13
• The framework identifies dynamic compression during training, optimal combinations with low-rank and quantization methods, and formal links between symmetry quotients and data hierarchy as open directions.The current approach performs static compression after training, leaving these engineering and theoretical relationships unresolved.Source: §3 · p. 13
• Emergent symmetries transform opaque networks into interpretable colored graphs whose learned regularities support theory-driven performance, generalization, and interpretability.The conclusion frames learning as structure formation rather than parameter tuning or brute-force scaling.Source: §3 · p. 13

4.1 Global and local symmetries in graphs: coverings, fibrations, and opfibrations

• The symmetry definitions extend from binary graphs to flavored connections by requiring tree isomorphisms to preserve connection flavors.When all connections share one flavor, the original binary definitions are recovered.Source: §4.1 · pp. 14, 17
• A fibration preserves input trees, an opfibration preserves output trees, and a covering preserves both through surjective graph homomorphisms.Each symmetry induces a corresponding fiber, opfiber, or cover partition.Source: §4.1 · p. 15
• Fiber, opfiber, cover, and orbit partitions are balanced colorings with refinement relations, while orbit membership additionally requires a graph automorphism between nodes.For undirected graphs, fiber, opfiber, and cover partitions coincide; dynamic cluster synchronization coarsens the fiber partition but not the opfiber partition.Source: §4.1 · p. 16
• Fibrations, opfibrations, and coverings generalize automorphisms by preserving input and/or output trees locally rather than enforcing global adjacency preservation.These symmetries form categories rather than groups, while color-preserving tree isomorphisms form groupoids.Source: §4.1 · pp. 16–17
• Nodes can share a covering symmetry even when no automorphism exchanges them, because covering preservation may hold within their local trees while adjacency elsewhere prevents a global permutation.In the example, nodes 9 and 10 form a cover, but restoring all affected connections would require an unavailable global permutation.Source: §4.1 · p. 17

4.2 Lifting property by fibration symmetry: Construction of fibration base

• Fibration bases are constructed by partitioning nodes into fibers, replacing node identifiers in in-neighborhoods with fiber colors, and connecting base nodes accordingly.A base node may receive multiple connections from another base node, representing summed single-connection inputs for weighted networks.Source: §4.2 · p. 18
• Opfiber bases use out-neighborhood lifting, whereas cover bases can use either lifting direction; this work selects fibration lifting because it preserves forward computation.Fibration and opfibration compression can otherwise produce different bases, so the compression direction must be chosen explicitly.Source: §4.2 · p. 18

4.3 Fibration lifting operation for weighted network

• Fibration lifting is the nonunique inverse of compression because multiple full weighted graphs can map to the same base.Different configurations can share the same minimal base, including the illustrated groups G1/G2 and G3/G4.Source: §4.3 · pp. 18–19
• Networks with the same minimal base produce identical outputs for every input and identical dataset losses, creating a degeneracy of the loss function.This applies to the example networks G1, G2, G3, and G4 and their bases.Source: §4.3 · p. 19

4.4 Fibration symmetry implies activity synchronization

• In weighted networks, two nodes share a fiber when their total incoming weight from every previous-layer color is equal, starting from distinct colors in the input layer.The equal-sum definition generalizes binary-graph fibers to continuous weights.Source: §4.4 · p. 20
• Fibration symmetry synchronizes the activities of fiber nodes for every network input, linking weight structure to network function through input-tree structure.The theorem follows recursively from each layer’s dependence on the preceding layer’s activities.Source: §4.4 · pp. 20–21

4.5 Input and output trees in weighted feedforward networks

• Fibration and opfibration symmetries impose weaker weight constraints than automorphisms, which require identical input and output weight vectors for permutable same-layer nodes.The distinction motivates defining fibers through input and output tree structure rather than full network-preserving permutations.Source: §4.5 · p. 21
• Input-tree isomorphism is defined recursively by aggregating connection weights over known fibers, yielding the equal-sum criterion for weighted fibers across layers.At each layer, summed weights reproduce the combined effects of nodes within a preceding fiber.Source: §4.5 · p. 21
• Flavored output-tree isomorphism defines opfibers by equal summed outgoing weights together with matching connection flavors, recursively across preceding layers.Flavors are assumed to depend only on the source node, and the criterion preserves their identity during aggregation.Source: §4.5 · pp. 21–22
• Flavoring weights with fibration colors couples forward and backward propagation, ensuring flavored opfiber nodes share σ′ across their full error-propagation trees.For linear networks, q = 1 and flavored opfibers reduce to regular opfibers, uncoupling forward and backward coloring.Source: §4.5 · p. 22

Activity synchronization

• Induction establishes activity synchronization in Eq. (15), first propagating the property from layer ℓ−1 to layer ℓ and then proving the base case at ℓ=1.Source: §4.6 · pp. 22–23
• For linear activation functions, observing synchronization between two nodes for all inputs implies that they belong to the same fiber.Source: §4.6 · p. 23

Error synchronization

• Induction also proves Eq. (17), propagating error synchronization backward from layer ℓ+1 to layer ℓ and verifying the base case at ℓ=N−1.Source: §4.6 · pp. 23–24

4.7 Quasi-synchronization of activity and error in quasi-symmetries

• Quasi-symmetries do not guarantee exact synchronization, but Lipschitz bounds constrain activity variation within quasi-fibers and error variation within quasi-opfibers.The bounds use quasi-fiber and quasi-opfiber thresholds and the activation-slope bound Kσ.Source: §4.7 · pp. 24–25

4.8 Equivalence between fibration and synchronization for linear activation

• For linear activations, synchronized activity implies fibration and synchronized errors imply opfibration, establishing a bidirectional structure–function relation.The equivalence relies on forward and backward propagation being analogous and decoupled in the linear case.Source: §4.8 · p. 25, §4.9 · p. 27

4.9 Flavored opfibration implies error synchronization

• In nonlinear networks, flavor-preserving opfibrations encode slope-dependent forward–backward coupling and imply error synchronization.The output-tree flavors inherit node colors from the forward fiber structure, and the opfiber partition is evaluated recursively by flavor-specific weight sums.Source: §4.9 · pp. 26–27

4.9.1 Alternative definitions to capture forward-backward coupling in nonlinear Feedforward Networks

• Alternative nonlinear definitions replace flavored structural conditions with hybrid synchronization requirements or recursive weight-based covers that couple fibers, opfibers, and activity.The hybrid alternatives may require synchronized slopes or explicit error synchronization, whereas the recursive definition uses forward fiber coloring followed by output-side weight constraints.Source: §4.9.1 · pp. 27–28
• The recursive cover alternative retains a coloring algorithm based on forward fiber partitions and output-starting cover partitions constrained by weights.The first two alternatives depend on activity and lack an obvious weights-only coloring algorithm; the recursive alternative is purely structural.Source: §4.9.1 · p. 28

4.9.2 Gradient Descent Rule - Eq. (4)

• Gradient descent updates each layer’s weights by subtracting the learning-rate-scaled loss gradient.For dataset inputs h(0)=x and predictions ŷ, the loss is L(ŷ,y), and the update uses learning rate α.Source: §4.9.2 · p. 28

4.9.3 Proof of Synchronized Learning - Eq. (5)

• The synchronized-learning proof considers nodes paired within the same fiber and opfiber and applies the synchronization relations to their propagated activity and errors.The setup uses nodes i,j in one fiber and m,n in one opfiber, as illustrated in Fig. 2a.Source: §4.9.3 · p. 29

4.10 Covering symmetries emerge during stochastic gradient descent

• A covering groups nodes that share both a fiber and a flavored opfiber, defining the network’s covering partition Ccov.The flavored opfiber accounts for connection dependence on the forward pass in nonlinear networks.Source: §4.10 · p. 29

4.11 Proof of Theorem 1: Cover Coarse-Graining Theorem

• Nodes in the same cover remain together under gradient descent, although distinct covers can merge into larger, coarser covers.This is the Cover Coarse-Graining Theorem’s central consequence and defines an invariant set of covers.Source: §4.10 · p. 29, §4.11.2 · p. 31

4.11.1 Opfiber, Fiber and Cover Stability Rules

• The opfiber, fiber, and cover stability rules respectively propagate synchronized structure backward, forward, and jointly across successive learning steps.Opfibers preserve opfibers toward the preceding layer, fibers preserve fibers toward the following layer, and their combination preserves covers.Source: §4.10 · p. 29, §4.11.1 · p. 30

4.11.2 Recursive proof of cover coarsening theorem

• The recursive proof applies fiber stability across all layers and opfiber stability from the output, yielding cover preservation at every layer.It establishes i ∼ cov,ℓ,t j ⇒ i ∼ cov,ℓ,t+1 j.Source: §4.11.2 · p. 31

4.11.3 Stability of covers

• Cover membership defines an invariant parameter set under the gradient-descent weight update.If network parameters satisfy a cover relation at time t, the same relation holds at time t+1.Source: §4.11.3 · p. 31
• Chen et al. [32] imply that invariant parameter sets of gradient descent are stable attractors of stochastic gradient descent under a sufficiently large learning constant.Transverse stochastic fluctuations are damped toward the symmetric set, while fluctuations along it vanish through synchronized learning.Source: §4.11.3 · pp. 31–32

4.12 Fibers coloring for sums, convolutions, gates and concatenations

• Fibration rules generalize to architectural motifs modeled as hypergraphs, including sums, convolutions, concatenations, and gates.Sums are handled by concatenating weights and inputs; concatenated vectors contribute their own colors.Source: §4.12 · p. 32
• Residual connections tend to break newly forming fibers because concatenated identity matrices are orthogonal, unless those fibers already exist in the input.Bias sums, weighted input sums, and residual sums are treated through concatenated weights and inputs.Source: §4.12 · p. 32
• Gate fibers require matching colors in both multiplicative factors, whereas attention preserves three interactions even when key and query fiber counts differ.The same coloring applies to LSTM gates and transformer key-query products, but their compression rules differ.Source: §4.12 · p. 32, Figure 6 · p. 33
• Convolutional fibration compression reduces channels from 3 to 2 and from 5 to 3, while pooling along time, pixels, or tokens leaves feature-channel colors unchanged.Convolution filters are treated as scalar weights after ignoring the non-channel dimensions.Source: §4.12 · p. 32, Figure 6 · p. 33
• Time, pixel, and token dimensions are not colored, so attention matrices do not affect colors; identifying symmetries in those dimensions remains future work.The attention matrix is square in the number of tokens and lies outside the current coloring scheme.Source: §4.12 · p. 33

4.13 Fibration compression rules

• Fibration compression derives base-network parameters that preserve node activity and forward computation, while analogous opfibration compression preserves backward error computation.For feedforward networks, weights with identical input colors are added and weights with identical output colors are averaged.Source: §4.13 · p. 33, §4.13.1 · p. 34

4.13.1 Fibration compression for a feedforward network

• Forward-and-backward compression into a covering base is possible only when all covers have the same cardinality |c′|, making the requirement strong and unexplored here.Compression can instead preserve either input trees or output trees.Source: §4.13.1 · p. 34

4.13.2 Fibration compression for a convolutional layer in an CNN

• The same fibration framework extends to CNN channels, treating each interchannel convolution kernel as the weight object used for compression.The MLP definitions of fibers and lifting are generalized to convolutional channels.Source: §4.13.2 · p. 34

4.13.3 Fibration compression for an Hypergraph

• For computational hypergraphs, nodes are equivalent when their interacting input nodes are equivalent, ensuring equal activity under multiplicative gating.This covers LSTM gate products and transformer key–query interactions.Source: §4.13.3 · p. 35
• LSTM gates share fiber distributions and therefore compress consistently across gates, whereas attention may reduce query and key nodes but must preserve the number of q ⊙ k interactions.The compressed LSTM dynamics are ẑ_c = x̂_c ŷ_c, while attention’s interaction layer cannot be reduced when query and key fibers differ.Source: §4.13.3 · p. 35

4.14 Datasets and tasks used in empirical tests

• Empirical tests use ten independent MNIST and ImageNet training runs, plus a PPO Atari Beam Rider agent with convolutional, linear, and LSTM components.The PPO setup uses 65 parallel environments, 30,720-transition batches, and a BPTT horizon of 2; transformer tests use a three-layer symmetric encoder–decoder with eight heads.Source: §4.14 · pp. 35–36, Table 1 · p. 36, Table 2 · p. 37

4.15 Refinement algorithm for balanced colorings in feedforward networks

• The feedforward refinement method converts approximate weight equalities into ε-tolerance distance constraints, clusters nodes agglomeratively, and treats resulting approximate symmetries as quasi-fibrations.The threshold satisfies ε ∈ (0, 2), and clustering defines the layerwise fibers.Source: §4.15 · pp. 38–39
• The refinement algorithm produces forward and backward colorings across all layers, with execution cost scaling linearly with each layer’s parameter size and the number of layers.The backward pass generates opfibration colorings, while recurrent networks require recursive iteration until convergence.Source: §4.15 · p. 39

4.16 Optimal quasi-cover thresholds for a given loss threshold

• The constrained optimization maximizes compression by minimizing the collapsed network’s parameter count P(ε⃗), whose dimensions decrease with threshold, subject to a fixed loss-variation threshold εLoss.The collapsed network is required to perform similarly to the original in terms of loss.Source: §4.16 · p. 39
• εLoss is defined where task performance A stabilizes despite continued loss reduction, and its value depends strongly on the network architecture and task.The relationship between A and L need not be exact, but performance is generally expected to improve as loss decreases before stabilizing.Source: §4.16 · pp. 39–40
• The method efficiently estimates compression-induced loss change using gradients G computed during training and the threshold-dependent weight update ΔW.The gradient does not depend on the compression thresholds, whereas ΔW does because the thresholds determine node coloring.Source: §4.16 · pp. 39–40

4.17 Redundant feature pruning is a special case of fibration compression

• Redundant-node pruning is a special case of covering symmetry: nodes with identical input and output weights form a trivial permutation symmetry within one layer.In the strict case sim = 1, equal weight vectors satisfy Win = Wjn for every n.Source: §4.17 · p. 40

4.18 Orthogonal regularization is a particular case of symmetry breaking

• Node collapse can signal emerging fibration symmetry through synchronization, while orthogonal regularization is an ad hoc strategy for preventing correlated node activity.Node collapse was reported in [24].Source: §4.18 · p. 40
• Orthogonal regularization prevents fiber formation because orthogonal output-node weights produce distinct clusters and therefore only trivial fibers.Its objective drives the weight-correlation matrix toward zeros on the diagonal and ones elsewhere.Source: §4.18 · pp. 40, 42

4.19 Continual learning

• Up to 88% initial accuracy deteriorated by the 2,000th task, showing that standard deep-learning methods lose plasticity in sequential ImageNet learning.This pattern was consistent across architectures, parameters, and optimizers.Source: §4.19 · p. 42
• Cover distributions show that continual backpropagation, and especially symmetry breaking, constrain cover sizes and maintain more covers than SGD across tasks and epochs.At initialization, all 512 nodes formed trivial fibers; larger covers emerged during training.Source: §4.19 · p. 42, Figure 8 · p. 43

4.20 Deep Continual Learning is a particular case of Fibration Symmetry Breaking

• In Deep Continual Learning, equal utility among nodes in a cover implies utility synchronization, so zero-utility nodes from the same cover are reset together.This operation breaks associated covers by reinitializing input weights and fixing output weights at zero.Source: §4.20 · p. 43
• Fibration symmetry breaking overcomes Deep Continual Learning because it targets all covers, whereas Deep Continual Learning resets only inactive, zero-utility covers.Deep Continual Learning reinitializes input weights while fixing output weights at zero, changing activity without immediately disrupting learned behavior.Source: §4.20 · pp. 43–44

4.21 Fibers are a refinement of clustering partitions

• Matching score=1 for small thresholds shows that fiber partitions perfectly match synchronization clusters under random inputs, while refinement score=1 validates fibers as a refinement of cluster intersections.The refinement relationship breaks down at larger cluster thresholds, as expected.Source: §4.21 · pp. 44–45

4.22 Hierarchical organization of the loss landscape under SGD dynamics

• Covers refine fibers, which refine synchronization clusters and class-conditional synchrony clusters, and merge into coarser partitions during training according to the Cover Coarse-Graining Theorem.Covers are located within intersections of class-conditional clusters and merge across transitions T1→2 and T2→3.Source: §4.22 · p. 45, Figure 11 · p. 47
• Once nodes i and j share a cover, the Cover Coarse-Graining Theorem makes their merger irreversible, followed by a second irreversible transition when node k joins them.The learning trajectory follows downhill paths through progressively coarser-grained symmetry quotients during SGD.Source: §4.22 · pp. 45–46
• The loss landscape forms a hierarchy in which configurations sharing a minimal base have the same loss, while different lifts of that base appear as multiple leaves and intermediate coarse-grainings.This hierarchy can produce disjoint local minima, flat valleys, or replicated loss structures in the full parameter space.Source: §4.22 · p. 46

4.23 Low-rank weight compression

• SVD compression factors W≈W^(2)W^(1) with k(d_out+d_in) parameters and is compact when k<d_outd_in/(d_out+d_in), while fibration compression can further shrink the factor matrices.The two procedures can be combined in either order, but SVD efficiency may be lost after fibration compression and additional collinear nodes may be found afterward.Source: §4.23 · pp. 46–47
• If a layer’s inputs lie in a linear subspace, SGD keeps the input weight matrix in that subspace, making the subspace an invariant stable attractor and explaining compressible low-rank weights.The gradient satisfies ∂L/∂W=ΔX^T, so rank-deficient representations can induce low-rank weight matrices.Source: §4.23 · p. 47
• SVD exposes geometric simplicity through rotations and scalings, whereas graph symmetries expose structural redundancies by partitioning structurally indistinguishable nodes.Thus, SVD symmetry is algebraic and distinct from the global-local hierarchy of graph symmetries.Source: §4.23 · p. 48

Funding

• The work received partial support from NIH grants R01CA247910 and R01EB028157 and Army Research Office grant WF911-NF-24-1-0031.The listed grants supported OMV, LCP, and HAM as specified in the passage.Source: Declarations · p. 49

Competing interests

• The authors declare no competing interests.Source: Declarations · p. 49

Data, materials, and code availability

• The code is available at Github/MakseLab/fibrations in dnns.Source: Declarations · p. 49

Author contributions

• HAM conceived the project, while OMV, HAM, and LCP designed experiments, contributed to mathematical proofs, and wrote the paper.Source: Declarations · p. 49
• All authors contributed to data analysis, and OMV and AH developed the analysis code.Source: Declarations · p. 49