PaperlayerView exact revision on arXiv

Source-linked AI summary

CosineGate: Semantic Dynamic Routing via Cosine Incompatibility in Residual Networks

Yogeswar Reddy Thota

arXiv:2512.22206v1cs.NEcs.AIcs.LG

TL;DR

Residual networks compute every residual block for every input, even when identity mappings suffice, and prior routing lacks a geometrically grounded criterion for deciding when computation is necessary. CosineGate uses CIR-based differentiable gating with progressive compute control. On CIFAR-10, its configurations span accuracy and efficiency trade-offs, including results that match or exceed ResNet-20 while reducing computation.

  • Problem

    Residual networks execute every block regardless of input redundancy, while a geometrically grounded criterion for deciding when residual computation is necessary is lacking.

    Source: §1.1 · p. 1, §1.2 · p. 2

  • Method

    CosineGate uses CIR, a geometric measure of semantic novelty between identity and residual representations, as a self-supervised signal for dynamic residual routing.

    Source: §1.3 · p. 2, §5 · p. 15

  • Results

    93.2% peak accuracy for the conservative configuration exceeds ResNet-20 (91.3%), while aggressive and balanced configurations yield up to 24–28% FLOPs savings with stable convergence.

    Source: §5 · p. 15

  • Takeaways & Limitations

    CIR provides a scale-invariant geometric signal that supports efficient conditional computation while retaining interpretability and robustness.

    Source: §5 · p. 15

    The gate design assumes γ < 0 to bias skipping by default, reflecting the observed redundancy of many residual blocks for many inputs.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

Modern deep residual networks perform substantial redundant computation by evaluating all residual blocks for every input, even when identity mappings suffice. We introduce CosineGate, an end-to-end differentiable architecture for dynamic routing in residual networks that uses cosine incompatibility between identity and residual feature representations as a self-supervised skip signal. CosineGate measures semantic redundancy through the Cosine Incompatibility Ratio (CIR), defined as 1 - cos(x, F(x)), and uses Gumbel-Softmax relaxation to enable per-sample, per-block gating during training. A progressive FLOPs regularization term controls average compute usage without destabilizing optimization. On CIFAR-10, CosineGate spans the accuracy-efficiency Pareto frontier: an aggressive configuration achieves 89.9 percent accuracy with 24.1 percent FLOPs savings, a balanced configuration achieves 91.3 percent accuracy with 28.5 percent savings at epoch 160, and a conservative configuration reaches a peak of 93.2 percent accuracy with minimal compute reduction. These results match or exceed ResNet-20 (91.3 percent) while reducing computation, without auxiliary supervision, distillation, or task-specific heuristics. Our results demonstrate that simple geometric measures of feature incompatibility provide a principled and effective signal for dynamic residual routing.

1 Introduction

The introduction frames fixed residual computation as a source of redundant work and asks whether a representation-level signal can support adaptive, differentiable routing. It proposes CIR-based gating and combines it with trainable gates and global compute control.

1.1 Motivation: Redundancy and Directionality in Residual Computation

  • Standard ResNets compute every residual block for every input, even when the residual adds little beyond identity, motivating adaptive routing for constrained devices.The introduction poses the central question of whether a representation-level signal can suppress such work during training and inference while preserving differentiability and accuracy.Source: §1.1 · pp. 1–2

1.2 From Static Pruning to Dynamic Routing

  • Prior dynamic methods adjust computation per input, but lack a geometrically grounded, representation-aware criterion for deciding when residual computation is necessary.Static pruning instead fixes the computation graph, while mixture-of-experts approaches can add routing overhead and architectural complexity.Source: §1.2 · p. 2

1.3 Geometric Redundancy and Cosine Incompatibility

  • CIR measures novelty as 1 − cosine similarity between identity and residual representations; low values indicate redundancy, while high values indicate complementary refinement.It is presented as self-supervised, per-sample and per-block, and requiring no auxiliary networks or supervision.Source: §1.3 · p. 2

1.4 Biological and Neuromorphic Motivation

  • Directional suppression and predictive coding motivate treating cosine incompatibility as novelty, with potential relevance to sparse, event-driven edge and neuromorphic computation.The introduction describes cosine similarity as a proxy for directional agreement and incompatibility as novelty-driven activation.Source: §1.4 · p. 3

1.5 Problem Formulation

  • CosineGate combines CIR with a lightweight controller, Gumbel-Softmax gates, and progressive FLOPs regularization to satisfy semantic grounding, differentiability, and global efficiency control.The introduction identifies classification accuracy and consistency regularization as additional components of its training objective.Source: §1.5 · p. 3, §1.6 · p. 3

1.6 Contributions

  • The contributions introduce CosineGate and CIR as a geometric residual-routing signal, with an end-to-end objective combining accuracy, consistency regularization, and progressive FLOPs control.The paper also reports that the approach spans the CIFAR-10 accuracy–efficiency frontier, matching or exceeding ResNet-20 accuracy while reducing FLOPs by up to 28.5%.Source: §1.6 · p. 3
  • CosineGate spans the CIFAR-10 accuracy–efficiency frontier, matching or exceeding ResNet-20 accuracy while reducing FLOPs by up to 28.5%.The introduction presents this result alongside a training objective combining classification accuracy, consistency regularization, and progressive FLOPs control.Source: §1.6 · p. 3

1.7 Paper Organization

  • The paper proceeds from related work to methodology, then experimental analysis, and finally conclusions and future directions.Those future directions include neuromorphic and TinyML implications.Source: §1.7 · p. 4

2 Background and Related Work

Prior work includes static pruning, dynamic residual skipping, conditional computation, feature-similarity routing, and biologically inspired approaches. The section positions CosineGate as combining geometric grounding, local residual-block gating, differentiable training, and global compute control.

2.1 Static Pruning and Model Compression

  • Static pruning can compress models, but its input-agnostic computation graph and iterative prune–retrain cycles limit adaptive inference and can destabilize optimization.Han et al. demonstrated magnitude-based pruning followed by retraining with minimal accuracy loss.Source: §2.1 · p. 4

2.2 Dynamic Residual Execution and Block Skipping

  • Dynamic block-skipping methods reduce computation, but their learned or shallow gating signals often lack semantic grounding and may overfit or collapse without careful regularization.CosineGate instead grounds gates in cosine incompatibility between identity and residual paths.Source: §2.2 · p. 4

2.3 Conditional Computation and Mixture-of-Experts

  • Mixture-of-experts routes inputs to selected experts, but selection, load balancing, and communication add overhead, and these systems often target transformers and distributed settings.CosineGate operates within standard residual blocks and computes its gate locally using dot products.Source: §2.3 · pp. 4–5

2.4 Attention-Based and Feature-Similarity Routing

  • Feature-similarity methods often support analysis or offline pruning, while CosineGate uses cosine similarity for per-sample routing with linear computational complexity.The related-work section contrasts this with attention mechanisms that typically incur quadratic channel complexity and lack a direct geometric interpretation of redundancy.Source: §2.4 · p. 5

2.5 Cosine Similarity in Neural Gating

  • Cosine similarity has been used for recurrent gating and feature agreement, but the section identifies cosine incompatibility as a new real-time residual-routing signal.Oguzie et al. [16] used cosine-gated LSTMs to modulate recurrent updates based on consecutive hidden-state similarity.Source: §2.5 · p. 5

2.6 Biological and Neuromorphic Inspirations

  • Biological suppression and predictive coding motivate suppressing expected signals while preserving novel ones; CosineGate is described as compatible with event-driven computation.The section connects neuromorphic platforms such as Loihi with sparse, event-driven energy reduction.Source: §2.6 · p. 5

2.7 Positioning and Novelty

  • CosineGate combines a CIR redundancy signal, lightweight local residual gating, Gumbel-Softmax differentiability, and progressive FLOPs regularization.The authors position the combination as supporting accurate, efficient, interpretable networks for edge and neuromorphic platforms.Source: §2.7 · p. 6

3 CosineGate Methodology

The methodology section introduces CosineGate by formalizing residual routing, defining CIR as its geometric skip signal, and outlining a gated residual block with controller augmentation. It defers training objectives and optimization details to Section 3.2.

3 CosineGate Methodology

  • The section lays out the architecture in order: residual routing problem, CIR skip signal, then gated residual block with controller augmentation.Training objectives and optimization details are deferred to Section 3.2.Source: §3 · p. 6
  • CIR is introduced as the core geometric signal for deciding whether a residual block should be routed or skipped.Source: §3 · p. 6
  • The gated residual block includes controller augmentation, while the section leaves its training and optimization details to a later section.Source: §3 · p. 6

3.1 Problem Formulation and Notation

The section frames residual routing as selectively executing residual transformations when they contribute meaningful information beyond the identity path. It asks how to determine redundancy in a self-supervised, input-adaptive way.

3.1 Problem Formulation and Notation

  • A standard residual block applies a residual transformation consisting of convolution, normalization, and nonlinearity.Source: §3.1 · p. 6
  • Standard residual networks execute each residual transformation for every input, even when its contribution may not be meaningful.Source: §3.1 · p. 6
  • The formulation seeks a learnable, self-supervised, input-adaptive gate that executes residual computation only when it contributes meaningful information.Source: §3.1 · p. 6

3.2 Cosine Incompatibility Ratio (CIR)

CosineGate uses the Cosine Incompatibility Ratio (CIR) as a continuous, self-supervised measure of residual novelty relative to the identity path. The measure informs routing based on directional alignment.

3.2 Cosine Incompatibility Ratio (CIR)

  • CIR ranges from 0 to 2: values near 0 indicate an aligned, redundant residual, while values near 2 indicate an orthogonal or opposing, novel refinement.Source: §3.2 · pp. 6–7
  • CIR provides a continuous, self-supervised signal quantifying how much new information a residual computation contributes beyond identity mapping.Source: §3.2 · p. 7
  • CIR is defined as 1 − cos(θ), where θ is the angle between the identity path and residual update.Source: §3.2 · p. 6, Figure 1 · p. 7

3.3 Geometric Interpretation

The geometric interpretation treats residual updates as directions in feature space: alignment with the identity path suggests redundancy, while orthogonality indicates new representational directions.

3.3 Geometric Interpretation

  • When a residual is collinear with the input, it rescales the existing representation without changing its direction, yielding negligible semantic benefit.Source: §3.3 · p. 7
  • An orthogonal residual introduces new representational directions and expands the network's expressive capacity.Source: §3.3 · p. 7
  • CIR captures directional novelty and, unlike magnitude-based criteria, is invariant to feature scaling and robust to normalization.Source: §3.3 · p. 7
  • The interpretation parallels predictive coding and neural inhibition, where redundant aligned signals are suppressed and orthogonal deviations trigger further processing.Source: §3.3 · p. 7

3.4 Gated Residual Block

The gated residual block uses CIR to determine how much of the residual update to apply. Its gate is sigmoid-based and initialized with a bias toward skipping.

3.4 Gated Residual Block

  • The gated residual block defines its gate value as a function of CIR.Source: §3.4 · p. 7
  • The gate logit is initialized as a scaled CIR, with γ < 0 biasing the model toward skipping by default.Source: §3.4 · pp. 7–8
  • Aligned residuals are treated as redundant and skippable, whereas orthogonal residuals add representational directions and should be computed.Source: Figure 2 · p. 8

3.5 Controller-Augmented Gating

CosineGate supplements its geometric incompatibility signal with a lightweight learned controller, allowing routing to adapt when task-specific context calls for computation.

3.5 Controller-Augmented Gating

  • A lightweight controller augments CIR to enable input-dependent adaptation in the routing signal.The paper describes CIR itself as task-agnostic.Source: §3.5 · p. 8
  • The controller contributes a learned adjustment term, c(xi), to the gate formulation.Source: §3.5 · p. 8
  • Global average pooling followed by a two-layer MLP adds less than 1% of parameters and can override geometric decisions when task context requires computation.Source: §3.5 · p. 8

3.6 Training Objective and Optimization

CosineGate learns differentiable routing, constrains compute at the network level, aligns skipped-path representations, and stages training to stabilize routing.

3.6.1 Differentiable Binary Routing via Gumbel-Softmax

  • Gumbel-Softmax relaxes binary skip-or-compute gates for gradient-based training, with deterministic thresholding at inference.During training, the relaxed gate is smooth; as temperature approaches zero, its distribution concentrates around hard decisions.Source: §3.6.1 · pp. 8–9, Figure 5 · p. 11

3.6.2 Global FLOPs Constraint via Progressive Regularization

  • A global FLOPs penalty acts on mean gate activation, approximating expected compute, and influences routing indirectly through backpropagation.The target sets the desired FLOPs ratio, and progressive pressure avoids directly penalizing individual gates.Source: §3.6.2 · p. 9, Figure 3 · p. 10

3.6.3 Consistency Regularization for Skipped Paths

  • Consistency regularization aligns gated outputs with full residual outputs, helping prevent semantic drift without a teacher model.Source: §3.6.3 · p. 10

3.6.4 Full Training Objective

  • The full objective combines cross-entropy for task performance, consistency loss for representational alignment, and FLOPs loss for computational efficiency.Source: §3.6.4 · p. 10

3.6.5 Training Dynamics and Stability

  • Training progresses from exploration to constraint enforcement and convergence, ending with stable routing while accuracy continues to improve.Each residual block has an independent gate, enabling adaptive FLOPs allocation across network depth.Source: §3.6.5 · p. 11, Figure 4 · p. 11
  • The progressive design avoids mode collapse and supports stable convergence without auxiliary supervision or reinforcement learning.Source: §3.6.5 · p. 11

4 Results and Experimental Analysis

The experiments evaluate CosineGate on CIFAR-10 and MNIST, reporting operating points, training dynamics, and efficiency comparisons.

4.1 Experimental Setup and Evaluation Protocol

  • CIFAR-10 experiments use a ResNet-20-style topology with a comparable ∼0.28M parameter budget, 160-epoch SGD training, and deterministic inference gates.Metrics include epoch-160 and peak Top-1 accuracy, FLOPs ratio approximated by mean gate activation, and skip percentage.Source: §4.1 · pp. 11–12

4.2 Primary Results on CIFAR-10

  • Balanced matches the ResNet-20 baseline at epoch 160 while reducing expected compute by 28.5%; Conservative reaches 93.2% peak accuracy and exceeds the baseline with modest efficiency gains.Aggressive shows graceful degradation under stronger compute pressure.Source: Table 2 · p. 13, §4.2 · p. 13

4.3 Training Dynamics and Efficiency Curves

  • Across configurations, training typically moves from exploration through constraint enforcement to convergence, while stronger consistency is correlated with higher peak accuracy, notably in Conservative.The supplied figure captions identify Balanced accuracy curves and FLOPs utilization and skip rates across training.Source: §4.3 · pp. 13–14, Figure 6 · p. 13, Figure 7 · p. 13

4.4 MNIST Validation (Extreme Redundancy)

  • On MNIST, CosineGate achieves 99.5% accuracy with 37% FLOPs savings within 10 epochs.Source: §4.4 · p. 14

4.5 Context Table

  • The context table presents representative CIFAR-10 comparisons of CosineGate operating points and their geometrically grounded skip signal.Source: §4.5 · p. 14, Table 3 · p. 15

5 Conclusion and Future Work

CosineGate spans an efficiency–accuracy Pareto frontier on CIFAR-10 and MNIST, with its conservative configuration surpassing ResNet-20 on peak CIFAR-10 accuracy. The conclusion connects directional incompatibility to biological novelty detection, edge deployment, and future tests of scale and generalization.

5 Conclusion and Future Work

  • 93.2% peak accuracy surpasses ResNet-20 (91.3%), while aggressive and balanced configurations yield up to 24–28% FLOPs savings with stable convergence.Results are reported across experiments on CIFAR-10 and MNIST; deterministic inference-time routing adds negligible overhead.Source: §5 · p. 15
  • CosineGate frames geometric directionality as an architecture-agnostic computation-utility signal, with CIR invariant to feature scaling and normalization.The conclusion presents this geometric incompatibility as a foundation for dynamic computation that unifies efficiency, interpretability, and biological plausibility.Source: §5 · p. 15, §5.3 · p. 16

5.1 Neuromorphic Perspective and Biological Grounding

  • CosineGate suppresses computation for residual features aligned with identity representations and enables it when residuals introduce orthogonal directions.The conclusion relates this directional-novelty mechanism to biological novelty detection; future work proposes mapping CIR gating to lateral inhibition and studying CIR distributions against neural sparsity.Source: §5.1 · p. 15

5.2 Edge Intelligence and TinyML Implications

  • Lightweight dot products, simple nonlinearities, and hard binary inference decisions give CosineGate negligible runtime overhead and static computation graphs compatible with embedded compilers and low-power accelerators.Microcontroller-class deployment, real-world energy measurements, and partially offline-compiled hybrid execution are proposed as future directions.Source: §5.2 · p. 16

5.3 Scaling, Generalization, and Open Directions

  • CIFAR-100, ImageNet, ConvNeXt, and Vision Transformers are proposed tests of scaling and generalization beyond the reported settings.The conclusion also identifies NAS integration as a possibility and convergence and stability analysis of CIR-gated residual dynamics as an open challenge.Source: §5.3 · p. 16
Loading 2512.22206v1…