PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Competition during Training For the Emergence of Specialized Modules

Baptiste Rossigneux, Karim Haroun

arXiv:2608.30978v1cs.LG

TL;DR

The paper addresses whether neural networks can develop meaningful, reusable modular organization rather than redundant or entangled representations. It introduces competition-based sparse routing with sharpness and balance objectives, and finds specialized modules that preserve accuracy, carry class information, and organize hierarchically. The approach also distinguishes sparse deployable specialists from dense co-adapted ones.

  • Problem

    Existing modular architectures do not guarantee meaningful functional roles, while benchmark accuracy does not distinguish reusable localized computations from distributed heuristics.

  • Method

    Sparse Competitive Modules induce usage-based modularity through input-dependent routing, per-sample sharpness, and population-level balance while leaving weights unconstrained.

  • Results

    Specialized modules emerge without module-level supervision, preserve near-baseline accuracy, carry class information, and form hierarchically refining semantic partitions.

  • Takeaways & Limitations

    Competition dynamics provide a loss-based mechanism for inducing functional modularity in standard architectures, including sparse modules that are individually deployable.

  • Takeaways & Limitations

    Experiments use a controlled setting and modularize only a small number of layers, so the paper does not claim full-network modular computation or complete circuit recovery.

Abstract

from arXiv · show

Modularity in deep neural networks has been proposed as a means of improving both interpretability and training by promoting disentangled representations and reducing redundancy. In this work, we study the emergence of modular structure through competition dynamics between groups of neurons during training. We introduce a method that (i) maintains near-baseline accuracy, (ii) induces usage-based modularity by sparsely routing inputs to neuron groups, and (iii) encourages specialization of these modules, such that their activations are correlated with input classes. We evaluate the proposed approach on ImageNet-100 and CIFAR-100 and show that with it, specialized modules emerge without module-level supervision. These modules capture a meaningful high-level structure in the data, with individual modules responding to semantic categories (e.g., dogs or vehicles). We also study the emergence of a hierarchical partition of sub-tasks depending on the number of modules. Our results suggest that competitive dynamics can serve as a simple mechanism for inducing functional modularity in standard architectures.

1 Introduction

The paper asks whether standard training can produce meaningful modular organization without sacrificing task performance. It introduces competition-based routing that yields specialized, hierarchically organized modules while preserving accuracy and enabling sparse deployment.

  • Architectural separation alone does not ensure meaningful module roles, because representations can remain redundant or entangled.
  • On ImageNet-100, sparse top-1 routing remains within roughly one percentage point of the vanilla baseline while classes with the same coarse label share modules approximately 3× more often than chance.
  • Sparse Competitive Modules induce usage-based modularity through per-sample sharpness and population-level balance, without module-level supervision.The approach is based on learned routing rather than weight-structure constraints.
  • The learned modules become class-informative and acquire substantially more class information than networks without modules or with clustered baselines.
  • Sparse competitive training is presented as necessary for sparse deployment, because dense training produces co-adapted specialists that rely on one another.
  • Increasing module count refines learned partitions hierarchically while preserving semantic coherence across module counts and layers.

Related Works

Prior work separates modularity for interpretability from conditional computation for efficiency, while sparse expert routing scales capacity. This paper instead uses information-maximization principles inside network layers to induce specialization rather than merely imposing structure.

  • Modularity research targets interpretable weakly interacting components, efficient conditional computation, or simplified descriptions based on shared connection patterns.
  • Mixture-of-Experts activate only a subset of parameters per input, increasing model capacity without increasing computation; vision variants can remain competitive with dense transformers while reducing inference cost.
  • SCM resembles information-maximization clustering because it combines confident assignments with population-level diversity, but uses these objectives to induce specialization inside a network layer.
  • Unlike circuit-discovery methods, the paper trains for emergent modular specialization as a possible basis for unsupervised, human-understandable circuits.
  • Sparse Competitive Modules route each input to one active module while balancing module usage, leaving weights unconstrained and using class labels only for evaluation.

3 Methodology

Sparse Competitive Modules induce usage-based modularity by selecting one module per input while balancing population-level usage, leaving weights unconstrained. The method uses activation-based routing, sharpness and balance objectives, and sparse top-1 training and inference to encourage specialization without class labels.

  • Sparse Competitive Modules: Sparse Competitive Modules select one module per routed layer for each input while preventing collapse through balanced module usage.The routing decision is unsupervised by class labels, which are used only through the standard classification loss.
  • Routing mechanism: Module energy determines routing, with hard masks selecting the highest-energy module while the weights remain unconstrained.The method partitions layer coordinates into modules, computes activation energy for each module, and induces modularity through input-dependent usage patterns.
  • Routing objective: The routing objective minimizes per-sample entropy and maximizes entropy of average usage, producing confident decisions while maintaining population-level diversity.A temperature parameter controls routing sharpness: smaller values approach winner-take-all decisions, whereas larger values provide smoother gradients.
  • Gaussian-ring capacity competition: Gaussian-ring masks replace equal-width hard blocks with soft, capacity-adaptive modules whose learnable widths share a fixed total budget.The circular arrangement avoids border effects and makes increasing one module’s capacity reduce capacity available to others.
  • Gaussian-ring capacity competition: The Gaussian ring does not substantially affect accuracy while consistently increasing class–module mutual information, motivating its use in subsequent experiments.Class–module specialization is quantified through mutual information between the class variable and selected module index.
  • Training and inference: During training and inference, hard top-1 routing activates only the selected module, while the auxiliary loss uses the differentiable pre-argmax energy profile.Classification gradients update only the selected module, and the routing loss can reshape energy patterns to modify assignments.

4 Experiments

Experiments show that Sparse Competitive Modules preserve near-baseline accuracy while producing class-informative, balanced, and hierarchically organized modules across datasets, module counts, layers, and architectures. Sparse competitive training is especially important when modules must remain functional under sparse inference.

  • Main results: SCM reaches 57.84±0.59% on ImageNet-100 CMC, matching the clustered baseline and remaining within roughly one percentage point of the vanilla model.The same qualitative accuracy pattern is observed on CIFAR-100.
  • Main results: SCM raises fc2 class-module mutual information to 0.649 ± 0.128, versus 0.036 ± 0.006 for vanilla and 0.049 ± 0.012 for clustered models.This corresponds to approximately 18× over vanilla and 13× over the clustered baseline.
  • Robustness: Across random ImageNet-100 label subsets, SCM preserves baseline-level accuracy while making module assignments more predictive of class identity.The specialization pattern is therefore not specific to the CMC label subset.
  • Module-count analysis: Gaussian-ring competition increases MI(Y;Mfc2) from 0.409 to 0.638 at K = 4, from 0.730 to 0.891 at K = 8, and from 1.150 to 1.297 at K = 16.Accuracy changes are small, so the ring is used to strengthen specialization rather than improve accuracy.
  • Sparse deployment: Sparse competitive training is required for modules to remain functional under sparse inference, because only the selected module receives the task gradient for each example.Dense training can produce class-structured routing scores, but top-1 routing applied only at evaluation does not reliably produce independently deployable modules.
  • Hierarchical specialization: SCM produces a coarse-to-fine module hierarchy, with pair retention of 28.32% for 4→8 and 27.15% for 8→16 versus random baselines of 13.85±0.86% and 6.40±0.89%.Weighted child purity is 69.0% and 72.0%, indicating finer specialists refine broader semantic specialists.

5 Conclusion

The paper presents competition-based training as a way to produce specialized neural modules while preserving the accuracy of a non-modularized network. Usage-based and capacity-based competition organize data at varying granularities without requiring monitored internal organization.

  • Sparse Competitive Modules produce meaningfully specialized modules through usage-based competition.The method uses information-theoretic metrics and does not require module-level supervision.
  • Capacity-based competition with Gaussian ring ownership learns module widths under a shared capacity budget.
  • The two competition mechanisms conserve non-modularized accuracy while grouping data at varying granularity as the module count changes.

6 Limitations and future works

The experiments are deliberately controlled and modularize only a small number of layers, limiting the scope of claims about larger models and full-network modular computation. Future work targets broader architectures, datasets, and model families.

  • The experiments use a controlled setting that makes module usage, specialization, and ablation easy to measure.
  • The paper identifies larger backbones, larger vision datasets, richer internal structures, language models, and vision-language models as next-step settings.
  • Most experiments route one layer, while multi-layer results cover only 1 to 3 layers.This scope is insufficient to claim full-network modular computation or complete circuit recovery.
  • An appendix experiment routes fc2 into four modules and fc3 into eight modules, suggesting structured module-to-module pathways may arise naturally.

Supplementary Material: Sparse Competition during Training For the

The supplementary material extends the paper’s analysis using the same ImageNet-100 scaled_cnn setup unless otherwise stated. It focuses on cross-layer assignment structure and detailed causal specialization diagnostics.

  • Unless otherwise stated, supplementary analyses use the ImageNet-100 scaled_cnn setup from the main paper.
  • The supplement visualizes whether assignments across two routed layers form a coherent cross-layer structure.
  • It also reports a full per-module breakdown of the causal specialization test.

A. Cross-layer modular structure

The cross-layer analysis compares unconstrained and hierarchically constrained routing between fc2 and fc3. Similar qualitative flows suggest that cross-layer modular organization can emerge from the training objective without an explicit parent–child hierarchy.

  • The analysis routes fc2 into K = 4 modules and fc3 into K = 8 modules, comparing unconstrained and constrained variants.The constrained variant imposes a hard hierarchical connectivity prior.
  • Each alluvial flow represents classes sharing the same dominant module at fc2 and fc3.
  • The two variants produce qualitatively similar flows, suggesting cross-layer organization can arise without explicitly imposing a parent–child hierarchy.The diagnostic is qualitative.

B. Per-module causal specialization

The per-module diagnostic tests whether modules matter more for their assigned classes than for other classes. SCM shows positive specialization across modules while maintaining relatively balanced class ownership, and cross-layer structure can arise without an explicit hierarchy prior.

  • Causal specialization: The diagnostic compares accuracy drops on a module’s owned classes versus all remaining classes after ablating that module.A positive gap means the ablated module is more important for its assigned classes.
  • Cross-layer structure: The unconstrained and hierarchy-constrained routed models have qualitatively similar cross-layer alluvial structure.The hierarchy-constrained variant restricts each fc2 module to two designated fc3 modules.
  • Causal specialization: SCM has a positive causal gap for every module at both fc2 and fc3.This indicates specialization across all modules in the evaluated routed layers.
  • Class ownership: SCM assigns classes nearly uniformly, with each module owning between 23 and 28 classes.The clustered baseline also has positive gaps but substantial ownership imbalance, including an fc3 module owning 86 of 100 classes.

C. Robustness of the ImageNet-100 CMC results

Robustness tests on two random ImageNet-100 label subsets examine whether the reported CMC structure depends on an unusually favorable semantic split. SCM preserves baseline-level accuracy while increasing class–module mutual information across both subsets.

  • Random label subsets: SCM preserves baseline-level accuracy across two random ImageNet-100 label subsets.The experiment tests whether the original split contains unusually favorable semantic structure.
  • Random label subsets: SCM substantially increases class–module mutual information across both random subsets.The comparison uses sparse top-1 routing, with bold values marking the best result within each metric group.
  • Experimental design: The robustness evaluation repeats the main routed-fc2 experiment on two random 100-class subsets.This directly probes whether the ImageNet-100 CMC findings generalize beyond the original label subset.

D. More details on the algorithms

The algorithms define Gaussian-ring module memberships and a sparse competitive training loop. Training computes routing through module energies and probabilities, applies top-1 selection, tracks usage, computes routed-layer and total losses, and updates parameters by backpropagation.

  • Gaussian-ring memberships: The Gaussian-ring procedure places fixed module centers on a unit ring and allocates neuron widths under a total width budget.Its inputs include layer width, module count, temperatures, a budget factor, a width-floor fraction, and learnable logits.
  • Gaussian-ring memberships: For each neuron, the Gaussian-ring procedure sets a ring position and computes smooth periodic squared distance and Gaussian log-affinity.Affinities are then normalized across modules to produce the membership representation G.
  • SCM training loop: SCM applies the routing update at every routed layer during each training batch.The algorithm requires data, a network, routed layers, routing temperature, and routing weight, and returns trained parameters with Gaussian-ring logits.
  • SCM training loop: At each routed layer, SCM computes pre-activations, module memberships, module energies, and routing probabilities before selecting a winning module.The selected winner enables sparse top-1 routing.
  • SCM training loop: The training loop computes average module usage, routed-layer loss, and total loss, then updates network parameters and Gaussian-ring logits by backpropagation.These steps connect routing behavior to optimization during training.
Loading 2608.30978v1…