PaperlayerView exact revision on arXiv

Source-linked AI summary

Modular Deep Learning

Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, Edoardo Maria Ponti

arXiv:2302.11529v2cs.LG

TL;DR

Transfer learning faces interference, forgetting, and difficulty generalising systematically across distribution shifts. This survey unifies modular deep learning methods and their applications, concluding that modularity supports systematic generalisation while learned routing remains challenging.

  • Problem

    Transfer learning can suffer from negative interference, catastrophic forgetting, and poor systematic generalisation when training and evaluation distributions differ.

    Source: §1 · p. 3

  • Method

    The survey presents a unified view of modular deep learning through the dimensions of computation, routing, aggregation, and training.

    Source: §1 · p. 5

  • Results

    The survey reports modularity’s role in causal inference and discovery, programme simulation, and hierarchical reinforcement learning.

    Source: §1 · p. 5

  • Takeaways & Limitations

    Modularity supports systematic generalisation to new tasks through module recombination and local updates.

    Source: §9 · p. 42

    Learned routing is challenging because of training instability, module collapse, and overfitting, and often underperforms fixed routing.

    Source: §9 · p. 42

Abstract

from arXiv · show

Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as autonomous parameter-efficient modules. Information is conditionally routed to a subset of modules and subsequently aggregated. These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer a survey of modular architectures, providing a unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such as cross-lingual and cross-modal knowledge transfer. Related talks and projects to this survey, are available at https://www.modulardeeplearning.com/.

1 Introduction and Motivation

Modular deep learning is presented as a response to transfer learning’s interference, forgetting, and weak systematic generalisation under distribution shifts. The survey unifies modular methods and describes their benefits, challenges, and applications.

1 Introduction and Motivation

  • Transfer learning can suffer from negative interference, catastrophic forgetting, and poor systematic generalisation when training and evaluation distributions differ.These limitations are presented as barriers to deployment where distribution shifts are common.Source: §1 · p. 3
  • Modules can be recombined for zero-shot transfer and updated locally, supporting robustness to local distribution shifts and sample-efficient adaptation.The survey illustrates recomposition with Guaraní-language dependency parsing, where the component modules can be trained separately despite a lack of annotated data for that combination.Source: §1 · p. 4
  • Modular architectures can reduce parameter and time costs by storing task-specific adapters, adjusting capacity incrementally, and activating only a small set of experts.The survey describes this selective activation as enabling language models to scale to more parameters while retaining the same time complexity.Source: §1 · p. 5
  • The survey organizes modular methods along computation, routing, aggregation, and training, and highlights connections between research threads that developed independently.It also surveys applications across transfer learning, causal inference and discovery, programme simulation, and hierarchical reinforcement learning.Source: §1 · p. 5
  • Modular designs trade off efficiency and performance across different implementations, while learned routing can face instability, module collapse, and overfitting.The survey also notes that learned routing often underperforms fixed routing.Source: §1 · p. 5

2 Modular Deep Learning

The survey defines modular deep learning through autonomous computation modules, routing, and output aggregation. It classifies methods by computation, routing, aggregation, and training, then gives a common notation for these components and their combinations.

2 Modular Deep Learning

  • Modular deep learning comprises autonomous computation modules, a routing function that controls information flow, and an aggregation function that combines module outputs.Modules may also be combined with fully shared parameters.Source: §2 · p. 6

2.1 Taxonomy

  • The taxonomy distinguishes computation implementation, active-module selection, output aggregation, and module training settings.Routing may be fixed or learned and hard or soft; aggregation may be deterministic or implemented by a learnable neural network.Source: §2.1 · p. 6

2.2 Notation

  • The notation represents a network as sub-functions with indexed parameters and modules, with modules modifying a sub-function through parameter, input, or function composition.Routing scores select modules, whose outputs are combined by an aggregation function to produce the output.Source: §2.2 · pp. 6–8

3 Computation Function

This section surveys modular computation through parameter, input, and function composition, including hypernetworks that generate modules from metadata. It describes how these approaches modify or compose a base model and contrasts their trade-offs in parameter use, efficiency, adaptation, and performance.

3.1 Parameter Composition

  • For pretrained models, fine-tuned weights often stay near their initial values, making magnitude pruning task-insensitive; gradient-based pruning better captures task relevance.Movement pruning learns a binary mask jointly with parameters, using straight-through estimators.Source: §3.1 · p. 10
  • In low-dimensional optimization, the intrinsic dimensionality is the minimum dimension reaching within 90% of full-parameter performance; it decreases during pre-training and is lower for larger models.Source: §3.1 · p. 11
  • Parameter composition uses sparse or low-rank modules, often updating less than 0.5% of model parameters, and leading methods achieve strong zero-shot and few-shot transfer.At inference, these methods keep model size constant or can reduce it, though sparse modules can require multiple retraining iterations.Source: §3.1 · pp. 8, 12

3.2 Input Composition

  • Input composition adds module vectors to a model’s input; prompts, learned continuous prompts, layer-wise prefixes, and retrieved context are examples.It adds few parameters but extends the context window, reducing training and inference efficiency; prompt tuning also requires large models for decent performance.Source: §3.2 · p. 12

3.3 Function Composition

  • Function composition adds task-specific sub-functions, including bottleneck adapters and learned rescaling, or auxiliary modules that provide knowledge, memory, or other capabilities.Adapters can be sequential or parallel; rescaling vectors can emphasize task-relevant representation dimensions.Source: §3.3 · pp. 12, 14
  • State-of-the-art function-composition methods match or outperform standard fine-tuning, while adding operations at training and inference time.Adapters typically require more parameters but avoid storing gradients for frozen parameters.Source: §3.3 · p. 14

3.4 Hypernetworks

  • Hypernetworks generate module parameters from task embeddings or inputs, enabling parameter sharing but entangling modules and intertwining computation with routing.They have generated classifier heads, continuous prompts, and adapter layers.Source: §3.4 · p. 15
  • Hypernetworks can condition on task and language embeddings to generate modules, enabling systematic generalisation to unseen task–language combinations at inference.They may generate one module from the combined embeddings or separate modules; task or language embeddings can be learned or fixed, and other metadata such as labels can also be incorporated.Source: §3.4 · p. 15

3.5 Unifying Parameter, Input, and Function Composition

  • The survey extends a framework to parameter, input, and general function composition, showing that modular computation functions reduce to function composition, where a model function’s output gains a learned-function-dependent term.Source: §3.5 · p. 15
  • Function-composition methods such as adapter and compacter layers achieve the best performance but add parameters, while (IA)3 composes a lightweight diagonal weight.Prompt tuning underperforms because of limited capacity; intrinsic dimensionality uses few parameters but has a large memory footprint and poor performance, while bias-only fine-tuning has a small footprint but lower performance.Source: §3.5 · p. 16

4 Routing Function

This section surveys routing functions that determine which modules contribute to computation and how information flows through them. It covers fixed or learned selection, task-specific parameter generation, routing granularity, and composition order.

4 Routing Function

  • Learned routing can underuse and less-specialise modules, increasingly so as task counts grow; Muqeeth et al. (2022) report similar results, but Ponti et al. (2022) find it can beat expert selection.The under-utilisation finding comes from synthetic data; Ponti et al. report their result even when procedurally constructed tasks require particular skills.Source: §4 · p. 17

4.1 Fixed Routing

  • Fixed routing selects modules in advance using metadata such as task identity, and composing task and language modules can generalise to unseen combinations.Modular policies can likewise combine robot-specific and task-specific modules to generalise to unseen robot–task combinations.Source: §4.1 · pp. 17–18

4.2 Learned Routing

  • When routing is unknown, a learnable network maps an example representation or metadata to routing scores, while modules and router are trained jointly.The router may be a linear projection or an MLP; the latter may ignore input features, and module specialisation is not known in advance.Source: §4.2 · p. 19

4.2.1 Challenges of Learned Routing

  • Learned routing faces instability, module collapse, and overfitting, driven by balancing exploration against exploitation and positive transfer against negative interference.Instability arises early when modules lack specialisation and the router lacks a principled basis for selection.Source: §4.2.1 · p. 19
  • Module collapse leaves most modules untrained; exploration, load-balancing losses, diversity incentives, metadata conditioning, diverse tasks, and temperature warm-up can help prevent it.Routing at token level can overfit noise and impair out-of-domain generalisation; metadata conditioning and combinatorial module behaviour can improve robustness.Source: §4.2.1 · p. 19

4.2.2 Hard Learned Routing

  • Hard routing makes binary module selections, requiring methods such as reinforcement learning, evolutionary search, or stochastic re-parameterisation rather than vanilla gradient descent.It can reduce time and space costs and localise parameter updates to active modules, but evolutionary search can be highly inefficient.Source: §4.2.2 · pp. 19–20

4.2.3 Soft Learned Routing

  • Soft routing weights and aggregates module outputs for end-to-end gradient training; top-k MoEs limit active modules, and top-1 routing can be competitive for language modelling.Using all modules increases compute and prevents local parameter updates; token-level load balancing may impede whole-example routing and lead similar concepts to be relearned across modules.Source: §4.2.3 · pp. 21–22

4.2.4 Hypernetworks

  • Hypernetworks generate task-specific parameters as linear combinations of shared module parameters, using unnormalised task-level routing scores.Unlike post-hoc tensor factorisation, they jointly learn the generator and task embedding (Yang & Hospedales, 2017).Source: §4.2.4 · p. 23

4.3 Level of Routing

  • Routing granularity ranges from global choices shared across layers to per-layer or hierarchical decisions; conditional independence makes per-layer routing scalable.Without that assumption, the architecture space grows as |M|^l and exact posterior computation would marginalise over every configuration; Kirsch et al. (2018) use Expectation Maximisation.Source: §4.3 · p. 23, Figure 5 · p. 23
  • Routing can also define module composition and order: Neural Module Networks build module graphs from parsed queries, while Modular Meta Learning samples compositional graphs with simulated annealing and updates parameters by gradient descent (Andreas et al., 2016b; 2017; Alet et al., 2018).Source: §4.3 · p. 23

4.2.3 Soft Learned Routing

  • Learned routing can make hard discrete selections or soft weighted combinations, while training may face instability, module collapse, and overfitting.Hard routing may require reinforcement learning or stochastic re-parameterization; soft routing often uses top-k strategies for computational efficiency.Source: §4.3 · p. 24

5 Aggregation Function

Aggregation combines information from selected modules at parameter, input, function, or representation level; in most current approaches, routing and aggregation happen together.

5.1 Parameter Aggregation

  • Parameter interpolation can be risky when architectures or hidden representations differ, though linear mode connectivity suggests it can work under certain conditions.Examples include adding language- and task-specific subnetworks to base parameters and interpolating soft prompts with input-conditioned attention.Source: §5.1 · pp. 24–25

5.2 Representation Aggregation

  • Weighted averaging combines active module outputs using learned scalar weights, while attention weights post-transformation representations according to their relevance to the input.With soft routing, averaging may require passing inputs through every module; top-k hard routing offers a more efficient alternative.Source: §5.2 · pp. 25–26

5.3 Input Aggregation

  • Input aggregation concatenates prompts, instructions, examples, or prefixes, while hypernetworks combine embeddings to infer parameters for unseen task–language combinations.Concatenating task and language embeddings can condition a hypernetwork that generates a classifier head.Source: §5.3 · pp. 26–27

5.4 Function Aggregation

  • Function aggregation composes modules sequentially or hierarchically; fixed routing often uses sequential order, while global routers can jointly select modules and tree structure.Disentangling language and task adapters supports zero-shot inference on a target language without annotated data.Source: §5.4 · pp. 27–28

6 Training Setting

The survey groups modular training into joint multitask learning, continual learning, and parameter-efficient transfer learning; these strategies can also be combined.

6.1 Joint Multitask Learning

  • Joint multitask learning integrates task-specific components into shared architectures to mitigate forgetting or interference, or uses fully modular architectures that share routing parameters.Modules may specialize through fixed routing, and multilingual pretraining can prepare a model for adding language-specific modules later without changing shared parameters.Source: §6.1 · p. 28

6.2 Continual Learning

  • Continual learning adds new data over time while seeking to prevent later updates from overwriting earlier knowledge, using new modules or task-specific subnetworks.Some methods freeze earlier task parameters and add modules; others identify unused subnetworks or use supermasks for additional tasks.Source: §6.2 · p. 29

6.3 Parameter-efficient Transfer Learning

  • Parameter-efficient transfer learning adapts pretrained models post hoc through parameter, input, or function composition, including sparse subnetworks, prompts, and adapter layers.Updating a small parameter set has been demonstrated to perform equally well as full-model fine-tuning.Source: §6.3 · p. 29

7 Applications in Transfer Learning

Most modular deep-learning applications concern transfer learning, particularly parameter-efficient fine-tuning and zero- or few-shot generalisation to new tasks.

7 Applications in Transfer Learning

  • Transfer-learning applications mainly target parameter-efficient fine-tuning and zero- or few-shot generalisation, with further uses in reinforcement learning, programme simulation, and causal inference.The survey focuses first on NLP applications, then draws analogies to speech, vision, and multimodal learning.Source: §7 · p. 30

7.1 Parameter-Efficient Fine-tuning

Across NLP, speech, vision, and related settings, modular fine-tuning stores specialized updates in modules rather than full task-specific model copies. The survey shows how modular designs vary with transfer goals and practical constraints.

7.1 Parameter-Efficient Fine-tuning

  • Lightweight modules store task-specific updates while the large base model stays frozen, reducing storage needs and supporting either better performance at a fixed budget or similar performance with fewer parameters.Modules can also be combined for zero-shot transfer; parameter-efficient fine-tuning is now a main way to update large language models.Source: §7.1 · p. 30

7.1.1 Machine Translation

  • Bilingual adapters specialize a 103-language NMT model for a translation direction, while monolingual encoder and decoder adapters need only 2n modules and support directions without parallel data.The latter can translate between languages without seeing a parallel sentence for that pair, including languages unseen by the base model.Source: §7.1.1 · pp. 30–31

7.1.2 Cross-Lingual Transfer

  • Cross-lingual systems combine language and task modules for zero-shot transfer, while Hyper-X conditions generated modules on task, language, and layer embeddings (Üstün et al., 2022).MAD-X replaces source-language modules with target-language modules at inference while retaining the task module.Source: §7.1.2 · pp. 31–32

7.1.3 Domain Adaptation

  • Domain adaptation combines shared parameters with dedicated domain modules, often adding adversarial, orthogonality, or similarity constraints to shape their representations.These constraints encourage domain-invariant shared parameters, distinct information across domain modules, or proximity between similar modules.Source: §7.1.3 · p. 33

7.1.4 Knowledge Injection

  • Knowledge modules store external sources in adapters, sometimes aggregated by task; retrieval modules similarly decouple knowledge fetching from language-model generation.Examples include adapters trained on ConceptNet-derived sentences and a retrieval module that can be improved without changing the rest of the model.Source: §7.1.4 · p. 33

7.1.5 Speech Processing

  • Speech work mainly uses bottleneck adapters for ASR; wav2vec-family encoders and CTC are typical, and mixture-of-experts also appears in multilingual and cross-lingual ASR.Adapters are also used for speech translation, text-to-speech, speaker and domain adaptation, and multilingual learning.Source: §7.1.5 · p. 34

7.1.6 Computer Vision and Cross-Modal Learning

  • Vision and multimodal work adapts modules to tasks, domains, and modalities; examples include Flamingo’s frozen pretrained vision and language models with trained adapters.The survey describes applications including multilingual vision-and-language tasks and image-to-video transfer, while noting its examples are not exhaustive.Source: §7.1.6 · pp. 34–35

7.1.7 Comparison and Design Principles

  • Design choices trade off direction-specific performance, transfer, compactness, and inference or storage constraints; AdapterHub supports module reuse, while comprehensive empirical comparison remains lacking.For example, bilingual modules may favor a particular direction, whereas monolingual modules may better support zero-shot transfer; function-composition methods can increase forward-pass computation.Source: §7.1.7 · pp. 35–36

7.2 Task Generalisation

Modular systems can generalize to new tasks by composing locally updated skills, even when those skills are learned as latent units rather than named knowledge categories. The survey situates this approach among methods for zero- or few-shot transfer to tasks whose test examples differ from training examples.

7.2 Task Generalisation

  • Latent skills can be learned end-to-end from mixed-task data and recombined in arbitrary subsets, supporting systematic generalization to new tasks despite limited interpretability.These skills may have arbitrary inventory sizes and are paired with a learned routing function.Source: §7.2 · p. 36
  • Zero- or few-shot task generalization evaluates models pretrained on training tasks and adapted to disjoint test tasks using zero or few examples.CrossFit, the T0 task suite, and Natural Instructions are cited as example benchmarks; instruction tuning and modular learning are strategies for this setting.Source: §7.2 · p. 36
  • Because test examples need not be identically distributed with training examples, modular deep learning is presented as a strong contender alongside instruction tuning for this transfer setting.The cited modular approaches include work by Alet et al. (2018), Kudugunta et al. (2021), and Ponti et al. (2022).Source: §7.2 · p. 36

8 Other Purposes of Modularity

Modularity extends beyond transfer learning and model scaling to hierarchical reinforcement learning, programme simulation, and causal inference. Across these uses, modular deep learning can induce latent structures end-to-end while jointly modeling low-level perception.

8 Other Purposes of Modularity

  • Modular deep learning also supports hierarchical reinforcement learning, programme simulation, and causal inference, with potential to induce options, programmes, or causal graphs end-to-end alongside low-level perception.These are presented as additional purposes of modularity beyond scaling large models and facilitating transfer learning.Source: §8 · p. 36

8.1 Hierarchical Reinforcement Learning

  • Options provide intermediate abstractions between high-level goals and low-level actions, but learning them faces specialization, training-cost, and non-stationarity challenges.Language-based options can share statistical strength across sub-policies and be reused in new tasks; hindsight relabelling can improve sample efficiency.Source: §8.1 · pp. 37–38

8.2 Programme Simulation

  • Programme simulation ranges from dynamically routed state transitions to globally composed module graphs and memory-augmented models.These approaches support compositional reasoning and reuse of modules for shared sub-problems; external memory can support reasoning over complex data structures.Source: §8.2 · pp. 38–40

8.3 Causal Discovery and Inference

  • Causal modularity assumes independent, reusable mechanisms, so local distribution shifts may require updating only the corresponding module.A key challenge is specializing modules from observational data when the number and nature of mechanisms are unknown; observational data alone is often insufficient for identifying causal models without assumptions.Source: §8.3 · pp. 40–41

9 Conclusions

The conclusions synthesize modular deep learning around specialized computation separated from routing and aggregation, with applications spanning transfer learning and other domains. They also identify open directions in routing, evaluation, modular representations, and extending modular models.

9 Conclusions

  • Modular deep learning separates computation functions from routing and aggregation, while modules may modify, concatenate with, or compose with the original model.Composition strategies offer different efficiency and performance trade-offs.Source: §9 · p. 42
  • Learned routing remains challenging because of instability, module collapse, and overfitting, and often underperforms fixed routing.Routing may be conditioned on input or metadata, and may operate globally or layer-wise; soft routing combines all module outputs, while hard routing activates a subset.Source: §9 · p. 42
  • Applications in NLP, computer vision, and speech processing use parameter-efficient fine-tuning with shared module types and fixed routing to prevent negative interference and enable zero-shot transfer.Modularity also supports systematic generalization by recombining modules and updating them locally.Source: §9 · p. 42
  • Modularity also supports community-driven sharing, expansion, reuse, and updating of modules, while deep models can learn options, programmes, and causal graphs end-to-end from perceptual stimuli.The survey connects these capabilities to hierarchical reinforcement learning, programme induction, and causal discovery.Source: §9 · p. 42

9.1 Future Work

  • Future work calls for standardized, forward-looking modularity benchmarks and clearer understanding of how computation, routing, aggregation, and training shape modular representations.Current evaluation often repurposes existing datasets, including for zero-shot transfer; proposed benchmarks should test skill composition and new reasoning abilities.Source: §9.1 · p. 43
  • Future work includes learned routing for pre-training without metadata, more flexible module structures, and combining different computation-function types.Fixed routing limits use on heterogeneous data without metadata, while learned routing has training difficulties.Source: §9.1 · p. 43

Modular Instruction Tuning

  • Modular instruction tuning could support acquiring, updating, and composing specialized knowledge, while modular multi-task models may be easier to extend and merge across settings.The survey also points to structured or sparse aggregation and learned aggregation methods as directions for improving efficiency or generalization.Source: Modular Instruction Tuning · p. 44

Related papers

Loading 2302.11529v2…