BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

1. Introduction

• The approach shares most model parameters across tasks while retaining a small number of task-specific parameters; the task is assumed to be known.The paper explores eight GLUE tasks, including question answering, sentiment analysis, and textual entailment.Source: §1 · p. 1
• Parameter sharing is motivated by battery, computation, and storage constraints when multiple task models must be run or stored.A shared transformation can be applied once and reused as input to task-specific functions.Source: §1 · p. 1
• Training-set sizes vary widely across the eight GLUE tasks, motivating a schedule that starts proportional to dataset size and gradually reduces its influence.The motivation includes avoiding undue preference for well-resourced tasks and overfitting low-resource tasks.Source: §1 · pp. 1–2
• PALs are low-dimensional multi-head attention layers added in parallel to BERT layers, alongside a training schedule that gradually deemphasizes dataset size.The paper also compares alternative adaptation modules for self-attention architectures.Source: §1 · p. 2
• ≈7 times fewer parameters let PALs match fine-tuned BERT-base on many GLUE tasks; RTE accuracy is 76.6%, above BERT-large (70.1%) and MT-DNN (75.5%).The paper also reports that sharing more parameters increased RTE performance.Source: §1 · p. 2

2. Background

• Hard sharing uses common hidden layers with task-specific outputs, while soft sharing keeps separate models and regularizes their parameters; this work explores hard sharing with adapters.The authors assume soft sharing of the whole BERT model would require too many parameters.Source: §2 · p. 2

2.1. Adaptation Parameters

• LHUC adapts hidden units with learnable scalars, while residual adapters use task-specific 1×1 filters that can be compressed with low-rank approximations.The paper draws on low-rank approximations of a model's key operation, including multi-head attention in transformers.Source: §2.1 · p. 2

2.2. Fine-tuning Approaches

• BERT uses bidirectional masked-word and sentence-adjacency pretraining on Transformer layers; concurrent Houlsby et al. (2019) adapters freeze BERT, unlike this paper's joint multi-task fine-tuning.The authors note that joint fine-tuning risks interference or forgetting and requires access to all tasks during training.Source: §2.2 · pp. 2–3

3. Adapting Self Attention

• BERT-base is a 12-layer bidirectional Transformer encoder with 110 million parameters; the study limits added parameters to 1.13× the base model.The final hidden state of [CLS] is used for the task output.Source: §3 · p. 3, §3.1 · p. 3

3.1. Model Architecture and Multi-head Attention

• Multi-head attention transforms each sequence element using the other elements, while BERT layers combine self-attention, residual connections, layer normalization, and feed-forward networks.The full model stacks 12 such layers and uses the final [CLS] representation for output.Source: §3.1 · p. 3

3.2. Adding Parameters to the Top

• Top-added task-specific functions act before classification, and a shared BERT representation can be computed once when several tasks use the same text.A low-dimensional encoder-decoder transformation allows a more complex function without requiring O(d_m^2) parameters.Source: §3.2 · p. 4

3.3. Adding Parameters within BERT

• PALs add task-specific multi-head attention in parallel with BERT layers, sharing encoder and decoder matrices across layers and using d_s = 204.Parallel connections performed better than serial connections in preliminary experiments; within-layer low-rank layers instead use the identity function with d_s = 100.Source: §3.3 · pp. 4–5

4.1. Sampling Tasks

• Table 1 compares parameter allocations for methods on a 12-layer base network; PALs omit the final W_o transform, while projected attention worked best with six layers.Source: Table 1 · p. 5
• Task-size ratios reached ≈158, and round-robin sampling risked overfitting smaller tasks while undertraining larger ones.The authors also note that training one task or subset for many steps can cause interference that harms other tasks.Source: §4.1 · p. 5
• Sampling shifts from probability proportional to dataset size toward more balanced choices with α = 0.5 or an annealed α schedule.Annealed sampling makes task selection more equal toward the end of training, when the authors are most concerned about interference; an epoch is 2,400 training steps.Source: §4.1 · p. 5
• Square-root or annealed sampling was particularly important when sharing a pooling layer, where the authors sought to guard against task interference.Source: §4.1 · p. 6

4.2. Setup

• Unless noted, training ran for 60,000 steps with batch size 32 and maximum sequence length 128, selecting the best model by average development-set score.The experiments used BERT-base architecture and twelve attention heads; the authors found no significant performance differences from changing the number of heads.Source: §4.2 · p. 6
• Pre-trained BERT-base weights were crucial: training from scratch performed significantly worse, although the authors note the reused hyperparameters might not suit scratch training.They concentrated on jointly fine-tuning all parameters, finding it took fewer parameters to approach matching separately fine-tuned BERT.Source: §4.2 · p. 6

4.3. Details of GLUE Tasks

• The evaluation covered eight of GLUE’s nine tasks, excluding Winograd NLI, where trained systems had performed below the 65.1 majority-class accuracy baseline.The included tasks span single-sentence acceptability and sentiment, sentence-pair similarity, and textual entailment.Source: §4.3 · p. 6

5. Experiments and Discussion

• PALs matched BERT-base on MNLI, QQP, and QNLI, while multi-task training achieved state-of-the-art RTE results; the fully shared model was slightly better on RTE.The authors note that fine-tuned BERT-large performed better except on RTE.Source: §5 · p. 6, Table 2 · p. 7

5.1. PALs and Alternatives

• PALs and low-rank layers adapting every BERT layer gave the best results, while sharing encoder or decoder matrices across layers or tasks hurt PAL performance.The authors also found that PAL performance increased with a larger hidden state.Source: §5.1 · p. 7
• Low-rank layers and PALs had similar mean performance, but the feedforward variant performed poorly; PALs did better on MNLI, QQP, and QNLI and were equivalent on other tasks.Low-rank layers' strong results were partly associated with better CoLA performance, a task scored by Matthews correlation coefficient rather than accuracy.Source: §5.1 · p. 7
• At 1.5 million parameters, PALs and low-rank layers performed similarly to the fully shared model, and LHUC did not improve on that baseline.The passage gives d_s = 10 for low-rank transforms and d_s = 60 for PALs at this parameter budget.Source: §5.1 · p. 8
• Low-rank layers remain an attractive option for their simplicity, and PALs can be combined with them across different network layers.The authors note that results may differ for question answering, which uses hidden states of every input token rather than only the final [CLS] state used by GLUE tasks.Source: §5.1 · p. 8

5.2. Where should we add Adaptation Modules?

• Within-network adaptations outperformed adding task-specific functions at the top, and adapting the final half performed slightly better than adding them to the top while retaining six shared layers.Adapting every layer performed best but left no operations shareable across tasks.Source: §5.2 · p. 8
• For within-network adaptations, parallel connections performed better than serial ones, which performed worse than using no adapters.The authors suggest parallel adaptations are less prone to losing knowledge stored in the base network.Source: §5.2 · p. 8

• Annealing the influence of dataset size on task sampling improved mean performance over proportional sampling when substantial parameters were shared, but increased variance across random seeds.The authors say several training runs may be needed to find the best single model.Source: §6 · p. 8
• The study did not test many training-method variations or methods for reducing interference between tasks; interactions with adaptation choices remain future work.Source: §6 · p. 8
• The best adaptation depends on constraints: PALs on every layer use few parameters but prevent sharing operations, while adapting only the final half offers a compromise.When sharing operations is not a concern, the authors say PALs on every layer perform well; when it is, adding a task-specific BERT layer on top is another option if parameters are less constrained.Source: §6 · p. 8

• Low-resource tasks varied more over training than high-resource tasks, while CoLA varied substantially early; late performance fell on CoLA and RTE with low-rank adapters but not PALs, with the reverse trend on MRPC.The authors suggest better scheduling or regularization might rectify these downward trends.Source: A. Performance on Tasks Over Time · p. 11

B. Squad and SWAG Performance

• SQuAD predicts answer-span locations in Wikipedia paragraphs; SWAG evaluates selection of a plausible continuation for a sentence.Source: B. Squad and SWAG Performance · p. 11
• PAL adapters show a slight improvement over the fully shared baseline and low-rank adapters on SQuAD and SWAG, though all approaches perform similarly.The authors suggest there may be less need for adapter flexibility when training on only two tasks.Source: B. Squad and SWAG Performance · p. 11
• The experiments use round-robin sampling, train for 24,000 steps, and set the maximum sequence length to 256 tokens.Source: B. Squad and SWAG Performance · p. 11
• Table 4 reports average development-set scores, using exact match and F1 for SQuAD and accuracy for SWAG.Source: Table 4 · p. 12