Source-linked AI summary
SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning
Mohammad Aref Jafari-Raddani, Morteza Mohajjel Kafshdooz
TL;DR
PEFT reduces the cost of adapting large language models, but existing parameter-sharing methods either tie layers uniformly or add dynamic masking overhead. SAPE uses boundary-isolated adapters with hard sharing among balanced intermediate groups, achieving strong low-parameter results while exposing a trade-off between semantic generalization and rigid arithmetic reasoning.
Problem
Existing PEFT sharing methods use uniform layer tying or dynamic masking, leaving Transformer hierarchy-inspired sharing patterns underexplored.
Method
SAPE applies a static sandwich topology with dedicated adapters at boundary layers and hard-shared adapters across balanced intermediate-layer groups.
Results
SAPE achieves state-of-the-art low-parameter performance across encoder-only and decoder-only evaluations, outperforming proPETL and AdaLoRA in reported comparisons.
Takeaways & Limitations
Hard sharing acts as a structural regularizer for semantic generalization while eliminating dynamic-sharing overhead, but layer-wise adaptation remains advantageous for peak multi-step arithmetic accuracy.
Takeaways & Limitations
On GSM8K, SAPE reaches 40.30% versus PAdapter’s 41.55%, indicating reduced step-by-step arithmetic capability relative to non-sharing adapters.
Abstract
from arXiv · showhide
While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
1 Introduction
Large-scale Transformer fine-tuning is costly, motivating PEFT and parameter sharing, but existing sharing methods are either homogeneous, slower to converge, or computationally expensive. SAPE introduces a static sandwich topology that isolates boundary layers and shares intermediate adapters, achieving strong low-parameter performance.
- Full fine-tuning of large Transformer models is prohibitively expensive because of their scale and associated memory and computation requirements.
- PEFT reduces memory costs but can retain a performance gap from full fine-tuning, while AdaLoRA adds overhead through importance scoring and iterative masking.
- Cross-layer sharing reduces parameter footprints, but uniform tying can delay convergence and dynamic masking introduces training overhead.
- SAPE uses a deterministic sandwich topology with isolated boundary adapters and shared intermediate-layer groups.The design aligns sharing with Transformer depth while avoiding dynamic mask generation.
- SAPE achieves state-of-the-art low-parameter results across encoder-only and decoder-only models, including improvements over AdaLoRA and proPETL.Reported results include +4.85% on GSM8K, +3.11% on CommonsenseQA, and RoBERTa-large performance using 10% of proPETL’s parameter budget.
2 Related Work
Related PEFT methods reduce trainable parameters through adapters, selective updates, low-rank decomposition, pruning, or parameter sharing. SAPE addresses a gap in these sharing strategies by replacing uniform depth-wise tying with a heterogeneous sandwich topology.
- Parameter-Efficient Fine-Tuning (PEFT): Adapter tuning inserts small nonlinear bottlenecks into Transformer sublayers, while SparseAdapter reduces adapter redundancy through initialization-time pruning.
- Parameter-Efficient Fine-Tuning (PEFT): Selective methods such as BitFit train only bias vectors, trading parameter efficiency for architectural dependence and limited capacity under complex domain shifts.
- Parameter-Efficient Fine-Tuning (PEFT): LoRA learns low-rank updates, whereas AdaLoRA dynamically allocates rank but incurs overhead from iterative importance scoring and pruning schedules.
- Parameter Sharing within PEFT: Cross-layer sharing methods including ALBERT, Compacter, proPETL, ShareLoRA, and MASA reduce parameters through shared components or prototypes.
- Parameter Sharing within PEFT: Existing sharing strategies tie parameters uniformly across depth, overlooking heterogeneous sharing patterns motivated by Transformer structure.
3 Methodology
SAPE combines bottleneck adapters with a sandwich topology that isolates boundary layers and hard-shares adapters across balanced intermediate groups. Its routing and optimization design reduces trainable parameters while encouraging shared transformations across multiple Transformer depths.
- Adapter architecture and placement: SAPE applies a bottleneck adapter after the final feed-forward sub-layer in each Transformer layer.The adapter maps the hidden state through a reduced bottleneck dimension, ReLU, dropout, and an up-projection.
- Sandwich topology: The sandwich topology isolates the first and final Transformer layers with dedicated adapter parameters and reserves intermediate layers for sharing.Boundary layers are treated separately from intermediate semantic routing.
- Sandwich topology: Intermediate layers are partitioned into G disjoint groups, with every layer in a group using the same adapter parameter set.The allocation distributes N −2 intermediate layers as uniformly as possible.
- Balanced grouping: Each group contains either s or s + 1 layers, while DenseEarly controls whether extra layers concentrate toward earlier or later Transformer layers.This preserves balanced sharing while allowing the denser grouping direction to be selected explicitly.
- Optimization and parameter complexity: Shared adapters receive aggregated gradients from all layers in their assigned group, coupling optimization across multiple representation depths.This encourages parameters to capture features useful across the group rather than specializing to one layer.
- Optimization and parameter complexity: Hard sharing reduces trainable parameters and acts as structural regularization, while the sandwich topology preserves expressive capacity for adaptation.The total parameter count is constrained across the G shared groups and two boundary adapters.
4 Experiments
SAPE is evaluated on encoder-only and decoder-only Transformers across language understanding, generation, reasoning, and architectural ablations. Across low-parameter settings, it achieves strong benchmark performance, while results reveal a trade-off between semantic generalization and rigid arithmetic reasoning.
- Experimental Setup: SAPE is evaluated on RoBERTa-large, DeBERTa V3-base, and LLaMA-3.2 (3B) across GLUE, generation, reasoning, and structural ablations.The study compares SAPE with full fine-tuning and standard PEFT baselines, and examines topology, group granularity, density allocation, and masking.
- Convergence: SAPE achieves higher early-epoch validation results than AdaLoRA because its fixed topology avoids iterative warmup and rank-reallocation overhead.Figure 3 compares epoch-wise validation trajectories across low-resource tasks using AdaLoRA’s original hyperparameter and initialization schedules.
- Natural Language Understanding: +2.20% average: SAPE exceeds BitFit under the 0.10M budget while using roughly one-third the parameters of HAdapter, PAdapter, and LoRA.At D = 16, SAPE reaches 96.10% peak accuracy on SST-2 and remains highly competitive on RTE.
- Natural Language Understanding: 88.75% overall average: SAPE leads RoBERTa-large PEFT methods with a 0.55M parameter budget, surpassing proPETL Adapter by +0.25%.ProPETL uses over 10× more trainable parameters and incurs additional mask-generation overhead; SAPE also reaches 92.25% on MRPC and 69.25% on CoLA.
- Natural Language Generation: 0.1496 ROUGE-L and 0.7376 BERTScore F1: SAPE outperforms LoRA and AdaLoRA on ConvAI2 under the highly constrained ∼0.6M budget.At ∼1.0M parameters, SAPE reaches a BERTScore of 0.7393 and outperforms comparable baselines.
- Reasoning Trade-off: SAPE retains 78.54% CommonsenseQA Exact Match at ∼0.6M parameters, while AdaLoRA drops to 75.43%; however, rigid arithmetic reasoning still favors layer-wise adaptation.On GSM8K, SAPE reaches 40.30%, below PAdapter’s 41.55%, despite outperforming AdaLoRA.
- Group Granularity and Density: 86.41% average: G = 2 captures over 99.4% of the unshared baseline’s performance while using 101K rather than 304K parameters.Density placement has only marginal, mixed effects across tested granularities, indicating structural robustness.
- Parameter Allocation: 86.41% versus 84.52% average accuracy: SAPE with G = 2 and d = 16 outperforms an unshared G = 10 baseline with d = 5 at the same ∼101K budget.The comparison isolates parameter-reduction mechanics from total trainable-parameter count.
5 Conclusion and Future Work
SAPE uses boundary-isolated sandwich sharing to reduce adapter redundancy and computational overhead while maintaining strong performance across NLP tasks. The paper identifies a trade-off: sharing supports semantic generalization but can smooth transformations needed for rigid multi-step arithmetic reasoning.
- 5 Conclusion and Future Work: SAPE isolates initial and final boundary transformations while sharing intermediate adapters through a sandwich-style topology.The design separates input and output adaptations from shared intermediate representations to prevent destructive gradient interference.
- 5 Conclusion and Future Work: SAPE achieves state-of-the-art efficiency in constrained-parameter natural language understanding, ConvAI2, and CommonsenseQA evaluations.The conclusion reports strong efficiency across encoder-only and generative task settings, including comparisons against dynamic methods.
- 5 Conclusion and Future Work: Cross-layer sharing acts as a structural regularizer but marginally smooths transformations required for highly rigid tasks.The authors associate this trade-off with respectable but non-peak performance on GSM8K multi-step arithmetic reasoning.
- 5 Conclusion and Future Work: Future work will deploy SAPE in Federated Learning and extend its boundary-isolated topology to other PEFT methods such as Shared LoRA.The proposed directions target communication bottlenecks in Federated Learning and broader parameter-sharing designs.
Statements and Declarations
The authors report no external funding or relevant competing interests, and the analyzed datasets are publicly available.
- Statements and Declarations: The authors report receiving no organizational support for the submitted work.
- Statements and Declarations: The authors disclose no relevant financial or non-financial competing interests.
- Statements and Declarations: GLUE, ConvAI2, CommonsenseQA, and GSM8K are available in the public domain.
Appendix A Comprehensive Hyperparameter Specifications
The appendix provides detailed configurations intended to support experimental transparency and reproducibility across evaluated benchmarks and model backbones.
- Appendix A Comprehensive Hyperparameter Specifications: Detailed hyperparameters, optimization dynamics, and decoding setups are provided for all evaluated benchmarks and model backbones.
A.1 GLUE Benchmark Setup: DeBERTa V3-Base
The appendix specifies task-specific optimization parameters for the primary GLUE experiments under two adapter bottleneck settings.
- A.1 GLUE Benchmark Setup: DeBERTa V3-Base: Task-specific optimization parameters are listed for primary GLUE experiments with D = 16 and D = 64 bottleneck constraints.
A.2 GLUE Benchmark Setup: RoBERTa-Large
The RoBERTa-large experiments use a 355M-parameter backbone, with detailed setup documented in Table A2.
- A.2 GLUE Benchmark Setup: RoBERTa-Large: The experiments use a 355M-parameter RoBERTa-large backbone.Table A2 provides the experimental setup details.
- A.2 GLUE Benchmark Setup: RoBERTa-Large: Table A2 documents the experimental setup for the RoBERTa-large backbone.
A.3 Generative & Reasoning Benchmarks: Llama-3.2-3B
The causal language model experiments use documented optimization hyperparameters and inference decoding parameters.
- A.3 Generative & Reasoning Benchmarks: Llama-3.2-3B: The experiments specify optimization hyperparameters for the causal language model.
- A.3 Generative & Reasoning Benchmarks: Llama-3.2-3B: The experimental setup includes inference decoding parameters.
- A.3 Generative & Reasoning Benchmarks: Llama-3.2-3B: Table A3 documents the optimization and decoding configurations used in the causal language model experiments.
A.4 Topology Ablation Study Setup
The topology ablation study compares four routing strategies under controlled configurations across DeBERTa V3-base, RoBERTa-large, and Llama-3.2-3B experiments.
- A.4 Topology Ablation Study Setup: The DeBERTa V3-base SAPE fine-tuning configuration fixes the random seed at 42, uses cosine annealing, and applies early stopping with patience 2.
- A.4 Topology Ablation Study Setup: The RoBERTa-large configuration uses D = 64 and G = 2, with weight decay fixed at 0.01, seed 42, and cosine annealing.
- A.4 Topology Ablation Study Setup: The Llama-3.2-3B configurations cover GSM8K, CSQA, and ConvAI2 with hyperparameter, optimization, and decoding settings.
- A.4 Topology Ablation Study Setup: The DeBERTa V3-base topology ablation compares Regular, Top-Slice, Bottom-Slice, and Sandwich routing strategies.All four strategies were trained under identical optimization conditions to ensure a fair parameter budget.