Source-linked AI summary
ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Xianming Li, Zongxi Li, Tsz-fung Andrew Lee, Jing Li, Haoran Xie, Qing Li
TL;DR
Full-parameter fine-tuning is costly, and LoRA-style PEFT uses independent local weight perturbations that remain tied to the backbone. ShadowPEFT centralizes adaptation through a depth-shared shadow network that maintains parallel states and refines hidden representations across layers. It achieves competitive or improved performance under comparable parameter budgets while enabling detachable deployment and shadow pretraining.
Problem
LoRA-style PEFT uses independent updates for selected layers, leaving adaptation fragmented and tied to the backbone’s internal weight structure.
Method
ShadowPEFT uses a shared shadow network that maintains and updates a parallel hidden state across transformer depth to refine frozen-backbone representations.
Results
Across generation and understanding benchmarks, ShadowPEFT achieves competitive or improved performance relative to LoRA and DoRA under comparable trainable-parameter budgets.
Takeaways & Limitations
ShadowPEFT supports reusable, detachable adaptation modules and extends PEFT toward modular, stateful function-level adaptation.
Takeaways & Limitations
Evaluation does not cover larger-scale LLMs or a more diverse set of architectures because of computational resource constraints.
Abstract
from arXiv · showhide
Parameter-efficient fine-tuning (PEFT) reduces the training cost of full-parameter fine-tuning for large language models (LLMs) by training only a small set of task-specific parameters while freezing the pretrained backbone. However, existing approaches, such as Low-Rank Adaptation (LoRA), achieve adaptation by inserting independent low-rank perturbations directly to individual weights, resulting in a local parameterization of adaptation. We propose ShadowPEFT, a centralized PEFT framework that instead performs layer-level refinement through a depth-shared shadow module. At each transformer layer, ShadowPEFT maintains a parallel shadow state and evolves it repeatedly for progressively richer hidden states. This design shifts adaptation from distributed weight-space perturbations to a shared layer-space refinement process. Since the shadow module is decoupled from the backbone, it can be reused across depth, independently pretrained, and optionally deployed in a detached mode, benefiting edge computing scenarios. Experiments on generation and understanding benchmarks show that ShadowPEFT matches or outperforms LoRA and DoRA under comparable trainable-parameter budgets. Additional analyses on shadow pretraining, cross-dataset transfer, parameter scaling, inference latency, and system-level evaluation suggest that centralized layer-space adaptation is a competitive and flexible alternative to conventional low-rank PEFT.
1 Introduction
ShadowPEFT addresses the fragmented, backbone-tied adaptation of LoRA-style PEFT with a shared shadow network that refines transformer layers and supports modular reuse. Across generation and understanding benchmarks, it achieves competitive or improved performance while enabling portable deployment modes.
- Motivation: LoRA adapts models through independent low-rank updates to selected linear projections, leaving adaptation fragmented across network depth.Its modules remain tied to the backbone’s internal weight structure and cannot be decoupled from the backbone.
- ShadowPEFT: ShadowPEFT centralizes adaptation in a lightweight shadow network reused across transformer layers to refine frozen-backbone hidden states.The shadow maintains a parallel hidden state and updates it across depth, replacing decentralized linear perturbations with layer-level refinement.
- Shadow Pretraining: Shadow pretraining substantially improves both attached and detached performance.The study compares randomly initialized and pretrained shadow modules.
- Evaluation: Across generation and understanding benchmarks, ShadowPEFT achieves competitive or improved performance relative to LoRA and DoRA under comparable trainable-parameter budgets.Evaluations include MMLU, GSM8K, and SQuAD V2, alongside analyses of transfer, scaling, efficiency, and system-level performance.
- Modularity: The shadow module is modular and portable: it can be attached or detached from the backbone and initialized from a smaller pretrained model for cross-scale adaptation.For example, Qwen-0.5B can serve as the shadow model for a Qwen-8B backbone.
2 Related Work
Related PEFT work progresses from prompt-based adaptation to adapters and low-rank weight updates, each retaining the frozen pretrained backbone while reducing trainable parameters. Prompt methods face dimensionality or context-related constraints, whereas LoRA uses constrained low-dimensional updates to preserve performance efficiently.
- Prompt-Based Methods: Prompt engineering avoids parameter updates but is limited by context length and lacks persistent task-specific adaptation.These constraints motivate parameter-efficient adaptation methods that update only a small subset of parameters while freezing the backbone.
- Prompt-Based Methods: Soft prompt methods optimize continuous prompt parameters while leaving the backbone unchanged, but their expressive capacity is constrained by prompt-vector dimensionality.Examples include Prompt Tuning, Prefix Tuning, and P-Tuning.
- Low-Rank Adaptation: LoRA learns low-rank matrices and injects updates alongside frozen pretrained weights instead of updating full weight matrices.This constrained low-dimensional subspace reduces trainable parameters while preserving performance.
3 Our ShadowPEFT Framework
ShadowPEFT centralizes parameter-efficient adaptation in a depth-shared shadow model that maintains a parallel hidden state and refines a frozen backbone across transformer layers. Its pipeline combines discrepancy-based low-rank injection, frozen base encoding, gated shadow updates, joint training, and attached or detached inference.
- Overview: ShadowPEFT reuses a centralized shadow model across transformer depth instead of distributing independent perturbations across linear weights.The shadow model maintains a parallel hidden state that evolves alongside the frozen backbone.
- Shadow Update Module: The frozen base layer encodes the corrected representation, after which a gated residual update advances the shadow state using the new base output.The gate controls interpolation between the previous shadow state and a candidate representation derived from the current base output.
- Shadow Injection Module: At each layer, ShadowPEFT computes the discrepancy between the previous backbone state and shadow state before injecting a learned low-rank refinement.The bottleneck filters useful components of the discrepancy rather than directly copying it into the backbone.
- Shadow Update Module: The gated update retains earlier task-specific context while incorporating information from later layers, supporting a depth-wise state-space adaptation process.The authors describe the GRU-style mechanism as helping prevent shadow collapse and improve optimization stability.
- Base-Shadow Joint Training: ShadowPEFT freezes the base model while training the shadow backbone, injection projections, update networks, prediction head, and task-specific objectives.The training setup includes causal language modeling and sequence classification objectives, with a joint loss using the base and shadow predictions.
- Inference: The trained adaptation supports shadow-attached inference and detached shadow-only inference, allowing the standalone shadow component to bypass the large base model.The shadow module can therefore be deployed together with the backbone or separately as a lightweight component.
4 Experiment
Across generation and understanding benchmarks, ShadowPEFT matches or improves upon LoRA and DoRA with comparable or slightly fewer trainable parameters. Additional analyses show benefits in out-of-distribution transfer, scalable shadow capacity, low latency overhead, and detachable edge-oriented deployment.
- Main Results: ShadowPEFT achieves the best average performance across Qwen3 0.6B, 4B, and 8B while using slightly fewer trainable parameters than LoRA and DoRA.Its average scores are 62.27 versus 61.81/62.08 on 0.6B, 75.43 versus 74.55/74.85 on 4B, and 76.92 versus 76.51/75.99 on 8B.
- Trainable Parameter Scaling: 82.12 GSM8K accuracy is ShadowPEFT’s peak when scaling trainable parameters to 0.4B, while performance marginally decreases to 81.80 at 0.5B.LoRA remains nearly flat at 80.52–81.28, whereas DoRA declines from 81.12 at 0.1B to 77.79 at 0.5B.
- Efficiency: 3.0 ms, 6.0 ms, and 5.9 ms are ShadowPEFT’s latency increases on Qwen3 0.6B, 4B, and 8B, respectively, corresponding to an average overhead of 4–6%.The shadow forward pass runs in parallel with base-model computation, while injection and update modules add a lightweight residual connection.
- System-Level Evaluation: ShadowPEFT attains the lowest latency and highest accuracy in the Unitree Go2 intent-understanding experiment compared with LoRA and DoRA.Detached shadow inference handles routine skills locally, while complex or open-domain requests are forwarded to the cloud model, reducing cloud usage.
- System-Level Evaluation: Detached shadow inference reliably handles routine robot skills and returns a [REMOTE] tag for complex or open-domain queries instead of generating hallucinations.The full ShadowPEFT model handles both routine and complex queries, while LoRA and DoRA occasionally produce incorrect actions or hallucinations.
5 Conclusion
ShadowPEFT centralizes adaptation in a shared, stateful shadow network reused across transformer depth instead of independent linear-wise weight perturbations. The resulting framework achieves competitive or improved performance while supporting detachable deployment and shadow pretraining.
- 5 Conclusion: ShadowPEFT replaces independent linear-wise weight perturbations with a shared transformer layer-level shadow network.The network maintains and updates a parallel shadow state across transformer depth to centralize adaptation in a reusable functional component.
- 5 Conclusion: ShadowPEFT achieves competitive or improved performance relative to strong low-rank baselines under comparable parameter budgets while enabling detachable deployment and shadow pretraining.The conclusion characterizes this as modular, stateful function-level adaptation rather than only lightweight parameter injection.
Limitation
The evaluation does not cover larger-scale LLMs or a more diverse set of architectures because of computational resource constraints.
- Limitation: The study does not evaluate ShadowPEFT on larger-scale LLMs or across a more diverse set of architectures.The authors identify these evaluations as future work because of computational resource constraints.
A More Designs of Centralized Shadow Model
ShadowPEFT supports implicit or explicit centralized shadow models, with optional embedding sharing and projection to align shadow and backbone representations.
- Implicit shadows derive from the base configuration by reducing layers and optionally narrowing width and attention heads.
- Explicit shadows accept independently provided compatible architectures, enabling cross-model knowledge transfer.
- The shadow reuses the frozen base embedding layer, eliminating a duplicate token-embedding lookup table.
- A learned bias-free projection aligns shadow outputs with the backbone representation space when hidden dimensions differ; otherwise it is an identity.
B Centralized Shadow Model Pretraining
ShadowPEFT attaches a smaller pretrained shadow model to a larger backbone through a learned projection, using pseudo-inverse initialization and continued causal-language-model pretraining.
- A Qwen3 0.6B shadow model is reused to attach to Qwen3 8B, requiring a projection between their hidden dimensions.
- The attached forward pass maps shadow hidden states through the projection before applying the frozen larger model’s language-model head.
- Moore–Penrose pseudo-inverse initialization provides a warm start that approximates the original output distribution and reduces subsequent alignment training.
- The shadow model and projection are then pretrained with causal language modeling on sampled English FineWeb-Edu and Chinese Wudao data.
C Dataset Prompt Templates
The benchmark prompts format each task according to its answer structure, mask prompt tokens during supervised training, and distinguish zero-shot from few-shot MMLU evaluation.
- Supervised prompts use the model’s chat template, append the assistant response after the prompt prefix, and compute loss only on answer tokens.
- MMLU: MMLU uses four-way multiple-choice prompts requiring a single answer letter, with zero-shot prompting used in the main experiment.
- MMLU: Few-shot MMLU prepends examples of correctly formatted multiple-choice responses to the zero-shot template for generalization testing.
- GSM8K: GSM8K prompts request the full gold solution, including intermediate reasoning and the final #### number line.
- SQuAD v2: SQuAD v2 prompts require an exact context span or the literal response unanswerable for questions lacking an answer.
D Gated Residual Update
The gated residual update uses lightweight transform and gate networks, while an auxiliary shadow loss regularizes the shadow and supports detached deployment.
- Both the transform and gate functions are implemented as lightweight two-layer MLPs.
- All projection matrices in the transform and gate networks are bias-free, with gate hidden size controlled by hg.
- The auxiliary shadow loss directly supervises shadow outputs, stabilizing optimization and encouraging task-relevant shadow representations.
- This regularization is especially important for detached inference, where only the shadow model is used at deployment.
F System-Level Evaluation Setup
The robot-dog instruction dataset contains bilingual examples covering predefined robot skills and a remote category. Evaluation uses detached shadow-only inference for on-device skills, forwarding out-of-scope or [REMOTE] requests to a cloud model.
- Dataset: The dataset contains 4,771 examples covering 34 predefined robot skills plus one [REMOTE] category.It includes 2,816 English examples and 1,955 Chinese examples.
- Dataset: The corpus is bilingual, with 2,816 English examples and 1,955 Chinese examples.
- Evaluation: Detached shadow-only inference executes recognized predefined skills directly on-device.
- Evaluation: Out-of-scope requests or predictions containing [REMOTE] are forwarded to the cloud model for further processing.
- Evaluation: LoRA and DoRA do not support the detached execution mode used in this evaluation.