Source-linked AI summary
LiST: Local-Simplex Test-Time LoRA Fusion
Yihua Shao, Jia Li, Siyu Chen, Xinyu Luo, Yang Liu, Kecheng Chen, Xinwei Long, Lingyu Zhu, Fanhu Zeng, Maolin Wang, Ziyang Yan, Jingcai Guo, Hao Tang, Nicu Sebe, Zhenyi Wang
TL;DR
Existing LoRA composition methods are largely static, leaving open how to adapt a frozen adapter bank to individual test inputs while reducing forgetting. LiST performs label-free, sample-specific search over a target-conditioned local simplex using branch-preserving fusion and constrained prompt-level energy. Across multimodal and language benchmarks, it outperforms static merging and conventional test-time adaptation baselines while improving unseen-task robustness and preserving task-specific adapter utility.
Problem
Existing adapter compositions are mostly static and cannot adapt a frozen task-specific LoRA bank to individual test inputs while addressing less forgetting.
Method
LiST retrieves neighboring adapters into a target-conditioned local simplex and optimizes low-dimensional fusion weights with branch-preserving fusion, constrained prompt-level energy, and safe fallback.
Results
LiST outperforms static merging and conventional test-time adaptation baselines on multimodal and language benchmarks while improving robustness on unseen tasks.
Takeaways & Limitations
LiST provides sample-specific, label-free LoRA composition while preserving task-specific adapter utility and keeping model parameters frozen.
Takeaways & Limitations
LiST assumes useful neighboring task-specific adapters, adds inference-time search cost, and may have less metric-aligned energy for highly open-ended generation.
Abstract
from arXiv · showhide
Task-specific LoRA adapters offer a modular way to specialize large language and vision-language models. However, existing adapter composition methods are mostly static and cannot adapt to individual test inputs. To address these issues, we propose \textbf{LiST}, a label-free test-time LoRA fusion framework that converts an existing LoRA bank into a target-conditioned local simplex and searches sample-specific fusion weights at inference time. LiST builds joint task representations from LoRA parameter anchors and prompt-level behavior vectors, retrieves neighboring adapters as a local search space, and performs branch-preserving fusion without updating the backbone or adapters. Candidate weights are selected by a prompt-level energy with prior, geometric, and stochastic-consistency constraints, and are deployed only when they pass a safe acceptance rule. Otherwise, LiST falls back to a target-conditioned prior. Experiments on multimodal and language benchmarks show that LiST outperforms static LoRA merging and conventional test-time adaptation baselines, while preserving task-specific adapter utility and improving robustness on unseen tasks.
1 Introduction
LiST addresses the limitations of static adapter composition with label-free, sample-specific test-time LoRA fusion over a target-conditioned local simplex. It preserves adapter branches, constrains optimization with prompt-level signals, and improves benchmark performance and unseen-task robustness.
- Motivation: Static adapter strategies cannot adapt compositions to individual test inputs and may distort LoRA branch structure through parameter averaging.Using only the target adapter ignores transferable knowledge, while fixed or similarity-based merging applies the same composition across inputs.
- Method: LiST constructs a target-conditioned local simplex by combining LoRA parameter anchors with prompt-level behavior vectors to retrieve neighboring adapters.The simplex restricts test-time search to a plausible task neighborhood.
- Method: LiST performs branch-preserving fusion at the LoRA branch-output level while keeping the backbone and adapter bank frozen.Only low-dimensional fusion weights are optimized at inference time.
- Method: A prompt-level energy combines prior, geometric, and stochastic-consistency constraints to select label-free candidate fusion weights.A safe acceptance rule rejects unreliable searches and falls back to the task-conditioned prior.
- Results: LiST outperforms LoRA merging and conventional test-time adaptation baselines on multimodal and language benchmarks while improving robustness on unseen tasks.The reported results cover both multimodal and language tasks.
2 Related Work
Prior work combines adapters through mostly static or learned coefficients and adapts models at test time using parameter or state updates. LiST instead treats LoRA composition as a training-free, sample-specific search over an adapter manifold.
- Adapter Composition: Parameter-space adapter merging can cause task interference, motivating conflict-aware and component-filtering approaches.AdaMerging and related methods learn task- or layer-wise coefficients from unlabeled data.
- Adaptive Fusion: LiST differs from mostly static fusion by performing continuous, training-free, sample-specific composition search over an adapter manifold.Its composition is adapted at test time rather than fixed globally.
- Test-Time Adaptation: Conventional test-time adaptation methods refine parameters or high-dimensional states using entropy, self-supervision, or consistency objectives.Recent approaches also reduce the adaptation space, including methods for constrained single-instance adaptation.
3 Methodology
LiST constructs a target-conditioned local simplex from task representations and searches low-dimensional, sample-specific fusion weights without changing the backbone or LoRA adapters. It evaluates candidates with constrained prompt-level energy and deploys them only when a safe acceptance rule is satisfied.
- Problem and Overview: LiST keeps the backbone and LoRA bank frozen, adapting each test input only through low-dimensional simplex-weight optimization.The online variable is a simplex-logit perturbation δ; all LoRA weights remain fixed.
- Task Representations and Local Simplex: Offline preparation builds target-conditioned local simplices from LoRA parameter anchors and prompt-level behavior vectors, then retrieves neighboring adapters.The resulting search space is restricted to the target adapter and a small set of related adapters.
- Task Representations and Local Simplex: The task-conditioned prior initializes the search and serves as fallback, while a local precision matrix defines a trust region around the target anchor.For unseen domains, LiST instead constructs the simplex from retrieved seen-domain adapters and uses a similarity-normalized prior.
- Branch-Preserving Fusion: LiST fuses adapter branch outputs rather than averaging LoRA parameters, preserving each branch’s structure while reducing adaptation to simplex weights.For each LoRA-augmented layer, the frozen base weight and adapter-specific low-rank branches are retained during fusion.
- Energy-Aware Search: Candidate weights are selected with a derivative-free prompt-level energy combining task uncertainty, prior deviation, geometric distance, and stochastic consistency.Energy components and constraint violations are normalized relative to the prior baseline or task statistics, and undefined consistency terms are omitted.
- Search and Safe Prediction: LiST deploys the searched candidate only when its energy improves over the prior and feasibility violations remain acceptable; otherwise, it uses the prior.The final prediction is generated by the frozen backbone with the accepted fusion weight.
4 Experiment
Experiments across multimodal and language benchmarks evaluate LiST against fusion and test-time adaptation baselines, with ablations examining its components, energy constraints, local simplex construction, and fallback rule. LiST achieves strong overall performance, including on unseen domains, while feasibility-aware search and target-conditioned retrieval reduce negative transfer.
- Experimental Setup: LiST is evaluated on multimodal MM-MergeBench and language GLUE using multiple pretrained multimodal and language backbones.The comparisons include model-fusion and test-time adaptation baselines.
- Main Results: LiST achieves the best overall multimodal performance, remaining close to or surpassing single-task LoRA on several datasets.The results indicate effective use of transferable knowledge while preserving target-task capability.
- Main Results: LiST also achieves strong language performance, outperforming single-task LoRA on most tasks while selectively reusing cross-task knowledge.The reported results support generalization to text-only LLM adaptation.
- Main Results: LiST achieves the best unseen-domain average on both evaluated multimodal models without target-domain LoRA adapters.It reuses seen-domain adapters through a local simplex, while the task-conditioned prior and feasibility-aware acceptance rule help reduce negative transfer.
- Ablations: Ablations show that sample-wise search, prior regularization, local geometry, and stochastic consistency jointly improve performance, with geometric constraints producing a larger gain.The full component combination achieves the best average result.
- Ablations: The full energy achieves the best average performance and lowest reported uncertainty, divergence, geometric-distance, and wrong-confident scores.The results characterize the energy as a feasibility-aware surrogate balancing confidence, stability, task identity, and local plausibility.
- Ablations: Joint anchor-and-behavior retrieval and the full local simplex outperform alternative construction strategies, with lower negative transfer and smaller geometric displacement.Global search underperforms local search, while random neighbors cause strong negative transfer.
- Ablations: The full acceptance rule achieves the best average performance and lowest negative transfer by requiring energy improvement together with low constraint violation.Its higher fallback rate reflects rejection of risky candidates in favor of the prior.
5 Analyses
LiST’s prompt-level energy tracks task-metric changes, but reliable deployment requires feasibility constraints and a safe acceptance rule.
- Candidates with larger energy decreases yield higher average metric gains, whereas energy-increasing candidates lead to negative changes.
- The prompt-level energy serves as an effective surrogate for test-time selection.
- Figure 2(b) represents simplex perturbations as points colored by their maximum normalized constraint violation.
- Some low-energy candidates with large constraint violations still degrade performance despite improving the task metric.
- The safe acceptance rule deploys a searched candidate only after sufficient energy improvement and feasibility; otherwise, LiST falls back to the task-conditioned prior.
- Prior, geometric, and stochastic-consistency constraints make energy decrease more reliable for final prediction.
6 Conclusion
LiST turns a frozen LoRA adapter bank into a target-conditioned local search space for label-free, sample-specific fusion. It uses branch-preserving, low-dimensional optimization with constrained energy selection and improves over static merging and conventional test-time adaptation baselines.
- LiST is a local-simplex framework for label-free test-time LoRA fusion.
- LiST converts a frozen adapter bank into a target-conditioned local search space and performs branch-preserving fusion.
- LiST optimizes low-dimensional simplex weights using prompt-level energy with prior, geometric, and consistency constraints.
- Experiments show that LiST improves over static merging and conventional test-time adaptation baselines while preserving task-specific adapter utility.
Limitations
LiST depends on related task-specific adapters and adds inference-time search cost; its surrogate energy may align less well with metrics for highly open-ended generation.
- LiST’s benefit may be limited when no useful neighboring adapters are available.
- Test-time search adds inference cost compared with directly using a single adapter, despite its low-dimensional search and frozen parameters.
- The prompt-level energy may be less aligned with task metrics for highly open-ended generation tasks.
Ethical Considerations
LiST uses publicly available pretrained models and benchmarks without new data collection, while inheriting standard misuse and bias concerns from base models.
- LiST operates on publicly available pretrained models and benchmarks and introduces no new data collection.
- Reusing frozen adapters reduces per-task compute compared with retraining and lowers the carbon footprint of multi-task deployment.
- The paper identifies no direct ethical risks specific to LoRA composition, while standard model misuse and bias concerns still apply.
A.1 Benchmark Details
The evaluation uses GLUE and MM-MergeBench, with MM-MergeBench further divided into seen and unseen domains. Unseen domains contain test samples only and provide no training samples.
- GLUE covers ten language-understanding tasks, including SST-2, MRPC, RTE, QNLI, QQP, CoLA, MNLI-m, MNLI-mm, STS-B, and WNLI.
- MM-MergeBench covers twelve multimodal tasks, including SciQA, Image, VQA, REC, OCR, VizWiz, Flickr, IconQA, AVQA, Image-R, S2W, and TabMWP.
- MM-MergeBench separates eight seen domains from four unseen domains: AVQA, Image-R, S2W, and TabMWP.
- Unseen domains contain only test samples and provide no training samples.
A.2 Hyperparameter Sensitivity
Sensitivity analyses identify moderate neighborhood size, sufficient stochastic evaluation, and a nonzero acceptance margin as important settings. Feasibility-aware energy selection improves the reliability of downstream performance gains.
- Neighbor Count: The best neighbor count is K = 3, while K = 5 performs nearly identically with slightly higher latency; larger neighborhoods hurt accuracy and negative transfer.
- MC Forward Count: Increasing stochastic evaluations from M = 2 to M = 8 improves performance and JS estimate stability, while M = 16 adds latency for marginal gains.
- Acceptance Margin: An acceptance margin of maccept = 3.0 achieves the best result while maintaining a moderate accept rate; zero margin causes low performance and high negative transfer.
- Energy–Performance Alignment: After high-violation candidates are filtered, larger energy decreases correspond more clearly to downstream metric gains.
B More Analyses
Feasibility filtering clarifies the relationship between prompt-level energy and downstream gains, while safe acceptance rules improve robustness through selective fallback. The full rule achieves the best average performance despite triggering more fallback.
- Energy–Performance Alignment: Removing high-violation candidates produces a cleaner positive association between energy decreases and metric gains.
- Energy–Performance Alignment: Prior, geometric, and stochastic-consistency constraints make prompt-level energy a more reliable test-time selection signal.
- Safe Acceptance Rules: Always accepting searched candidates improves over prior-only inference, indicating that test-time search can find useful sample-specific LoRA compositions.
- Safe Acceptance Rules: The full acceptance rule achieves the best average performance while triggering more fallback, rejecting unreliable candidates whose energy improvement lacks feasibility support.