Source-linked AI summary
Agents unlock new capabilities through Switching LoRA Adapters as a Tool (SLAaaT)
Kenneth Ge
TL;DR
Agents must compose specialized capabilities without losing generality across long trajectories. This paper enables mid-trace LoRA-adapter switching and finds that agents can autonomously route adapters to outperform fixed-adapter, fused-adapter, and subagent baselines.
Problem
Specialization can improve task performance while causing forgetting elsewhere, limiting evidence for composing capabilities across long agent trajectories.
Method
The agent repeatedly hotswaps specialized LoRA adapters within a trajectory, selecting among adapters through an autoswitch tool.
Results
The autoswitch agent outperformed all baselines, learned routing from one in-context example, and beat human heuristic switching on two tasks.
Takeaways & Limitations
The results establish that agents can effectively switch their own LoRA adapters under routing policies that sometimes exceed human switching decisions.
Takeaways & Limitations
Evaluation is limited to synthetic tasks, one base model and training seed, one sample per task, and a routing library with two adapters.
Abstract
from arXiv · showhide
Post-training can unlock new capabilities and improve performance on specialized tasks, but sometimes at the cost of catastrophic forgetting in other domains. This poses a problem in long agent trajectories that compose different capabilities. We reject this tradeoff by giving an agent a tool to switch between specialized LoRA adapters mid-trace. To test its effectiveness, we compose two synthetic coding tasks that are logically simple but require specialization. We find that this allows the model to solve problems it previously could not, that the model is able to switch autonomously (and find a new strategy that beats our human heuristic baseline on one task), and that this incurs an up to an 18x reduction in capability tax compared to an agent using only one specialized adapter. Our approach also substantially outperforms spawning subagents in both task capabilities and token usage.
1 Introduction
SLAaaT addresses the specialization–generality tradeoff by letting agents repeatedly hotswap LoRA adapters within a trajectory. The agent can learn when to switch, compose adapters to solve new synthetic tasks, and avoid sacrificing overall capacity.
- Motivation: Small models face a specialization–generality tradeoff: they absorb new capabilities readily but may forget older ones.Fine-tuning can match massive models on specific tasks, but specialist models typically have limited functionality and are switched only once in agentic settings.
- Approach: SLAaaT lets an agent repeatedly hotswap its own LoRA adapters within the same trajectory.The approach enables composing adapter swaps rather than restricting the agent to a single specialist configuration.
- Results: With one in-context example, the model learned when to use adapter switching, nearing human-heuristic performance on most tasks and exceeding it on two.The agent composed swaps to solve new synthetic tasks without sacrificing overall capacity.
2 Method
The method tests whether post-trained skills compose by using a verifiable, multi-stage synthetic pipeline whose subtasks require distinct, disjointly trained capabilities. It compares adapter switching with heuristic and subagent mechanisms across controlled processing tasks.
- Task design: The task design requires absent, hard-to-elicit skills, conditional multi-stage success, and fully disjoint training corpora.These properties isolate composition of post-trained skills rather than in-context elicitation or cross-subtask training overlap.
- Synthetic pipeline: The synthetic pipeline translates YAML into JSON or Fauxjson, then uses Python or Fauxthon to process the data and output XML.Fauxjson and Fauxthon cycle JSON and Python keywords or symbols, preserving the tokenizer while interfering with strong priors.
- Evaluation: 5 processing task types were tested, with 10 tasks per type and 10 YAML inputs each; success required correct translation and results on all 10 inputs.The task types were lossless conversion, counting, aggregation, sorting, and grouping, with binary scoring.
- Adapter selection: Autoswitch lets the model select base, Fauxthon, or Fauxjson, while the heuristic switches based on file extensions and subagents use host-written prompts.The harness switches adapters according to the model’s selection in Autoswitch.
3 Results
The results show that separately trained skills can compose through adapter switching, with autonomous routing discovering an effective strategy while largely avoiding the capability tax of single specialized adapters. Switching also uses fewer tokens than subagents.
- Composition: Each adapter performs well on its respective Fauxjson or Fauxthon task, while the fused adapter performs best on per-task knowledge but fails all agentic tasks.These tests removed agentic tool-calling confounds by having the model emit only a codeblock.
- Composition: 8/50 keystone success under the human heuristic nears the predicted 18/50 × 30/50 ≈11/50, indicating that switching largely preserves continuity across the trace.The result supports a close relationship between end-to-end success and success on each subtask.
- Routing: Autoswitch beats the heuristic on Fauxjson-Python by using the Fauxjson adapter to write Python code, suggesting autonomous discovery of effective routing policies.On Fauxjson–Fauxthon, Autoswitch scores 4/50 versus the predicted 37/50 × 30/50 ≈22/50, leaving substantial capability gains.
- Costs: Autoswitch uses 1.3x fewer tokens for the median success and 46.1x fewer tokens for the median failure than the subagent condition.Autoswitch incurs approximately the same capability tax as subagents, while both route unnecessarily.
- Costs: The fused and Fauxthon adapters incur significant capability tax, whereas switching largely avoids it.The tax is most severe on HumanEval+, where retention is 3.7% for the fused adapter and 7.3% for the Fauxthon adapter; routed conditions stay close to base.
4 Limitations and Future Directions
The evaluation is limited by synthetic tasks, narrow experimental coverage, and a two-adapter routing library. Future work includes reinforcement-learned routing, KV-cache reuse across adapters, and scaling to production.
- Limitations: The tasks are synthetic by design, isolating the composition question rather than establishing performance on broader tasks.The tasks are Fauxjson and Fauxthon.
- Limitations: All results use one base model, one training seed, and one sample per task, with binomial confidence intervals reported.The base model is Qwen3.6-35B-A3B.
- Limitations: The routing library contains only two adapters, so scaling to many adapters may produce different behavior.The paper identifies adapter-count scaling as a limitation.
- Limitations: The fixed file-extension human heuristic is not an optimal policy, qualifying comparisons that report beating the heuristic.The passage explicitly describes the heuristic as fixed rather than optimal.
- Future directions: Future work could use reinforcement learning for stronger routing, reuse KV caches across LoRA adapters, and scale the method to production.KV-cache reuse addresses the paper's biggest technical challenge.
A Training and Evaluation Details
Adapters were trained for one epoch on Qwen3.6-35B-A3B via the Tinker API. Evaluation used 20-turn attempts with a 4,096-token budget and no thinking, on held-out tasks with one shown YAML input and nine scoring inputs.
- Training: All adapters were trained for one epoch via the Tinker API on Qwen3.6-35B-A3B.The training procedure used the Tinker API and a single epoch for every adapter.
- Evaluation: Evaluation used 20 turns and a 4,096-token budget per attempt, with thinking disabled.These evaluation settings are specified in Section 2.3.
- Evaluation: Test tasks were held out from all training data.This separates evaluation tasks from the data used during training.
- Evaluation: Each condition included 50 tasks, each with one YAML input shown to the model and nine held-out inputs used for scoring.The displayed YAML input was distinct from the nine held-out scoring inputs.
B Compute
Training and evaluation used the Tinker API, with the Arrow baseline as the sole exception running locally on one NVIDIA DGX Spark. Producing Table 1 took roughly two days of end-to-end wall-clock time.
- Compute: Training and evaluation ran on the Tinker API, except for the Arrow baseline, which ran locally on a single NVIDIA DGX Spark.The Tinker API was provided by Thinking Machines.
- Compute: Roughly two days of end-to-end wall-clock time were required to produce the main results table, Table 1.
C Statistical Notes
Table 1 reports pass counts from 50 independent tasks in a single seeded run, with Wilson intervals indicating substantial sampling uncertainty. Despite this, fusion and pinned adapters fail outright, while autoswitch’s total of 116 exceeds subagent’s 59 by a margin well outside the intervals.
- Sampling basis: 50 independent tasks underpin every Table 1 cell, using one seed=5 and one sample per task.Each cell is a pass count out of n = 50 from a single run.
- Uncertainty: At most ≈14 percentage points is the 95% Wilson interval halfwidth, occurring at p = 0.5.The interval halfwidth shrinks toward the extremes, including 45/50: [79%, 96%] and 0/50: [0%, 7%].
- Comparative conclusions: 116 versus 59 is the autoswitch total compared with subagent’s, with conclusions separated by margins well outside these intervals.The stated conclusions include fusion and pinned adapters failing agentic tasks outright.
D Assets and Licenses
The work uses openly licensed models and datasets to create transpiled training data and releases the adapters, tooling, tasks, prompts, and evaluation harness for reproduction.
- Sources and releases: Training data is sourced from Qwen3.6-35B-A3B (Apache-2.0), MBPP (CC-BY-4.0), and Magicoder (MIT).The released assets include adapters, transpiler, task suite, prompts, and evaluation harness, with setup and reproduction instructions in the README.
E Broader Impact
SLAaaT makes it cheaper to equip small models with many narrow skills without a general capability tax, lowering the barrier to deploying capable agents on modest hardware. It could also ease adding undesirable specializations to open models, but the released synthetic-language adapters pose no incremental risk.
- Benefits: SLAaaT lowers the cost of giving small models many narrow skills without paying a general capability tax.This lowers the barrier to deploying capable agents on modest hardware.
- Risks: The same property could make it easier to add undesirable specializations to open models, although full fine-tuning already enables this.The released adapters target synthetic languages with no real-world use and pose no incremental risk.
F AI Usage
LLMs supported appendix drafting, LaTeX formatting, captions, and minor textual edits, without affecting the methodology, results, or conclusions.
- LLMs assisted with appendix drafting, LaTeX formatting, figure and table captions, and minor text edits, but did not affect the methodology, results, or conclusions.