PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Decide, Not to Reason: Parameter-Efficient Decision Operators via Low-Rank Activation Steering

Ran Li, Lei Chen

arXiv:2610.06950v2cs.LGcs.AI

TL;DR

Training useful skills for frozen language models currently entails a large operator and an RL pipeline. DecSteer uses a low-rank residual operator trained with behavior cloning; across tasks and backbones it matches or exceeds the larger RL-trained operator at much lower parameter cost. The paper also analyzes trainability, decision length, and where steering gains occur.

  • Problem

    Existing skill operators for frozen language models require 1.33M trainable parameters and a GRPO rollout pipeline, making skill training costly.

    Source: §1 · pp. 1–2

  • Method

    DecSteer applies a shared low-rank residual operator at eight decoder depths and trains it with answer-token behavior cloning.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across five tasks and three backbones, DecSteer matches or exceeds a 1.33M-parameter RL-trained operator using 23K–330K parameters and behavior cloning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The operator’s gains track base-model headroom, amplify its existing late-layer answer pathway, and allow skills to be added, interpolated, and hot-swapped at inference.

    Source: §1 · p. 2, §8 · p. 10

    The operator is best suited to single-forward-mapping tasks, binds to backbone width and depth, and cannot add accuracy on saturated bases.

    Source: §8 · p. 10

Abstract

from arXiv · show

Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline. We introduce DecSteer, a System-1 decision operator trained by behavior cloning that lowers this cost by roughly two orders of magnitude. The default operator uses 330K parameters to match a 1.33M-parameter operator trained with reinforcement learning, exceeds or achieve comparable performance, while collapsing 3,685-token deliberation into a 6-token decision with no loss in accuracy. A rank-4 variant with 23K parameters, 1/58 of the strongest published skill operator, suffices for SearchQA and near-suffices for LiveMath, where higher rank still helps; the same recipe transfers across five tasks and three backbones, with out-of-distribution gains persisting on LiveMath problems released months after training. The gap to prior work is trainability, and it is set jointly by initialization and architecture. The initialization of prior operators zeroes the gradient of both large factor matrices at the first optimization step, whereas our zero-initialized output projection inside a shared low-rank backbone receives a gradient immediately, which a gradient-flow probe confirms directly. The gain isn't chain-of-thought compression. 23 of 57 LiveMath points beat the base model's best-of-8 sampling, and a logit-lens probe shows the operator amplifies the answer along the model's existing late-layer pathway, not writing it earlier. Gains track the base model's headroom across 13 base-task pairs, and skills compose as approximately linear operators that can be added, interpolated, and hot-swapped at inference time.

1 INTRODUCTION

DecSteer asks whether a frozen language model’s skill can be made cheaper to train and deploy. It uses behavior cloning and a low-rank operator to achieve strong task results while shortening deliberation and exposing how initialization affects trainability.

1 INTRODUCTION

  • A 1.33M-parameter operator, rollout server, and minutes-to-hours of RL tuning make training skills costly.The paper frames cheaper skill training as a way to make a library of skills more feasible.Source: §1 · pp. 1–2
  • The prior BC control fails because its V =0 anchor freezes both large factor matrices initially, while DecSteer’s zeroed output projection receives gradient immediately.The paper attributes the difference jointly to initialization and architecture, and reports a gradient probe of the two paths.Source: §1 · p. 2, Figure 2 · p. 3
  • At least 23 of LiveMath’s 57 gain points exceed the base model’s best-of-8 reach; a logit-lens probe finds late-layer amplification, not earlier answer injection.Amplification follows the model’s existing late-layer output pathway.Source: §1 · p. 2
  • The gain tracks base-model headroom across 13 base–task pairs, and trained skills can be added, interpolated, or hot-swapped, with a forbidden zone for negative coefficients on mode-switching skills.The operator’s approximate linearity is reported across all 18 tested cells.Source: §1 · p. 2
  • Across five tasks and three backbones, DecSteer matches or exceeds a 1.33M-parameter RL-trained operator with 23K–330K parameters and behavior cloning.The default 330K configuration reaches 80.9 on LiveMath versus 79.0 for the RL baseline; a 6-token decision replaces a 3,685-token median reasoning chain.Source: Figure 1 · p. 1, §1 · p. 2

2 BACKGROUND: SKILL OPERATORS AND IDENTITY ANCHORING

The background describes skill operators as external residual-stream modules and explains how identity initialization preserves the base model before training. It distinguishes two behaviorally identical anchors by their optimization starting points.

2 BACKGROUND: SKILL OPERATORS AND IDENTITY ANCHORING

  • KV-Skill perturbs a frozen language model’s residual stream using shared down/up projections, a slot-retrieval bottleneck, and an input-dependent gate.With ds=256, r=32, the operator has 1,329,668 parameters and is trained with GRPO over rollout batches.Source: §2 · p. 3
  • Identity anchoring makes the operator an exact identity before training, either by zeroing KV-Skill’s V matrix or by zero-initializing the output projection.The anchors are behaviorally identical at initialization but differ as optimization starting points.Source: §2 · p. 3
  • The output-projection anchor guarantees zero perturbation regardless of upstream parameters, while V =0 guarantees it because the perturbation is proportional to V.The surrounding architecture and anchor together determine whether the large matrices train.Source: §2 · p. 3

3 DECSTEER: SYSTEM-1 DECISION OPERATORS

DecSteer applies a shared low-rank residual operator at eight decoder depths, using normalization and gates to retain depth-specific control. It is trained with answer-token behavior cloning and can be deployed as a steering vector or swapped skill bundle.

3 DECSTEER: SYSTEM-1 DECISION OPERATORS

  • At each of eight depths, DecSteer perturbs the block output using a shared low-rank backbone and a per-depth input-dependent gate.RMS normalization decouples the operator from residual-stream magnitude, while the gate restores depth- and input-specific amplitude.Source: §3 · pp. 3–4
  • The zero-initialized output projection makes the operator an exact identity, and depth sharing reduces the parameter count to 330,248 for dm=2560 and ds=64.Independent per-depth projections at the same rank require 2.62M parameters to reach the same score.Source: §3 · pp. 3–4
  • DecSteer at 0.33M parameters reaches or exceeds every 1.33M-parameter KV-Skill variant, while its median output falls from 3,685 to 6 tokens on LiveMath and from 289 to 11 on DocVQA.Accuracy is unchanged or higher; SearchQA already answers directly.Source: Figure 3 · p. 4
  • Training uses answer-token behavior cloning on the same gold examples as KV-Skill’s BC control, without rollouts, a verifier loop, or a reference policy.LiveMath training takes 87 steps and 12 minutes on one A100-40GB.Source: §3 · p. 4
  • At inference, the per-depth operator is materialized as a steering vector, while a scalar strength knob and bundle swapping support deployment-time control.The implementation uses no custom serving path and runs at full generation speed.Source: §3 · p. 4

4 EXPERIMENTS I: DECISIONS ARE CHEAP

Across evaluated tasks, behavior-cloned DecSteer delivers strong benchmark performance with substantially fewer parameters than RL-trained KV-Skill, while producing much shorter outputs. Results also include transfer to later LiveMath releases and gains beyond base best-of-8 sampling.

Main results

  • With 0.33M parameters and behavior cloning, DecSteer reaches 80.9 on LiveMath versus 79.0±3.9 for best KV-Skill RL, and 81.1 on SearchQA versus 80.1.It also reaches 94.8 on DocVQA and keeps saturated tasks within a point of the best RL numbers.Source: §4 · p. 4, Table 1 · p. 5
  • At 7.7K parameters, rank-1 already exceeds every published SearchQA number; the rank-4 LiveMath result is within the seed spread of the 1.33M RL baseline.Source: §4 · p. 5
  • On smaller backbones, the same BC recipe lifts LiveMath by +54 to +60 points and SearchQA by +15 to +37 points over base, beating a faithful KV-Skill reimplementation in 6 of 8 cells.There are no published RL baselines for these smaller-backbone comparisons.Source: §4 · p. 5

Training signal

  • On LiveMath, GRPO from scratch reaches 71.8, while GRPO started from the BC operator reaches 80.7, indicating that the starting point matters in this comparison.With GRPO matched to KV-Skill’s protocol, DecSteer matches published numbers using one quarter of the parameters.Source: §4 · p. 5

Inference cost

  • On LiveMath, median output falls from 3,685 reasoning tokens to a 6-token answer, with generation time dropping from 175 s to 4.8 s per batch (36×) and accuracy unchanged or higher.On DocVQA, output falls from 289 to 11 tokens (10.3× wall-clock); on SearchQA, direct-answer rate rises from 95.4% to 98.8%.Source: §4 · p. 5

Out-of-distribution generalization

  • On new LiveMath releases from April–June 2026, zero-shot DecSteer scores 79.0 versus base 25.4, with zero id overlap and gains of 50–61 points in each month.The figure reports a 7.5-point advantage over the official released KV-Skill RL artifact under the identical evaluation protocol.Source: §4 · p. 5, Figure 4 · p. 6

Training cost

  • DecSteer skills train in 12–33 minutes of BC on one GPU, while one baseline GRPO run costs 45–77 minutes plus a resident rollout engine.The entire comparison table of skills costs less to produce than one RL baseline.Source: §4 · p. 6

Gain analysis

  • The LiveMath gain exceeds base best-of-8 sampling: DecSteer scores 80.7 versus 57.3, and is correct on 43 problems where all eight base samples are wrong.The logit-lens readout finds answer readability only in the last 1–6 layers in both arms; the operator amplifies the existing late-layer signal by 3.4–6.4×.Source: Figure 6 · p. 7

5 EXPERIMENTS II: WHY IT WORKS

DecSteer’s gains depend on an optimization path that lets its operator parameters train, and low-rank operators can suffice despite rank-dependent needs across tasks. Controls indicate the gain comes from injecting answers along the model’s existing late-layer pathway, not merely shortening reasoning.

5.1 WHY THE SAME BC SIGNAL FAILED BEFORE

  • With the same signal and data, the zero-initialized output projection receives a first-step gradient, unlike the V = 0 anchor, whose large factor matrices have zero gradients initially.The comparison reaches 81.1 versus 68.4 on SearchQA and 80.9 versus 20.2 on LiveMath; initialization and architecture jointly determine trainability.Source: §5.1 · p. 7

5.2 HOW MUCH RANK DOES A DECISION NEED?

  • 23K parameters suffice for a rank-4 operator; rank-1 saturates SearchQA, while LiveMath gains 7.9 points from rank 2 to rank 4 and is flat beyond.The rank-4 solution is a different path from high-rank training: truncating it to rank 2 loses 4.8 points, and more training data does not move the knee.Source: §5.2 · p. 7

5.3 THE GAIN IS INJECTION, NOT COMPRESSION

  • At least 23 of LiveMath’s 57 gain points lie beyond the base model’s best-of-8 reach, indicating that the gain is not only suppressed reasoning.The operator reaches 80.7, while the base model’s best-of-8 sampling reaches 57.3; a direct-answer prompt recovers 15.3 points.Source: §5.3 · p. 8

5.4 WHERE THE ANSWER IS WRITTEN

  • Answer information becomes readable only in the last 1–6 layers, with the onset at the same layer in 6 of 6 backbone–task combinations.The operator amplifies the existing late-layer answer direction by 3.4× on the 4B base, 5.1× on Llama-3.2-3B, and 6.4× on the 0.8B base.Source: §5.4 · p. 8

6 EXPERIMENTS III: WHEN TO STEER

Measured skill gains increase with base-model headroom, and skill operators can be combined under a positive-coefficient boundary. Their strength is also adjustable at inference without retraining.

6.1 AN EMPIRICAL HEADROOM TREND

  • Across 13 base–task pairs on three backbones, skill gain rises with base headroom; saturated 4B bases show gains within about a point, while weaker bases gain +2.4 to +10.0 points on those tasks.The authors treat the trend as an empirical screening heuristic, not a law; the 13 points are not independent.Source: §6.1 · pp. 8–9

6.2 AN OPERATOR ALGEBRA FOR SKILL LIBRARIES

  • Across 18 combination-study cells, adding, interpolating, or reordering skills preserves the anchor skill’s full gain, but a negative coefficient on a mode-switching skill reverses the switch.Reordering changes decisions by at most 4 points; the authors recommend positive coefficients for composition and reserve subtraction for content-shifting skills.Source: §6.2 · p. 9

6.3 STRENGTH AS AN INFERENCE-TIME KNOB

  • The inference-time strength scalar α modulates performance without retraining; a uniform scale of 1.0 supports all reported conclusions.LiveMath prefers 0.75 and CSQA 0.5, each by about one point over scale 1.0; the authors characterize α as a deployment control, not a hidden tuning axis.Source: §6.3 · p. 9

7 RELATED WORK

DecSteer relates to work on portable skills, activation steering, and System-2-to-System-1 distillation, while differing in how it represents and deploys learned decisions.

Externalizing skills

  • Unlike portable skill files, trajectory-level lessons, and behavioral soft prompts, DecSteer uses a residual-stream operator that composes algebraically.The paper identifies Han et al. (2026) as its closest work and reference point, retaining its injection site and identity-anchor discipline while using a smaller operator trained with behavior cloning.Source: §7 · p. 9

Activation steering and parameter-efficient adaptation

  • Where activation-steering work often targets concepts or refusal behaviors, DecSteer steers task decisions using answer-token training and a task-level verifier.The paper also distinguishes prompt and continuous-prompt methods, which act through the input or prompt rather than the residual stream.Source: §7 · p. 9

Distilling System 2 into System 1

  • Unlike approaches that distill reasoning into weights or route when to think, DecSteer keeps the frozen model’s System-2 capability and places compiled decisions in a swappable, composable side operator.The paper describes that operator as having 23K parameters.Source: §7 · p. 10

8 LIMITATIONS AND CONCLUSION

DecSteer is suited to tasks with a single forward mapping; when reasoning itself carries information, gains depend on the base model’s headroom. The conclusion attributes its efficiency to identity anchoring, amplification of the model’s existing answer pathway, and low-rank operators.

8 LIMITATIONS AND CONCLUSION

  • The operator’s natural habitat is single-forward-mapping tasks; when reasoning itself carries information, gains are governed by the headroom trend.The base model can still be prompted to deliberate, and saturated bases should not be expected to gain accuracy; the operator’s value there is inference speedup.Source: §8 · p. 10
  • Operator parameters bind to backbone width and depth, so transferring a skill across model sizes requires retraining on the target at a cost of minutes.Negative coefficients are safe for content skills but not for mode-switching skills.Source: §8 · p. 10
  • 23K parameters, 12 minutes of behavior cloning, and a 6-token output reproduce a 1.33M-parameter RL pipeline across five tasks and three backbones, with gains persisting out of distribution.Rank 4 suffices on SearchQA and near-suffices on LiveMath, where higher rank still helps.Source: §8 · p. 10
  • Identity anchoring at the output projection keeps the large matrices trainable, while the operator amplifies the base model’s own late-layer answer pathway.The headroom trend indicates where this amplification will pay.Source: §8 · p. 10
  • Experiments use public models and datasets with Han et al. (2026)’s splits, prompts, decoding, and evaluators; code and detailed protocols are supplied in the supplementary material.Appendix A and Appendix D specify architectures, initialization, training hyperparameters, seeds, identity-anchor checks, contamination screens, and strength sweeps.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL SETUP

The experiments evaluate DecSteer on five tasks and three backbones using task-native metrics and a shared evaluation protocol. Protocol checks, multi-seed runs, strength sweeps, and controlled comparisons define how to interpret the reported results.

A EXPERIMENTAL SETUP

  • Experiments cover LiveMath, SearchQA, CommonsenseQA, OpenBookQA, and DocVQA on Qwen3.5-4B, Llama-3.2-3B-Instruct, and Qwen3.5-0.8B, using each task’s native evaluator.LiveMath also includes 2026-04/05/06 releases; splits and metrics follow KV-Skill exactly.Source: A EXPERIMENTAL SETUP · p. 13
  • The operator-strength scalar α was swept across three seeds per task, with mild overshoot on LiveMath and CSQA but none on SearchQA or OpenBookQA.The main table uses each dataset’s best strength; uniform scale 1.0 changes no conclusion.Source: A EXPERIMENTAL SETUP · p. 13, Figure 8 · p. 14
  • Identity anchors were verified item-by-item, LiveMath train–test overlap was zero, and the imported KV-Skill RL artifact reproduced its self-reported score within one problem.Gold labels were near-uniform, predictions showed no collapse, and budget audits were among the integrity checks.Source: A EXPERIMENTAL SETUP · p. 13, Table 2 · p. 14
  • Cross-backbone blocks have no other published baselines, the 0.8B LiveMath base was not measured, and rank scans cover only LiveMath and SearchQA.KV-Skill operators use 0.54–1.59M parameters against 0.13–0.40M for DecSteer in those blocks.Source: A EXPERIMENTAL SETUP · p. 14
  • Key arms use three training seeds with mean ± SD, and run-to-run variance on small-data tasks is dominated by bf16 drift after the first steps.Conclusions therefore use multi-seed aggregates rather than single runs.Source: A EXPERIMENTAL SETUP · p. 14
  • Behavior cloning trains on answer tokens only; GRPO comparisons follow KV-Skill’s hyperparameters, with βent set to 0 because the published bonus diverged on long-CoT tasks.BC uses 35 LiveMath examples for 12 minutes, 400 SearchQA examples for 33 minutes, and 107 DocVQA examples for 29 minutes; Table 3 reports the multi-seed comparison under the identical protocol.Source: A EXPERIMENTAL SETUP · p. 15, Table 3 · p. 15

B ADDITIONAL RESULTS

Additional controlled comparisons find that behavior cloning matches the RL reference with fewer parameters, while a faithful cross-carrier comparison favors DecSteer. Calibration improves on SearchQA, and auxiliary controls characterize training-signal stability.

B ADDITIONAL RESULTS

  • Under GRPO from scratch on LiveMath, performance is 71.8 versus 80.7 from the BC warm start, indicating the starting point matters more than the algorithm; under KV-Skill’s main protocol, DecSteer matches published numbers with 1/4.02 parameters.Source: B ADDITIONAL RESULTS · p. 15
  • DecSteer is higher in 6 of 8 cells and never lower in a faithful cross-carrier comparison, using 0.13–0.40M parameters versus 0.54–1.59M.The comparison uses identical data, prompts, and decoding and reimplements KV-Skill’s V =0 identity initialization.Source: B ADDITIONAL RESULTS · p. 15
  • Inference accelerates generation by up to 36× on LiveMath and 10.3× wall-clock on DocVQA; the BC recipe trains in minutes without rollouts.The table’s GRPO rows are protocol-matched controls from Table 3.Source: Table 5 · p. 16
  • On SearchQA, ECE drops on all three backbones, while answer-ranking AUROC improves on 4B and Llama and stays flat on 0.8B.Figure 9 reports ECE changes of 0.29 to 0.12 on Llama, 0.18 to 0.16 on 4B, and 0.14 to 0.12 on 0.8B; AUROC changes are 0.71 to 0.78 and 0.64 to 0.67 on 4B and Llama.Source: B ADDITIONAL RESULTS · p. 15, Figure 9 · p. 16

C RL STABILITY: THE ENTROPY-BONUS DIVERGENCE

  • The RFT control indicates direct-answer collapse is structural rather than an artifact of short targets, while the GRPO trace shows the RL baseline requires careful recipe control that BC avoids.Source: C RL STABILITY: THE ENTROPY-BONUS… · p. 16

D RANK SCAN AND TRAINABILITY DETAILS

The rank scan shows task-dependent capacity needs, while controls indicate that low-rank solutions are distinct optimization paths rather than compressed versions of higher-rank ones. DocVQA transfer and saturated-base results provide additional scope checks.

D RANK SCAN AND TRAINABILITY DETAILS

  • LiveMath accuracy rises from 65.0 at rank 1 to 74.8 at rank 4, while SearchQA EM is already 81.1 at rank 1 and saturates across ranks.LiveMath’s rank-2-to-rank-4 increase is 7.9 points; the full scan reports 77.4 at rank 16, while SearchQA reaches 81.7 at rank 16.Source: D RANK SCAN AND TRAINABILITY DETAILS · p. 16
  • All ranks fit training data to zero loss, and increasing LiveMath training examples from 35 to 53 leaves the rank knee unchanged, so data and fit are not the bottleneck.Gradient statistics grow smoothly with ds and show no inflection at the knee; truncating a rank-4 solution to rank 2 drops zero-shot performance by 4.8 points.Source: D RANK SCAN AND TRAINABILITY DETAILS · p. 16
  • On DocVQA, the identical BC recipe raises 4B ANLS from 92.5 to 94.8 and 0.8B ANLS from 57.9 to 81.8; on saturated 4B bases, skill arms remain within about a point of the base.The DocVQA gains are higher in all three seeds for 4B, with mild overshoot at full strength; saturated-base paired flips are nearly symmetric.Source: Figure 10 · p. 17

E DIAGNOSTIC PROBES

The probes distinguish tasks with remaining skill headroom from saturated tasks using commitment-curve height, while gate activations do not provide per-sample confidence. Controls also show that direct answers persist with full-chain-of-thought targets and that removing the entropy bonus restores RL stability.

E DIAGNOSTIC PROBES

  • Commitment-curve height at k=0 separates injectable tasks (≥0.20: LiveMath 0.68, SearchQA 0.21) from saturated tasks (≤0.15), but is only a secondary heuristic.Curve height tracks eventual skill gain across four probed tasks; its height, not its onset, is informative.Source: E DIAGNOSTIC PROBES · p. 17, Figure 12 · p. 18
  • Gate activations vary only 1.7–3.6% across samples and correlate with prompt length (ρ up to 0.9), so they cannot serve as per-sample confidence.The authors recommend a trained head or external signals such as self-consistency if internal skill confidence is needed.Source: E DIAGNOSTIC PROBES · p. 17

F LLM USAGE

  • Large-language-model tools assisted with manuscript editing and formatting; the authors report that the research and claims were their own.They state that all reported numbers came from the described experimental pipeline.Source: F LLM USAGE · p. 17
  • Full-chain-of-thought targets still produced 6-token direct answers without accuracy loss, while removing the entropy bonus restored RL stability.The published entropy bonus overwhelms the KL anchor by roughly 100× as entropy rises, with reward falling from 0.86 to 0.08.Source: Figure 11 · p. 18

Related papers

Loading 2610.06950v2…