PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Task Vectors for LLM Behavioral Control

Gabriel J. Perin, Lucas Boscaini, André Araujo, Nina S. T. Hirata

arXiv:2609.09054v1cs.LGcs.AI

TL;DR

Existing task vectors require fine-tuned checkpoints, making training-free behavioral editing desirable. TFTVs map activation steering directions into rank-one weight-space edits using forward-pass statistics, and experiments show strong, compositional trait control with utility largely preserved. The method supports persistent addition, subtraction, and multi-trait editing, while its performance depends on layer selection and retains a trait-control–utility trade-off.

  • Problem

    Task vectors typically require fine-tuning to discover meaningful weight-space directions, making post-training behavioral editing costly and less practical.

  • Method

    TFTVs map activation steering directions into rank-one weight-space edits using only forward-pass statistics, without auxiliary fine-tuning or optimization.

  • Results

    TFTVs achieve strong behavioral control across addition, subtraction, and multi-trait composition while matching or improving utility preservation relative to recent steering and editing methods.

  • Takeaways & Limitations

    TFTVs provide persistent, compositional post-training edits that amplify, suppress, and combine target behaviors without training.

  • Takeaways & Limitations

    Performance depends on the edited modules and layers, and stronger or composed edits retain a trade-off between trait control and utility.

Abstract

from arXiv · show

Task vectors enable post-training model editing by identifying semantically meaningful directions in weight space, typically computed as the difference between a fine-tuned model and its pretrained initialization. However, this reliance on fine-tuning makes discovering such directions costly and limits the practicality of post-training model editing. To address this limitation, we introduce Training-Free Task Vectors (TFTVs), a novel method to compute task-vector-like directions without requiring fine-tuning. Our method maps activation steering vectors to rank-one weight-space edits using only forward-pass statistics, while satisfying arithmetic properties that directly support learning via addition, forgetting via subtraction, and the composition of multiple edits. Empirically, we evaluate TFTVs on large language model behavioral control tasks and show that they consistently amplify, suppress, and compose target behaviors while preserving general knowledge and problem-solving skills. We also validate our method against other editing and steering baselines, experimentally demonstrating that TFTVs achieve stronger trait control with better or competitive utility preservation. We hope our work opens new directions for the community in post-training model editing and broader training-free model control. Code is available on the project website: tftv-llm.github.io.

1 INTRODUCTION

The paper introduces Training-Free Task Vectors (TFTVs) to overcome the need for fine-tuned checkpoints when editing model behavior. TFTVs map activation-space steering directions into compositional weight-space edits using only forward-pass statistics.

  • 1 INTRODUCTION: Task vectors encode behavioral directions in weight space and support addition, subtraction, and composition for post-training editing.They are traditionally computed as the difference between fine-tuned and pretrained weights.
  • 1 INTRODUCTION: TFTVs support learning through addition, forgetting through subtraction, and composition of multiple behavioral traits.These arithmetic operations provide persistent model edits rather than requiring inference-time intervention.
  • 1 INTRODUCTION: Fine-tuning-based task vectors require a completed fine-tuning run and a separate checkpoint for each new trait.This requirement limits their practicality for behavioral control.
  • 1 INTRODUCTION: TFTVs identify semantically meaningful weight-space directions from forward-pass statistics without auxiliary fine-tuning or additional optimization.The method maps activation steering directions into rank-one weight updates.
  • 1 INTRODUCTION: Across behavioral-control tasks, TFTVs provide stronger trait control while matching or improving utility preservation relative to recent steering and editing methods.The evaluated traits include evil, hallucination, and sycophancy.

2 RELATED WORK

Related work spans model merging, activation steering, and persistent weight-space editing. TFTVs connect these directions by turning steering information into training-free parameter edits.

  • 2 RELATED WORK: Model merging combines multiple models into one parameter set to preserve or improve source capabilities without additional inference cost.In language models, merging often integrates abilities from task-specific fine-tunings.
  • 2 RELATED WORK: Activation steering controls language-model behavior by injecting representation-space directions during the forward pass.These directions can be constructed with contrastive mean differences or sparse autoencoders and composed for combined behavior.
  • 2 RELATED WORK: Persistent model-editing methods apply direct weight-space changes, but existing approaches either require auxiliary training or target factual associations.Examples include learned editors such as KnowledgeEditor and MEND, and structured editors such as ROME and MEMIT.

3 TRAINING-FREE TASK VECTORS

TFTVs recover editable weight-space directions without auxiliary fine-tuning by mapping activation steering vectors into persistent parameter updates. Their arithmetic supports trait amplification, suppression, and composition, while adapting edits to layer-specific scales.

  • Task vectors traditionally use fine-tuned-minus-pretrained weights, whereas TFTVs seek comparable editable directions without an auxiliary fine-tuned model.
  • 3.3 PROPERTIES OF TFTV: Adding a TFTV increases its target trait, subtracting it decreases that trait, and adding multiple TFTVs increases multiple traits jointly.These properties are intended to reproduce task-vector arithmetic without training.
  • 3.1 PRELIMINARIES: STEERING VECTORS: A steering vector is added to a hidden representation to move model behavior toward a target trait, using contrastive prompt-completion data filtered for coherence and trait consistency.Positive and negative prompt sets produce D+ and D− examples whose activation means define the steering direction.
  • 3.2 BUILDING TFTVS FROM STEERING VECTORS: TFTVs map normalized steering directions into parameter-space updates using expected module inputs and the weight matrix's singular structure.The update is designed to induce the desired residual-space displacement directly, turning activation steering into a persistent edit.
  • 3.2 BUILDING TFTVS FROM STEERING VECTORS: The resulting module updates are rank-one, making edits compact and suitable for weight-space combination.Figure 2 summarizes the contrastive-prompt, judged-completion, steering-vector, and weight-edit pipeline.
  • 3.3 PROPERTIES OF TFTV: The update norm matches the underlying weight matrix's Frobenius norm, while a scalar coefficient controls edit magnitude across layers.This normalization adapts the construction to layer-specific parameter and activation scales.
  • 3.3 PROPERTIES OF TFTV: Composition is exactly linear only when composed steering directions share the same expected input; independently estimated traits are therefore only approximately linear.Experiments evaluate this practical regime and report that the resulting edits remain effectively compositional.

4 MAIN EXPERIMENTS

TFTV is evaluated for trait amplification, suppression, and composition across behavioral-control tasks while preserving general utility. Across these settings, it provides strong trait–utility trade-offs relative to training-free and fine-tuned baselines.

  • Experimental setup: The experiments evaluate evil, hallucination, and sycophancy on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, using trait, coherence, MMLU, and GSM8K metrics.The benchmark tests behavioral expression alongside general knowledge and mathematical problem-solving utility.
  • Learning via addition: Across six amplification settings, TFTV beats the strongest training-free baseline in trait score in five by 5.57–53.38 points, while keeping Llama MMLU within 0.15 and GSM8K within 3.71 points of the base model.For Qwen sycophancy, steering is 0.68 points higher; TFTV generally preserves utility better than fine-tuned methods.
  • Forgetting via subtraction: For suppression, TFTV reduces target trait scores by 7.74–77.28 points relative to steering while keeping Llama MMLU above the base model and GSM8K within 0.98 points in five of six settings.TFTV also achieves stronger suppression than Steer2Edit in three of six settings and the strongest suppression against fine-tuned baselines in four of six.
  • Composing multiple traits: Composed TFTV edits preserve or improve individual suppression effects, with lower trait scores indicating stronger suppression and higher MMLU or GSM8K indicating better utility.The compositions combine corresponding trait edits and compare them with the best single-trait edit within each composition.
  • Composing multiple traits: On Llama 3.1, the three-way composition improves sycophancy suppression by 12.68 points, while GSM8K changes by at most 0.83 points downward and up to 1.82 points upward.Against steering, the three-way edit adds 54.68 points of hallucination suppression while improving MMLU by 1.32 and GSM8K by 24.11 points.

5 ADDITIONAL ANALYSIS

Additional analyses test TFTV under out-of-distribution tasks and composed-edit comparisons. The reported effects largely transfer to OOD evaluations, while composed TFTV edits outperform training-free alternatives under matched settings.

  • Out-of-distribution evaluation: The OOD evaluation tests evil on Moral Stories, hallucination on TruthfulQA, and sycophancy on tasks covering opinions in NLP, philosophy, and politics.These tasks evaluate moral judgment, resistance to common misconceptions, and responses to user opinions.
  • Composed-edit comparison: Table 4 compares composed negation edits on Llama 3.1 using trait scores, MMLU, and GSM8K, with lower trait scores and higher utility scores preferred.Pairwise compositions report only target traits, and fine-tuned methods are excluded from the bold training-free comparison.
  • Qualitative composition examples: Figure 3 visualizes progressive suppression from no edit to evil, evil-plus-hallucination, and three-trait TFTV compositions under trait-eliciting prompts.The rows show how responses change as edits are progressively composed.
  • Out-of-distribution evaluation: TFTV moves 11 of 12 OOD scores in the expected direction: addition reduces morality and truthfulness while generally increasing sycophancy, and negation reverses these effects.The only exception is a 0.81-point decrease for addition on political sycophancy.
  • Composed-edit comparison: TFTV yields a stronger trait–utility trade-off than inference steering under matched layer and coefficient sweeps.The comparison is summarized in Figure 4.

6 CONCLUSIONS, LIMITATIONS, AND MISUSE

TFTVs provide persistent, compositional weight-space edits using forward-pass statistics, supporting behavioral control while largely preserving utility. Their performance depends on edited layers and modules, and stronger or composed edits retain a trait-control–utility trade-off; persistent trait manipulation also raises dual-use risks.

  • TFTVs map activation steering directions into rank-one weight-space edits using only forward-pass statistics, enabling persistent edits with addition, subtraction, and multi-trait composition.
  • TFTVs achieve strong behavioral control while largely preserving utility.
  • Performance depends on the edited modules and layers, making automatic selection an important direction for future work.
  • Stronger or composed edits retain a trade-off between trait control and utility.
  • Persistent trait amplification or suppression raises dual-use risks, including possible degradation of safety alignment, motivating safeguards and auditing.

A PROOF OF TFTV PROPERTIES

The appendix proves TFTV arithmetic properties using outer-product norm factorization, singular-vector orthonormality, and the fact that the normalization vector is independent of the steering vector. It also adopts sign(0) = 1, noting that exact inner-product zeros are rare in floating-point arithmetic.

  • A.1 Norm Matching: The proof establishes norm matching by factorizing the Frobenius norm of an outer product and relating the resulting vector norm to the weight matrix’s singular values.
  • The proof uses sign(0) = 1 and notes that exact inner-product zeroes are very rare under floating-point arithmetic.
  • A.1 Norm Matching: Orthonormal right singular vectors make the squared norm of qℓ equal to the sum of squared singular-value coefficients, regardless of their signs.
  • A.2 Steering: The steering proof treats qℓ as fixed for given Wℓ and μℓ, so the TFTV construction yields the stated steering property.
  • A.3 Linearity: Linearity follows for arbitrary β and γ because qℓ does not depend on the steering vector.

B EXPERIMENTAL DETAILS

The experiments evaluate TFTV behavioral control on Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct across evil, hallucination, and sycophancy, using judge-based trait and coherence metrics plus MMLU utility. Comparisons cover inference steering, Steer2Edit, fine-tuned Task Vectors, and CWS across addition, subtraction, and composition settings.

  • Evaluation framework: Experiments use Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, focusing on evil, hallucination, and sycophancy.
  • Evaluation framework: Trait manifestation and coherence are judged on 0–100 scales, with coherence serving as the generation-quality and utility metric.
  • Steering-vector construction: Each dataset contains 20 questions, five trait-inducing prompts, and five trait-suppressing prompts, with 100 prompts and 10 sampled completions per set.
  • Utility and decoding: Utility is assessed with zero-shot MMLU accuracy, while decoding uses temperature 1, top-p = 1, and at most 1000 new tokens.
  • Baselines: TFTV is compared with Persona Vector inference-time steering, Steer2Edit, Task Vectors, and Contrastive Weight Steering, selecting configurations with trait manifestation subject to coherence of at least 70.
  • Subtraction and composition: For subtraction and composition, evaluations use trait-eliciting prompts and the same judge and metrics as learning via addition.

C QWEN COMPOSING MULTIPLE TRAITS ADDITIONAL RESULTS

On Qwen 2.5 composition tasks, TFTV provides stronger suppression than inference steering on all nine measured trait scores while maintaining competitive utility. It outperforms other training-free methods overall and remains competitive with fine-tuned baselines.

  • TFTV achieves stronger suppression than inference steering on all nine measured trait scores while keeping MMLU within 1.25 points and improving GSM8K by 13.72–63.38 points.
  • Compared with Steer2Edit, TFTV has stronger suppression on seven scores, ties one, and obtains higher MMLU and GSM8K for every composition.
  • Task Vectors yields slightly higher utility but weaker suppression on all nine scores, while CWS matches or improves suppression on seven scores.
  • Overall, TFTV provides the strongest suppression–utility trade-off among training-free methods and remains competitive with fine-tuned baselines.

D TRAIT SCORE STANDARD DEVIATIONS

Trait-score variability is computed across prompt-level averages, not individual generations or runs. Intermediate trait means tend to show greater variability, while coherence and out-of-distribution results characterize practical robustness.

  • Standard-deviation computation: Trait-score standard deviations are computed over prompt-level averages after averaging runs for each prompt.They therefore measure variability across prompts rather than across individual generations or runs.
  • Variability pattern: Intermediate mean trait scores often have larger standard deviations than near-zero or near-saturated scores.The pattern is consistent with some prompts eliciting strong trait expression while others elicit little or none.
  • Coherence: Most coherence scores remain above 70, although aggressive composed edits can reduce generation quality.TFTV exceptions include Llama 3.1 sycophancy compositions with coherence scores of 62.29 and 61.14.
  • Additional behaviors: Additional TFTV edits substantially increase humorous and optimistic behavior while largely preserving MMLU and GSM8K performance.For humor, scores rise from near zero to 79.16 on Llama and 88.39 on Qwen.

F.3 ADDITIONAL MODEL ARCHITECTURES

TFTV generalizes beyond the primary Llama and Qwen evaluations to Gemma and Ministral, increasing target behaviors while largely preserving general capabilities. Module and layer ablations further show that edit location shapes the behavior–utility trade-off.

  • Additional model architectures: TFTV consistently increases all three target traits on Gemma and Ministral while largely preserving MMLU and GSM8K performance.These results extend TFTV beyond Llama and Qwen to additional architectures and model scales.
  • Module selection: Attention output projections provide the best behavior–utility trade-off, while MLP-only edits yield weaker control and greater coherence degradation.Combining attention and MLP edits does not consistently improve over attention-only edits, and MMLU remains largely stable except for a small hallucination-related drop.
  • Layer selection: Middle-layer edits produce the strongest trait manifestation, whereas later layers better preserve MMLU utility.Larger coefficients can further degrade coherence without substantially improving trait scores in some windows.
Loading 2609.09054v1…