PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation

Dongfang Zhao

arXiv:2609.11739v1cs.CLcs.AIcs.LG

TL;DR

Verbose preference-aligned responses increase autoregressive serving costs, motivating token-efficient post-training without sacrificing utility. LOCUS selects a task-aware low-rank adaptation subspace while retaining the native preference objective and frozen backbone. On Anthropic HH-RLHF, it reduces continuation length by up to 39.84% on Pythia-2.8B and 14.87–17.58% on Qwen2.5-3B while updating only 0.24–0.28% of parameters and leaving the internal preference diagnostic materially unchanged.

  • Problem

    Preference alignment can inflate response length, increasing serving cost, while existing length-control methods modify the objective or inference interface.

  • Method

    LOCUS selects a task-aware low-rank adaptation subspace under a utility constraint, training adapters with the native preference objective while freezing the pretrained backbone.

  • Results

    LOCUS reduces continuation length by up to 39.84% on Pythia-2.8B and by 14.87–17.58% on Qwen2.5-3B while updating 0.24–0.28% of model parameters.

  • Takeaways & Limitations

    Low-rank update parameterization can improve token efficiency while preserving the native preference objective and producing no material change in the internal preference diagnostic.

  • Takeaways & Limitations

    Evaluation covers only two roughly 3B decoder-only backbones under greedy decoding; larger models and sampling-based inference remain untested.

Abstract

from arXiv · show

Large language model serving costs scale directly with output sequence length, yet standard preference alignment often inflates response verbosity without improving utility. We study whether the parameterization of post-training updates affects generation length: low-rank subspaces alter sequence length without modifying the alignment loss. We present LOCUS, a method that selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint. Within this subspace, post-training retains the native preference objective with a frozen backbone. Across Anthropic HH-RLHF dialogue preferences, we evaluate two $\sim$3B decoder backbones, Pythia-2.8B and Qwen2.5-3B, against protocol-matched full-parameter DPO and DrDPO branches and the released SamPO checkpoint. LOCUS reduces continuation length by up to 39.84\% on Pythia-2.8B and by 14.87--17.58\% on Qwen2.5-3B while updating only 0.24--0.28\% of model parameters, with no material change in the internal preference diagnostic.

1 Introduction

Preference alignment can increase response verbosity, raising serving cost without necessarily improving utility. LOCUS addresses this by selecting a task-aware low-rank update subspace while preserving the native alignment objective, achieving shorter generations with nearly unchanged preference accuracy.

  • Motivation: Serving cost grows with autoregressively generated tokens because each additional token requires another full-depth forward pass and expands key-value cache storage.The motivation is to produce concise generations without sacrificing answer quality.
  • Motivation: Prior length-control approaches modify the objective or inference interface, whereas LOCUS studies output-length effects through update parameterization instead.Heuristic penalties can distort preference learning, and brevity prompting can degrade under distribution shifts.
  • Method: LOCUS selects a task-aware low-rank adaptation subspace to minimize output-token cost subject to a utility constraint while retaining the native post-training objective.The design variable spans adapter rank, module placement, layer scope, and training checkpoint step.
  • Empirical results: LOCUS reduces continuation tokens by 20.73% under DPO, 25.29% under DrDPO, and 39.84% under SamPO on Pythia-2.8B, with accuracy changes no greater than 0.13 percentage points.The Qwen2.5-3B DPO and DrDPO comparisons show 14.87% and 17.58% reductions, respectively.
  • Empirical results: The method updates only 0.24–0.28% of model parameters across evaluations on Pythia-2.8B and Qwen2.5-3B.Evaluation uses protocol-matched full-parameter DPO and DrDPO branches and the released SamPO checkpoint.
  • Contributions: The paper analyzes trainable-parameter counts and exact inference-time equivalence between merged and unmerged low-rank updates.This complements the empirical evaluation with parameter accounting and deployment analysis.

2 Related Work

Related work covers preference alignment, output-length bias, and parameter-efficient fine-tuning. Existing conciseness methods adjust rewards, objectives, or prompts, while LOCUS uses low-rank parameterization to target token efficiency without changing the native preference objective.

  • Preference Optimization and Alignment: Preference alignment trains language models from pairwise comparisons, including DPO and later variants that alter preference-learning objectives.DPO removes the separate reward-modeling stage through a closed-form policy objective.
  • Length Bias and Output Conciseness: Preference-optimized language models can develop length bias, producing unnecessarily long responses to maximize preference rewards.LR-DPO and SamPO address this through score adjustment or reference-length down-sampling.
  • Length Bias and Output Conciseness: Prior conciseness methods use length regularization, reward rescaling, or heuristic prompting, which can distort the target preference distribution.LOCUS instead seeks token efficiency through subspace parameterization without modifying the native objective.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning updates a small subset of parameters while freezing base weights, including through adapters, prefix tuning, prompt tuning, and low-rank adaptation.LOCUS builds on this family by applying low-rank adaptation to preference post-training.

3 Methodology

LOCUS freezes the pretrained backbone, trains candidate low-rank adapters with the native preference objective, and selects the shortest candidate that satisfies a utility constraint. The selected adapter can then be verified and deployed either merged into the backbone or separately for multi-tenant serving.

  • Objective-Preserving Adaptation: The method attaches low-rank adapters to selected modules, freezes the backbone, and trains them with the original preference objective without adding a length penalty.Candidate configurations vary by rank, target modules, target layers, and training step.
  • Task-Aware Constrained Selection: LOCUS selects the shortest candidate adapter whose preference utility remains within the allowed tolerance of the protocol-appropriate baseline.Candidates are evaluated on selection data and, when available, checked on a separate confirmation split before held-out testing.
  • Task-Aware Constrained Selection: LOCUS defines continuation length under greedy decoding as the cost metric and chosen-versus-rejected preference accuracy as the task utility.These quantities determine feasibility and selection for each task.
  • Task-Aware Constrained Selection: The utility tolerance is set to 1.0 percentage points below the protocol-appropriate baseline in this work.The baseline utility is measured on the same task and serves as the constraint reference.
  • Complexity and Serving Modalities: Low-rank adaptation exposes r(d1 + d2) trainable entries for a targeted module instead of d1d2 full-parameter entries, reducing gradient and optimizer-state storage.The reported primary adapters contain 7.86M entries for Pythia-2.8B or 7.37M for Qwen2.5-3B under a two-state optimizer accounting.
  • Complexity and Serving Modalities: Merged adapters add no adapter-induced latency or parameter overhead in single-tenant serving, while separate adapters support multiple tasks with unbenchmarked bookkeeping and memory overhead.Total training memory and FLOPs still require separate assessment because the frozen backbone remains computationally active.

4 Analysis

LOCUS’s low-rank parameterization reduces exposed trainable parameters while preserving the native objective, and its adapter can be merged exactly at inference under stated conditions.

  • 4.1 Trainable Parameterization: r(d1 + d2) trainable factor entries replace d1d2 full-parameter entries, with fewer entries exactly when r(d1 + d2) < d1d2.The proposition distinguishes exposed factor entries from the intrinsic dimension of the represented update set.
  • 4.1 Trainable Parameterization: The parameter-count identity does not establish changes in generation length, preference accuracy, total memory, or training speed.Those quantities require empirical measurement rather than inference from parameter reduction alone.
  • 4.1 Trainable Parameterization: 0.24–0.28% of model parameters are updated by the reported Pythia-2.8B and Qwen2.5-3B adapters.The cited configurations use 7,864,320 and 7,372,800 trainable parameters, respectively.
  • 4.2 Inference-Time Merge Equivalence: With adapter dropout disabled, unmerged and merged LoRA representations produce exactly the same linear-module output for every input.The merged weight is W0 + (α/r)BA.
  • 4.2 Inference-Time Merge Equivalence: Equivalent module outputs do not guarantee equivalent deployment costs, because unmerged adapters add computation and bookkeeping.Finite-precision or quantized implementations may also introduce small numerical differences.

5 Evaluation

LOCUS evaluates task-aware low-rank adaptation against matched preference-optimization baselines across two roughly 3B decoder backbones and multiple tasks. It consistently reduces continuation length while preserving internal preference diagnostics, using a small trainable parameter subset.

  • 5.2 Baseline Comparisons: 20.73% and 25.29% continuation-token reductions under DPO and DrDPO on Pythia-2.8B changed preference accuracy by −0.01 and −0.13 percentage points.LOCUS used 0.2826% of parameters for the DPO comparison and selected step 750 for both comparisons.
  • 5.2 Baseline Comparisons: 39.84% fewer continuation tokens were achieved on Pythia-2.8B with SamPO, while preference accuracy remained unchanged at 53.52%.The reduction was from 132.77 to 79.88 tokens using a low-rank adapter.
  • 5.3 Rank Sensitivity: Increasing adapter rank changed measured length effects from an 11.74% increase at rank 4 to a 25.10% reduction at rank 32 without a monotonic diagnostic trend.Ranks 16 and 32 produced 15.95% and 25.10% reductions, while the diagnostic was 45.31% through rank 16 and 46.88% at rank 32.
  • 5.4 Target-Module Ablation: Attention-All adaptation produced a 15.95% reduction, compared with 13.88% for All-Linear adaptation despite 2.7× more trainable parameters.The internal preference diagnostic remained 45.31% across all tested target-module placements.
  • 5.5 Cross-Task Generality: Across dialogue, safety, and instruction-following evaluations, mean continuation length fell by 20.73%, 25.29%, and 79.97%, respectively.The corresponding internal preference diagnostics changed by −0.01, +1.17, and +13.67 percentage points.
  • 5.6 Cross-Backbone Transfer to Qwen2.5-3B: 14.87% and 17.58% token reductions transferred to Qwen2.5-3B under controlled DPO and DrDPO comparisons, with preference-accuracy changes of −0.05 and −0.09 percentage points.The adapter updated 0.2383% of Qwen2.5-3B parameters, and both comparisons met the stated utility criterion.

6 Conclusion

LOCUS freezes the pretrained backbone and selects a low-rank adaptation subspace while preserving the native preference objective. Across matched HH-RLHF comparisons, it reduces continuation length with fewer than 0.3% of model parameters.

  • LOCUS reduced continuation length by 25.29% and 17.58% for protocol-matched DrDPO on Pythia-2.8B and Qwen2.5-3B, respectively, while updating fewer than 0.3% of parameters.It also achieved a 39.84% reduction with continued SamPO adaptation on Pythia-2.8B.
  • LOCUS freezes the pretrained backbone and selects a low-rank adaptation subspace while preserving the native preference objective.

Limitations

The evaluation is limited to two roughly 3B decoder-only backbones under greedy decoding, with larger models and sampling-based inference untested. The candidate pool also uses discrete ranks and coarse projection groupings, and cross-task evaluations do not predict unseen-task reductions.

  • The study covers only Pythia-2.8B and Qwen2.5-3B under greedy decoding; larger models and sampling-based inference remain untested.
  • The candidate pool uses four discrete ranks and coarse projection groupings, leaving continuous rank allocation, finer layer selection, and gradient-informed pruning outside the study.
  • Cross-task evaluations cover dialogue, safety, and instruction following but do not predict reductions on an unseen task.

A Formal Proofs of Propositions 1 and 2

The formal appendix proves parameter-count reduction for low-rank updates and exact inference equivalence when an adapter is merged into a linear module. It also limits what parameter reduction and algebraic equivalence establish empirically.

  • Proposition 1: LoRA exposes r(d1 + d2) trainable entries instead of d1d2 full-parameter entries, with represented updates constrained to rank at most r.
  • Proposition 1: The factorization has change-of-basis redundancy, giving a local represented-manifold dimension of r(d1 + d2 −r), while implementation-level trainable storage remains r(d1 + d2).
  • Proposition 1: Low-rank parameterization strictly reduces exposed parameters precisely when r(d1 + d2) < d1d2.
  • Scope: Parameter-count reduction alone does not imply changes in generation length, preference accuracy, memory, or training speed, which are measured empirically.
  • Proposition 2: Merged and unmerged inference compute the same function for a deterministic downstream network when adapter dropout is disabled and the scalar α is fixed.

B Experimental Reference Tables

The reference tables distinguish backbone-specific attention parameterizations, comparison-branch update schemes, and dataset roles across training, development, confirmation, and evaluation splits.

  • Pythia uses fused query-key-value projections while Qwen uses separate projections, exposing 7.86M and 7.37M attention-adapter parameters, respectively.
  • Controlled DPO and DrDPO branches share a full-parameter SFT checkpoint and native objective, whereas LOCUS updates a low-rank adapter and SamPO uses a separate released checkpoint.
  • Controlled HH comparisons select with development data before evaluation on 8,552 frozen test pairs, while SamPO and additional datasets use their specified 256-example evaluation scopes.

C Additional Experimental Results

This appendix consolidates primary Anthropic HH comparisons across two backbone families and three preference objectives, alongside cross-task development evaluations. It reports token changes and internal chosen-versus-rejected preference diagnostics without adding an evaluation protocol.

  • Primary comparisons: Table 4 consolidates primary Anthropic HH comparisons across both backbone families and three preference objectives.DPO and DrDPO share an SFT starting checkpoint, while LOCUS adapts the released SamPO checkpoint.
  • Cross-task evaluations: Table 5 reports cross-task token and accuracy comparisons using one frozen HH test result and two 256-example development evaluations.The evaluations cover Anthropic Helpful and Harmless, Harmless, and Orca DPO rows.
  • Measurement scope: Preference values in these comparisons are internal chosen-vs-rejected diagnostics rather than external quality measurements.The appendix describes measured token and internal preference-diagnostic changes without introducing an additional evaluation protocol.
  • Backbones: The evaluated backbones are Pythia-2.8B and Qwen2.5-3B, with architecture-specific LOCUS parameterizations.Pythia uses GPT-NeoX-style fused attention modules, whereas Qwen uses separate projection modules and grouped-query attention.
  • Experimental organization: The appendix organizes the experimental setup across backbone parameterizations, comparison branches, datasets, development splits, and frozen evaluation sets.Tables 1–3 cover backbones, full-parameter and low-rank branches, and evaluation data organization.
Loading 2609.11739v1…