PaperlayerView exact revision on arXiv

Source-linked AI summary

Tuning Language Models by Proxy

Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, Noah A. Smith

arXiv:2401.08565v4cs.CL

TL;DR

Large language models are costly to adapt and may expose only predictions rather than weights, creating a need for efficient customization. Proxy-tuning uses a smaller tuned and untuned model to steer a larger model at decoding time, closing 88% of the tuning gap for Llama2-70B and extending to code, task adaptation, and GPT-3.5 temporal adaptation.

  • Problem

    Adapting increasingly large language models is resource-intensive and may be impossible when their weights are private.

  • Method

    Proxy-tuning adds the prediction difference between a tuned small expert and an untuned anti-expert to a larger black-box model’s output logits.

  • Results

    Proxy-tuning closed 88.1% of the performance gap between LLAMA2-70B and its directly tuned CHAT version across five evaluation settings.

  • Takeaways & Limitations

    Small tuned models can efficiently customize larger pretrained or proprietary models through decoding-time guidance when output probabilities are available.

  • Takeaways & Limitations

    The method assumes access to the base model’s output logits and, in its standard formulation, a shared vocabulary with the proxy models.

Abstract

from arXiv · show

Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.

1 Introduction

Proxy-tuning addresses the cost and privacy barriers to adapting large language models by steering them at decoding time with predictions from a smaller tuned model. Across instruction-following, domain adaptation, task finetuning, token-level analysis, and black-box temporal adaptation, it approaches or improves on direct tuning while retaining large-model benefits.

  • Core idea: Proxy-tuning applies a small model’s tuning signal to a larger black-box model using only output predictions, without accessing the larger model’s weights.The method contrasts a tuned small expert with its untuned anti-expert and uses their prediction difference to guide the larger base model.
  • Domain and task adaptation: 17–32% absolute coding-benchmark improvement came from proxy-tuning LLAMA2-13B with CODELLAMA-7B, while question-answering and math proxy-tuning improved LLAMA2-70B by 31% over untuned 70B and 9% over the tuned 7B model.The method also enabled untuned models to follow strict syntactic constraints learned by the small expert.
  • Analysis: Proxy-tuning most strongly promotes reasoning and stylistic tokens, and optional runtime guidance scaling lets users control the amount of steering.These token-level effects are consistent with evidence that alignment mainly affects style rather than knowledge.
  • Black-box adaptation: Using only top-5 log probabilities, proxy-tuning improved GPT-3.5 accuracy on recent-event questions, extending the method to a truly black-box model.The result supports decoding-time customization when model weights are private but output probabilities are available.

2 Method

Proxy-tuning steers a pretrained model by adding the logit difference between a tuned small expert and an untuned small anti-expert to the base model’s logits. The method requires output-logit access and a shared vocabulary, but does not modify or access the base model’s parameters.

  • Requirements: The base model requires full-vocabulary output logits, while the small tuned and untuned models need only share its vocabulary and need not belong to the same model family.The method is therefore compatible with black-box models when their output distributions are accessible.
  • Empirical setup: Proxy-tuning closes 91.1% and 88.1% of the performance gap with directly tuned CHAT models at 13B and 70B, respectively.The proxy-tuned larger model also outperformed the small expert in every scenario except a 0.1% ToxiGen difference.
  • Proxy-tuning mechanism: At each decoding step, the method adds the expert logits minus the anti-expert logits to the base model logits before applying softmax.This transfers the smaller model’s learned tuning direction to the larger base model.
  • Design goal: The objective is to approach the directly tuned version of a base model without modifying or accessing its parameters.The method can be viewed either as applying a small-scale tuning difference to a large model or as giving the tuned small model larger-scale pretraining benefits.

3 Instruction-Tuning Experiments

Proxy-tuning steers larger base models with guidance from smaller tuned and untuned models, targeting instruction-following, reasoning, and safety without directly tuning the large models. Across five evaluation settings, it closes most of the performance gap to directly tuned CHAT models while preserving benefits from larger pretraining.

  • 3 Instruction-Tuning Experiments: Proxy-tuning uses 7B-CHAT as the expert and 7B-BASE as the anti-expert to steer larger LLAMA2 BASE models.The experiments target 13B and 70B models using smaller tuned and untuned models for decoding-time guidance.
  • 3.1 Datasets: The evaluation covers arithmetic word problems, open-ended instructions, toxicity, and misleading-question truthfulness in multiple-choice and open-ended settings.The datasets are GSM, AlpacaFarm, Toxigen, and TruthfulQA.
  • 3.2 Results: 88.0% AlpacaFarm win rate and 32.0% GSM accuracy result from proxy-tuning 70B-BASE, versus 3.7% and 9.6% for the untuned base model.At AlpacaFarm, the proxy-tuned model is only 2.4% below the CHAT model at that scale.
  • 3.2 Results: Proxy-tuning reduces toxic generations from 67–70% for BASE models to 0% at both 13B and 70B scale.This result comes from the Toxigen evaluation of instruction-tuned models.
  • 3.2 Results: On TruthfulQA open-ended evaluation, proxy-tuning exceeds CHAT models at both scales, producing 3.2% and 6.5% higher truthfulness.It is 1.0% and 1.4% less informative than CHAT at 13B and 70B, respectively.
  • 3.2 Results: 91.1% of the instruction-tuning gap is closed at 13B scale and 88.1% at 70B scale across five evaluation settings.The gap compares each BASE model with its directly tuned CHAT counterpart.
  • 3.2 Results: Proxy-tuning a larger model outperforms the small expert in every scenario except a 0.1% ToxiGen difference.The comparison indicates that guidance from the small model can retain the advantages of larger pretraining scale.

4 Code Adaptation Experiments

Proxy-tuning adapts pretrained language models to code using a smaller specialized code model as an expert, improving coding performance while revealing a domain-specific limitation of larger-scale pretraining.

  • Setup: Proxy-tuning uses CODELLAMA-7B-PYTHON as the expert, LLAMA2-7B as the anti-expert, and LLAMA2-13B or 70B as the base model.The code expert was trained on general and Python code after initialization from LLAMA2-7B.
  • Setup: The evaluation measures functional correctness on CodexEval and DS-1000 using pass@10, the chance that at least one of ten sampled solutions is correct.CodexEval asks for Python functions, while DS-1000 contains Python programming problems from StackOverflow.
  • Results: 32.0% and 16.6% absolute improvements at 13B on CodexEval and DS-1000, respectively, demonstrate substantial gains from proxy-tuning pretrained models on code.At 70B, the improvements are 8.6% and 6.7% on the same benchmarks.
  • Limitation: Proxy-tuning a larger model usually does not outperform the tuned 7B-CODE expert, unlike instruction tuning where larger pretraining scale tends to provide more knowledge.The authors hypothesize that generic larger-scale pretraining is not helpful once the model has already been tuned for a particular domain.

5 Task Finetuning Experiments

Task-specific proxy-tuning applies small task experts to larger pretrained models for TriviaQA and GSM, substantially improving performance and preserving strict task-specific output formats.

  • Setup: Proxy-tuning combines larger pretrained base models with small task-specific experts trained on TriviaQA or GSM.The task expert is a finetuned LLAMA2-7B model, while the anti-expert is LLAMA2-7B.
  • Tasks: TriviaQA uses exact-match answer accuracy, while GSM trains models to generate step-by-step solutions with a specified final-answer format.TriviaQA contains 88K training examples, and GSM contains 7.5K.
  • Results: 19.1% and 37.3% absolute gains at 13B, and 17.5% and 44.3% at 70B, occur on TriviaQA and GSM, respectively.These gains close 84.0% of the task-expert gap at 13B and 86.9% at 70B on average.
  • Conclusion: The results indicate that proxy-tuning combines the benefits of larger pretraining scale and task-specific tuning.

6 Analysis

The analysis shows that proxy-tuning primarily changes early, reasoning- and style-related tokens, while a tunable guidance strength creates a smooth truthfulness–informativeness tradeoff.

  • 6.1 What kinds of tokens are most influenced by proxy-tuning?: The analysis measures each selected token’s probability change as the proxy-tuned probability minus the base-model probability.For GSM, tokens are compared on the left and right sides of intermediate equations.
  • 6.1 What kinds of tokens are most influenced by proxy-tuning?: 0.131 average probability change for GSM reasoning-side tokens versus 0.056 for answer-side tokens indicates stronger influence on formulating reasoning steps.The difference is statistically significant with p-value < 0.0001 under a t-test.
  • 6.1 What kinds of tokens are most influenced by proxy-tuning?: Proxy-tuning most strongly promotes reasoning and stylistic tokens, consistent with instruction-tuning influencing reasoning and style rather than increasing knowledge.TruthfulQA increases include tokens associated with challenging assumptions, correcting misconceptions, declining answers, and acknowledging complexity.
  • 6.2 Can a hyperparameter provide more granular control over steering?: Increasing α consistently improves truthfulness, whereas informativeness peaks at α = 0.4 before excessive tuning increases answer refusals.The smooth tradeoff lets users adjust guidance strength for their application.

7 Case Study: Proxy-Tuning GPT-3.5 for the Present

The case study proxy-tunes GPT-3.5 for temporal adaptation using a smaller language model and only limited top-5 log probabilities. This improves accuracy on recent-event questions despite both proxy models being weaker than GPT-3.5.

  • Evaluation selects the highest-probability token among A, B, C, and D, with proxy-tuning reweighting only those four tokens.Questions lacking all answer choices are excluded, representing 1.8% of questions.
  • 2.3% accuracy improvement on REALTIMEQA was statistically significant for proxy-tuned GPT-3.5 (p < 0.0001).The result uses a t-test and is reported in Table 7.
  • GPT-3.5 reaches 54.2% accuracy, outperforming random and the small expert, while proxy-tuning still extracts a positive signal from weaker models.The authors characterize this as weak-to-strong generalization.
  • The API limitation prevents conditioning on partial responses, restricting proxy-tuning to tasks involving only one generated token in this setting.The API always starts a new conversational turn rather than continuing a partial response.

8 Related Work

Related work places proxy-tuning among efficient finetuning, decoding-time control, and logit-arithmetic methods. Its distinction is using decoding-time guidance to emulate tuning large models without white-box access and evaluating that approach empirically.

  • Efficient Finetuning: Existing efficient-finetuning methods update a small number of parameters but still require white-box access to model weights.This motivates methods that can adapt models whose weights are unavailable.
  • Logit Arithmetic: Proxy-tuning shares DEXPERTS-based decoding-time logit arithmetic with prior work, but evaluates its effectiveness on established benchmarks rather than only analyzing scaling and instruction-tuning effects.The paper identifies this as a key distinction from Mitchell et al. (2024).
  • Efficient Finetuning: Prompt-based instruction-following can be competitive with instruction-tuning, but long prompts increase inference cost and constrain generation length.These limitations distinguish prompting from the paper’s decoding-time tuning approach.
  • Controllable Generation: Controllable-generation methods also modify output logits, but typically target attributes such as toxicity or sentiment and may require additional control parameters.Proxy-tuning instead targets model adaptation through a tuned expert and anti-expert.

9 Conclusion

The conclusion presents proxy-tuning as decoding-time customization of large language models through output logits, including proprietary models whose weights remain private. It also highlights an unresolved comparison between direct weight updates and decoding-time guidance.

  • Proxy-tuning modifies output logits to make large language models more accessible for users lacking resources to train them.The paper frames this as a way to adapt proprietary models to diverse use cases.
  • The paper leaves open how direct tuning through weight updates compares with proxy-tuning through decoding-time guidance.This comparison is presented as a question for future exploration rather than a resolved result.
  • The supplied evaluation materials cover prompts, answer extraction, toxicity scoring, truthfulness metrics, coding evaluation, and runtime settings across the experiments.These materials include datasets such as GSM, TruthfulQA, AlpacaFarm, CodexEval, and DS-1000.

A.2 Code Adaptation Experiments

The code-adaptation experiments evaluate CodeLlama-guided proxy-tuning on programming benchmarks using standardized prompts, sampling, stopping rules, and pass@10 evaluation. The appendix also specifies task-specific finetuning and runtime configurations.

  • Code Adaptation: Code adaptation uses CodeLlama outputs with the Llama2 tokenizer, enabling the models’ outputs to be combined for evaluation.
  • Code Adaptation: Programming evaluation samples 20 generations with top p = 0.95 and temperature = 0.8 to compute pass@10, while limiting outputs to 512 tokens.The setup also bans “pass” and “...” tokens and applies postprocessing.
  • Code Adaptation: CodexEval contains 164 Python function-completion problems, whereas DS-1000 provides Python programming problems sourced from StackOverflow and evaluates 200 sampled problems.The two benchmarks use task-specific stopping tokens and left-to-right completion.
  • Task Finetuning: Task-specific finetuning covers GSM arithmetic problems and TriviaQA question answering, using Question/Answer prompts and task-appropriate answer extraction.TriviaQA requires exact match with the target answer or an alias, while GSM extracts the final number.

C.1 Runtime Analysis

Proxy-tuning increases runtime when models execute sequentially, but parallel deployment can approach true-tuning speed. Its influence is concentrated early in generation, while applying it only to initial tokens is insufficient because repetition can return.

  • Runtime overhead: At 13B, proxy-tuning increases runtime by ∼2.4×, while at 70B it increases runtime by ∼1.5×.The increase mostly comes from sequential forward passes through the base model, expert, and anti-expert.
  • Runtime overhead: Parallel execution across GPUs can make proxy-tuning’s runtime similar to a true tuned model, though the pilot implementation uses three GPUs instead of one.Each model runs its forward pass concurrently before logits are gathered and distributed.
  • Prediction changes: Proxy-tuning changes the base model’s top-token prediction in 17.3% of AlpacaFarm positions, 24.6% of ToxiGen positions, 13.5% of GSM positions, and 18.0% of TruthfulQA positions.The largest influence occurs on the earliest generated tokens, with the same positional pattern across datasets.
  • Prediction changes: Applying proxy-tuning only to the first few tokens has limited effectiveness because the base model can return to endless repetition when unchecked.The authors suggest selecting time steps more efficiently as a possible way to reduce runtime cost.

D Comparison with LORA for Task Finetuning

Proxy-tuning is a viable alternative to LoRA for task finetuning, especially in training efficiency, but their relative performance depends on the task and model size. LoRA is stronger on TriviaQA, whereas proxy-tuning leads at 13B on GSM and trails at 70B.

  • Implication: Even in white-box settings, the authors find proxy-tuning valuable, especially when training efficiency is important.LoRA requires access to all parameters of the base model, whereas proxy-tuning uses a separately tuned 7B expert.
  • Task performance: For TriviaQA, LoRA outperforms full finetuning by 6.5% at 13B and 12.2% at 70B.LoRA’s performance depends on both task and model size.
  • Task performance: On GSM, proxy-tuning outperforms LoRA by 11.5% at 13B, while LoRA outperforms proxy-tuning by 9.1% at 70B.The relative effectiveness of the methods is therefore mixed across model sizes.
  • Interpretation: The authors hypothesize that LoRA’s task inconsistency reflects the shift between pretraining and finetuning data, with GSM’s formatted answers farther from base-model predictions than TriviaQA’s answers.This explanation is presented as a hypothesis rather than an established causal result.
  • Training efficiency: Full finetuning of a 7B expert is 1.3× more train-efficient than LoRA at 13B and 15× more train-efficient than LoRA at 70B.The comparison uses the same hardware for all tuning runs.
Loading 2401.08565v4…