PaperlayerView exact revision on arXiv

Source-linked AI summary

Following Length Constraints in Instructions

Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, Jing Xu

arXiv:2406.17744v1cs.CL

TL;DR

Length-biased evaluation and training can favor longer responses, while many prompts leave the desired length ambiguous. The paper trains models with length instructions and reports fewer violations and improved win rates on length-instructed benchmarks. It also identifies limits in the length units, wording, and types of constraints studied.

  • Problem

    Length-biased preferences and ambiguous desired response lengths complicate evaluation and can lead training methods to produce longer responses.

    Source: §1 · p. 1

  • Method

    LIFT augments instruction-following prompts with length limits and preference pairs reflecting both length constraints and response quality, then fine-tunes with DPO.

    Source: §1 · p. 1

  • Results

    LIFT models show fewer length-constraint violations and improved overall win rates on length-instructed benchmarks compared with existing instruction-following models.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Length instructions provide a way to compare models without length bias from simply increasing response length.

    Source: §7 · p. 8

    The study uses word-count limits and a fixed instruction template, and does not address minimum-length instructions such as requiring at least 100 words.

    Source: §8 · p. 9

Abstract

from arXiv · show

Aligned instruction following models can better fulfill user requests than their unaligned counterparts. However, it has been shown that there is a length bias in evaluation of such models, and that training algorithms tend to exploit this bias by learning longer responses. In this work we show how to train models that can be controlled at inference time with instructions containing desired length constraints. Such models are superior in length instructed evaluations, outperforming standard instruction following models such as GPT4, Llama 3 and Mixtral.

1 Introduction

Length ambiguity complicates evaluation and can reinforce preferences for longer responses; the paper proposes explicit length instructions and trains models to follow them. On length-instructed benchmarks, LIFT models reduce violations and improve win rates.

1 Introduction

  • Evaluation is difficult when queries leave desired response length ambiguous, and length-biased preferences favor longer responses that training methods then tend to produce.The intended length may depend on context, such as the downstream application or interface.Source: §1 · p. 1
  • GPT4-Turbo violates maximum word-length constraints almost 50% of the time, exposing a length-control weakness in existing instruction-following models.The paper evaluates length-instructed versions of AlpacaEval 2 and MT-Bench.Source: §1 · p. 1
  • LIFT augments instruction-following prompts with length instructions and constructs preference pairs that reflect both length constraints and response quality for DPO fine-tuning.The paper trains both Llama 2 and Llama 3 models using LIFT-DPO.Source: §1 · p. 1
  • LIFT models show fewer length-constraint violations and higher overall win rates than existing instruction-following models on the length-instructed benchmarks.The paper also presents example generations from a LIFT Llama-3-8B-Instruct model under different limits.Source: §1 · p. 2, Table 1 · p. 2

2 Related Work

Prior work links length bias in evaluation and training to longer outputs, while length-aware methods often assume a preferred output length. This paper instead treats desired length as context-dependent and asks models to follow explicit length instructions.

2.1 Length Bias in Model Alignment

2.2 Length-aware Model Training

  • Length-aware approaches balance preferences, disentangle length from quality, or regularize DPO, but assume an optimum response length rather than one determined by context.Prior work also reports that mitigations do not uniformly help [Singhal et al., 2023]; the paper notes that length-penalized scoring may still be gameable.Source: §2.2 · pp. 2–3, §2.1 · p. 2
  • Although some production prompts reference output length, systematic evaluations have not been reported, and released post-training data suggests length-following examples are generally scarce.Preference pairs often imply a target length without stating it in the prompt.Source: §2.2 · p. 3

3 AlpacaEval-LI & MT-Bench-LI: New Length-Instructed Benchmarks

The paper introduces AlpacaEval-LI and MT-Bench-LI by adding maximum-length instructions to existing tasks, then evaluates both instruction following and response quality. GPT4-0409 and Claude3-Opus frequently exceed limits in the AlpacaEval-LI examples.

3.1 Augmenting General Instructions with Length Constraints

  • The benchmarks add maximum-length limits to existing instructions, testing whether models can answer the query while honoring the specified cap.Targets are set per prompt to the shortest generation among GPT4 Turbo, Claude 3 Opus, and Mistral Large, making them challenging for current models.Source: §3.1 · p. 3, §3.1.1 · p. 3, §3 · p. 3

3.1.2 Length Instruction-Following Baseline

  • The AlpacaEval-LI baseline selects the shortest response from GPT4-1106, Claude3-Opus, and Mistral Large to meet the limit while retaining high generation quality.Each tested model is compared with this baseline in a pairwise setting.Source: §3.1.2 · p. 3

3.1.3 Metrics

  • Evaluation reports violation rates and LI win rates; a response exceeding the hard length limit automatically loses, while compliant responses receive a pairwise LLM-as-a-Judge comparison.Violation rates count responses exceeding the word limit, and LI win rates measure response quality against the baseline.Source: §3.1.3 · p. 4

3.2 Length-Instructed AlpacaEval

  • AlpacaEval-LI adapts AlpacaEval 2’s 805 prompts, excluding the three already carrying explicit length limits and leaving 802 evaluated prompts.The prompt set spans creativity, brainstorming, writing, question answering, math, and reasoning tasks.Source: §3.2 · p. 4

3.1.2 Length Instruction-Following Baseline

  • GPT4-0409 exceeds target length limits almost 50% of the time on AlpacaEval-LI, especially above 200 words, and Claude3-Opus shows a similar trend.Figure 2 plots target length against the ratio of generated length to target length; red dots mark violations.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2 Length-Instructed AlpacaEval

  • The AlpacaEval-LI baseline models’ standard AlpacaEval 2 win rates are 50% for GPT4-1106, 40.5% for Claude3-Opus, and 32.7% for Mistral Large.The paper describes the resulting baseline as high-quality while consistently meeting the length constraint.Source: §3.2 · p. 4

3.3 Length-Instructed MT-Bench

  • MT-Bench-LI uses first turns from 80 challenging multi-turn questions across eight prompt categories, with three sampled length constraints per prompt yielding 240 prompts.Categories include writing, roleplay, extraction, reasoning, math, coding, STEM, and humanities/social science.Source: §3.3 · p. 5

4 Length-Instruction Fine-Tuning (LIFT)

LIFT creates length-instructed preference pairs from existing instruction-following preferences, so models can learn to respect length limits without discarding response-quality preferences. LIFT-DPO trains on both original and augmented data.

4 Length-Instruction Fine-Tuning (LIFT)

  • LIFT constructs preference pairs with length instructions from existing pairwise preferences, producing data usable with RLHF or other preference-optimization methods.The original prompt is augmented with a target-response-length instruction.Source: §4 · p. 5
  • The method filters out pairs whose chosen and rejected responses differ by fewer than T words, with T set to 10 in the experiments.The retained pairs are used to construct a length-instructed augmented dataset.Source: §4 · p. 5
  • When the original winner is longer, LIFT creates cases where both responses satisfy the cap and cases where the winner loses because it exceeds the limit.The resulting preferences make length constraints take priority over original preferences when necessary.Source: §4 · p. 5
  • When the original winner is shorter, the augmented examples retain the original preference under length constraints that both responses satisfy.The procedure adds two samples with different target-length constraints.Source: §4 · p. 5
  • LIFT-DPO trains on the union of original and augmented datasets, allowing the model to handle prompts with and without length instructions.The augmented dataset is constructed from preference pairs after the length-gap filtering step.Source: §4 · p. 5

5 Experimental Setup

The experiments compare standard DPO, length-regularized DPO, and LIFT-DPO using OpenAssistant preference data, then evaluate length instruction following and standard instruction following.

5.1 Train Dataset & Baselines

  • The standard dataset uses 3,200 OpenAssistant examples; LIFT creates 5,954 length-instructed preference pairs and trains on their union with the original data.Among the original pairs, 1,083 change preference when length limits are applied.Source: §5.1 · p. 6

5.2 Training Details

  • Training sweeps learning rates from 5e−7 to 5e−6 with batch size 16, dropout 0.1, and DPO β of 0.1.Llama 2 trains for up to 2,000 steps and Llama 3 for up to 20 epochs, with checkpoint selection for early stopping.Source: §5.2 · p. 6

5.3 Evaluation Method

  • Models are evaluated on AlpacaEval-LI and MT-Bench-LI as well as standard AlpacaEval 2 and MT-Bench; GPT-4 judges pairwise winrates on the LI benchmarks.Table 2 summarizes state-of-the-art model results on AlpacaEval-LI and MT-Bench-LI.Source: §5.3 · p. 6, Table 2 · p. 6

6 Experimental Results

LIFT-DPO reduces length-instruction violations while improving benchmark winrates over standard DPO, and its performance remains comparable on standard AlpacaEval 2. Its length-control results also hold under tighter constraints, while Length-Controlled AlpacaEval winrates vary with instruction scaling.

6 Experimental Results

  • For GPT-4 0409, violation rates are 49.3% on AlpacaEval-LI and 44.2% on MT-Bench-LI; Llama3-8B-Instruct records 7.0% and 20.0%, respectively.Table 2 reports length instruction-following results for state-of-the-art models on both benchmarks.Source: §6 · p. 6
  • On AlpacaEval-LI, Llama-2-70B-Base LIFT-DPO lowers Vlt(%) from 65.8% to 7.1% and raises Win(%) from 4.6% to 13.6%.Table 3 reports Llama 2 results across AlpacaEval-LI and MT-Bench-LI.Source: Table 3 · p. 7, §6 · p. 7
  • On MT-Bench-LI, Llama-2-70B-Base LIFT-DPO lowers violation rate from 60.8% to 10.0% and raises win rate from 5.0% to 11.0%.For Llama-2-70B-Chat, the violation rate falls from 24.2% to 6.7%, while win rate rises from 10.8% to 12.5%.Source: §6 · p. 7
  • On standard AlpacaEval 2, LIFT-DPO shows no overall performance degradation, with some model winrates increasing and others decreasing slightly.The reported Llama-2-70B-Base win rate increases from 8.6% with DPO to 9.9% with LIFT-DPO.Source: §6 · p. 7
  • As target-length scale decreases, DPO violations rise from below 50% to almost 100% at 0.1, while LIFT-DPO stays below 10% across tested scales.R-DPO follows trends similar to DPO; the paper reports similar LIFT-DPO trends on MT-Bench-LI.Source: Figure 4 · p. 8, §6 · p. 8

Robustness of Length Controlled AlpacaEval

  • AlpacaEval LC winrate varies from 23% to 29% as length instructions are scaled; the reported optimal LIFT-DPO length choice scores similarly to Mistral Medium and above original Llama-3-8B-Instruct.The paper notes that changing instruction lengths can manipulate LC winrate.Source: Figure 5 · p. 8, Robustness of Length Controlled… · p. 8

7 Conclusion

The paper proposes length instructions for assessing whether models respond within given limits, and LIFT augments instruction-following examples with those limits. The authors report improved length control while maintaining response quality, with added user control in real-world use cases.

7 Conclusion

  • Length instructions assess whether models generate responses within given length limits.The paper presents them as an approach to addressing length bias in general instruction following.Source: §7 · p. 8
  • LIFT augments general instruction-following examples with varying length limits, and LIFT-DPO improves output-length control while maintaining high response quality.Source: §7 · p. 8
  • Adding length limits to general instructions provides users more control in real-world use cases.Source: §7 · p. 9

8 Limitations

The paper limits length constraints to word counts, while noting alternatives and unanswered questions about desired response lengths and the effect of computation on longer outputs.

8 Limitations

  • Length limits use word counts, though character counts or other measures could generalize the approach.The paper also identifies different instruction wording and minimum-length instructions as directions not addressed here.Source: §8 · p. 9
  • Human preferences for longer, more detailed responses and increased computation from additional tokens may also contribute to longer outputs.The authors suggest studying desired response lengths across instructions and analyzing the role of computation.Source: §8 · p. 9

A Word Count Function We Use

B Additional Results on SOTA models’ length following measurements

These additional results describe occasional constraint violations by two existing models, compare training and evaluation constraint distributions, and explain checkpoint selection.

B Additional Results on SOTA models’ length following measurements

C Training and test length distribution

D Checkpoint Selection

  • Checkpoints are saved every 200 steps and at each epoch's end, then early stopping uses the average win rate across standard and length-instructed validation sets.The length-instructed comparisons use the baseline model whose generation length matches the prompt's limit.Source: D Checkpoint Selection · p. 10

E MT-Bench Results

The section specifies evaluation settings and visualizes constraint distributions and model behavior; under progressively tighter limits, LIFT-DPO has substantially lower violation rates than DPO and R-DPO.

E MT-Bench Results

  • MT-Bench-LI sampling used temperature 0.7 across categories, while standard MT-Bench restored category-specific temperatures and evaluated 80 unique questions.Source: E MT-Bench Results · p. 10
  • Figure 6 plots target length against actual-to-target length ratios for Mistral Large and LLAMA3-70b-Instruct on 802 examples; Figure 7 compares constraint distributions, including scaled AlpacaEval-LI.Source: Figure 6 · p. 11, Figure 7 · p. 11

G Additional Length Instruction Following Results

F Decoding Parameters

  • Llama2 decoding used temperature 0.7 and a 2048-token limit; Llama3 used temperature 0.6, top-p 0.9, and a 4096-token limit.Top-p was 0.9 for AlpacaEval and AlpacaEval-LI, and 1.0 for MT-Bench and MT-Bench-LI.Source: F Decoding Parameters · p. 12

H AlpacaEval Results & MT-Bench Results

On standard, non-length-instructed benchmarks, LIFT-DPO models show no performance degradation compared with standard DPO models.

H AlpacaEval Results & MT-Bench Results

Loading 2406.17744v1…