PaperlayerView exact revision on arXiv

Source-linked AI summary

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

Zhen Zhang, Changyi Yang, Zijie Xia, Zhen Yang, Chengzhi Liu, Zhaotiao Weng, Yepeng Liu, Haobo Chen, Jin Pan, Chenyang Zhao, Yuheng Bu, Alkesh Patel, Zhe Gan, Xin Eric Wang

arXiv:2604.27039v2cs.CL

TL;DR

Existing length modeling is largely sequence-level despite generation length affecting inference cost and reasoning performance. LenVM estimates remaining length token by token through a bounded discounted value target with scalable supervision, and experiments show broad support for control, performance–efficiency steering, prediction, and interpretation. Its limitations include policy-conditioned calibration and slower guided decoding.

  • Problem

    Existing length modeling and control remain coarse-grained, while generation length affects both inference cost and reasoning performance.

  • Method

    LenVM predicts a bounded discounted return from each decoding state, using constant per-token negative rewards and annotation-free, dense, unbiased, scalable supervision.

  • Results

    LenVM supports precise length control, smooth performance–efficiency steering, generation-length prediction, and interpretation across evaluated settings.

  • Takeaways & Limitations

    Generation length functions as a scalable token-level value signal for control, prediction, and future value-based training.

  • Takeaways & Limitations

    LenVM is policy-conditioned, so calibration under unrelated generators remains open, and guided decoding is 1.9× slower than vanilla decoding in the reported setup.

Abstract

from arXiv · show

Tokens are the fundamental units of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches model length primarily at the coarse sequence level. We introduce the Length Value Model (LenVM), a token-level framework that estimates the remaining generation length at every decoding step. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that is a monotone proxy for the remaining generation horizon. This value formulation provides annotation-free, dense, unbiased, and scalable supervision. Experiments on LLMs and VLMs show that LenVM supports exact control, continuous performance--efficiency steering, length prediction, and interpretation. On LIFEBench-token, it raises the exact-length score of Qwen2.5-7B-Instruct from $30.9$ to $64.8$ in one pass and reaches $83.6$ when combined with LCG. Near $200$ tokens on GSM8K, LenVM retains about $63\%$ Pass@1 versus $6\%$ under a hard token budget. Across other settings, LenVM improves matched-length quality over prompting and EOS calibration, while its length-prediction accuracy improves with scale and exceeds specialized predictors. Its token-level values also identify markers of shifts between longer and shorter trajectories. These results establish generation length as an effective scalable token-level value signal for control, prediction, and future value-based training.

1. Introduction

Generation length affects both inference cost and reasoning performance, yet existing methods model it mainly at the sequence level. LenVM addresses this gap with token-level value estimation and scalable supervision for control, prediction, and efficiency steering.

  • Motivation: Generation length affects reasoning performance and inference cost, but existing penalties, prompts, and predictors operate on whole sequences or fixed pre-generation decisions.The missing capability is a token-level model of remaining generation length.
  • Method: LenVM predicts a scalar value at each decoding step as a bounded, discounted, monotone proxy for remaining generation length.A constant negative reward per generated token and discounted future steps reduce length modeling to value estimation.
  • Scalable value pretraining: LenVM supervision is annotation-free, dense, unbiased under a fixed rollout policy, and scalable with prompts, completions, and model size.Targets are constructed automatically from sampled completions, with every token position contributing supervision.
  • Capabilities: LenVM exposes a smooth performance–efficiency trade-off, supports explicit length constraints, predicts length from prompts and prefixes, and reveals shifts between reasoning regimes.These capabilities are evaluated across control, prediction, and interpretation settings.

2. Related Work

Prior work controls or predicts output length through prompting, constrained decoding, fine-tuning, entropy signals, hidden representations, and online feedback. LenVM differs by providing a standalone token-level value model with bounded regression targets for stable prediction and control across tasks and length scales.

  • Inference-time control: Prompting and constrained decoding regulate length without retraining, while MCMC-style methods satisfy lexical or structural constraints during inference.Examples include countdown prompts, plan-first scaffolds, and Metropolis–Hastings-based constrained sampling.
  • Training-based control: Training-based approaches modify internal length representations or fine-tune termination behavior to achieve more precise control.LDPE and Hansel represent this model-modification direction.
  • Length prediction: Length prediction methods estimate completion length statically before decoding or progressively during generation using entropy signals and hidden representations.Related approaches include entropy-guided token pooling, progressive length prediction, and frozen layerwise hidden states.
  • Online prediction and serving: Online length control can also use dynamic feedback, while prediction helps scheduling, batching, load balancing, and reducing padding or fragmentation.These approaches target serving efficiency and stochastic sampling alongside generation control.
  • LenVM's distinction: LenVM models length regularization as a dense per-step negative reward with a discounted return, providing a standalone value signal compatible with PPO-style value baselines.The paper focuses on inference-time decoding control and analysis rather than empirical RL fine-tuning.

3. Length Value Model

LenVM models remaining generation length as a token-level value over decoding states, using a bounded discounted return derived from constant per-token costs. Its targets support dense, annotation-free Monte Carlo training and standard value-learning properties.

  • Model formulation: LenVM assigns a scalar value to each decoding state to estimate the remaining generation horizon under a rollout policy.The model starts from the last prompt token and produces values at successive decoding steps.
  • Return construction: A constant negative reward is assigned at every non-terminal step, with zero reward at EOS and normalization by 1 − γ.The normalization keeps the discounted return in a fixed range compatible with sigmoid-bounded outputs.
  • Return properties: The resulting return lies in (−1, 0), preserves the ordering of remaining horizons, and satisfies a Bellman recursion.States nearer termination have values closer to 0, while longer continuations have values closer to −1; γ controls long-horizon compression.
  • Training: LenVM trains by regressing token-level predictions against realized returns from trajectories sampled under a fixed generator and decoding policy.The targets are computed exactly from each sampled completion and provide dense supervision over the induced rollout-state distribution.
  • Training: The formulation is annotation-free, dense, unbiased for the policy-conditioned value target, and scalable through multiple sampled completions per prompt.It also supports an optional GAE variant, although the experiments find λ = 1 performs best.
  • Model implementation: LenVM uses a scalar value head on final-layer hidden states, with a two-layer SiLU MLP constrained to output values in (−1, 0).Training uses token-averaged mean squared error over prompt-completion trajectories.

4. Experiments: Validating LenVM as a Token-Level Length Signal

Across control, efficiency steering, prediction, and scaling evaluations, LenVM provides a token-level length signal that improves adherence and matched-length quality while supporting scalable training.

  • Length-controlled generation: LenVM predicts token-level generation horizons and supports exact length control through candidate scoring for Equal To, At Least, and At Most targets.Hard-constraint decoding compares predicted next-state values with target values or favors longer or shorter continuations.
  • Length-controlled generation: 64.8 Length Score versus 30.9 for Qwen2.5-7B-Instruct, while eight-call LCG plus LenVM reaches 83.6 versus 71.3.The corresponding average absolute deviation falls from 71% to 44% in the one-pass comparison and from 14.0% to 5.8% with eight-call LCG.
  • Performance–efficiency trade-off: 63% Pass@1 at roughly 200 tokens versus 6% for hard truncation on GSM8K with Qwen2.5-3B-Instruct.LenVM-guided decoding improves the performance–efficiency frontier without modifying the base generator.
  • Performance–efficiency trade-off: Varying β smoothly changes the performance–efficiency frontier, providing a continuous knob for balancing response quality and token budget.At matched average lengths, LenVM also improves task accuracy over budget-aware prompting and EOS calibration on MATH500 and GSM8K.
  • Generation length prediction: LenVM’s prompt-boundary length prediction improves with scale and reduces prompt MRE from 32.56 to 26.90 and progressive MRE from 26.71 to 15.10.At 32B, MRE is 9.8% on math, 14.9% on code, and 17.1% on instruction following.
  • Scalability of LenVM: Validation loss decreases consistently with larger models, more training questions, and more sampled completions per question, supporting scalable value pretraining.The training mixture uses multiple sampled completions converted into dense per-token regression targets.

5. Qualitative Case Study: Length Tokens as Markers of Length Shifts

LenVM’s token-level values identify tokens associated with shifts toward longer or shorter predicted generation horizons. Reasoning pivots tend to mark longer horizons, while closure tokens tend to mark shorter ones.

  • Positive and negative temporal-difference residuals mark longer and shorter predicted horizons, respectively.
  • Reasoning pivots such as wait, think, and try contrast with closure tokens such as therefore, clearly, and perfect.

6. Ablations

Ablations show that LenVM benefits from discounted-return targets, shuffled batches, common floating-point precisions, and sufficiently large candidate sets. The discount factor trades earlier-generation accuracy against accuracy near termination.

  • Discount Factor: Larger γ improves prediction earlier in generation, whereas smaller γ improves prediction closer to termination.The discount factor determines where prediction resolution is concentrated along the trajectory.
  • Length-Space Representation: Discount Return + Sigmoid achieves the lowest average absolute length error throughout training among the four target parameterizations.Raw and normalized length perform worse, while log length is stronger but still underperforms discounted return.
  • Batch Construction: Shuffling consistently improves evaluation loss compared with grouping multiple completions from the same prompt.Grouping samples from one prompt is unnecessary and can mildly hurt generalization under the tested setup.
  • Numerical Precision: fp16, bf16, and fp32 produce nearly identical loss curves, convergence, and final performance.These results indicate numerical stability under the tested common floating-point precisions.
  • Candidate-Set Size: Increasing K from 1 to 128 raises the LIFEBench-token score from 29.3 to 72.1 and reduces deviation from 100% to 19%.On MATH500 with β = -100, most changes occur between K = 1 and K = 2; beyond K = 2, average length and Pass@1 remain stable.

7. Conclusion

The paper concludes that LenVM is a useful and scalable token-level value model for generation length. It supports control, efficiency steering, prediction, and interpretation across the evaluated settings.

  • LenVM models remaining generation length with a bounded, dense, annotation-free discounted return induced by a constant per-token reward.The conclusion presents this formulation as useful and scalable for value pretraining.
  • LenVM supports precise length control, smooth performance–efficiency steering, expected-horizon prediction, and token-level interpretation.It complements iterative controllers, matched-length baselines, specialized predictors, and length-penalized RLVR steering.
  • Entropy skipping reduces LenVM overhead, while empirical RL fine-tuning with LenVM remains future work.

A. LIFEBench-token Evaluation Details

LIFEBench-token evaluates token-level adherence to explicit length constraints across diverse tasks and target scales. It preserves the benchmark’s tasks, target grid, and metrics while applying constraints to model-specific token counts.

  • Benchmark Overview: LIFEBench comprises 10,800 instances spanning question answering, summarization, reasoning, and creative generation.The benchmark covers diverse English and Chinese data sources and task types.
  • Length Constraints: LIFEBench-token applies Equal To, At Most, and At Least constraints to model-specific token counts across ten target lengths from 16 to 8,192.The tasks and target grid are preserved from the original benchmark while the length unit changes to tokens.
  • Evaluation Metrics: Length Deviation measures relative output-length error, with positive values indicating over-generation and negative values indicating under-generation.For Equal To, the reported value is the mean absolute relative deviation.
  • Evaluation Metrics: Length Score is bounded in [0, 100], with 100 indicating perfect adherence and lower scores indicating larger deviations.
  • Implementation Details: Each prompt combines an instruction with a natural-language length requirement, such as requiring an answer equal to 512 tokens.

B. Additional Experimental Details

LenVM experiments use controlled sampling and decoding settings across exact-control, matched-length, RLVR-comparison, and candidate-pool evaluations. The results show improved length matching, quality–length steering, and dense token-level signals for training and inference.

  • γ is set to 0.997 for Qwen2.5-Instruct and Qwen2.5-VL-Instruct, and 0.9998 for Qwen3-Instruct, with λ = 1 for GAE.
  • The experiments sample up to 16 completions per prompt, use 8k validation examples, and train for 2 epochs with learning rate 2 × 10^-5 and batch size 1024.
  • LenVM improves over EOS calibration and budget-aware prompting on MATH500, and over budget-aware prompting on GSM8K at nearest-length operating points.
  • Across two, four, and eight rounds, inserting LenVM into LCG consistently lowers average absolute deviation and raises the length score.The compared methods do not have equal inference cost because a round is one generation or refinement call.
  • 54.17 Pass@1 at 5611 tokens is achieved by Base + LenVM, compared with 51.25 at 5599 tokens for ART.Applying stronger steering to ART shortens responses from 5599 to 4949 tokens while accuracy decreases from 51.25 to 46.25.
  • Increasing K from 1 to 128 raises the LIFEBench-token score from 29.3 to 72.1 and reduces deviation from 100.2% to 18.5%.Intermediate values are not strictly monotone, and soft steering remains anchored to the generator distribution.
  • LenVM can serve as a potential for policy-invariant reward shaping under fixed-potential assumptions, or as a direct length-specific value for quality–efficiency optimization.Its token-level signal can redistribute terminal information for denser learning, but the paper does not claim empirical RL gains.

E. Finite-Precision Analysis of Relative Length Resolution

The analysis examines finite-precision resolution and the population objective used to train LenVM. It identifies short horizons as the main numerical-risk region and shows why future-dependent weighting shifts the regression target.

  • LenVM models horizons from 1 to 32k, making relative perturbation ∣δl∣/l more informative than absolute error.
  • The proxy f(z, k) measures local relative length resolution under finite precision.The analysis models local logit perturbations using an effective precision parameter k.
  • Finite-precision error is concentrated in the small-length region, while large horizons approach an approximately constant relative-resolution floor.This behavior is also reflected in the complementary views summarized in Figure 7.
  • Token-uniform weighting recovers the state-conditional mean of the return proxy that LenVM aims to estimate.Every non-terminal decoding step receives equal weight in the adopted token-averaged objective.
  • State-dependent weights change the global importance of states without altering the per-state regression target.
  • Future-dependent weighting, such as trajectory-level weighting by 1/L(τ), shifts the per-state regression target away from the token-uniform objective.Longer trajectories receive smaller weights while producing more negative return targets, yielding a future-length-reweighted variant.

G. Why Inverting the Transformed Horizon Underestimates Expected Remaining Length

The transformed horizon is useful for value prediction, but directly inverting its conditional mean does not generally recover the expected raw remaining length. Concavity and Jensen’s inequality explain the resulting downward bias.

  • The appendix analyzes length prediction in transformed space rather than by directly inverting predicted horizons into raw length estimates.
  • Because the transformed horizon is strictly increasing and strictly concave, Jensen’s inequality produces the upper bound on the inverted estimate.
  • The inverted length estimate is always less than or equal to the true expected remaining length E[L].
  • Equality holds only when remaining length is deterministic; with nonzero variance, inversion systematically underestimates expected remaining length.The underestimation arises from conversion into length space, not from operating directly in value space.

H. Derivation of the Exponential Tilting Solution in the Performance–Efficiency Trade-off Experiment

The performance–efficiency trade-off derives a KL-regularized token distribution that balances shorter predicted horizons against fidelity to the base model. The solution is an exponentially tilted Gibbs distribution.

  • The objective minimizes expected LenVM value while penalizing deviations from the base distribution through KL divergence.
  • β > 0 makes the KL-regularized objective strictly convex and guarantees a unique global minimizer.Using β > 0 in the stated convention is necessary to avoid an ill-posed objective.
  • The normalization constraint determines the constant in the Lagrangian solution.
  • With β < 0, the Gibbs form assigns more probability to tokens with lower predicted values, while ∣β∣ controls deviation from the base model.
Loading 2604.27039v2…