PaperlayerView exact revision on arXiv

Source-linked AI summary

Quality Determines Direction, Length Shapes Magnitude: Length Control for Open-Ended Reinforcement Learning

Zijun Weng, Zhongan Bi, Xuanang Gao, Xiaohui Hu, Shuangyong Song, Yongxiang Li, Kaidong Yu, Xuanjing Huang

arXiv:2609.34718v1cs.LGcs.CL

TL;DR

Open-ended RL makes length control difficult because length is entangled with quality, there is no natural success boundary for efficiency, and dense rewards can make quality advantages sensitive to length shaping. QGLAS preserves the reinforcement direction set by quality while adapting bounded bonuses for shorter positive-advantage responses. At approximately 30% compression, it retains more of quality-only RL’s macro-average quality gains than GR3 and GRLC at comparable compression.

  • Problem

    Open-ended RL length control must balance efficiency with useful content because response length is entangled with quality, no natural success boundary guides efficiency, and dense rewards can create small within-group quality margins.

    Source: §1 · p. 1

  • Method

    QGLAS computes advantages from quality rewards alone and adds bounded bonuses only to shorter positive-advantage responses, adapting bonus strength to their quality separation.

    Source: §1 · p. 2

  • Results

    At approximately 30% compression, QGLAS retains 98.4–102.0% of quality-only RL’s macro-average quality gain over the base model, versus 68.3–75.5% for GR3 and GRLC at comparable compression.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across evaluated models, benchmarks, and reward sources, QGLAS consistently achieves a stronger quality–length trade-off than representative baselines.

    Source: §1 · p. 2, §5 · p. 9

    The reward-granularity comparison uses fixed rollouts and does not reproduce RLVR training.

    Source: §4.3.1 · p. 7

Abstract

from arXiv · show

Reinforcement learning (RL) changes not only what language models say, but also how much they say, often increasing response length at the cost of token efficiency. Controlling this length growth is particularly challenging in open-ended RL because (i) response length is entangled with quality, (ii) open-ended tasks lack a natural success boundary for deciding when efficiency should be prioritized, and (iii) dense, graded rewards often yield small within-group quality margins, making quality-induced advantages especially sensitive to reward-level length shaping, which can perturb their magnitudes and even reverse their signs. We therefore adopt an asymmetric principle: quality should determine the direction of reinforcement, while length should only shape its magnitude. We instantiate this principle with Quality-Gated Length Advantage Shaping (QGLAS), which first computes advantages from quality rewards alone, then adds bounded bonuses only to shorter positive-advantage responses, leaving all other advantages unchanged. The bonus strength is further adapted to within-group quality separation, allowing conciseness to matter more when quality-favored responses are similar and less when their quality differences are clear. Across different model families, open-ended benchmarks, and reward sources, QGLAS consistently achieves a stronger quality--length trade-off than representative baselines. At approximately 30% compression, QGLAS retains 98.4--102.0% of the macro-average quality gains achieved by quality-only RL over the base model, compared with 68.3--75.5% for these baselines at comparable compression.

1 INTRODUCTION

Open-ended RL length control must balance efficiency against useful content because response length is entangled with quality and graded rewards lack a natural success boundary. The paper introduces QGLAS, which preserves quality-driven reinforcement direction while adapting length shaping, and reports stronger quality–length trade-offs than GR3 and GRLC.

1 INTRODUCTION

  • Open-ended RL complicates length control because extra tokens can be either useful or redundant, no natural success boundary prioritizes efficiency, and dense rewards may leave small quality margins.Length shaping can therefore perturb quality-induced learning signals, while indiscriminate penalties risk suppressing useful content.Source: §1 · pp. 1–2
  • GR3 applied to identical rollout groups has an 11.73% macro-average sign-reversal rate with dense rewards versus 0.30% averaged over four binarized variants.The controlled comparison isolates sensitivity to reward representation on fixed rollouts and does not reproduce RLVR training.Source: §1 · p. 2
  • QGLAS computes advantages from quality rewards alone, then adds bounded bonuses only to shorter positive-advantage responses, preserving reinforcement polarity by construction.Its shaping strength adapts to reward spread and favored-response quality separation, emphasizing conciseness more when those responses have similar rewards.Source: §1 · p. 2
  • At approximately 30% compression, QGLAS retains 98.4–102.0% of quality-only RL’s macro-average quality gain over the base model, versus 68.3–75.5% for GR3 and GRLC at comparable compression.The reported comparison covers Qwen3-4B and GLM-4.7-Flash; Figure 1 shows the Qwen3-4B quality–length trade-off.Source: Figure 1 · p. 1, §1 · p. 2
  • Across policy models and reward sources, QGLAS improves quality–length trade-offs, with ablations supporting complementary benefits from polarity-preserving constraints and adaptive shaping.Repeated runs and alternative reward sources further support robustness.Source: §1 · p. 2

2 RELATED WORK

Prior RL length-control approaches include methods for verifiable rewards and extensions to continuous quality feedback. The paper identifies remaining challenges in graded open-ended settings: preserving quality-induced reinforcement polarity and adapting shaping to the quality separation among favored responses.

Length control with verifiable rewards

  • Verifiable-reward length-control methods use correctness, solve rate, or estimated difficulty to choose which responses to shorten and how strongly; DDCA uses the correct-response subset and group pass rate.Binary feedback supplies an eligibility criterion and a shared task reward among successful responses, unlike graded open-ended feedback.Source: §2 · p. 3

Length control with continuous quality rewards

  • GRLC adds group-relative length adjustments and quality-gated bonuses to quality rewards, while GR3 uses multiplicative reward rescaling, length normalization, and advantage-aware calibration.Both support continuous feedback and include safeguards against indiscriminate length reduction.Source: §2 · p. 3
  • Both GRLC and GR3 incorporate length into rewards before computing relative advantages, without requiring every response to preserve its quality-only reinforcement polarity.Their gating and calibration also do not explicitly adapt shaping strength to quality separation among positive-advantage responses in the current group.Source: §2 · p. 3
  • QGLAS treats quality-induced advantages as the primary signal, restricting bonuses to shorter positive-advantage responses and adapting their strength to reward spread and favored-response quality separation.Conciseness receives greater weight when quality-favored responses differ little.Source: §2 · p. 3

3 QUALITY-GATED LENGTH ADVANTAGE SHAPING

QGLAS computes the quality-induced advantage first, then shapes only shorter quality-favored responses and adapts the shaping strength to local reward structure. These choices isolate the length signal, preserve quality-driven reinforcement polarity, and vary conciseness pressure with the separation among favored responses.

3 QUALITY-GATED LENGTH ADVANTAGE SHAPING

  • QGLAS starts from advantages computed using original quality rewards, then adds length shaping while keeping the quality-induced advantage as the primary learning signal.Its asymmetric design lets quality determine reinforcement polarity while length modulates strength.Source: §3 · p. 3

3.1 ISOLATING THE LENGTH SIGNAL

  • Reward-level length perturbations can shift centered rewards for responses receiving no direct perturbation, and dense-reward small margins increase sign-reversal risk; QGLAS leaves untargeted advantages unchanged.For responses with h_i = 0, the shaped advantage equals the quality-induced advantage.Source: §3.1 · p. 4

3.2 RESTRICTING THE SCOPE OF LENGTH SHAPING

  • QGLAS shapes only shorter quality-favored responses; if a group has no positive-advantage responses, shaping is skipped, and longer favored responses are not penalized.The reference length is defined using only quality-favored responses, so short quality-disfavored responses do not set the efficiency target.Source: §3.2 · p. 4

3.3 ADAPTING THE SHAPING STRENGTH

  • Shaping strength increases with overall reward variation and decreases as quality separation among favored responses grows, making conciseness more influential when favored responses are similar.The overall reward scale uses a lower-quartile estimate to reduce sensitivity to anomalously low-reward responses.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Experiments compare QGLAS with quality-only RL, GR3, and GRLC across open-ended benchmarks, model families, and reward sources. They report stronger quality–length trade-offs for QGLAS, alongside analyses of sign reversals, structural constraints, and adaptive scaling.

4.1 EXPERIMENTAL SETUP

  • Experiments use Qwen3-4B and GLM-4.7-Flash, 13K open-ended prompts, and instruction-following, dialogue, and creative-writing benchmarks.The main setup uses GSPO for 1,000 steps and Skywork-Reward-V2-Llama-3.1-8B as the quality reward model.Source: §4.1 · pp. 5–6

4.2 QUALITY–LENGTH TRADE-OFFS

  • At matched compression, QGLAS retains 102.0% and 98.4% of quality-only RL’s macro-average gains on Qwen3-4B and GLM-4.7-Flash, versus 75.5% and 70.2% for GR3 and 74.9% and 68.3% for GRLC.QGLAS also has the highest mean quality among length-controlled methods on every evaluated benchmark for both policy models.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • Across overlapping compression rates, QGLAS achieves higher QGR than GR3 and GRLC, retaining essentially all quality-only RL gains up to roughly 32% compression.The benchmark-level curves show that the advantage is not driven by a single benchmark or control coefficient.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3.1 SENSITIVITY TO REWARD GRANULARITY

  • Under fixed rollouts, GR3’s macro-average sign-reversal rate is 11.73% with dense rewards versus 0.30% averaged across four binarizations.The analysis holds responses, lengths, and rollout groups fixed; it does not reproduce RLVR training.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2 TARGETED CORRECTION OF SIGN REVERSALS

  • On Qwen3-4B, ZERO, SIGN, and RESTORE improve GR3 quality retention at similar compression, but all remain below QGLAS’s trade-off curve.ZERO achieves 86.82% QGR at 32.53% compression, compared with approximately 75.50% and 32.9% for unmodified GR3.Source: §4.3.2 · pp. 7–8, Figure 3 · p. 7

4.4.1 STRUCTURAL CONSTRAINTS FOR QUALITY-ALIGNED SHAPING

  • At comparable compression, removing all structural constraints reduces QGR from 102.0% to 60.0%; reward-level shaping yields 87.9%, and removing positive-only gating yields 74.6%.Two-sided shaping yields 83.4% QGR; these ablations support applying conciseness bonuses only to quality-favored responses while leaving longer responses unpenalized.Source: §4.4.1 · p. 8, Table 2 · p. 8

4.4.2 QUALITY-ADAPTIVE SCALING BEYOND SIGN PRESERVATION

  • With structural constraints retained, fixing shaping strength reduces QGR from 102.0% to 82.0%; fixing s+ yields 88.9%, and fixing sall yields 84.7%.QGLAS reorders fewer than 1% of pairs in the largest pre-shaping advantage-gap bin, with reordering concentrated among small-gap pairs.Source: §4.4.2 · p. 9

4.5 GENERALIZATION ACROSS REWARD SOURCES

  • Across Learned RM, Rubric-based Judge, and LLM-as-a-Judge, QGLAS reduces average response length by 28.2–32.4% while retaining 99.4–102.0% of quality-only RL gains.The reward-source comparisons use unchanged QGLAS hyperparameters.Source: §4.5 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion frames QGLAS as a way to make quality determine reinforcement direction while length adjusts its magnitude, and reports better quality–length trade-offs across evaluated settings. The appendix details training and length-control configurations, including how rarely QGLAS reaches its clipping threshold.

5 CONCLUSION

  • QGLAS achieves a better quality–length trade-off across evaluated models, benchmarks, and reward sources, retaining nearly all macro-average quality improvement at around 30% compression.The conclusion presents conciseness as a secondary preference under quality rather than a competing objective.Source: §5 · p. 9

A.1 TRAINING CONFIGURATION

  • Qwen3-4B matched-compression comparisons use three shared seeds, while each GLM-4.7-Flash configuration is trained once because of its higher cost.Each trained policy is evaluated three times.Source: §A.1 · p. 13

A.2 LENGTH-CONTROL CONFIGURATION

  • With c = 0.5, a 10% relative shortening corresponds to h_i = 0.2, and effective shaping strength in the linear regime depends on β_g/c.The configuration fixes c and varies β_g to control overall length-shaping strength.Source: §A.2 · p. 14
  • Only 0.719%, 0.790%, and 1.286% of bonus-receiving responses reach the clipping threshold under Learned RM, Rubric-based Judge, and LLM-as-a-Judge, respectively.Pooled across reward sources, 3,043 of 331,491 responses (0.918%) reach the threshold.Source: §A.2 · p. 14, Table 5 · p. 14
  • GR3 selects α as large as possible while maintaining a sufficiently high constraint-satisfaction rate; all evaluated coefficients achieve at least 99.9%.The matched-compression comparison uses α = 0.3.Source: §A.2 · pp. 14–15
  • GRLC centers component-wise, group-relative shortness weights, sets them to zero when component lengths are equal, and gives bonuses to the shortest reasoning trace and final answer.A response shortest in both components may receive both bonuses.Source: §A.2 · p. 15

A.3 EVALUATION CONFIGURATION

The evaluation uses benchmark-specific protocols and reports average quality and response length. In repeated Qwen3-4B runs, QGLAS retains higher macro-average quality than GR3 and GRLC at comparable response lengths across all three seeds.

A.3 EVALUATION CONFIGURATION

  • Each trained policy is evaluated three times, with average quality score and average response length reported.IFBench uses its prompt-level strict metric; Arena-Hard-v2 uses GPT-4.1 for judge-based evaluation.Source: §A.3 · pp. 15–16

B.1 REPEATED QWEN3-4B RUNS

  • At macro-average lengths of 2824, 2877, and 2845 tokens, QGLAS scores 32.28, versus 29.20 for GR3 and 29.14 for GRLC.The QGLAS quality advantage is reproduced across all three training seeds.Source: §B.1 · p. 16

B.2 FULL QUALITY–LENGTH TRADE-OFF RESULTS

Across the overlapping compression range, the Qwen3-4B sweep shows QGLAS retaining more quality gain than GR3 and GRLC at comparable compression. The appendix reports full benchmark quality–length results and averages matched-compression configurations across three seeds.

B.2 FULL QUALITY–LENGTH TRADE-OFF RESULTS

  • The Qwen3-4B sweep varies β_min for QGLAS, α for GR3, and (λ, β) jointly for GRLC, holding other method-specific hyperparameters fixed.Matched-compression values additionally average across three training seeds.Source: §B.2 · pp. 16–17
  • Near 32% compression, QGLAS retains 102.0% of the quality gain at 32.4% compression, versus 75.5% for GR3 at 32.9% and 74.9% for GRLC at 31.6%.The full sweep shows the separation is not specific to the matched-compression operating point.Source: Table 7 · p. 17, §B.2 · p. 17

B.3 ADDITIONAL BENCHMARK-LEVEL ANALYSIS

The benchmark-level analysis shows that some task-level changes reflect the underlying quality objective, while creative writing can show simultaneous quality improvement and response shortening under mild QGLAS settings. Aggregate quality-gain retention does not establish non-degradation on every benchmark.

B.3 ADDITIONAL BENCHMARK-LEVEL ANALYSIS

  • QGR is an aggregate retention metric and does not imply non-degradation on every individual benchmark.The quality-only RL policy may itself improve some benchmarks while degrading others relative to the base model.Source: §B.3 · p. 17
  • Under Learned RM, Qwen3-4B IFBench falls from 29.22 for Base to 26.89 for NoBonus, then to 25.96 with QGLAS at 32.4% compression.The passage reports most of this degradation occurs under quality-only RL, before length control.Source: §B.3 · p. 17
  • Under Rubric-based Judge and LLM-as-a-Judge rewards, QGLAS scores 33.44 and 32.67 on IFBench while substantially reducing response length.The corresponding NoBonus scores are 33.11 and 32.88, both above the base score of 29.22.Source: §B.3 · p. 17
  • The two mildest QGLAS settings improve Creative Writing quality over NoBonus while also reducing response length.Source: §B.3 · p. 17

B.4 FULL ABLATION RESULTS AND COMPRESSION MATCHING

The ablations test QGLAS’s structural constraints and adaptive strength while matching compression to make quality retention comparisons more direct. Removing constraints or fixing adaptive statistics yields lower quality retention than full QGLAS.

B.4 FULL ABLATION RESULTS AND COMPRESSION MATCHING

  • Compression is matched by varying only scalar shaping strength, with other settings fixed and evaluation quality excluded from operating-point selection.This matching is used because removing structural constraints can substantially change compression and confound direct quality comparisons at the original strength.Source: §B.4 · p. 18
  • Adaptive ablations retain structural constraints but replace the overall strength or one input statistic with a constant; every variant preserves original advantage signs.The constants are calibrated on stored rollout groups without using evaluation quality or evaluation response length.Source: §B.4 · pp. 18–19
  • At 31.2% CR, the joint structural ablation retains 60.0% QGR, while individual structural ablations retain 74.6–87.9%, all below full QGLAS’s 102.0%.The reported individual ablations remove positive-only gating or one-sided shaping; without compression matching, removing all structural constraints raises CR from 32.4% to 58.3% and reduces QGR to 33.8%.Source: §B.4 · p. 19
  • Fixing overall strength, s+, or sall yields 82.0%, 88.9%, and 84.7% QGR, respectively, at 29.6–30.2% compression, below full QGLAS.Retaining either statistic’s group-specific variation improves quality retention over fixed overall strength, but neither variant recovers full QGLAS’s performance.Source: §B.4 · p. 19

B.5 OPTIMIZATION INTERPRETATION AND PAIRWISE REORDERING

QGLAS treats polarity preservation as fidelity to a quality-only reference optimization, while allowing conciseness to redistribute reinforcement among quality-favored responses. Its pairwise reorderings are concentrated among near-ties, and rollout groups typically contain multiple responses eligible for shaping.

B.5 OPTIMIZATION INTERPRETATION AND PAIRWISE REORDERING

  • Overall positive-set reordering is 3.48% (Learned RM), 3.02% (Rubric-based Judge), and 1.91% (LLM-as-a-Judge), concentrated among weak quality gaps.The table defines overall rates over eligible pairs and gap-quintile rates conditional on each quintile.Source: §B.5 · p. 20
  • For Learned RM, reversal falls from 12.30% in the smallest-gap quintile to 0.95% in the middle and 0% in the largest; other sources stay below 1% in the largest.The quintile rates are conditional on pairs within each quintile.Source: §B.5 · p. 20
  • Polarity preservation does not freeze quality-induced ordering; it allows conciseness to redistribute reinforcement mainly among near-tied quality-favored responses.The method is framed as a conservative intervention conditional on a useful quality-only reference optimization; correcting a poorly specified quality reward or training procedure is a separate problem.Source: §B.5 · pp. 19–20

B.6 DISTRIBUTION OF QUALITY-FAVORED RESPONSES

  • More than 97% of rollout groups contain at least four positive-advantage responses, averaging 8.16–8.67 positive responses out of 16.Groups with no or only one quality-favored response are rare across reward sources.Source: Table 10 · p. 20, §B.6 · p. 21

C EVALUATION ROBUSTNESS

  • Arena-Hard-v2 robustness checks include matched-compression head-to-head comparisons, built-in length-controlled evaluation, and re-evaluation with an alternative LLM judge.These are three complementary checks on the evaluation.Source: C EVALUATION ROBUSTNESS · p. 21

C.1 DIRECT PAIRWISE EVALUATION AT MATCHED COMPRESSION

At closely matched response lengths, seed-matched Arena-Hard-v2 comparisons prefer QGLAS over both GR3 and GRLC across Hard Prompts and Creative Writing.

C.1 DIRECT PAIRWISE EVALUATION AT MATCHED COMPRESSION

  • The policies have closely matched average lengths: 5379, 5296, and 5447 tokens on Hard Prompts, and 1723, 1774, and 1668 on Creative Writing for QGLAS, GR3, and GRLC.The comparisons use matched-compression Qwen3-4B configurations.Source: §C.1 · p. 21
  • Seed-matched comparisons pair policies trained with the same seed; both answer orders are judged and aggregated relative to the baseline policy.Scores above 50 indicate an overall preference for QGLAS.Source: §C.1 · p. 21
  • QGLAS is preferred in all twelve seed-by-subset comparisons, with mean scores of 58.7 and 65.9 against GR3, and 55.1 and 70.1 against GRLC.The scores correspond to Hard Prompts and Creative Writing, respectively; 50 is the indifference point.Source: §C.1 · p. 21

C.2 LENGTH-CONTROLLED ARENA-HARD EVALUATION

QGLAS remains ahead of competing length-control methods in Arena-Hard-v2 after length-controlled aggregation and under an alternative judge. Its differences from NoBonus are not interpreted as statistically significant.

C.2 LENGTH-CONTROLLED ARENA-HARD EVALUATION

  • Longer NoBonus responses can confound preference-based evaluation because response length may be associated with systematically different judgments.This motivates controlling for response length in aggregation.Source: §C.2 · p. 21
  • After length control, QGLAS scores 17.30 on Hard Prompts, versus 16.60 NoBonus, 14.07 GR3, and 16.93 GRLC.The length-controlled evaluation adjusts Bradley–Terry aggregation using a normalized relative-response-length feature while keeping responses and judge decisions fixed.Source: §C.2 · p. 22, Table 12 · p. 22
  • On Creative Writing, length-controlled QGLAS scores 48.33, versus 46.20 NoBonus, 44.47 GR3, and 41.17 GRLC.QGLAS is highest among these methods on this subset.Source: §C.2 · p. 22, Table 12 · p. 22
  • QGLAS–NoBonus differences are not statistically significant; after accounting for response length, QGLAS remains close to quality-only RL and exceeds GR3 and GRLC on both subsets.The bootstrap intervals for QGLAS and NoBonus overlap.Source: §C.2 · p. 22

C.3 ROBUSTNESS TO THE CHOICE OF LLM JUDGE

  • Changing the judge alters absolute score calibration but preserves method ordering: QGLAS is close to NoBonus and exceeds GR3 and GRLC on both subsets.The alternative judge evaluates the same model outputs.Source: §C.3 · p. 22, Table 13 · p. 22

Related papers

Loading 2609.34718v1…