PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It

Jonathan Williams, Esin Tureci, Karthik R. Narasimhan

arXiv:2610.00991v2cs.LG

TL;DR

Majority voting and RLVR are commonly composed by training one policy and sampling it repeatedly, but RLVR can narrow the diversity that voting relies on. This paper splits the same training budget across independently trained adapters and finds that the resulting thickets outperform a fully trained single adapter across all tested model–K vote settings.

  • Problem

    The paper asks whether spending an RLVR budget on one policy improves its majority vote when the inference budget is fixed, despite RLVR’s potential to narrow sampling diversity.

    Source: §1 · p. 1

  • Method

    The paper trains K independent LoRA adapters on disjoint shards, dividing the rollout budget among them while keeping total data, training compute, and inference votes matched to the single-adapter setup.

    Source: §1 · pp. 1–2

  • Results

    Thickets beat the fully trained adapter in all 16 model–K majority-vote settings and, for K≥4, stay within 0.8 p.p. of the base or above it on every model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When the goal is to sample and vote, splitting the RLVR budget across disjoint shards avoids much of the voting damage associated with concentrating it in one policy.

    Source: §5 · p. 9

    The study covers one RL algorithm, GRPO; models up to 8B; K≤16; and vote budgets of 16–160, and thickets do not always outperform the base.

    Source: §5 · p. 9

Abstract

from arXiv · show

Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy so that its samples increasingly make the same mistakes. With every method drawing exactly $160$ completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test ($1.5$B-$8$B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to $4.8$ points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to $7.0$ points. The cause is concentration, not RLVR itself. We split the same data and training budget across $K$ LoRA adapters, each trained on its own random disjoint shard, and call the result an adapter thicket. Thickets out-vote the fully trained adapter in all $16$ (model, $K$) settings, and for $K{\geq}4$ they stay within $0.8$ points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small $K$. For $K{\geq}8$, thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from $16$ to $160$ votes, the thicket's lead over the fully trained adapter widens from $1.3$ to $3.3$ points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.

1 INTRODUCTION

The paper asks whether composing RLVR training with majority voting improves the vote at a fixed inference budget, given that RLVR can narrow sampling diversity. It finds that splitting the training budget across adapters avoids much of the voting damage caused by concentrating it in one policy.

Motivation and approach

  • A vote benefits when voters’ errors are decorrelated, while RLVR sharpening can narrow sampling diversity; random disjoint shards vote at least as well as subject-specialist shards on all four models at K=7.The thicket splits the same training pool and rollout budget across K adapters, each trained on its own shard.Source: §1 · pp. 1–2

Findings

  • The full-budget single adapter raises single-sample accuracy on every model but votes below the untrained base on three of four models, by up to 4.8 p.p.It also covers less of the answer space than the base in seven of eight model–suite settings.Source: §1 · p. 2
  • Along the single-adapter run, voter error correlation rises monotonically on every model, while majority-vote accuracy peaks by step 125 and then falls by 1.4–7.0 p.p.On Qwen2.5-3B and 7B, no checkpoint out-votes the base.Source: §1 · p. 2
  • Thickets beat the fully trained single adapter in all 16 model–K vote settings; at K≥4, they stay within 0.8 p.p. of the base or above it on every model.Against the step-matched control, thickets win 10 of 16 settings and tie one.Source: §1 · p. 2
  • At K≥8, thickets win six of eight step-matched vote comparisons and retain more single-sample gain than the matched run.Where RLVR adds pass@1 on Qwen2.5-1.5B and Llama, thicket members retain 63–78% of it at K≥8, versus 8–66% for the step-matched run.Source: §1 · p. 2
  • From 16 to 160 votes, the thicket’s lead over the fully trained adapter grows from +1.33 to +3.28 p.p.The fully trained adapter moves from level with the base (+0.15 p.p.) at 16 votes to 2.58 p.p. below it at 160 votes.Source: §1 · p. 2

2 RELATED WORK

The paper studies how RLVR’s effect on sampling diversity bears on majority voting, and distinguishes its budget-splitting approach from prior diversity and adapter-ensemble methods.

Prior work on diversity

  • Prior work links RLVR to narrower sampling distributions and reduced test-time coverage, measured through pass@k; this paper studies the consequence for majority voting at a fixed inference budget.The reported consequence can be a vote below the untrained model’s.Source: §2 · p. 3

Distinction from prior approaches

  • Rather than recover diversity within one trained policy, the paper preserves it by splitting the RLVR training budget across independent policies.It replaces Neural Thickets’ perturb-and-select procedure with reward-directed RLVR training in LoRA space.Source: §2 · p. 3
  • Unlike much LoRA ensemble work targeting uncertainty or calibration, this work studies allocating a fixed RLVR budget between one deeply trained adapter and many shallowly trained adapters.The paper does not consider extra voters produced from a single run, such as snapshot ensembles, because those apply to both sides of its comparison.Source: §2 · p. 3

3 ADAPTER THICKETS AND EXPERIMENTAL SETUP

The experiments compare four ways of fielding voters under a shared training recipe and fixed inference budget. They span four instruction-tuned models and math and code tasks, with majority-vote accuracy reported for math.

Compared methods

  • The four voter pools are the untrained base, a full-budget single adapter, its step-matched checkpoint, and K independently trained adapters on disjoint random shards.Every training problem is seen exactly once in both trained arms.Source: §3 · p. 3

Inference and measures

  • Every method pools exactly 160 completions per problem, with each voter contributing 160/K; single-policy voters are grouped streams from one pool.The study reports majority vote, coverage, pass@1, and mean pairwise Pearson correlation of voter error indicators.Source: §3 · p. 3

Seeds

  • Main experiments use three seeds, with means and standard deviations reported; the subject-versus-random shard ablation uses one seed.The seed controls data partition or ordering and completion-sampling seeds.Source: §3 · p. 3

Central comparison

  • Table 1 reports math-suite majority-vote accuracy as macro means over five benchmarks under the fixed 160-vote budget, with mean and standard deviation over three seeds.Base and fully trained single-adapter rows are constant across K by construction.Source: Table 1 · p. 4

Training and evaluation

  • Experiments use Qwen2.5 models from 1.5B to 7B and Llama-3.1-8B, across math and code problem pools with verifiable-answer or unit-test rewards.The math suite contains five benchmarks and the code suite three; code is excluded from majority-vote analysis because programs lack canonicalizable final answers.Source: §3 · p. 4

4 RESULTS

Full-budget single-adapter RLVR improves single-sample accuracy but can weaken majority voting; thickets outperform the fully trained adapter and preserve more of the vote as the budget grows.

4.1 CONCENTRATED RLVR HURTS THE VOTE

  • Full-budget single-adapter RLVR raises pass@1 on every model, yet its vote trails the base on three of four models by up to 4.8 p.p.It covers less of the answer space than the base in seven of eight model-suite settings.Source: §4.1 · p. 4

4.2 THE DAMAGE ACCRUES WITH TRAINING

  • Error correlation rises monotonically on all four models as voter accuracy rises overall, while the 160-vote majority peaks by step 125 and then declines.Llama's vote falls 7.0 p.p. from 58.5 at step 125 to 51.5 at the final checkpoint.Source: Figure 2 · p. 5, §4.2 · p. 5

4.3 SPLITTING THE BUDGET AVOIDS THE DAMAGE

  • Thickets beat the fully trained adapter in all 16 vote settings (+0.8 to +6.3 p.p.) and stay within 0.8 p.p. of the base or above for K≥4.They also exceed the adapter's coverage in all 32 model-suite-K settings.Source: §4.3 · p. 5, Figure 3 · p. 6

4.4 . . . WHILE KEEPING MOST OF THE SINGLE-SAMPLE GAIN

  • At K≤4, step-matched early stopping performs comparably; at K≥8, thickets win six of eight vote comparisons and retain more pass@1 gain.At K≥8, thicket voters are 0.9 to 5.3 p.p. more accurate on three of four models overall.Source: §4.4 · p. 7

4.5 THE COST OF CONCENTRATION GROWS WITH THE VOTE BUDGET

  • At 16 votes, GRPO-single-adapter is +0.15 p.p. versus base; at 160, it is −2.58 p.p., while the thicket lead grows +1.33 to +3.28 p.p.Margins are averaged over 16 model-K settings.Source: §4.5 · p. 7

4.6 CONFIDENCE-FILTERED VOTING

  • Confidence filtering adds +0.53 ± 0.10 p.p. for thickets but +0.01 ± 0.14 for the fully trained adapter, widening their lead from +3.28 to +3.80 p.p.The fully trained adapter's change is reported as no gain.Source: §4.6 · p. 8

4.7 RANDOM SHARDS ARE ENOUGH

  • Random shards vote at least as well as subject shards across four models (+0.2 to +0.8 p.p.); coverage, voter accuracy, and error correlation are nearly identical.The comparison used one seed.Source: §4.7 · p. 9

5 DISCUSSION, LIMITATIONS, AND CONCLUSION

The discussion argues that concentrating RLVR in one policy can reduce majority-vote accuracy, while splitting the same budget across disjoint shards preserves more of the diversity voting needs. The study’s conclusions are bounded by its tested models, algorithms, vote budgets, and sharding schemes.

5 DISCUSSION, LIMITATIONS, AND CONCLUSION

  • Across tested budgets of 16–160 votes, thickets’ lead over the base and matched GRPO-single shrinks as budget grows; at 160 votes, they beat the base in 9 of 16 settings.Source: §5 · p. 9
  • The study covers one RL algorithm (GRPO), K≤16, models up to 8B, budgets of 16–160 votes, math voting, and random or subject-based sharding.The subject-shard ablation uses a single seed; all arms train without a KL penalty, whose effect on the single adapter’s vote is untested.Source: §5 · p. 9
  • The conclusion recommends splitting the training budget across disjoint shards when the deployment plan is to sample and vote.The discussion notes that thickets use the same total data and training compute, and that K≥4 is a forgiving choice; at larger vote budgets their lead may shrink.Source: §5 · p. 9

A TRAINING DETAILS

Training uses frozen base models with only LoRA parameters trained, and partitions a fixed training pool into disjoint shards for thickets. The runs share a GRPO recipe and hyperparameters, with matched checkpoints evaluated at thicket-member step counts.

A TRAINING DETAILS

  • All four instruction-tuned models are frozen during training; only their LoRA parameters train.The models are Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B, and Llama-3.1-8B.Source: A TRAINING DETAILS · p. 12
  • Each thicket partitions the fixed 2,000-problem training pool uniformly at random into K disjoint shards, sampled without replacement.The union of the shards is exactly the single adapter’s training set.Source: A TRAINING DETAILS · p. 12
  • All runs use the same GRPO implementation and hyperparameters, with the training file as the only difference between arms.Single-adapter runs train for about 500 optimizer steps; thicket members train for one epoch on their shard, about 500/K steps.Source: A TRAINING DETAILS · p. 12, Table 4 · p. 12
  • The matched single-adapter control uses the last saved checkpoint at or before each thicket member’s step count.The matched steps are 250, 125, 50, and 25 for K=2, 4, 8, and 16, respectively.Source: A TRAINING DETAILS · p. 12

B EVALUATION PROTOCOL

The evaluation holds sample budgets fixed across methods and voter counts, using prefixes of completion pools for smaller budgets. It specifies answer canonicalization, voting, and a cross-validated confidence-filtering protocol for math responses.

B EVALUATION PROTOCOL

  • Every method and K uses exactly 160 completions per problem, with each thicket member contributing 160/K completions.Single-policy pools are regrouped into K nominal voters; the base and fully trained single-adapter plain votes are therefore identical across K.Source: B EVALUATION PROTOCOL · p. 12
  • At budget B, evaluation uses each pool’s first B completions, apportioned as B/K completions per thicket member.Smaller budgets are prefixes of larger ones, and every B is a multiple of 16.Source: B EVALUATION PROTOCOL · pp. 12–13
  • Votes use canonical answer-equivalence classes, drop null responses, and score plurality ties by expected accuracy under uniform tie-breaking.Confidence filtering ranks each voter’s responses by mean per-token log-likelihood and selects q by benchmark-level four-fold cross-validation, with the held-out fold used for evaluation.Source: B EVALUATION PROTOCOL · p. 13

C PER-BENCHMARK MAJORITY VOTE AND CONFIDENCE-FILTERED VOTING

The appendix provides per-benchmark breakdowns and visual comparisons for majority voting and confidence-filtered voting. Its figures describe comparisons and encodings, while the supplied table passages specify scope and format rather than cell-level outcomes.

C PER-BENCHMARK MAJORITY VOTE AND CONFIDENCE-FILTERED VOTING

  • Figure 5 compares per-model math-suite majority-vote accuracy for all four methods at K=2, 4, 8, and 16.Its red–blue connector labels give the thicket-minus-GRPO-single-adapter accuracy difference.Source: C PER-BENCHMARK MAJORITY VOTE AND… · p. 13, Figure 5 · p. 14
  • Tables 5–8 report per-benchmark accuracy for plain majority voting and confidence-filtered voting at K=2, 4, 8, and 16, using fixed 160-vote pools.Table 9 reports thicket accuracy under confidence-filtered voting, with q chosen by per-benchmark five-fold CV. Figure 7 reports the most frequently selected CV filter strength by setting and method.Source: Table 5 · p. 14, Table 6 · p. 15, Table 7 · p. 15
  • Figure 6 plots majority-vote accuracy against voter count at a fixed 160-vote budget, with suite macro and benchmark panels across models.Lines show seed means and bands show ±1 standard deviation; the base and fully trained single-adapter curves are flat by construction.Source: Figure 6 · p. 17

D PER-BENCHMARK COVERAGE

The section breaks suite-level coverage results down by benchmark and plots coverage against voter count for math and code.

D PER-BENCHMARK COVERAGE

  • Tables 13–16 report per-benchmark coverage for math and code across K values from 2 to 16.Tables 13–14 cover math; Tables 15–16 cover code.Source: D PER-BENCHMARK COVERAGE · p. 22, Table 13 · p. 22, Table 14 · p. 22
  • Figures 10 and 11 plot coverage against voter count using fixed 160-completion pools, with seed means and ±1 standard-deviation bands.Figure 10 covers math and Figure 11 covers code.Source: Figure 10 · p. 24, Figure 11 · p. 25
  • In the Knights & Knaves row, coverage reaches 100%, so coinciding lines obscure one another.This display caveat applies to Figure 10’s math-suite plot.Source: Figure 10 · p. 24

E VOTER ACCURACY, ERROR CORRELATION, AND A MARGIN DECOMPOSITION

This section examines voter accuracy and error correlation, then decomposes the thicket’s vote margin over the fully trained adapter. The decomposition distinguishes retained RLVR vote gain from damage avoided by splitting training.

E VOTER ACCURACY, ERROR CORRELATION, AND A MARGIN DECOMPOSITION

  • Per-benchmark error correlation uses each voter’s 160/K stored completions; suite-level values are per-voter macro means across benchmarks.The complete accuracy-versus-correlation plot sets cover both suites and K ∈ {2, 4, 8, 16}.Source: E VOTER ACCURACY, ERROR CORRELATION,… · p. 26, Figure 13 · p. 27, Figure 14 · p. 28
  • At K=16 and 160 votes, avoided damage supplies most of the thicket margin on Qwen2.5-3B (+4.8 p.p.) and Llama (+4.1 p.p.).On Qwen2.5-7B, avoided damage (+1.7 p.p.) exceeds the full margin (+1.2 p.p.); on Qwen2.5-1.5B, retained gain (+2.9 p.p.) yields a +2.6 p.p. margin after −0.3 p.p. offset.Source: E VOTER ACCURACY, ERROR CORRELATION,… · p. 26, Table 17 · p. 26
  • At K=16, fully trained GRPO-single-adapter voters move up and far right of the base in the accuracy–correlation plane; thicket members add less correlation in every panel.Thicket members gain accuracy over the base in all but Qwen2.5-7B math, where the difference is −0.1.Source: Figure 12 · p. 26, Table 18 · p. 27

F THE VOTE BUDGET: FROM 16 TO 160 VOTES

The section compares majority-vote performance as the vote budget grows from 16 to 160. Thickets gain more from added votes than the fully trained adapter, and their 16-vote results also show advantages over the base and matched control.

F THE VOTE BUDGET: FROM 16 TO 160 VOTES

G DISJOINT RANDOM SHARDS VS. SUBJECT-SPECIALIST SHARDS

The ablation compares random disjoint shards with subject-specialist shards at K=7. Across benchmark cells, random shards win more often, while subject specialists show only limited benchmark-specific advantages.

G DISJOINT RANDOM SHARDS VS. SUBJECT-SPECIALIST SHARDS

  • The comparison uses seven subject-based shards and seven random disjoint shards of equal size, with each adapter trained for 46 steps.Both arms draw from the same 2,000-problem MATH pool and use the same GRPO recipe.Source: G DISJOINT RANDOM SHARDS VS.… · p. 33
  • Across 20 model–benchmark cells, random shards have the higher majority vote in 11, subject shards in 6, and 3 tie.Subject shards lead on Countdown for Qwen2.5-3B (+2.2 p.p.); random shards lead on Knights & Knaves by +1.4 to +3.8 p.p. on three models.Source: G DISJOINT RANDOM SHARDS VS.… · p. 33, Table 22 · p. 33
  • The specialists’ only consistent edge is on GSM8K, where subject shards are at least as good as random shards on all four models, by at most 0.3 p.p.On MATH500, random shards are never worse despite the matching subject labels.Source: G DISJOINT RANDOM SHARDS VS.… · p. 33
Loading 2610.00991v2…