Source-linked AI summary
Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers
Pedro Tabacof, Sagar Joglekar
TL;DR
Small models are attractive for recurring agentic tasks, but evidence on whether RL can train competent negotiators and how gains vary by size is limited. The paper trains four Gemma 4 sellers with GRPO and evaluates them against held-out buyers. Shared-rate gains rise with size, while tripling the rate improves on the shared rate at every size; the authors caution that size and architecture are confounded and each size is trained once.
Problem
Small models may be more cost-efficient for agentic sub-tasks, but it is unclear whether RL can make them competent negotiators or how RL gains vary with size.
Source: §1 · p. 1Method
Results
Shared-LR gains over base rise from +0.001 at 2.3B to +0.078 at 31B, while the 3× LR improves on the shared rate at every size, with its largest point estimate at 4.5B (+0.081).
Source: §1 · p. 2, §5 · pp. 5–6Takeaways & Limitations
The results suggest tuning learning rate before concluding that a small model cannot learn to negotiate, and evaluating against buyers from more than one model family.
Source: §6 · p. 7Each size was trained once, and architecture co-varies with size, so the study cannot isolate size effects or estimate run-to-run variance.
Source: §5 · p. 5
Abstract
from arXiv · showhide
LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate ($10^{-6}$) for every size, the gain of the RL model over its base rises from $+0.001$ at 2.3B to $+0.078$ at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by $+0.032$ (2.3B) to $+0.081$ (4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.
1 Introduction
The paper asks whether reinforcement learning can make small language models competent sellers, and whether gains vary with model size and learning rate. It compares shared-recipe training across sizes and highlights that learning-rate choice and buyer-family coverage qualify the results.
1 Introduction
- Under the shared GRPO recipe, the gain over base rises from +0.001 at 2.3B to +0.078 at 31B, but each size is a single run and architecture co-varies with size.The point-estimate rise is not treated as a scaling law; about half coincides with the architecture change.Source: §1 · pp. 1–2
- At early checkpoints, tripling the learning rate improves on the shared-rate arm at every size, with the largest point estimate at 4.5B (+0.081).Against a shared-rate run trained for 104 steps, the tripled-rate arm stays ahead at 2.3B and 4.5B; differences are not detectable at 12B and 31B.Source: §1 · p. 2
- A 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer sharing a model family with the training pool.The passage also reports formatting degeneration for this arm.Source: §1 · p. 2
- The evaluation protocol uses matched scenario cells, two buyers excluded from training, a held-out domain, seed-cluster bootstrap intervals, and a deviation log.One evaluation buyer shares a model family with the training pool, while the other does not.Source: §1 · p. 2
2 Related work
Prior work studies LLM bargaining, RL for negotiation, and scaling, but the cited studies differ from this paper in task, comparison, or training setup. The paper positions its contribution as comparing RL gains across sizes and learning rates in multi-issue negotiation.
LLM negotiation and its evaluation
- Bergemann et al. [7] study single-issue, distributive bargaining, unlike our multi-issue setting; their RL stage recovers deal rate while eroding surplus, whereas our 2.3B shared-rate arm does the opposite.Source: §2 · p. 2
- Pre-LLM neural negotiators split a dialogue-act strategy layer from generation [23] [18], whereas this seller learns strategy inside the language policy, without an act layer or dialogue supervision.Source: §2 · p. 2
RL for negotiation, and how the gain scales
- Prior scaling results concern other settings: RL post-training on single-turn mathematical reasoning [33] and learned-reward RLHF [19], while multi-agent scaling remains unstudied in the authors’ account.The cited reasoning study finds larger models more compute- and data-efficient; the cited RLHF study finds larger policies benefit less from a fixed reward model.Source: §2 · p. 2
3 Environment, reward, and training
The study trains a seller policy for bilateral, multi-issue bargaining in which parties have private preferences and can trade across integrative terms. Sellers communicate in prose and update a persistent contract ledger through structured actions; training uses a terminal programmatic utility reward and GRPO.
3 Environment, reward, and training
- Scenarios pair a trained seller with a frozen LLM buyer over 7 to 10 issues, combining opposed distributive terms with integrative terms that can create joint gains through logrolling.Each party has private weights and sees its own preferences, not the other party’s.Source: §3 · p. 2
- Each turn pairs two to three sentences of seller prose with a JSON COUNTER or ACCEPT action; named counter terms overwrite a persistent ledger that the buyer sees.Negotiations alternate for at most 12 rounds, and the buyer sees the prose and resulting ledger rather than the JSON action.Source: §3 · pp. 2–3, Figure 1 · p. 3
- The seller receives its normalized utility for a closed deal or a fixed no-deal penalty, minus small malformed-turn and episode-length penalties, with no per-turn shaping or model in the reward loop.Reward constants were fixed before training and were identical across sizes; the no-deal penalty is below the seller’s worst feasible-contract utility.Source: §3 · p. 3
- Training uses a GRPO-style estimator with group-normalized advantages, a leave-one-out baseline, and no learned value model; buyer tokens are masked from the loss.Apart from learning rate and evaluated checkpoint, arms use the same recipe, including 128 episodes per step and a KL penalty of 0.04 to the initial policy.Source: §3 · p. 4
4 Experimental setup
The experiment compares four Gemma 4 sizes, shared- and higher-learning-rate arms, and two held-out buyers across matched evaluations. Its design controls scenario-level comparisons but does not isolate parameter count from architecture or compute, and several contrasts are exploratory.
4 Experimental setup
- The four dense Gemma 4 checkpoints span 13× in effective parameters, but architecture changes between the two E-series and conventional checkpoints, so cross-size contrasts do not isolate parameter count.Arms are matched on optimizer steps, meaning larger arms use more FLOPs and cross-size comparisons also confound size with compute.Source: §4 · p. 4
- Each size receives 104 shared-rate steps and a separate 3×-rate arm read at step 56; a further 2.3B arm uses 10× the shared rate.The 3× checkpoint is step-matched to the shared-rate initial runs at 4.5B and 31B, and eight steps short at 2.3B and 12B.Source: §4 · p. 4
- Evaluation uses Claude Sonnet 5 and GPT-5.6-Terra, neither in training, but Terra shares the training pool’s GPT model family; a used-car domain is also withheld.With one model per family, a same-family gain cannot be separated from a model-specific effect.Source: §4 · p. 4
- All arms face the same 1,152 episodes, with paired within-scenario contrasts and seed-cluster bootstrap intervals based on 64 seed clusters, not 1,152 independent episodes.The protocol also reports deal rate, deal-only utility, joint welfare, mean turns, and parse-failure rate, alongside prespecified behavioral guards.Source: §4 · p. 4
- The shared-LR gain over base is significant at 4.5B, 12B, and 31B, but not at 2.3B; these four prespecified contrasts receive Holm correction.All other intervals, including 3× LR and frontier-seller comparisons, are exploratory and unadjusted.Source: §4 · p. 4
- Figure 2 separates arm-level attained seller scores from paired trained-minus-base differences, and marks the E-series/conventional architecture boundary.The paired-difference intervals are not comparable with the arm-level intervals; intervals beyond the primary family are exploratory and unadjusted.Source: Figure 2 · p. 5
- Some held-out analyses may be underpowered because a prespecified doubling of held-out-domain episodes was not run due to cost.Evaluation checkpoints were fixed by step-matching rather than selected on validation.Source: §4 · p. 5
5 Results
At the shared learning rate, RL gains over base rise from +0.001 at 2.3B to +0.078 at 31B, while tripling the rate improves results at every size. Exploratory seller comparisons favor the 12B 3× LR seller over two frontier sellers, while the 4.5B seller shows no detectable difference from either.
Shared learning rate
- +0.001 at 2.3B and +0.078 at 31B are the shared-learning-rate gains over base, with point estimates rising across the four sizes.Each size was trained once, and size co-varies with architecture between 4.5B and 12B; the study fits no scaling law.Source: §5 · p. 5
Learning-rate comparison
- +0.081 at 4.5B is the largest early-checkpoint improvement of the 3× LR arm over the shared-rate arm, which it beats at every size.The 3× LR arm also gains over base at every size, with its largest gain at 4.5B (+0.089).Source: §5 · p. 6
- +0.036 at 2.3B and +0.065 at 4.5B are the 3× LR gains over the 104-step shared-rate arm; at 12B and 31B, intervals include zero.The shared-rate arm ran 104 steps versus 56 for the 3× LR arm, so this comparison is not compute-matched.Source: §5 · p. 6
Frontier reference sellers
- +0.086 over Claude Sonnet 5 and +0.097 over GPT-5.6-Terra are exploratory 12B 3× LR seller comparisons; the 4.5B seller shows no detectable difference from either.The untrained 12B base already scores as high as both reference sellers, and the 4.5B seller fits on one 48 GB accelerator.Source: §5 · p. 6
10× learning rate
- At 2.3B, the 10× LR arm gains +0.050 [+0.028, +0.073] over base, but its buyer split is +0.175, versus +0.032 to +0.079 for other 2.3B arms.Its format degenerates: 91.6% of seller turns leak a fragment of the JSON action into prose, and 53.4% begin with “I appreciate” (17.5% for the 3× LR arm).Source: §5 · p. 6
Held-out domain
- On the held-out car domain (n = 128), only two of eight gains exclude zero, and the full size ordering does not replicate.The 12B and 31B arms still score highest there, but 2.3B arms score above 4.5B arms under both rates.Source: §5 · p. 6
Utility decomposition
- At 12B and 31B, distributive terms account for 75% to 81% of seller gains, while integrative terms have the larger point estimate at 2.3B and 4.5B.Under the shared LR at 12B and 31B, buyer losses are 88% and 93% of the seller gain and joint welfare does not move detectably; the integrative-term gap excludes zero only at 4.5B under the shared LR.Source: §5 · p. 6
Post-hoc strategy analysis
- At 12B and 31B, 3× LR sellers name terms at turn 1 in 76.6% and 99.7% of episodes, but the analysis cannot establish that openings cause gains.Those openings name 6.6 and 6.2 terms on average; the 4.5B 3× LR seller opens with terms in 20.7% of episodes despite its largest pooled gain over base.Source: §5 · p. 6, Table 2 · p. 7
6 Discussion
Across four once-trained Gemma 4 checkpoints, shared-rate gains rise with size, while tripling the learning rate improves every size at early checkpoints. The authors recommend tuning learning rates and testing across buyer families, while noting important limits in the reward proxy and compute comparisons.
Discussion
- The coded reward avoids learned reward-model approximation error, but the policy can exploit its frozen opponent pool when pushed hard enough.The 10× arm’s gain concentrates on the same-family buyer; its no-deal episodes include counters without concessions.Source: §6 · p. 7
- At 2.3B and 4.5B, raising the rate improved score more than extending shared-rate training, but the compute explanation remains open.The study tested only two rates, did not tune per size, and larger sizes still receive more FLOPs at matched steps.Source: §6 · p. 7
- The study uses one environment class, one model family and RL algorithm, one run per arm, and two evaluation buyers across one held-out domain.Arms are not compute-matched, size and architecture co-vary between 4.5B and 12B, and reference sellers decode differently.Source: §6 · p. 7
- Tripling the learning rate improves on the shared rate at every size, while the 10× arm’s pooled gain concentrates on the same-family buyer.The conclusion recommends tuning learning rates before concluding that small models cannot learn to negotiate, and evaluating against buyers from multiple model families.Source: §6 · p. 7
A Limitations in full
The appendix identifies limits in the bargaining setup, model and training comparisons, evaluation coverage, and strategy analysis. These boundaries constrain which broader conclusions the experiments support.
One environment class
- The setting tests bilateral, multi-issue bargaining with a seller that always opens; buyer-side learning and co-adaptation are untested.The seller-side results concern frozen opponents rather than equilibrium behavior; single-issue and multi-party settings may behave differently.Source: A Limitations in full · p. 11
Size and architecture co-vary
- Size and architecture co-vary across 4.5B and 12B, so the cross-boundary gain difference cannot isolate a parameter-count effect.Within-pair gain rises are not well resolved, and the paper says separating size from architecture requires a second family with conventional small models.Source: A Limitations in full · p. 11
One model family, one seed per size
- Within one Gemma 4 release, each size was trained once, so the size trend cannot be separated from run-to-run training variance; comparisons match training steps, not FLOPs.The intervals quantify scenario sampling only.Source: A Limitations in full · p. 11
Two learning rates, not a sweep
- Only two learning rates and one RL algorithm/reward shape were tested, not per-size optima or alternative credit-assignment methods.The 3× rate improved early-checkpoint scores at every size, but the authors say a full sweep at four sizes needs substantially more training budget.Source: A Limitations in full · p. 11
Two evaluation buyers
- Evaluation covers two buyers and one held-out domain at n = 128, limiting evidence about opponent-conditional behavior and domain transfer.The held-out panel is underpowered, and the prespecified extension was not run.Source: A Limitations in full · p. 11
Partial strategy analysis
- Strategy descriptors were chosen after results were known, and no Nash distance or Pareto efficiency was computed.Joint welfare is the only reported efficiency proxy; testing whether opening behavior causes gains would require fixed openings.Source: A Limitations in full · p. 11
No scripted baseline
- No scripted seller baseline tests how much score a fixed strategy could reach; frontier sellers also differ in decoding and self-play exposure.Each frontier seller meets a same-model buyer on half its episodes, unlike Gemma arms, though removing self-play would barely move either reference score.Source: A Limitations in full · pp. 11–12
Off-schema terms
- Off-schema terms can enter the ledger without being scored, and training/evaluation prompts and ACCEPT handling differ.All arms are evaluated under the re-rendered evaluation scaffold, but only trained arms were optimized on the training prompt; evaluation ignores terms attached to ACCEPT.Source: A Limitations in full · pp. 11–12
B Full evaluation grid and the 2.3B deal-rate decomposition
Figure 3 tracks shared-recipe gain over base across training steps and decomposes the 2.3B outcome. The accompanying tables document the evaluation grid and outcome measures.
Evaluation grid
- Table 3 decomposes headline arms, while Table 4 gives the full Figure 2 grid; every arm uses the same 1,152 paired episodes.Table 3 reports deal rate, utility per closed deal, and joint welfare; Figure 3 shows shared-recipe gains against steps.Source: B Full evaluation grid and the 2.3B… · p. 12
Figure 3
- Added-step gain point estimates rise with size: −0.004, +0.015, +0.038, and +0.055 for 2.3B, 4.5B, 12B, and 31B.Figure 3a plots paired gain over base against combined optimizer steps with 95% intervals; continuation adds 40 steps at 2.3B and 12B and 48 at 4.5B and 31B.Source: Figure 3 · p. 12
- At 2.3B, deal rate falls from 0.939 to 0.909 while utility per closed deal rises from 0.469 to 0.496, leaving score flat.Figure 3b shows the shared-recipe change; the caption notes Table 5 shows a larger learning rate moves both measures.Source: Figure 3 · p. 12
C What the held-out panel separates
The held-out car domain shows limited and uneven evidence across arms, while exploratory panel comparisons distinguish some pooled and held-out rankings. The planned adaptive extension was not run despite its prespecified condition being met.
Held-out car results
- Under 3× learning rate, held-out car gains are +0.059, +0.042, +0.020, and +0.006 from 2.3B through 31B; only the 2.3B interval clears zero beyond the Monte Carlo band.The 104-step 12B arm is the other of the eight reported intervals that excludes zero.Source: C What the held-out panel separates · p. 12
- The 31B 3× arm has the highest pooled score but the smallest held-out gain among the four 3× arms.In car transcripts, it opens in the buyer’s voice on 78.1% of episodes, versus 14.1% for the 104-step 31B arm.Source: C What the held-out panel separates · p. 13
Held-out panel comparisons
- Sonnet 5 leads Terra by +0.068 on the held-out panel, despite no detectable difference on training domains or pooled results.The 31B-versus-12B ranking also flips between pooled and held-out panels at the specified unequal steps; the authors call these exploratory contrasts at n = 128.Source: C What the held-out panel separates · p. 13
Adaptive evaluation
- The prespecified panel-doubling condition was met for the largest trained model, but the extension was not run; n = 128 intervals have about ±0.03 half-width.The protocol required applying the extension to both buyers and all ten frozen-matrix seller arms, beyond either evaluation round’s budget.Source: C What the held-out panel separates · p. 13
D Training diagnostics: the small arm was moved least
Training diagnostics indicate that the 2.3B shared-recipe policy remained close to its base while retaining higher entropy than the conventional arms. The higher learning rate moved its measured KL into the larger arms’ range, while checkpoint read points are not interchangeable across runs.
D Training diagnostics: the small arm was moved least
- The 2.3B shared-recipe deal-rate decline is concentrated in acquisition, job_offer, and partnership_dissolution, where rates fall to 0.74–0.80.Five of the other six domains stay above 0.93; media_rights is 0.90.Source: D Training diagnostics: the small arm… · p. 13
- At the end of shared-recipe continuations, KL rose with size from 0.0088 at 2.3B to 0.0485 at 31B, while the 2.3B policy had higher entropy than either conventional arm.The 2.3B entropy was 0.269 →0.261; 4.5B was higher still at 0.466 →0.443.Source: D Training diagnostics: the small arm… · p. 13
- Table 4 covers 1,152 paired episodes per arm; its S score is a paired score with 95% seed-cluster bootstrap intervals.The table’s held-out domain has 128 episodes; its bad-parse rate counts seller and buyer turns.Source: Table 4 · p. 14
- 0.047 was the 2.3B policy’s measured KL after a three-fold learning-rate increase, alongside a gain of +0.037 [+0.017, +0.057].Its KL was 0.0088 at the end of the shared-recipe continuation.Source: D Training diagnostics: the small arm… · pp. 13–14
- At step 56, 31B’s 3× learning-rate KL was 0.437 versus 0.027 for its initial run, while 12B’s was 0.126 and 10× LR’s was 0.313.These KL readings compare arms measured at step 56 and are not interchangeable with other read points.Source: D Training diagnostics: the small arm… · p. 14
E Guard failures: the 10× LR arm and the 31B 3× LR arm
The higher-rate arms have different behavioral profiles: the 2.3B 10× arm raises pooled score but fails guards, while the 31B 3× arm passes prespecified guards yet shows post-hoc style concerns. The section also clarifies the episode ledger and serving requirements.
E Guard failures: the 10× LR arm and the 31B 3× LR arm
- +0.112 [+0.084, +0.141] was the 2.3B 10× arm’s gain over base against GPT-5.6-Terra, versus −0.012 [−0.040, +0.017] against Claude Sonnet 5.Its buyer split was +0.124 [+0.089, +0.160]; the arm also failed three guards, including parse failures and unfilled placeholders.Source: E Guard failures: the 10× LR arm and… · p. 15
- The 31B 3× arm passed all four prespecified guards but failed two post-hoc style guards: opener concentration and held-out role confusion.It had 1.000 deal rate, zero walk-aways, zero cap episodes, and joint welfare 1.239 versus 1.202 for the 104-step arm; the authors describe the style checks as post-hoc.Source: E Guard failures: the 10× LR arm and… · pp. 15–16
- The 31B 3× arm’s style costs occurred against an arm it did not measurably beat, with an interval of −0.014 to +0.020.Its step-56 KL was 0.437, compared with 0.313 for the 10× arm, which games the opponent pool; policy-change size did not track Goodharting in these runs.Source: E Guard failures: the 10× LR arm and… · pp. 15–16
F The quoted episode
- In the quoted episode, COUNTER terms overwrite the ledger, while terms attached to ACCEPT are ignored and the deal is scored on the current ledger.After the seller’s turn 7, the ledger had five of ten terms; the buyer later changed accessories, and the seller accepted that ledger on turn 11.Source: F The quoted episode · p. 16
G Serving cost of each arm
- The 31B arms required four 48 GB accelerators for a 16k-context KV cache, while every smaller arm fit on one.The 12B 3× seller was the smallest arm whose paired contrasts against both frontier reference sellers excluded zero; the comparison is serving capacity, not metered cost or throughput.Source: G Serving cost of each arm · p. 16
H Training recipe
The training recipe holds most choices constant across sizes and rates, with learning rate and evaluated checkpoint varying by arm. Compute accounting is approximately proportional to parameter count for the matched-step comparison, and the models retain their own chat templates.
H Training recipe
- The 3× and 10× arms scale both the peak learning rate and cosine floor by the same factor, while hardware-forced differences do not enter the objective, data, or optimizer hyperparameters.The sizes retain their own chat templates by design.Source: H Training recipe · pp. 16–17
- GRPO uses group-normalized advantages and a leave-one-out baseline without a learned value model; each arm trains on 128 episodes per step.The recipe uses a 0.04 KL penalty against the initial policy, warmup–cosine scheduling, and one training seed per size-by-rate arm.Source: H Training recipe · pp. 16–17
I FLOP accounting for the matched-step budget
- At matched steps, token counts differ by at most about 30% while parameter count differs by 13×, making per-step compute approximately proportional to parameter count.The 30% accounting applies to base and shared-recipe Gemma arms, not the learning-rate-variant arms.Source: I FLOP accounting for the matched-step… · pp. 17–18
- The study does not report measured FLOPs or wall-clock GPU-hours, and the 31B increase from 0.564 at 56 steps to 0.620 at 104 could reflect a schedule-restart artifact.The authors cannot separate restart effects from extra training.Source: I FLOP accounting for the matched-step… · p. 18
J Chat-template divergence across sizes
- The E-series prompts are 10 tokens and conventional-size prompts are 14 because their published templates differ when thinking mode is disabled.Each base counterpart uses the same template as its trained counterpart, leaving the base-versus-RL contrast unaffected.Source: J Chat-template divergence across sizes · p. 18
K Evaluation protocol deviations
The authors document deviations from the frozen evaluation protocol, including a harness correction whose effect exceeded the reported contrasts. Checkpoint substitutions and read-point choices qualify several comparisons; a planned held-out-panel extension was not run.
K Evaluation protocol deviations
- The corrected harness changed arm-level S by up to about 0.4; every reported result uses that harness, with no results from the earlier version mixed in.The earlier harness stripped seller prose, shortened the seller horizon, and used different decoding settings.Source: K Evaluation protocol deviations · p. 19
- The 31B 104-step arm was read at step 48 of its continued run, not its validation peak or final step, so its reported +0.078 gain is the conservative choice.Validation was 0.645 at step 48, 0.676 at the peak, and 0.659 at the final step.Source: K Evaluation protocol deviations · p. 19
- The 2.3B 3× and 10× arms were read eight steps earlier than the shared-LR arm; the 10× arm’s step-64 validation score was higher, so +0.050 may understate it.The authors base their case against the 10× arm on its behavior.Source: K Evaluation protocol deviations · p. 19
- The 12B 3× arm was read at step 56 to match other 3× arms, although its validation peaked at step 32 and declined to 0.449 at step 56.A step-64 read would probably score lower; none of the five higher-rate arms was in the frozen arm matrix.Source: K Evaluation protocol deviations · p. 19
- The prespecified held-out-panel extension was not run, and Nash distance and Pareto efficiency were not computed; joint welfare is the sole efficiency proxy.Strategy descriptors and the buyer-split interval were added at camera-ready and are exploratory.Source: K Evaluation protocol deviations · pp. 19–20