PaperlayerView exact revision on arXiv

Source-linked AI summary

General Decision Models: Benchmarking and Insights Beyond Jev

Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, Zhongyu Wei

arXiv:2610.03935v2cs.CL

TL;DR

The paper asks which structured decisions general decision models can make reliably and how their behavior changes when decisions compose into larger systems. It introduces JEVal and evaluates decision models across local and system-level tasks, finding strengths and limitations across both settings. It also proposes open-weight models distilled from an LLM’s reasoning, with SII-Decision-27B performing comparably to Jev on JEVal.

  • Problem

    How reliably general decision models handle different structured decisions, and how their behavior changes when individual decisions compose into larger systems, remains underexplored.

    Source: §1 · p. 2

  • Method

    The paper introduces JEVal and evaluates general decision models across benchmark tasks, interactive agent workflows, and social simulations, then distills an open-weight LLM’s reasoning into single-pass decision models.

    Source: §1 · pp. 2–3, §5.4 · p. 12

  • Results

    General decision models are strongest when decisions are resolved by available evidence but weaker on specialist knowledge and faithful uncertainty, while system-level evaluations reveal reliability and social-simulation limitations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SII-Decision-27B performs comparably to Jev on JEVal, and the evaluations show that local decision performance alone does not characterize behavior in larger systems.

    Source: §1 · p. 2

    The evaluation covers only a limited set of agentic workflows and interaction patterns, and Jev is its primary representative among general decision models.

    Source: §9 · p. 18

Abstract

from arXiv · show

General decision models, such as Jev, have recently emerged as efficient alternatives to LLMs for structured judgment and selection. But what kinds of decisions can these models reliably make, and how does their behavior change when individual decisions are composed into larger systems? To study this, we introduce JEVal, a bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, and evaluate 25 model configurations spanning general decision models and generative LLMs. Our results show that (1) general decision models are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation: they can often identify the most likely outcome while substantially overstating its probability. (2) In more dynamic and realistic systems involving long-horizon, multi-step interactions, the advantages of fast local decision making are offset by reliability failures at the system level. on $\tau$-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories. (3) In large-scale social simulation, decision models approach strong generative LLMs on individual response prediction at substantially lower inference cost, yet remain weaker in user profiling and exhibit larger aggregate estimation errors and systematic bias. Finally, we propose SII-Decision-4B and SII-Decision-27B, which internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation, with SII-Decision-27B performing on par with Jev on JEVal while answering a typical query in about 0.1 s.

1 Introduction

The paper examines where general decision models succeed on structured decisions and how their behavior changes in larger systems. It introduces JEVal, reports capability boundaries across local and system-level tasks, and proposes open-weight models distilled from an LLM’s reasoning.

1 Introduction

  • Decision models are most competitive when input evidence resolves the task, but weaker on specialist knowledge and calibrated uncertainty, long-horizon reliability, and social-simulation profiling and aggregate estimation.They can identify the most likely outcome while overstating its probability; on τ-bench, faster local decisions coincide with lower task success as errors accumulate.Source: §1 · p. 2
  • Reasoning-to-Readout Self-Distillation trains a first-answer-token decision head on an open-weight LLM’s own reasoning distributions, without labeled answers or a stronger teacher.SII-Decision-4B and SII-Decision-27B answer a typical query in about 60 ms and 110 ms, respectively, on a single H100 GPU.Source: §1 · pp. 2–3
  • SII-Decision-27B performs comparably to Jev on JEVal, while SII-Decision-4B surpasses decision models of the same size.The models provide open-weight alternatives at 4B and 27B scales.Source: §1 · p. 3
  • JEVal comprises 11,257 bilingual instances from 36 datasets across 10 application domains.The benchmark supports a systematic analysis of decision-model capabilities and failure modes.Source: §1 · p. 3
  • The paper evaluates decision models in interactive agent workflows and large-scale social simulations, where system-level behavior is not captured by isolated decision accuracy alone.In social simulation, models approach strong generative LLMs on individual response prediction at lower inference cost, but show weaker profiling and larger aggregate estimation errors.Source: §1 · pp. 2–3

2 Related Work

Prior work spans task-specific predictors, natural-language decision formats, and benchmarks for distinct capabilities and reliability dimensions. The paper positions JEVal as a unified evaluation of general decision models beyond conventional classification and isolated tasks.

From task-specific predictors to general decision models

  • Jev extends earlier task-specific predictors and natural-language label approaches toward request-time choices, binary judgments, and scores.The breadth of its generalization beyond conventional classification remains unclear, motivating JEVal.Source: §2 · p. 3

Evaluation of decision models

  • Existing benchmarks cover knowledge reasoning, long-context understanding, interactive tool use, and joint accuracy, calibration, and efficiency evaluation.Examples include MMLU-Pro, LongBench, τ-bench, and HELM.Source: §2 · p. 3
  • Prior Jev studies examine individual applications and reliability aspects, while JEVal unifies diverse decision tasks and evaluation dimensions to characterize capability breadth.The cited work includes annotation, response judging, legal-document evaluation, agent delegation, output validity, context sensitivity, and confidence calibration.Source: §2 · p. 3

3 JEVal Benchmark

JEVal represents structured judgments as decisions over finite outcome spaces and assembles compatible datasets into a normalized benchmark. Its 11,257 instances span multiple domains, task formats, and input lengths.

3 JEVal Benchmark

  • JEVal presents a benchmark overview covering task formulation, data processing, and benchmark statistics.Figure 2 depicts the processing pipeline, domain shares, and input-length distribution.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Task formulation

  • JEVal tasks use a finite decision space and three formats: selecting an option, making a true-or-false judgment, or predicting an ordered rating level.Inputs organize the state, instruction, and criteria; criteria determine the valid decision space.Source: §3.1 · pp. 3–4

3.2 Data Processing

  • Dataset collection prioritizes explicit decision objectives and well-defined outcome spaces; experts verify references, normalize samples, and cap inputs at 20,480 tokens and options at 255.The team deduplicates questions and samples at most 200 instances per subtask to improve balance across domains and subtasks.Source: §3.2 · pp. 4–5

3.3 Benchmark Statistics

  • JEVal contains 11,257 instances across 36 datasets, 58 subtasks, and 10 domains, with 9,457 English and 1,800 Chinese instances.Choice tasks account for 85.1% of instances, noul for 12.0%, and score for 2.8%; the median choice set has five candidates.Source: Figure 2 · p. 4, §3.3 · p. 5

4 Experiments

The experiments compare 25 decision-model and generative-LLM configurations using JEVal accuracy, calibration, and latency measures. Decision models match thinking LLMs in overall accuracy and are strongest when evidence is explicit, but domain-specific gaps, probability-calibration limits, and input-length-dependent latency qualify that result.

4.1 Models and Evaluation Protocol

  • The evaluation covers 25 configurations across decision models, non-thinking generative LLMs, and thinking generative LLMs, with open-weight models run locally and selected models accessed through hosted APIs.Accuracy counts invalid responses as incorrect, while ECE and Brierraw use valid responses; latency is measured on a separate 200-question test stratified by input length.Source: §4.1 · p. 5
  • JEVal evaluation measures classification accuracy, expected calibration error across ten equal-width confidence bins, raw probability-vector squared error, and response latency.ECE compares each bin's accuracy with its mean confidence; Brierraw evaluates candidate probabilities against one-hot labels.Source: §4.1 · pp. 5–6

4.2 Main Results

  • The seven most accurate configurations fall within three points of one another, including SII-Decision-27B and Jev alongside five generative LLM configurations.Open-weight generative LLMs reach this group only with thinking; SII-Decision-27B matches its thinking backbone, while SII-Decision-4B outperforms thinking Qwen3.5-4B and other 4B decision models.Source: §4.2 · p. 6, Table 1 · p. 6
  • Decision models lead on long-context and document understanding but trail the best generative LLMs by three to six points in medicine, finance, and law.SII-Decision-27B is within half a point of the best configuration on agent and tool use; no configuration exceeds 54% on personalization.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Jev, Kev-27B, Intern-Decision-4B, and SII-Decision-27B have lower ECE than every generative configuration, while ECE does not capture full probability distributions.SII-Decision-27B has the lowest ECE in Table 1, about one fifth of the best-calibrated generative LLM's; Jev and SII-Decision-27B also have the lowest Brierraw.Source: Table 1 · p. 6, §4.2 · p. 8
  • On inputs up to 1K tokens, every local model is faster than hosted Jev; SII-Decision-4B answers in under 0.1 s and SII-Decision-27B in about 0.2 s.Local-model latency grows with input length; 27B models become slower than Jev at about 2K tokens, and most 4B models between 4K and 8K tokens.Source: §4.2 · p. 8, Figure 4 · p. 8

5 Analysis: From Local Decisions to Systems

This section examines where local decision accuracy breaks down and how those decisions behave in long-context, agentic, and social-simulation settings. Across these settings, strong choice prediction does not guarantee accurate probabilities or reliable aggregate outcomes.

5.1 Struggles with Exact Computation and Probability Estimation

  • On prior items, Jev assigns the true mode 89.35% of mass versus 35.60% in the true distribution, despite choosing the most likely label.Jev answers 44% of division items and 39% of area items exactly, but 97% and 73% fall within one integer of the gold value.Source: Table 2 · p. 9, Figure 5 · p. 9, §5.1 · p. 10

5.2 Long Contexts Still Degrade Constrained Decisions

  • Jev’s LongBench accuracy falls from 93.5% at 4–8k tokens to 74.5% around 20k, and all three models use middle-position evidence least.Jev’s accuracy reaches 80.4% when supporting evidence is in the middle-position bin.Source: Figure 6 · p. 10, §5.2 · p. 11, Figure 7 · p. 11

5.3 Fast Local Decisions in Agentic Workflows

  • Jev + LLM succeeds on 66.1% of episodes versus 77.6% for the baseline, while median episode time falls from 267.1 s to 160.6 s.P95 time rises from 695.5 s to 816.0 s; local action-selection and argument errors can trigger retries, recovery attempts, and timeouts.Source: Table 3 · p. 11, §5.3 · pp. 11–12

5.4 From Individual Decisions to Population Simulation

  • Social simulation spans user annotation, individual survey-response prediction, and population outcomes, where many constrained decisions must be made at low cost.The evaluation finds competitive individual response prediction alongside weaker profiling and aggregate estimation limitations.Source: §5.4 · p. 12, §5.4.2 · p. 13, §5.4.3 · p. 13

5.4.1 Agreement in User Interest Profiling

  • Jev’s mean Jaccard similarity with the reference profiler is 53.88%, versus 65.24% and 65.86% for non-thinking and thinking Qwen3.8-27B.The reference is a frozen Qwen3-Max profile, and none of the systems’ outputs is treated as ground truth.Source: §5.4.1 · p. 12, Table 4 · p. 12

5.4.2 Individual-Level Social Survey Prediction

  • SII-Decision-27B reaches 39.87% accuracy versus 40.41% for thinking DeepSeek-V4-Flash-0731, at 8.36 s versus 211.65 s mean latency.Jev is comparable to non-thinking DeepSeek-V4-Flash-0731 at 37.69% versus 37.90%, with 0.50 s mean latency.Source: Table 5 · p. 13, §5.4.2 · p. 13

5.4.3 Population-Level Election Simulation

  • SII-Decision-27B calls 47 of 51 states and 12 of 15 battleground states, one more of each than the original pipeline, but its vote-share errors are about twice as large.Every run overestimates the Democratic share by two to six points on average, and the four state misses are Trump-won states called for Harris.Source: §5.4.3 · pp. 13–14, Figure 8 · p. 14

6 Distilling Reasoning into Decision

The section builds open-weight decision models by reading decisions from an LLM’s first answer token and distilling its own post-reasoning distribution into that readout. SII-Decision-27B performs on par with Jev on JEVal at a fraction of its latency.

6.1 Reading Decision from the First Token

  • The First-Token Decision Readout restricts the next-token distribution at the answer position to valid option labels, producing choice, noul, and score outputs in one forward pass.Its pretrained output-layer rows act as a decision head without new parameters or generated tokens; an untrained Qwen3.8-27B readout comes within a few points of Jev.Source: §6 · p. 14, §6.1 · p. 15

6.2 Learning from the Model’s Own Reasoning

  • Reasoning-to-Readout Self-Distillation trains the same model without reasoning to match the distribution it reaches at the end of its reasoning chains.The teacher averages distributions from two independent chains; gold labels do not enter the target, and a KL term anchors the student to the original model’s readout to limit forgetting.Source: §6.2 · p. 16

6.3 Results

  • SII-Decision-27B is nearly two points more accurate than Jev on JEVal and answers in a fraction of its time, placing both SII models on the accuracy–latency frontier.SII-Decision-4B has about 60 ms median latency and SII-Decision-27B about 110 ms; SII-Decision-27B also has the lowest ECE and Brierraw, while SII-Decision-4B is less calibrated than Intern-Decision-4B.Source: §6.3 · pp. 16–17, Figure 10 · p. 17
  • Full-rank and LoRA versions land within a point of each other at both model scales, while SII-Decision-27B improves about seven points over the untrained readout on newly constructed questions.The comparison supports the reported gain from learning the model’s post-reasoning distribution rather than from the weight-update method.Source: §6.3 · p. 17

6.2 Learning from the Model’s Own Reasoning

  • Reasoning improves open LLM readouts, but distillation transfers option weighing more successfully than intermediate computation.It recovers most of the reasoning gain on label-selection tasks, but a quarter or less on GPQA and MMLU-Pro; on TruthfulQA, distillation lowers accuracy when the reasoning teacher remains below Jev.Source: §6.2 · p. 15, §6.3 · p. 17

7 Future Directions

The future-directions section considers the capability frontier of general decision models and how they might be composed with generative LLMs. It points to both unresolved capability gaps and potential system-level trade-offs.

Scaling Toward Collective Intelligence

  • Fast decision models may support lower-cost, large-scale studies of collective behavior and emergent intelligence.The motivation is that generative LLM latency and computational cost limit large-scale parallel deployment in settings such as social simulation and multi-agent coordination.Source: §7 · p. 17

The Capability Frontier of General Decision Models

  • Finite output spaces do not confine decision models to simple pattern matching, but a capability gap relative to state-of-the-art LLMs remains and its extent is unclear.The paper identifies characterizing this progression as an important direction for understanding the limits of general decisional intelligence.Source: §7 · pp. 17–18

System-Level Composition of Heterogeneous Intelligence

  • Generative LLMs suit open-ended reasoning, planning, and complex generation, while fast decision models suit frequent structured decisions; combining them may balance capability, latency, and cost.The paper frames heterogeneous components as a potential path for future AI systems.Source: §7 · p. 18

8 Conclusion

The paper studies general decision models from isolated decisions through their behavior in larger systems, using JEVal and agentic settings. It also constructs open-weight models through first-token readout and self-distillation.

8 Conclusion

  • General decision models approach strong thinking LLMs on broad decision tasks, especially when available evidence resolves the decision, but remain weaker in several capability areas.The reported weaknesses include specialist knowledge, faithful uncertainty estimation, exact computation, and long-context evidence use.Source: §8 · p. 18
  • In interactive agent workflows, faster local decisions coincided with lower task success, while the trajectory audit found action errors and integration-level empty responses without isolating a dominant cause.In social simulation, efficient inference enables scalable population modeling but remains vulnerable to profiling errors, inaccurate aggregate estimates, and systematic bias.Source: §8 · p. 18
  • First-token decision readout and Reasoning-to-Readout Self-Distillation produce SII-Decision-4B and SII-Decision-27B with competitive single-pass performance.The conclusion presents these as open-weight models constructed from an LLM’s own reasoning.Source: §8 · p. 18

9 Limitations

The limitations section bounds the paper’s evaluation, construction approach, and interpretation of JEVal findings. It also identifies privacy and inherited-bias concerns in the benchmark’s task content.

9 Limitations

  • Agentic evaluation covers only tool-using agents and social simulation, leaving broader workflows and interaction patterns untested.The paper calls for evaluation in richer tool environments and multi-agent coordination.Source: §9 · p. 18
  • The evaluation covers a limited set of decision models, with Jev as the primary representative, so generalization across model families and training paradigms remains unexamined.The paper calls for broader open- and closed-source model coverage.Source: §9 · p. 18
  • Reasoning-to-Readout Self-Distillation is one construction paradigm, not an exhaustive approach to training general decision models.Other objectives, architectures, broader training data, and more scalable strategies remain future work.Source: §9 · p. 18

A Ethics

  • JEVal results should be interpreted only within the sampled tasks, not as representative evidence about real-world populations or settings.The authors note that bias may persist through data collection and annotation; sensitive examples are included to evaluate model behavior, not endorse their source assumptions or labels.Source: A Ethics · p. 23

B.1 Datasets by Domain

JEVal converts heterogeneous source tasks into shared decision formats and retains 11,257 instances from 36 datasets across ten domains; the appendix details task conversions, benchmark composition, and representative cases.

B.1 Datasets by Domain

  • The benchmark samples differ by source split and availability of labels; SciFact uses labeled dev examples because its released test questions lack gold labels.Other examples include 200 test examples for AI2 ARC and 37 evidence-grounded test questions for FinanceBench.Source: §B.1 · pp. 23–24

B.2 Benchmark Composition

  • Choice decisions dominate the set; the 9,582 choice instances span candidate spaces from two to 255 options.Binary and ordered-score decisions provide distinct output structures, and Figure 11 depicts decision-format and candidate-space composition.Source: Figure 11 · p. 25, §B.2 · p. 27

B.1 Datasets by Domain

  • Source tasks are mapped to choice, binary, or score decisions, including long-context retrieval, intent classification, and participant-preference prediction.For example, HELMET key–value retrieval uses a long dictionary as state and listed values as choices; CLINC150 uses 151 intent labels, including an out-of-scope class.Source: §B.1 · pp. 23, 26

B.2 Benchmark Composition

  • 36 datasets, 58 subtasks, and 11,257 instances make up the final evaluation set, with source language assigned by subtask.Each upstream source counts as one dataset, each evaluation configuration as one subtask, and each retained decision record as one instance.Source: Table 6 · p. 25, §B.2 · p. 26

B.3 Representative Test Cases

  • Six representative test cases illustrate how source tasks appear in the shared interface, with state, instruction, criteria, and gold label shown for each.The examples are drawn from AI2 ARC, MedQA, LexGLUE, HaluEval, LongBench, and JevBench.Source: Table 8 · p. 27, §B.3 · p. 28

C More JEVal experimental settings

The experimental settings standardize task content across model configurations while accommodating their different input and output interfaces. Prompts, response schemas, validation rules, and inference budgets specify how answers are produced and evaluated.

C.1 Input and Output Protocol

  • All configurations answer the same 11,257 instances, but decision models read probabilities from candidate heads while generative LLMs return JSON answers and probabilities.SII-Decision models use a softmax over candidate labels at the first answer position; other decision models use their native interfaces or prediction heads.Source: §C.1 · p. 28

C.2 Prompt Templates and Response Protocol

  • Generative LLM prompts preserve candidate identifiers, descriptions, order, and language while placing task state, instructions, and options or score levels in user templates.Figure 12 shows the shared system message and user templates for choice, noul, and score tasks.Source: §C.2 · p. 28, Figure 12 · p. 30
  • Strict question-specific JSON schemas require task-specific fields, and malformed or incomplete responses are rejected without repair or probability-based replacement of the returned answer.Probability vectors need not sum to one, and the independent confidence field need not equal the selected-option probability.Source: §C.2 · p. 28

C.3 Model Configurations

  • Inference configurations specify reasoning controls, token budgets, and temperature, with reported context and token limits serving as experimental budgets rather than native capacities.The settings table notes model-specific input limits and that SII-Decision’s one-token allowance supports candidate-logit readout.Source: §C.3 · p. 28, Table 9 · p. 29

D Benchmark-Level Results

This section presents benchmark-level evaluation across JEVal and a controlled τ-bench comparison, covering classification and score metrics, an LLM posterior-probability task, and JEV’s effect on agent action selection.

D Benchmark-Level Results

  • The benchmark-level results cover 25 model configurations across 58 JEVal benchmarks in ten domains, with pooled domain summaries.The section reports accuracy, ECE, and Brierraw for classification questions, MAE for score questions, and mean latency over valid responses across question types.Source: D Benchmark-Level Results · p. 30
  • Accuracy counts skips and final failures as incorrect, while ECE and Brierraw use only valid classification responses.Bold and underline mark the best and second-best distinct displayed values across all 25 configurations.Source: D Benchmark-Level Results · p. 30
  • For score questions, MAE is computed on valid score responses, and invalid outputs are excluded rather than imputed.Each question retains its defined rating scale.Source: D Benchmark-Level Results · pp. 30–31

4. Posterior probability — LLM

  • The task asks the LLM to estimate each label’s conditional probability given the state, then select the label with the highest estimate.The four labels are mutually exclusive and exhaustive; each probability must represent P(hidden label = k | information provided in state), not the probability that k is most likely.Source: §4 · p. 54

E.2 Agentic Workflow Evaluation

  • The evaluation compares the original and JEV-based τ-bench agents on 115 Retail and 50 Airline tasks, totaling 330 episodes under matched settings.Both use DeepSeek-V4-Flash as the base agent and user simulator at temperature 0, with identical task configurations, tools, simulation, execution, and reward verification; episodes allow up to 30 turns.Source: §E.2 · p. 54
  • JEV replaces only action selection: it chooses among available tools or responding, while the unchanged base LLM generates tool arguments or response text.The same τ-bench environment executes the resulting action and updates interaction state; Figure 14 illustrates the matched workflows.Source: §E.2 · p. 54, Figure 14 · p. 55

E.3 ElectionSim Results

The appendix defines how ElectionSim errors and decision-model latency are measured, and documents the first-token readout and training settings for SII-Decision. It also gives example readouts for binary and score decisions.

E.3 ElectionSim Results

  • ElectionSim reports correctly called states and battleground states alongside MAE, RMSE, and bias for predicted Democratic two-party share.Positive bias indicates overestimation of Harris; the accompanying table summary counts bias closer to zero as better.Source: §E.3 · p. 55, Table 40 · p. 55
  • Table 41 reports per-state Harris two-party vote shares and prediction errors in percentage points, marking the 15 battleground states and incorrectly called states.The archived original pipeline run covers 331,836 voters with nearly the same per-state counts as SII-Decision-27B.Source: §E.3 · p. 55, Table 41 · p. 55

F.1 Readout Template

Example: noul question

Example: score question

  • For score questions, the example converts option probabilities into an expected rating level using the ordered level values.The example’s calculation is 0·p(A) + 1·p(B) + 2·p(C) + 3·p(D).Source: Example: score question · p. 57

F.2 Training Details

  • SII-Decision-4B and SII-Decision-27B share training settings, while full-rank and LoRA variants differ in updated weights and learning rate.Table 42 summarizes the shared settings for both backbones.Source: §F.2 · p. 57, Table 42 · p. 57

F.3 Accuracy and Latency of Decision Models

  • 58.7 ms for SII-Decision-4B and 112.2 ms for SII-Decision-27B are the reported median latencies for sampled JEVal questions.Latency is measured on one H100 GPU with one request at a time, including tokenization, forward pass, and readout; Jev’s comparison value comes from its hosted API and includes network transport.Source: §F.3 · p. 57

F.4 Effect of Training Set Size

Development performance rises with training-set size at both model scales, but the observed curves support a trend rather than a scaling law. The larger set adds no further early gain, and the authors identify question quality as a key constraint on further scaling.

F.4 Effect of Training Set Size

  • Checkpoints are selected on a frozen 7,506-question development set that weights nine capability groups equally and sources equally within each group.Its score uses accuracy for classification and one minus normalized MAE for soft-nou and score questions.Source: §F.4 · p. 58
  • Sets from 5,000 to 29,326 questions are nested and keep the same mixture; the 39,279- and 48,358-question sets add sources and are drawn independently.The 48,358-question set also includes questions written by Qwen3.8-27B; SII-Decision-27B uses the 39,279-question set.Source: §F.4 · p. 57, Table 43 · p. 58
  • Development performance rises steadily with training-set size at both model scales, and full-rank and LoRA follow nearly the same curve; for Qwen3.8-27B, expanding data to 39,279 questions yields no further gain.Source: §F.4 · p. 58
  • Growth is roughly linear in the logarithm of training-set size, with a smaller gain per doubling for Qwen3.8-27B, whose scores are already higher.The authors caution that these curves show a trend, not a law, and do not support fitting a scaling law.Source: §F.4 · pp. 58–59
  • The authors attribute the plateau mainly to added-question quality: self-written science questions were too easy, while flawed hard questions can yield wrong targets.Programmatic template questions taught the template rather than the skill, so gains did not transfer to real questions; further scaling depends on hard, well-posed questions.Source: §F.4 · pp. 58–59
  • The curves have limits: development-set checkpoint selection makes absolute scores optimistic, expanded sets change the mixture, and larger-set runs had uneven seeds or incomplete training.The authors say neighboring-size gains are unsupported because some 27B sizes use one seed and 48,358-question runs stopped about halfway through an epoch.Source: §F.4 · p. 59

Related papers

Loading 2610.03935v2…