An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

1 Introduction

• Team scaling is a compute-allocation choice: extra calls can generate candidates, critique or revise answers, or synthesize reports, and 30-call architectures differ 2.1× in observed tokens.Most prior evidence fixes team size and base model while changing graph, roles, and synthesis together.Source: §1 · p. 1
• Scaling from three to thirty calls raises accuracy by up to 17 points on GSM8K and GSMHard, but at most four on ARC, GPQA, and MMLU; Proposer-Critic captures arithmetic gains while ranking among the weakest elsewhere.Task-averaged reporting hides this split.Source: §1 · p. 1
• An exact generate–transform decomposition separates the coverage added by more proposals from changes in how downstream computation transforms available information.The paper uses it to diagnose whether scaling is limited by missing coverage or by failure to convert coverage.Source: §1 · pp. 1–2

Debate, layers, and graphs

• Debate, layered aggregation, and workflow graphs represent distinct collaboration designs, while self-consistency and Self-Refine provide proposal-aggregation and serial-revision limiting cases.These works motivate comparing how teams communicate and process answers.Source: §2 · p. 2

Related architecture comparisons

• Broad architecture comparisons examine interactive coordination or fixed-scale tool-using agents, whereas this work targets full budget trajectories and coverage-versus-transformation accounting.Adaptive systems selecting workflows per query motivate architecture selection.Source: §2 · p. 2

Scaling and diversity

• Scaling studies connect topology to performance curves or heterogeneous channels to effective agent counts, while repeated-sampling and voting work characterize proposal generation in isolation.Critics and synthesizers produce dependent intermediate answers, so the paper uses agreement only in a matched Star/Persona-Star comparison.Source: §2 · p. 2

Oracle bounds, selection, and generative aggregation

• Candidate-pool oracles and selection-versus-synthesis work distinguish choosing from available answers from generatively combining them, which can produce answers absent from the initial pool.The paper measures coverage, loss, and recovery across heterogeneous fixed workflows.Source: §2 · p. 2

Compute-normalized evaluation

• Compute-normalized work reports that single-agent systems can match or exceed some multi-agent designs under matched reasoning-token budgets, motivating reports of both calls and observed tokens.Source: §2 · p. 2

3 Orchestration Architectures and Decomposition Framework

• The framework treats the requested node budget as calls and defines L as the first tier’s parseable candidate answers, excluding planners instructed not to answer and all later agents.Source: §3 · p. 2

3.1 Orchestration architectures

• At matched call budgets, architectures allocate proposals and later operations differently: parallel stars expand proposals, hierarchical designs filter or synthesize, and chains refine serially.At N = 30, the first tier has 29 proposals for each star, 15 for Proposer-Critic and Tournament, 22 for Tree, 23 for Diamond, and 1 for each chain.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Proposal coverage and agreement

• Exact-answer pairwise agreement measures agreement among valid proposals, but is undefined below two proposals and used only when the proposal boundary is held fixed.Role-asymmetric architectures are compared by loss, recovery, and final accuracy instead.Source: §3.2 · p. 3

3.3 Generate–Transform decomposition

• Transfer loss counts correct proposals followed by incorrect outputs, while recovery counts incorrect proposal pools followed by correct outputs; their difference gives the proposal-coverage gap to final accuracy.Recovery can occur because critics, refiners, and managers are generative reasoners rather than passive selectors.Source: §3.3 · p. 3
• Final accuracy is accounted for by proposal coverage, covered-case success, and generative recovery, allowing scaling changes to be split exactly into coverage and transformation terms.The paired analysis partitions trials by coverage transitions to attribute each stratum’s contribution; a deterministic plurality control distinguishes voting from active synthesis in QA.Source: §3.3 · pp. 3–4

Models and tasks

• The evaluation covers five 7–9B instruction-tuned models, five short-answer benchmarks, and all 164 HumanEval problems scored with HumanEval+ tests.The QA sets include ARC-Challenge, GPQA, GSM8K, GSMHard, and a hard-subject MMLU subset.Source: §4 · p. 4

Inference protocol

• Each condition has three runs with prescribed node seeds; models use temperature 0.4 and nucleus probability 0.95, and the manager’s parsed output is the final answer.HumanEval+ candidates execute in the EvalPlus sandbox.Source: §4 · p. 4

Statistics and controls

• The primary sweep analyzes 4,179,735 QA and 154,980 code team trials, averages runs within item, and uses 1,000 item-clustered bootstrap replicates for 95% intervals.Additional controls include direct N = 1 conditions, a 10×-cap long-reasoning control, and dense-communication controls; these are excluded from primary-sweep counts.Source: §4 · p. 4

5.1 Scaling returns concentrate on arithmetic

• From N = 3 to N = 30, accuracy rises by up to 17 points on GSM8K and GSMHard, 4 to 6 on code, and at most 4 on ARC, GPQA, and MMLU.A task-averaged number obscures this sharp split.Source: §5.1 · pp. 4–5, Figure 2 · p. 5

5.2 Scaling is architecture-, model-, and cost-dependent

• At N = 30, mean QA token cost ranges from 15.1k for Tournament to 31.3k for Cascading-Chain, a 2.1× spread.Proposer-Critic reaches 59.1% accuracy at 17.1k tokens; model selection matters more than architecture on multiple-choice, while arithmetic architecture spreads are larger.Source: §5.2 · pp. 5–6
• At N = 30, Proposer-Critic surpasses every other architecture in aggregate on arithmetic but ranks last or near-last on ARC, MMLU, and code.No architecture wins across tasks; Proposer-Critic is also on the accuracy–cost frontier.Source: §5.2 · p. 6

5.3 Scaling gains split into coverage and transformation

• Coverage increases in every category, but accuracy gains depend on whether the downstream transform converts that added coverage.Across six proposal-expanding designs, the extensive dividend is positive in all 150 cells, ranging from +4.9 to +14.1 points by category and design.Source: §5.3 · pp. 6–7
• On multiple-choice, the intensive term is negative in 87 of 90 cells, while Proposer-Critic’s arithmetic intensive term is +5.55 and its net gain is +14.6 points.For arithmetic, the transform sheds less coverage overall; Proposer-Critic converts coverage and adds recovery.Source: §5.3 · p. 6
• On HumanEval+, generative recovery is 0–1% for Star, Persona-Star, and Diamond, and all architectures finish below their proposal oracle.Chains instead recover more than they lose; width adds candidate opportunity, whereas serial depth repeatedly transforms one evolving solution.Source: §5.3 · pp. 6–7

5.4 Choosing an architecture: resolution beats prediction

• Testing several architectures on a labeled development set at N = 10 and deploying the best at N = 30 lands within 1.2 points of the best of eight.Routing rules based on decomposition signals do no better than the task’s historically strongest architecture.Source: §5.4 · p. 7

5.5 Controls and robustness

• On 15 hard-QA cells, fixed N = 30 architectures improve over a stronger single-agent control by +3.0 to +6.6 points; dense debate is costly and task-dependent.On code, one round of peer exchange captures debate’s benefit, while debate ties the best sparse design at twice the calls.Source: §5.5 · p. 7

• The study covers fixed, homogeneous 7–9B non-thinking teams on short-answer and executable-code tasks, not frontier or heterogeneous models, tool use, or long-form generation.Source: §6 · p. 7
• Because graph structure and role prompts change together in most conditions, the comparisons rank orchestration bundles rather than graph structure alone.Source: §6 · p. 7
• With only one code benchmark, the vanished-recovery effect cannot be separated from domain, answer format, or execution-based scoring.Source: §6 · p. 7
• The generate–transform decomposition is exact accounting, not causal mediation or a parametric forecast, and its intensive term can mix composition with transformation.Source: §6 · p. 7

• Scaling returns concentrate on arithmetic word problems, where Proposer-Critic converts added coverage at scale, while ARC, GPQA, and MMLU gain little for any architecture.Source: §7 · p. 7
• No architecture wins across models and tasks, so scaling is not a uniform lever.Source: §7 · p. 7
• The generate–transform decomposition distinguishes the coverage a workflow adds from whether its transform converts it.Source: §7 · p. 7

Supplementary Material

• The supplement documents evidence for task-dependent scaling returns, the generate–transform decomposition, and the absence of a universal architecture winner, alongside additional controls and stress tests.Source: Supplementary Material · p. 10

A Architecture Specification

• The eight conditions are directed acyclic workflows ending in an active LLM manager, with most architectures jointly defined by graph and role prompt.Source: A Architecture Specification · p. 10
• Proposer-Critic pairs proposals with critics, while Tournament duels reports; Chain and Cascading-Chain transform one evolving solution, and Diamond uses plan-conditioned solvers.Source: A Architecture Specification · p. 10

B Prompts, Inference, and Scoring

• Role prompts specify how critics, refiners, synthesizers, duelists, planners, and managers use parent reports; QA outputs are parsed, while HumanEval+ candidates must pass both test suites.Source: B Prompts, Inference, and Scoring · p. 10

C Statistical Protocol

• The primary sweep contains 4,334,715 team trials and 50,296,455 agent responses, and proposal metrics use only the first answer-producing tier.Source: C Statistical Protocol · p. 10
• The protocol uses item-level averaging and bootstrap intervals, with aggregate tables weighting model–task cells equally; intervals do not treat the tested models or tasks as random samples from wider populations.Source: C Statistical Protocol · p. 11
• Direct-baseline and long-reasoning controls use different prompts and caps, so the latter does not isolate a pure token-budget effect.Source: C Statistical Protocol · p. 11
• Prompt templates, workflow implementations, analysis code, and per-cell result files are released, with tables and figures generated by named scripts.Source: C Statistical Protocol · p. 11

D Exact Two-Margin Scaling Decomposition

• Every one of the 200 model–task–budget cell rows satisfies ∆Y = E + I to numerical precision, and item-clustered intervals show positive extensive terms for width designs but negative intensive terms for five eroding designs.Proposer-Critic’s intensive interval [−0.02, 1.05] straddles zero; the primary evidence is the equal-cell intervals, not the uncorrected per-cell significance count.Source: D Exact Two-Margin Scaling Decomposition · p. 11, Table S2 · p. 12
• On items covered at both N = 3 and N = 30, accuracy rises for Star, Persona-Star, Proposer-Critic, and Tree, is flat for Diamond, and falls only for Tournament.The aggregate covered-case decline is dominated by newly covered, harder items that convert weakly, rather than a general decline on previously covered items.Source: D Exact Two-Margin Scaling Decomposition · p. 12
• Proposer-Critic’s mean intensive term is positive because positive recovery outweighs a small negative covered-case component, despite negative intensive terms in 18/25 cells.Chain’s gain is almost entirely increased recovery; Cascading-Chain has smaller positive contributions from both intensive components.Source: D Exact Two-Margin Scaling Decomposition · p. 12
• 96.0% of cell labels at N = 30 agree with labels calibrated at B = 10, but this is held-out-budget persistence, not a deployable router.Agreement is 76.0% at B = 3 and 93.0% at B = 7.Source: D Exact Two-Margin Scaling Decomposition · p. 12

D.1 Selecting an architecture under strict hold-out

• A Star probe routed by the net oracle gap has a 1.55-point gap to the held-out hindsight best, a statistical tie with the 1.59-point training-best baseline.Coverage-based probe rules have gaps of 4.02 and 2.46 points, significantly worse than that baseline; a labeled N = 10 pilot across all architectures reaches 1.22 points but does not make the choice predictable from a cheap probe.Source: §D.1 · p. 12
• Proposer-Critic realizes 108.1% of its extensive dividend, while the other positive-extensive designs realize 25.6%–53.9%.The ratio is left undefined for the chains because their extensive terms are numerically near zero.Source: §D.1 · p. 12

E Per-model level accuracy

• At N = 30, the within-model architecture spread averages 14.0 points on arithmetic, versus 4.3 on multiple-choice and a small spread on code; no architecture wins across all five models in any category.Proposer-Critic leads four of five models on arithmetic, while every multiple-choice design lands within a few points.Source: E Per-model level accuracy · p. 12
• In paired QA scaling from N = 3 to N = 30, accuracy on both-covered items rises for Star, Persona-Star, Proposer-Critic, and Tree, is flat for Diamond, and falls only for Tournament.The negative intensive term is consistent with weak conversion of harder newly covered items, rather than a transform discarding answers it previously returned.Source: E Per-model level accuracy · p. 13, Table S3 · p. 13
• With item difficulty held fixed in the both-covered stratum, accuracy rises for the proposal-expanding Star, Persona-Star, Proposer-Critic, and Tree designs, is flat for Diamond, and falls only for Tournament.Source: E Per-model level accuracy · p. 13
• At N = 30, the non-dominated QA accuracy–cost set is Tournament (55.4%, 15.1k), Star (56.4%, 15.3k), Tree (57.3%, 16.0k), and Proposer-Critic (59.1%, 17.1k).Proposer-Critic is both the most accurate design overall and the most accurate point on this frontier; QA cost ranges from 15.1k to 31.3k tokens at the same 30-call budget.Source: E Per-model level accuracy · p. 13

F Within-Task Structure and Task-Type Specialization

• From N = 10 onward, Proposer-Critic significantly leads the field on arithmetic, and at N = 30 it beats all seven other architectures by +5.7 to +10.2 points.Its advantage is below zero at N = 3 and indistinguishable from zero through N = 7; scaling gains on multiple-choice tasks are small for every design.Source: F Within-Task Structure and Task-Type… · p. 14, Table S9 · p. 14
• Architecture spread rises from about 6 to 20 points across GSM8K difficulty bins but stays at 7–9 points across GSMHard bins.The GSM8K widening reflects easy-item ceiling effects near 80%, not evidence that architecture matters more on harder problems; 94.4% of GSMHard items matched to GSM8K for step-count analysis.Source: F Within-Task Structure and Task-Type… · p. 14
• For Star, ARC coverage rises only from 89.6 to 91.2% from N = 3 to 30, while GPQA and MMLU coverage rises 13.7 and 10.0 points but final accuracy changes by 0.0 and +1.5.On arithmetic, coverage rises 23.9 and 14.8 points and final accuracy rises +5.4 and +4.3 for Star; gains are larger for Proposer-Critic.Source: F Within-Task Structure and Task-Type… · pp. 14–15
• On MMLU-hard, accuracies span 54–62% and Chain leads in four of five subjects, so the no-universal-winner pattern is not an artifact of pooling subjects.The table reports the five subject-level results pooled over five models.Source: F Within-Task Structure and Task-Type… · p. 14, Table S10 · p. 15

G.1 Persona-Star intervention

• The matched Star–Persona-Star prompt intervention yields +5.08 extensive points and −3.72 intensive points, leaving +1.36 final points.Persona newly covers 10.53% of trials but gains only +0.94 point there; on the 59.14% covered by both systems, Persona-Star is +1.19 points more accurate.Source: §G.1 · p. 15
• Persona-Star’s proposal coverage gain is positive in 23/25 cells, while loss rises by 5.85 points and recovery changes by −0.91.The paired means close as ∆Y = ∆O + ∆r − ∆ℓ = 1.36 points; agreement intervals are negative in all 25 cells, but only seven final-accuracy intervals are positive and none negative.Source: §G.1 · p. 15
• Lost cases have lower proposal agreement in all 25 Star and all 25 Persona cells, but item difficulty remains an uncontrolled common cause of the association.The agreement-gap interval excludes zero in 24/25 Star cells and 25/25 Persona cells; these comparisons do not identify why marginal discard rates differ.Source: §G.1 · p. 15

G.2 Downstream transformation controls

• On Diamond, tally–decide and critique–synthesize changed proposal coverage by −0.02 and +0.08 points, but final accuracy fell by 3.18 and 1.46 points, mostly from reduced recovery (−2.67, −1.27).Coverage status agreed on 97.7% and 96.4% of paired trials, while complete proposal-answer vectors agreed on 70.8% and 60.0%; exact proposal replay was unavailable.Source: §G.2 · p. 15

H Direct Baseline and HumanEval+ Robustness

• The direct-baseline grid gives positive equal-cell means for every architecture, but only 9–12 of 15 individual cells improve over the stronger single-agent comparator.The best-observed row is selected after seeing the grid, making it an upper envelope rather than a fair fixed-policy estimate.Source: H Direct Baseline and HumanEval+… · pp. 15–17
• At N = 30, the Star–Persona-Star intervention’s +5.08-point extensive dividend is mostly offset by a −3.72-point intensive change, leaving +1.36 final points.The figure’s coverage-transition split shows weak conversion on newly covered cases and slightly higher Persona-Star accuracy where both systems cover.Source: Figure S1 · p. 16
• The ordinary-to-long-reasoning single-agent comparison gains +9.18 points while observed tokens rise from 528 to 1,995 per item; every fixed N = 30 sparse architecture is less accurate on those same items.The post-hoc sparse upper envelope is effectively tied at −0.16 points while using 13.5× as many observed tokens on average; this does not show that the generation cap alone causes the gain.Source: H Direct Baseline and HumanEval+… · p. 17
• On HumanEval+, generative recovery is 0–1% for the wide star-family and at most 9% elsewhere, so no architecture finishes above its proposal oracle.The width designs have positive extensive terms offset by negative intensive terms, while chains add essentially no coverage; final accuracy stays close to coverage minus loss.Source: H Direct Baseline and HumanEval+… · p. 17

I Dense Debate and No-Peer Revision

• Debate runs three rounds with each agent exposed to the other 29 reports, while Self revises only each agent’s own prior answer; both aggregate 30 answers by plurality.Source: I Dense Debate and No-Peer Revision · p. 18
• Debate R3 beats one direct call by 8.3 points on average and in 14/15 cells, yet trails Self R3 by 0.69 point on average and wins only 5/15.Debate consumes 2.5–6.7× as many cumulative tokens; the sparse upper envelope is higher in 10/15 cells and is a post-hoc, non-deployable comparison.Source: I Dense Debate and No-Peer Revision · p. 18, Table S21 · p. 19
• On HumanEval+, N = 30 Debate rises from 73.6% initial to 76.1% after one round and remains 76.1% after two; versus Self, it gains 2.4 points across all five models.One-round Debate uses 60 calls and ties the best sparse architecture at 30 calls, with a mean +1.0 point; it wins on Llama and Ministral and loses on the other three models.Source: I Dense Debate and No-Peer Revision · p. 19

J Prompt Templates

• Task-specific templates enforce distinct multiple-choice, numeric, and code output contracts, while role prompts specify strategies for personas and downstream agents.The six personas are adapted for numeric reasoning and coding; role-specific prompts cover agents such as critics, synthesizers, refiners, judges, and planners.Source: J Prompt Templates · pp. 20–21
• The response must use `FINAL: <integer> CONF: <0-100> RATIONALE: ...`, with one critique line per distinct answer and at most eight lines.The rationale must include the surviving decision.Source: J Prompt Templates · p. 23