PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Organizing Agent Teams Learn to Reason Together

Aneesh Pappu, Mirac Suzgun, Yongchan Kwon, Federico Bianchi, Batu El, Mykel J. Kochenderfer, Hancheng Cao, James Zou

arXiv:2609.22682v1cs.AIcs.MA

TL;DR

AI teams’ achievements depend on organization as well as member capability, and this paper asks whether teams can learn reusable coordination strategies from prior collaboration. SAT learns such strategies; across two domains, teams achieve the highest average accuracy among tested methods, with math and physics teams also outperforming a perfect router over independent answers.

  • Problem

    The paper asks whether fixed agent teams can learn effective, reusable ways to coordinate from collaboration, given that team achievements depend on how members organize their work.

    Source: §1 · p. 2

  • Method

    SAT uses a team member to reflect on prior collaborations, develop and evaluate coordination strategies, and select reusable strategies for inference.

    Source: §1 · p. 2

  • Results

    Across two domains, teams achieve the highest average accuracy among tested methods; math and physics teams average 66.7% versus 59.0% for the routing oracle, while knowledge-and-logic accuracy remains below its 79.6% oracle.

    Source: §1 · p. 4, §7 · pp. 13–14

  • Takeaways & Limitations

    Math and physics teams’ gains over a perfect router indicate interaction can construct solutions unavailable from selection alone, while knowledge-and-logic results show that generating a correct candidate does not ensure recognizing it.

    Source: §1 · p. 4, §7 · p. 14

    On knowledge and logic, 87.9% coverage yields only 72.8% accuracy, and demonstrability is measured post hoc rather than used to guide teamwork reflection.

    Source: §7 · p. 14

Abstract

from arXiv · show

Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknown, useful roles and divisions of labor cannot be specified in advance; teams must learn from experience how to organize reasoning as it unfolds. Human teams routinely adapt this way, while existing AI agent teams rely on fixed protocols, explicit task decomposition, or routing. We introduce Self-Organizing Agent Teams (SAT), fixed teams of AI agents that learn reusable strategies from prior collaborations to organize roles, conversational phases, participation, and information flow. These strategies enable what we call collaborative computation: agents exchange, challenge, repair, and synthesize partial reasoning into solutions no member produced independently. In two independent settings, we learn teamwork strategies that transfer unchanged to unseen benchmarks, using only 15 mathematics and 25 graduate-level knowledge problems. Across five mathematics and physics benchmarks, self-organizing teams average 66.7% accuracy, versus 48.8% for their strongest member, 58.7% for compute-matched inference by that agent, and 59.0% for a perfect router over members' independent answers; on AIME 2026, they exceed this router by 13.4 points. Because gains vary across benchmarks, we ask when self-organizing collaboration helps. Across eight benchmarks, demonstrability (the organizational-psychology construct of whether a team can distinguish correct from incorrect reasoning) strongly tracks improvement over the strongest member (Spearman $ρ=0.90$, $p=0.005$): teams benefit most when correct reasoning can be recognized once it appears. More broadly, these results suggest that organization itself can become an agent capability: agent teams can learn how to reason together and produce solutions their members could not reach independently.

1 INTRODUCTION

When useful task divisions are unknown, existing agent-team approaches generally rely on predefined candidates or subtasks rather than learning organization through collaboration. SAT learns reusable coordination from prior interactions, enabling teams to compose partial reasoning into solutions no member produced independently.

1 INTRODUCTION

  • Human teams can learn organization through collaborative experience, while existing AI approaches typically depend on predefined candidate answers or subtasks.The paper targets settings where useful divisions of reasoning are unknown and must emerge through interaction.Source: §1 · p. 2
  • Collaborative computation lets agents exchange, challenge, repair, and synthesize partial reasoning, sometimes producing a correct solution absent from every member’s independent answer.The routing oracle provides a stricter comparison than selecting among individual answers: exceeding it indicates interaction produced a solution no member supplied independently.Source: §1 · pp. 2–3
  • Across five mathematics and physics benchmarks, SAT averages 66.7% accuracy versus 48.8% for the strongest member, 58.7% compute-matched inference, and 59.0% for the routing oracle.On AIME 2026, the team exceeds the routing oracle by 13.4 percentage points.Source: §1 · p. 4
  • Across three knowledge-and-logic benchmarks, the team achieves 72.8% final-answer accuracy but 87.9% coverage, while the routing oracle reaches 79.6% coverage.The team’s coverage exceeds the routing oracle on every benchmark, but generating a correct candidate does not ensure it is selected.Source: §1 · p. 4
  • Across eight benchmarks, demonstrability—whether correct reasoning can be distinguished from incorrect reasoning—strongly tracks how much teams improve over their strongest member.The reported association is Spearman ρ = 0.90, p = 0.005.Source: §1 · p. 4
  • The findings suggest that organization itself can become an agent capability, changing what a fixed collection of models can solve.The paper reports transfer across problems, competitions, and domains, and solutions unavailable from members’ independent answers.Source: §1 · p. 4
  • SAT uses fixed teams that learn reusable strategies for roles, conversational phases, participation, information flow, and synthesis, without prescribing task-specific decompositions.Strategies are learned from prior collaborations and transferred unchanged to held-out problems and benchmarks.Source: §1 · pp. 2, 4

2 LEARNING GENERALIZABLE TEAMWORK STRATEGIES

SAT represents team organization as reusable strategies that specify conversational phases, roles, participation, and information flow. The strategies are learned through reflection and testing, then frozen for deployment on new problems.

2.1 A LANGUAGE FOR TEAMWORK STRATEGIES

  • A strategy specifies an ordered sequence of communication steps, shared teamwork norms, and persistent role prompts for a fixed roster.Each step sets participants, rounds, information flow, and shared or agent-specific prompts; strategies do not assign problem-specific subtasks.Source: §2.1 · p. 5

2.2 LEARNING TEAMWORK STRATEGIES

  • Teamwork reflection evolves strategies by inspecting transcripts, member answers, outcomes, and validation results, then proposing targeted mutations.For each source problem, the designated member runs six mutation rounds and may propose up to three candidates per round.Source: §2.2 · p. 5
  • Validation probes test whether a mutation that solves its source problem transfers to five other training problems.Probe outcomes return to the archive and guide later mutations.Source: §2.2 · pp. 5–6
  • After reflection, training-set performance guides greedy selection of up to ten complementary strategies, which are frozen before evaluation.At deployment, each strategy produces a candidate solution and certificate; a single judge audits the pool and selects the answer.Source: §2.2 · p. 6, Figure 3 · p. 6
  • A separate semantic audit screens candidates for answer values, problem-specific facts, configurations, and source-derived solution recipes.Validation rewards transfer beyond the source problem, while the audit guards against source-specific content.Source: §2.2 · p. 6

3 EXPERIMENTAL SETUP

The evaluation tests frozen, problem-independent strategy banks on held-out source problems and transfer benchmarks. It compares SAT with controls aimed at separating collaboration gains from selection, extra inference, and fixed multi-agent protocols.

EXPERIMENTAL SETUP

  • The evaluation covers five competition-mathematics and physics benchmarks and three scientific-knowledge, broad-domain question-answering, and logical-reasoning benchmarks.The same fixed, problem-independent strategies are evaluated on held-out source problems and transfer benchmarks.Source: §3 · p. 7
  • The math-and-physics bank is learned from 15 AIME-2024 training problems and tested on a disjoint split plus four transfer benchmarks.The knowledge-and-logic bank is learned from 25 GPQA Diamond training problems and evaluated on 100 disjoint GPQA problems, MMLU-Pro (n=100), and five BBEH subtasks (n=75).Source: §3 · p. 7
  • Controls include the best member, a perfect router over independent answers, fixed multi-agent methods, homogeneous teams, and compute-matched single-agent linearization.Linearization serially executes the learned roles and phases at approximately the team’s total inference budget.Source: §3 · p. 7

4 RESULTS

SAT’s learned strategies achieve the highest average accuracy across both evaluated suites, with math-and-physics gains exceeding perfect routing and knowledge-and-logic accuracy remaining below routing-oracle coverage. Case studies show how teams repair and combine partial reasoning, while coverage results reveal that generating correct certificates and selecting them are distinct challenges.

4 RESULTS

  • 66.7% math-and-physics accuracy and 72.8% knowledge-and-logic accuracy are the highest average results among tested methods in their respective suites.The averages cover five mathematics-and-physics benchmarks and three knowledge-and-logic benchmarks.Source: §4 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 9
  • The AIME-trained bank transfers unchanged across years, competitions, and physics, including 71.2% accuracy on AIME 2026.On AIME 2026, SAT is 29.0 percentage points above the best member and 13.4 percentage points above the routing oracle.Source: §4 · p. 8
  • 72.8% knowledge-and-logic accuracy is the suite’s highest average, but remains below the routing oracle’s 79.6% coverage.SAT trails Mixture of Agents on MMLU-Pro, while the homogeneous team leads on BBEH.Source: §4 · p. 8, Table 2 · p. 9
  • 75.2% math-and-physics pool coverage exceeds 59.0% routing-oracle coverage, while knowledge-and-logic coverage reaches 87.9% but selected accuracy is 72.8%.Across all three knowledge-and-logic benchmarks, pool coverage exceeds routing-oracle coverage, but this advantage does not carry through to final-answer accuracy.Source: §4 · pp. 9–10
  • On HMMT February 2026 problem 6, cross-member repair yields 3840, absent from all three initially wrong answers.o3-mini provides the invariant and factorization, DeepSeek-V3 repairs the counting error, and Claude Sonnet 4 audits the correction.Source: §4 · p. 9, Figure 5 · p. 10

5 WHEN DOES LEARNED ORGANIZATION IMPROVE TEAM PERFORMANCE?

The section examines whether demonstrability—how reliably correct reasoning is distinguished from incorrect reasoning—tracks when teams improve over their strongest member. Across eight benchmarks, the two rankings are strongly associated, but whether increasing demonstrability improves accuracy remains for future work to test.

5 WHEN DOES LEARNED ORGANIZATION IMPROVE TEAM PERFORMANCE?

  • Demonstrability is measured as the balanced rate at which a ten-model panel selects correct over incorrect team reasoning.The analysis includes eight benchmarks with at least ten eligible problems each.Source: §5 · p. 11, Figure 6 · p. 11
  • Spearman ρ = 0.90 (exact permutation p = 0.005) across eight benchmarks: demonstrability strongly tracks the rank order of team improvement over the strongest member.The rank association remains ρ = 0.86–0.96 when any single benchmark is left out.Source: Figure 6 · p. 11
  • The authors hypothesize that collaboration creates value when correct reasoning can redirect or survive later deliberation and selection.They leave testing whether task-level interventions that increase demonstrability improve team accuracy to future work.Source: Figure 6 · p. 11

6 RELATED WORK

The related work is organized around how teams compute, whether their collaboration scaffold is reusable across problems, and whether it is learned or hand-designed. SAT is distinguished by learning reusable strategies for multi-phase exchanges that let agents revise reasoning, rather than relying only on voting, one-way aggregation, or problem-specific workflows.

6 RELATED WORK

  • Debate repeatedly revises independent responses, but prior findings report that voting over initial answers accounts for much of its gains and that debate does not reliably beat single-agent methods.The section characterizes debate gains as often arising from sampling diversity and selection rather than new joint inferences.Source: §6 · p. 12
  • Feed-forward aggregation systems such as Mixture of Agents combine proposals through a synthesizer, but their one-way structure does not let agents challenge and repair reasoning through back-and-forth deliberation.Figure 7 compares this fixed interaction structure with SAT’s learned participation, roles, and repair.Source: §6 · p. 12, Figure 7 · p. 12
  • Workflow and topology methods optimize computational graphs, routing, prompts, or agent programs, whereas SAT learns reusable strategies for multi-phase conversations that exchange and revise reasoning.The closest comparison, OPTAGENT, learns interaction edges while retaining manually specified roles, a fixed exchange-and-revision operation, and majority-vote selection.Source: §6 · pp. 12–13
  • Problem-conditioned systems adapt task organization to each problem, while SAT freezes strategies learned from training problems and deploys them without prescribing a new problem’s decomposition.Conductor emits problem-conditioned workflows, and LATTE lets agents construct and revise shared task graphs during execution.Source: §6 · pp. 12–13
  • The comparison framework distinguishes joint reasoning, problem-agnostic reuse, and whether a collaboration scaffold is learned rather than hand-designed.Figure 7 contrasts debate’s repeated symmetric exchange and voting, Mixture of Agents’ one-way aggregation, and SAT’s variable participation, persistent roles, and targeted repair.Source: §6 · pp. 11–12, Figure 7 · p. 12, Table 3 · p. 13
  • Virtual Lab organizes a PI-led team with human feedback, open agent platforms enable free collaboration, and reflective-search methods optimize artifacts other than reusable multi-agent teamwork strategies.SAT learns strategies from prior collaborations and deploys them unchanged on held-out problems and benchmarks.Source: §6 · p. 13

7 DISCUSSION AND LIMITATIONS

Across two domains, learned teamwork strategies transfer unchanged to held-out problems and benchmarks, and teams outperform their strongest member and matched linearization on average. The results also identify a limitation in selecting among generated solutions and frame organization as an agent capability that can construct solutions unavailable to individual members.

7 DISCUSSION AND LIMITATIONS

  • Both teams achieve the highest average accuracy among tested methods, outperform their strongest member and matched linearization, and the mathematics-and-physics team exceeds a perfect router over independent answers.The router comparison indicates gains beyond selecting among members’ individual answers.Source: §7 · pp. 13–14
  • On knowledge and logic, the team pool reaches 87.9% coverage while team accuracy is 72.8%, so richer generation does not guarantee correct selection.The authors identify better selection or more legible certificate formats as needed to turn generated solutions into final answers.Source: §7 · p. 14
  • Future work proposes testing whether optimizing strategies for demonstrability improves accuracy and whether distilling successful multi-agent reasoning strengthens individual members for another learning cycle.The proposed recursive cycle would reassemble distilled members and learn new teamwork strategies.Source: §7 · p. 14
  • The results position organization itself as an agent capability: a fixed set of models can learn to reason together and construct solutions unavailable to any member independently.Source: §7 · p. 14

A DEPLOYED STRATEGY BANKS

The appendix records the complete learned strategy banks as deployed unchanged at test time. It specifies the phase structure and prompts, the fixed team rosters, and the held-out benchmarks on which each bank is used.

A DEPLOYED STRATEGY BANKS

  • Each strategy bank specifies phase participants, rounds, and information-flow mode, along with any persistent per-agent roles and shared team-level collaboration prompt.The notation identifies local and summary-broadcast information flow, role prompts α, and teamwork prompt τ.Source: A DEPLOYED STRATEGY BANKS · p. 18
  • The AIME-2024 bank uses o3-mini, Claude Sonnet 4, and DeepSeek-V3, while the GPQA-Diamond bank uses Llama-4-Maverick, GPT-4.1, and Gemini-2.5-Flash.Source: A DEPLOYED STRATEGY BANKS · p. 18
  • The AIME-2024 bank is deployed unchanged on the remaining mathematics-and-physics benchmarks, and the GPQA-Diamond bank unchanged on MMLU-Pro and BBEH.No phase, role, or teamwork prompt names a test problem.Source: A DEPLOYED STRATEGY BANKS · p. 18

A.1 AIME-2024 STRATEGY BANK (10 STRATEGIES)

The strategy bank organizes collaboration around independent reasoning, audits, verification, and synthesis, with distinct strategies for handling disagreement, constraints, method choice, and consensus.

A.1 AIME-2024 STRATEGY BANK (10 STRATEGIES)

  • Mechanistic audits expose load-bearing steps, while a designated dissenter challenges synthesis steps that could suppress correct minority reasoning.The team addresses challenges, revises, and finalizes its synthesis.Source: §A.1 · p. 18
  • Persistent divergences are independently recalculated and either reconciled or downweighted, including a strategy that reduces agent 2’s influence if divergence persists.The reconciliation strategy calls for verifying disputed steps; the weighted strategy specifies lower influence for agent 2 when divergence remains.Source: §A.1 · p. 18
  • Constraint-focused strategies derive necessary conditions, check candidates backward, and inventory constraints before accepting an answer.The listed checks include bounds, divisibility, modular residues, integrality, parity, and case coverage.Source: §A.1 · p. 19
  • Independent-solve synthesis compares separate reasoning chains, while method diversification assigns distinct problem-suited approaches before synthesis.Answers are resolved using mathematical steps, and the team favors independent routes or the clearest complete derivation.Source: §A.1 · p. 19
  • A consensus challenger tests early agreement for alternative derivations, missing cases, arithmetic errors, or constraint violations before deciding whether to revise.Agents are directed to support responses to the challenge with mathematical evidence.Source: §A.1 · pp. 19–20
  • Component-recombination validation checks whether independently computed quantities imply components that consistently recombine to satisfy the problem’s overall constraints.An inconsistency triggers review of intermediate arithmetic for possible overestimation.Source: §A.1 · p. 20

A.2 GPQA-DIAMOND STRATEGY BANK (10 STRATEGIES)

This section presents ten GPQA-DIAMOND teamwork strategies centered on structured critique, verification, and adjudication of claims, including P3’s protocol for adjudicating minority evidence. It also notes that forced role reversal and overlapping pairwise verification were not selected for either deployed bank.

constructive_challenge_and_preservation

  • Constructive challenge and preservation uses a designated challenger to propose repairs or alternatives and ensure well-supported claims are retained or explicitly rejected before synthesis.The team decides whether challenged reasoning survives, is repaired, or is abandoned, then synthesizes the surviving claims into a robust solution.Source: §A.2 · p. 20

option_aware_claim_critique

  • Option-aware claim critique tests scientific claims against the available answer choices and documents unresolved disagreements or pragmatic choices made despite ambiguity.Agents defend claims with evidence, assess whether weaknesses alter the answer choice, and justify any selection made under practical constraints.Source: §A.2 · pp. 20–21

discrepancy_and_contradiction_audit

  • A discrepancy and contradiction audit has a designated auditor identify inconsistencies, conflicting solutions, and unmet problem constraints for structured resolution.Agents respond to the findings, trace their causes, and resolve them through scientific reasoning against the problem statement.Source: §A.2 · p. 21

provisional_consensus_cross_validation

  • Provisional consensus cross-validation compares independent solutions, tests each strong contender against necessary conditions, and audits assumptions before finalizing.The team checks reasoning chains for units, signs, mechanisms, limiting cases, and consistency with the question’s conditions.Source: §A.2 · pp. 21–22

neglected_effects_challenge; precision_matched_effects_audit

  • Neglected-effects and precision-matched audits quantify secondary physical effects and compare their impact with the precision needed to distinguish answer options.The team incorporates effects that matter, justifies excluding negligible ones, and uses precise numerical comparison to select the closest option.Source: §A.2 · pp. 22–24

final_auditor_claim_recovery; minority_evidence_adjudication

  • Final-claim recovery and minority-evidence adjudication resurface distinct, well-supported claims outside the emerging consensus and require the team to integrate or refute them with evidence.The final auditor revisits overlooked claims, while the adjudicator explicitly determines whether dissenting claims are scientifically sound.Source: §A.2 · pp. 22–24

option_conditions_and_absence_audit

  • Option-conditions and absence auditing derives necessary conditions for each answer, evaluates supporting evidence, and scrutinizes eliminations based on presumed absence.The team demands structural or mechanistic justification that an absent signal or feature is truly absent rather than overlooked or misinterpreted.Source: §A.2 · p. 23

corrective_step_audit

  • Corrective-step auditing checks specific claims, requires scientific corrections and acknowledgement of errors, and preserves evidence-backed minority claims in the final reasoning.The final answer incorporates agreed corrections and explains why validated individual claims are retained.Source: §A.2 · p. 23

A.2 GPQA-DIAMOND STRATEGY BANK (10 STRATEGIES)

  • P3 assigns Agent 2 to adjudicate minority claims, integrating scientifically sound insights and refuting incorrect ones with evidence before the team synthesizes its answer.Agents 0 and 1 engage with the adjudication, while the team preserves independent reasoning until submissions and evidence have been compared.Source: §A.2 · p. 24
  • Forced role reversal and overlapping pairwise verification appeared in earlier searches but were selected for neither deployed bank.Source: §A.2 · p. 24

B INITIALIZATION AND COMPARISON WITH LEARNED STRATEGIES

The comparison evaluates SAT against the debate-like initialization used to seed evolutionary search; SAT has higher average accuracy in both task suites.

B INITIALIZATION AND COMPARISON WITH LEARNED STRATEGIES

  • SAT achieves higher average accuracy than Pinit in both task suites.The comparison is with the search initialization, which uses independent solutions, two rounds of debate-like exchange, and majority vote.Source: B INITIALIZATION AND COMPARISON WITH… · p. 24
  • Pinit gives each member an independent solution, runs two debate-like exchange rounds, and selects the final answer by majority vote or randomly if there is no strict majority.The protocol has no teamwork prompt, member-specific roles, or learned phase instructions, and is evaluated once per problem.Source: B INITIALIZATION AND COMPARISON WITH… · p. 24

C ADDITIONAL MECHANISMS OF COLLABORATIVE COMPUTATION

The additional cases illustrate how team interaction can repair and compose partial reasoning, preserve correct minority answers, and construct solutions absent from initial outputs.

C ADDITIONAL MECHANISMS OF COLLABORATIVE COMPUTATION

  • Earlier searches also found forced role reversal and overlapping pairwise verification, but neither strategy entered the deployed banks.Role reversal has members advocate for alternatives; pairwise verification checks all three member pairs and can trigger full-team re-solving.Source: Figure 8 · p. 25
  • The paper reports higher average accuracy for SAT than Pinit in both task suites.Table 4 compares SAT with the debate-like initialization used to seed evolutionary search.Source: Table 4 · p. 25
  • On HMMT 2026 problem 6, the team composes a correct answer from three incorrect initial answers by combining an invariant and factorization with repaired counting.o3-mini supplies the invariant and factorization but miscounts; DeepSeek repairs the count, and Sonnet audits it.Source: Figure 9 · p. 26
  • On GPQA Diamond 132, the learned peer-audit strategy repairs atom tracking after debate and aggregation preserve incorrect product counts.The team identifies aniline and two deuterioaniline products, yielding the correct answer B.Source: Figure 10 · p. 27
  • On AIME 2026 II-08, a failed integer check becomes a divisibility constraint for constructing 245; on II-01, a step audit preserves a correct minority answer.DeepSeek verifies minimality in II-08, while two wrong members revise in II-01.Source: Figure 11 · p. 28, Figure 12 · p. 28

D EVALUATION DETAILS

The evaluation details specify benchmark-specific baselines and grading, then define certificate-audit prompts for selecting among candidate answers. The prompts distinguish free-response mathematics and physics from multiple-choice knowledge and logic.

D EVALUATION DETAILS

  • Baselines select each benchmark’s highest-accuracy member; self-consistency draws K=10 samples at temperature 0.5 and uses benchmark-specific grading.HMMT 2026 uses MathArena’s answer checker; other mathematics uses math-verify, MMLU-Pro exact-matches the gold option, and BBEH uses its official evaluation logic.Source: D EVALUATION DETAILS · p. 29

D.1 FINAL-ANSWER SELECTION PROMPTS

  • The selection prompts ask a judge to audit candidate certificates rather than solve anew, then choose the answer with the strongest written support.The judge receives the problem and every candidate certificate in one prompt.Source: §D.1.1 · p. 29, §D.1 · p. 29

D.1.1 FREE-RESPONSE MATHEMATICS AND PHYSICS

  • For free-response certificates, judges record specific local defects, audit without answer frequency, then use frequency to break defect-free ties and choose the least-bad candidate if all are flawed.Defects include unsupported steps, arithmetic errors, missing cases, unjustified assumptions, or an unsupported final transition.Source: §D.1.1 · p. 29

D.1.2 MULTIPLE-CHOICE KNOWLEDGE AND LOGIC

  • For multiple-choice certificates, audits flag unsupported calculations, factual claims, eliminations, or option transitions; frequency enters only afterward, with a least-bad fallback if all are flawed.When defect-free candidates tie in answer frequency, the prompt prefers the certificate with the most complete, explicit support.Source: §D.1.2 · p. 30

E DEMONSTRABILITY SENSITIVITY ANALYSIS

The leave-one-benchmark-out analysis tests whether the demonstrability–improvement association depends on a single benchmark; every seven-benchmark subset retains a strong positive association.

E DEMONSTRABILITY SENSITIVITY ANALYSIS

  • Every seven-benchmark subset retains a strong positive association between demonstrability and SAT improvement (Spearman ρ = 0.86–0.96; exact permutation p = 0.003–0.024).The analysis recomputes the rank association after omitting each benchmark in turn.Source: Table 5 · p. 30
  • Demonstrability is the balanced rate at which a ten-model panel selects correct over incorrect reasoning; SAT improvement is accuracy minus the best member’s accuracy, in percentage points.Source: Table 5 · p. 30
  • The reported exact two-sided permutation p-values enumerate all rank permutations: 8! for the full analysis and 7! for each leave-one-out analysis.Source: Table 6 · p. 30

Related papers

Loading 2609.22682v1…