Source-linked AI summary
Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines
Jingjie Ning, Xueqi Li, Chengyu Yu
TL;DR
Multi-LLM revision gains may reflect stronger-model re-solving rather than genuine error correction, but existing end-to-end results do not distinguish these sources. The paper introduces a four-condition decomposition and finds that task structure and draft quality determine whether re-solving, scaffolding, or content contributes most. These findings support targeted pipeline designs instead of blanket revision strategies.
Problem
End-to-end revision gains do not reveal whether improvement comes from transferred draft information or the reviewer’s independent re-solving.
Method
A four-condition experiment additively decomposes second-pass gains into re-solving, scaffold, and content effects across two model pairs, three benchmarks, and role directions.
Results
Revision gains are task-dependent: MCQ gains are mostly consistent with stronger-model re-solving, while code gains are scaffold-driven and weak draft content can be harmful; strong drafts benefit weak reviewers.
Takeaways & Limitations
Task structure and draft quality should guide routing and collaboration design, with targeted routing for constrained reasoning and scaffold-centered pipelines for structured generation.
Takeaways & Limitations
The evaluation controls model identity in prompts and uses a fixed MCQ answer format with the answer letter last to reduce anchoring.
Abstract
from arXiv · showhide
Multi-LLM revision pipelines, in which a second model reviews and improves a draft produced by a first, are widely assumed to derive their gains from genuine error correction. We question this assumption with a controlled decomposition experiment that uses four matched conditions to separate second-pass gains into three additive components: re-solving, scaffold, and content. We evaluate this design across two model pairs on three benchmarks spanning knowledge-intensive MCQ and competitive programming. Our results show that the gains of multi-LLM revision are not monolithic, but depend on task structure, draft quality, and the type of draft information. On MCQ tasks, where the answer space is constrained and drafts provide little structural guidance, most gains are consistent with stronger-model re-solving, and directly routing queries to the stronger model can be more effective than revising a weak draft. On code generation tasks, however, two-stage prompting remains useful because even semantically null drafts can provide substantial structural scaffolding, while weak draft content can be harmful. Finally, role-reversed experiments show that strong drafts clearly benefit weak reviewers. Ultimately, our findings demonstrate that the utility of multi-LLM revision is dynamically bottlenecked by task structure and draft quality, necessitating more targeted pipeline designs rather than blanket revision strategies.
1 Introduction
The paper challenges the assumption that revision gains reflect error correction by decomposing second-pass improvements into re-solving, scaffold, and content effects. Across tasks, these mechanisms vary: MCQ gains are mostly consistent with stronger-model re-solving, whereas code gains rely more on scaffolding and can be harmed by weak draft content.
- End-to-end revision gains cannot reveal whether improvement comes from transferred draft information or the reviewer’s independent re-solving.This distinction determines whether direct routing to the stronger model or multi-stage collaboration is preferable.
- The four-condition design additively separates second-pass gains into re-solving, scaffold, and content effects across two model pairs and three benchmarks.The evaluation covers GPQA Diamond, HLE, and LiveCodeBench, with an additional role-reversed setting.
- On knowledge-intensive MCQ tasks, content effects are non-significant and most observed gains are consistent with stronger-model re-solving.This pattern holds across both model pairs and datasets.
- On code generation, weak-draft content performs significantly worse than structure-only scaffolding, with gains driven mainly by scaffold and the disadvantage increasing with problem difficulty.
- Strong-model drafts significantly help weaker reviewers on both MCQ and code tasks, showing that draft utility depends on draft quality and information type.
- The recurring task-dependent regimes motivate distinguishing transferred semantics from structural coordination and additional downstream computation in collaborative systems.
2 Related work
Prior work develops revision, critique, self-correction, and multi-agent collaboration methods, but often leaves the source of their gains unresolved. This paper positions its contribution as attributing revision improvements to distinct mechanisms rather than treating revision as a single process.
- Iterative refinement methods use critique, feedback, memory, structured intermediate reasoning, decomposition, execution results, or tool validation to improve model outputs.
- The paper addresses an attribution gap by identifying which part of a revision procedure produces its gain rather than only measuring end-to-end improvement.
- Research on intrinsic self-correction reports that revision without reliable external feedback can be ineffective or degrading, while sampling and selection can explain apparent gains.
- Multi-agent debate and capability-asymmetric collaboration improve some outcomes, but recent analyses indicate that ensembling and draft quality complicate interpretations of those gains.
- Test-time scaling studies show that abandoning flawed trajectories and re-solving can outperform patching them, motivating explicit separation of re-solving from revision effects.
3 Method
The method uses four matched conditions to decompose second-pass accuracy gains into re-solving, scaffold, and content contributions. It controls draft format and evaluates the decomposition across model pairs, capability directions, and MCQ and code benchmarks.
- The four-condition framework decomposes total second-pass gain x2 − x1 into three interpretable components within a fixed revision pipeline.
- The study evaluates two model pairs in weak-to-strong and role-reversed settings across GPQA Diamond, HLE, and LiveCodeBench.The benchmarks contain 198, 451, and 1,054 questions or problems, respectively.
- The generator baseline x1 answers directly, while standard revision x2 gives the reviewer the question and generator output under a review prompt.
- The re-solving control x3 gives the reviewer only the question, isolating direct-answer capability while leaving review framing to x2.
- The scaffold control x4 supplies a semantically null placeholder draft in the same review format, separating structural framing from task-specific content.
- A negative content effect indicates that the generator’s draft misleads the reviewer, while scaffold measures review framing and re-solving measures capability difference under direct answering.
- The accounting permits joint variation among effects as task structure, reviewer capability, and draft quality change, rather than assuming statistical independence.
4 Results and Analysis
Second-pass gains separate into task-dependent mechanisms: stronger-model re-solving dominates MCQ, while scaffold drives code gains and weak draft content can harm. Role reversal shows that sufficiently strong drafts can transfer useful content to weaker reviewers, making draft quality central to pipeline utility.
- MCQ: Re-solving-dominated gains: On MCQ tasks, most second-pass gains are consistent with stronger-model re-solving, while content remains near zero across datasets and model pairs.The constrained answer space and limited structural guidance let stronger reviewers discard weak rationales and reconstruct answers independently.
- MCQ: Draft-quality dependence: In the strong→weak setting, re-solving becomes negative, while sufficiently strong drafts provide positive content that compensates for the weaker reviewer’s capability loss.Content is significantly positive in both GPQA pairs and HLE Pair 2, but HLE Pair 1 remains weak and non-significant because its supplementary generator reaches only 18.0% accuracy.
- Code: Scaffold-dominated gains: On LiveCodeBench, scaffold rather than re-solving dominates gains, and weak-draft content is significantly negative in both model pairs.The null scaffold outperforms standard revision, indicating that code-shaped structure can help while concrete weak implementations can anchor the reviewer to brittle artifacts.
- Code: Structural scaffolding: A null code draft can improve the second pass by shifting it from free-form solving toward structured submission repair despite carrying no algorithmic solution.The scaffold supports wrapper completion, I/O normalization, and production of a complete executable artifact; Null Scaffold and True-Null Scaffold differ negligibly.
- Mechanistic case studies: Diagnostic cases are dominated by re-solving on MCQ but shift toward scaffold-positive and content-negative cases on LiveCodeBench.Representative MCQ cases show both independent recovery from a weak premise and drift away from a correct weak answer, whereas code cases reflect artifact repair and implementation anchoring.
- General framework: The decomposition treats each handoff as a reusable unit for attributing gains to downstream capability, structural scaffolding, and transmitted semantic content.The framework is intended to extend beyond pairwise revision to cascades, planner–executor–verifier systems, debate, and reflection architectures.
5 Conclusion
The decomposition framework shows that second-pass gains arise from different sources across tasks and reviewer directions. These findings support designing collaboration around task structure, draft quality, and the distinction between semantics and structural coordination.
- On MCQ tasks, apparent improvements are overwhelmingly driven by stronger-model re-solving, while content effects remain near zero.
- On code generation, semantically empty scaffolds provide substantial value, whereas weak-draft content can actively degrade performance.
- Strong-to-weak experiments show that high-quality drafts transfer useful capability downstream, while re-solving becomes costly for weaker reviewers.
- The same qualitative regimes recur across both heterogeneous model pairs, supporting task- and draft-aware pipeline design rather than blanket revision.
- The framework audits sequential computation through downstream capability, interface structure, and upstream semantics across cascades and agent graphs.
A Experimental Setup and Prompt Templates
The experiments use matched model-role assignments, controlled revision conditions, semantically null drafts, and selective caching to isolate re-solving, scaffold, and content effects. Prompt templates are separated by task and function to preserve comparable condition logic.
- Experimental Setup: Two model pairs are evaluated in primary weak→strong and supplementary strong→weak directions, with roles swapped within each pair.
- Experimental Setup: x1 is generator-only, x2 is standard revision, x3 is question-only re-solving, and x4 uses the review prompt with a semantically null draft.
- Prompt Templates: The x2 and x4 conditions share the critique template, while x1 and x3 share the direct-answer template, isolating draft and framing effects.
- Null Drafts: The coding ablation x5 replaces x4’s format-matched placeholder with a maximally neutral stub to test identifier leakage and format matching.
- Null Drafts: MCQ null drafts preserve the three-line answer format, while code null drafts are syntactically valid Python stubs without task-specific semantics.
- Execution Controls: Generator outputs are cached and reused across downstream variants, whereas critique and re-solving calls bypass the cache and run freshly.
- Prompt Templates: Direct-answer, review, and null-draft templates are maintained separately by task and condition.
B Complete Experimental Results
The complete results report accuracies and percentage-point decompositions across all experimental conditions, datasets, settings, and model pairs. The tables organize performance by condition and quantify re-solving, scaffold, content, and total effects.
- Table 3 reports accuracy for all 48 experimental conditions across both experiment pairs.
- Pair 1 uses Gemini Flash Lite as generator and GPT-5-mini as reviewer, while Pair 2 uses GPT-4o-mini as generator and Gemini Flash as reviewer.
- Table 4 reports effect decompositions in percentage points for re-solving, scaffold, content, and total second-pass gain.
B.3 McNemar’s Test Results
The statistical results use McNemar’s tests for key condition comparisons and report LiveCodeBench accuracy and content effects by difficulty tier. These analyses provide significance testing and difficulty-stratified decomposition views.
- McNemar’s Tests: Tables 5–7 report full McNemar’s test statistics for key comparisons on GPQA, HLE, and LiveCodeBench.
- McNemar’s Tests: Significance is marked at p < .05, p < .01, and p < .001, with discordant-pair counts defined by condition correctness.
- Difficulty Analysis: Table 8 breaks down LiveCodeBench accuracy and the content effect across difficulty levels in the Pair 1 primary setting.
B.5 Scaffold Ablation on Identifier Retention
The scaffold ablation tests whether identifier retention explains scaffold-control performance by replacing task-shaped placeholders with a fixed generic stub. Results show negligible differences, while the benefit remains tied to presenting a syntactically valid, code-shaped object rather than trivial lexical cues.
- Ablation design: The ablation compares Null Scaffold and True-Null Scaffold conditions that differ only in the placeholder shown to the reviewer.The True-Null Scaffold replaces the task-shaped placeholder, including an extracted function name when available, with a fixed generic stub.
- Results: 87.0% versus 86.4% accuracy for Pair 1 showed a negligible Null-versus-True-Null difference (p = 0.617).
- Results: 86.0% versus 85.1% accuracy for Pair 2 likewise showed a negligible difference (p = 0.481).
- Interpretation: The results provide no evidence that scaffold-control performance is primarily driven by retained function names or simple task-specific lexical cues.
- Interpretation: The ablation does not show that all structural signals are irrelevant or that the two scaffold variants are behaviorally identical in every respect.Its narrower conclusion is that scaffold gains are not reducible to identifier retention alone; the main benefit appears to come from a syntactically valid, code-shaped intermediate object under review framing.