LLMs Can Generate a Better Answer by Aggregating Their Own Responses

1 Introduction

• Standard LLM training supports text generation but lacks explicit supervision for discriminative judgment, while specialized training requires extensive data curation and computational resources.The introduction links these constraints to the difficulty of relying on model-generated feedback or selection criteria.Source: §1 · p. 2
• External feedback requires task-specific design, while self-consistency uses majority voting and is limited to tasks with specific, verifiable answer tokens.Self-consistency also aggregates final answers rather than the reasoning processes behind them.Source: §1 · pp. 2–3
• GSA samples diverse responses, then prompts the model to generate an improved aggregate using their reasoning, without judging candidates or requiring additional training.The method uses the responses as context for a new solution.Source: §1 · p. 3
• GSA extends aggregation to open-ended problems where majority voting is not applicable.It uses the generative process and reasoning paths rather than relying only on final answers.Source: §1 · p. 3
• Across tasks and model scales, GSA outperforms self-correction and choose-from-N, and matches or exceeds self-consistency on tasks with verifiable answers.Self-consistency cannot be applied to open-ended tasks.Source: §1 · p. 3
• Likelihood analysis finds models more confident generating new responses than selecting among existing ones, providing empirical support for GSA’s framework.The introduction also reports robustness across different sampling strategies.Source: §1 · p. 3

2.1 Self-consistency

• Self-consistency samples diverse reasoning paths and majority-votes final answers, but its voting mechanism is difficult to apply to open-ended tasks.The method is described as improving Chain-of-Thought prompting through aggregation.Source: §2.1 · p. 3

2.2 Self-correction

• Intrinsic self-correction asks models to critique and refine outputs, but self-assessments can be inaccurate and training-based methods require human-annotated data.The cited discussion reports inaccurate assessments by LLMs relying on their own discriminative judgments (Huang et al. (2024)).Source: §2.2 · p. 4

2.3 Choose-from-N Methods

• Choose-from-N methods select among generated responses using assessments, but individual LLM judgments may be suboptimal and open-source alignment falls short of human-to-human agreement.GSA takes an alternative approach that does not require LLM discriminative criteria.Source: §2.3 · pp. 4–5

2.4 Multi-model Collaboration

• Multi-model collaboration uses distinct roles and iterative aggregation, whereas GSA improves a single model’s own responses without simultaneous multi-agent deployment and its higher resource costs.The compared approaches aim to exploit complementary strengths across heterogeneous models.Source: §2.4 · p. 5

3 Methodology

• Generative Self-Aggregation combines diverse response generation with context-enriched synthesis, improving answers without discriminative judgments or additional training.Unlike self-consistency’s majority voting, it synthesizes a new solution from diverse attempts.Source: §3 · p. 5

3.1 Diverse Response Generation

• Diverse candidates are sampled from the model’s distribution; differing reasoning paths or knowledge can provide context that a deterministic generation might miss.The paper names temperature and nucleus sampling as possible strategies.Source: §3.1 · pp. 5–6

3.2 Context-Enriched Response Synthesis

• After generating candidates, the method provides the query and responses as context and prompts the model to generate a new response.The paper links this synthesis to the model’s ability to identify and combine useful reasoning patterns or knowledge from the attempts.Source: §3.2 · p. 6
• In the illustrated math example, synthesis used three sampled solutions—two incorrect and one correct—and produced a response with the correct answer.The resulting solution also used a more direct approach without introducing variables.Source: §3.2 · p. 6

4.1 Experimental Setup

• The experiments evaluate four LLMs on mathematical reasoning, knowledge, and open-ended tasks.The setup includes GSM8K, Math, and SVAMP for mathematics; MMLU and GPQA for knowledge; and open-ended evaluation including MT-bench and Alpaca Eval.Source: §4.1 · p. 7

4.2 Main Results

• On math and knowledge tasks, GSA matches or surpasses Self-Consistency using three rather than four candidates, and improves open-ended Alpaca-Eval and MT-bench over greedy decoding.The reported open-ended gains are with GPT-4o Mini; Self-Consistency is not applicable to those tasks.Source: §4.2 · pp. 8–9
• GSA outperforms Choose-from-N and Self-Refine across tasks and models, supporting generative aggregation over model-based selection or feedback.The comparison spans results reported for Llama 3 and GPT-4o mini as well as Gemma 2 and Qwen 2.5.Source: §4.2 · p. 8, Table 1 · p. 9, Table 2 · p. 10

4.3 Ablation Studies

• With GPT-4o mini, GSA and Self-Consistency continue benefiting from more responses while Choose-from-N declines slightly; with Llama 3 8B, GSA peaks at N = 5 and then dips.The Llama 3 decline is attributed to difficulty processing longer contexts with too many responses.Source: §4.3 · p. 10
• On GSM8K and GPQA with Llama 3 8B, performance improves as temperature rises from 0.2 to 1.0, then declines at higher temperatures; at 1.0, GSA matches Self-Consistency on GSM8K, exceeds it on GPQA, and beats Choose-from-N.The passages attribute the gains at 1.0 to increased response diversity.Source: §4.3 · pp. 10–11
• The ablation explores prompt-template variation for distinct response formats and multilingual generation, alongside temperature-based sampling.Multilingual sampling prompts the model to answer in different languages.Source: §4.3 · p. 11
• With Gemma 2 9B on GSM8K, GSA maintains strong performance across temperature, prompt-template, and multilingual sampling, indicating robustness to sampling choice.The table compares the strategies with N = 3.Source: §4.3 · p. 11, Table 3 · p. 12

4.4 Discussions

• GSA has lower normalized negative log-likelihood than choose-from-N on Alpaca eval, indicating higher model confidence when generating responses than selecting them.The comparison uses Llama 3 8B; lower NLL indicates higher confidence in token generation.Source: §4.4 · pp. 11–12, Figure 4 · p. 12
• Lower NLL alone is insufficient for optimal response selection; choosing the lowest-NLL response performs substantially worse than self-consistency (Wang et al., 2023).The passage notes that lower NLL may correlate with improved response quality, but does not suffice to select responses optimally.Source: §4.4 · p. 12
• When only one of three candidates is correct, GSA succeeds on more samples than choose-from-N, and it can solve some cases when none is correct.When all three candidates are correct, both methods consistently succeed; both also perform well when two are correct.Source: §4.4 · p. 12, Figure 5 · p. 13
• GSA synthesizes correct solutions from incorrect examples, whereas choose-from-N cannot succeed when all candidates are incorrect.The analysis groups GPQA and MBPP samples by the number of correct responses among three candidates.Source: §4.4 · pp. 12–13, Figure 5 · p. 13

4.5 Case Study

• In the coding case study, GSA combines the simplicity of one candidate with the counting approach of another to produce an improved solution.The candidates use different strategies with distinct limitations.Source: Figure 6 · p. 13, §4.5 · p. 14
• In the math case study, GSA preserves a straightforward calculation path and derives the correct final answer despite errors in candidates’ age-difference calculations.All candidates identify the cousin’s age in eight years, but two miscalculate the current age difference and a third uses a less intuitive algebraic approach.Source: Figure 6 · p. 13, §4.5 · p. 14

• GSA aggregates multiple solutions through generative processes, using diverse reasoning paths to produce enhanced responses.Source: §5 · p. 14
• GSA requires neither additional training nor external feedback and is applicable across different model architectures and domains.Source: §5 · p. 14
• Across diverse tasks, GSA outperforms existing self-correction and choose-from-N methods that rely on LLMs’ discriminative capabilities.The conclusion describes the evaluation as extensive and spanning diverse tasks.Source: §5 · p. 14
• The authors suggest using GSA to generate supervised data for fine-tuning and exploring specialized training to improve aggregation capabilities.They state that LLMs’ aggregation capabilities may further improve performance.Source: §5 · p. 14

• The work’s stated goal is to advance the field of machine learning.Source: Impact Statement · p. 14
• The authors note potential societal consequences but identify none as needing specific emphasis.Source: Impact Statement · p. 14

• Experiments use inference-only settings with A6000 40GB GPUs and vLLM; open-source models use top-p 0.95.GPT-4o-mini is accessed through the OpenAI API service.Source: A Implementation Details and Parameters · p. 19
• Maximum new-token lengths are 2048 for mathematical reasoning and 4096 for GPQA and MMLU; open-ended benchmarks use their default evaluation conventions.Source: A Implementation Details and Parameters · p. 19
• Candidate-generation temperatures are empirically tuned across models and tasks: LLaMA-3 uses 1.0 for GSM8K and GPQA, 0.7 for MATH, SVAMP, MMLU, and MT-bench, 0.5 for MTbench, and 0.8 for Alpaca eval.GPT-4o-mini uses 0.7 across tasks except Alpaca eval at 1.0; Gemma-2 uses 0.7 uniformly. Qwen-2.5 uses 0.7 for GSM8K and MT-bench and 1.0 for GPQA.Source: A Implementation Details and Parameters · p. 19
• Aggregation uses greedy decoding at temperature 0 for closed-ended tasks and task-specific temperatures for open-ended benchmarks.Source: A Implementation Details and Parameters · p. 19

B.1 GSM8K and SVAMP

• For GSM8K and SVAMP, prompts generate step-by-step numeric answers, aggregate solutions into a reasoned answer, or select a correct response by index.The prompts request consistency when aggregating and an index in the choose-from-N format.Source: §B.1 · pp. 19–20
• A GSM8K prompting variation asks for a concise student-oriented explanation or step-by-step error checking, with an optional language placeholder.The final answer is still requested in the specified numeric format.Source: §B.1 · p. 20

B.2 MATH

• For MATH, candidate and aggregated solutions use step-by-step reasoning and boxed answers, while choose-from-N prompts return the selected solution’s index.The aggregation prompt asks the model to consider consistency and provide a correct solution with clear reasoning.Source: §B.2 · pp. 20–21

B.3 GPQA

• For GPQA, candidate prompts request a single-letter answer, aggregation asks for consistency and correctness analysis, and choose-from-N returns the correct answer’s index.The aggregation prompt requests clear reasoning; final answers are formatted as a single letter.Source: §B.3 · pp. 21–22

B.4 MMLU

• For MMLU, candidate-generation and aggregation prompts request a single-letter answer, while choose-from-N asks for the index of the correct response.The aggregation prompt asks the model to review the given solutions before answering.Source: §B.4 · pp. 22–23

B.5 MT-bench

• For MT-bench, aggregation asks the model to generate a better response, whereas choose-from-N asks it to identify the best response by index.The index is placed at the end of the response.Source: §B.5 · p. 23

B.6 Alpaca Eval

• For Alpaca Eval, aggregation reviews reference solutions and produces an answer with brief reasoning, while choose-from-N returns the best solution’s index with reasoning.The aggregation prompt first asks the model to review the problem and supplied solutions.Source: §B.6 · pp. 23–24

B.7 MBPP

• For MBPP, aggregation reviews the supplied solutions and generates a Python function with a brief explanation in a Python code block; choose-from-N selects a solution by index.The prompt also asks the model to review the problem and provide a brief explanation of its reasoning.Source: §B.7 · p. 24
• The cited passage identifies Listing 19 by number and title as a Choose-from-N prompt for MT-bench.Source: §B.7 · p. 24