PaperlayerView exact revision on arXiv

Source-linked AI summary

SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses

Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, Daniel Khashabi

arXiv:2404.04298v3cs.AIcs.CLcs.LG

TL;DR

Whether LLMs can improve their own responses depends on judging prior generations better than generating directly, but the evidence for that advantage is unclear. The paper compares both capabilities in a unified framework and finds that discrimination is not reliably better across the evaluated models and tasks. These results raise concerns about self-improvement based solely on a model’s own judgment.

  • Problem

    The paper asks whether LLMs can reliably discriminate the quality of their own prior generations better than they can generate good responses directly, a capability relevant to self-improvement.

    Source: §1 · p. 1

  • Method

    The paper proposes a unified framework that compares generation and discrimination using the same model, task, and evaluation metric.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    Across diverse evaluated tasks, LLMs were not consistently better at discriminating among their own generated alternatives than generating initial responses.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    The findings raise concerns about the potential for LLM self-improvement on any task when it relies on the model’s own judgment.

    Source: §8 · p. 11

    The study’s experimental scope is limited, and evaluating more models and tasks could further validate SELF-[IN]CORRECT.

    Source: §7 · p. 11

Abstract

from arXiv · show

Can LLMs consistently improve their previous outputs for better results? For this to be true, LLMs would need to be better at discriminating among previously-generated alternatives, than generating initial responses. We explore the validity of this hypothesis in practice. We first formulate a unified framework that allows us to compare the generative and discriminative capability of any model on any task. In our resulting experimental analysis of several open-source and industrial LLMs, we observe that models are not reliably better at discriminating among previously-generated alternatives than generating initial responses. This finding challenges the notion that LLMs may be able to enhance their performance only through their own judgment.

1 Introduction

The paper asks whether LLMs can judge their own earlier outputs better than they can generate good responses directly. It proposes a shared framework for comparing these capabilities and finds no reliable discrimination advantage across its evaluated tasks.

1 Introduction

  • The paper tests whether LLMs can reliably distinguish the quality of their own prior generations well enough to support self-improvement.The motivating premise is that such improvement requires discrimination to surpass direct generation.Source: §1 · p. 1
  • The unified framework compares generation and discrimination by scoring a randomly selected output against the model’s choice among its own generated alternatives.The two phases use generation and discrimination prompts, respectively.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Across mathematics, world knowledge, truthful question answering, and instruction following, LLMs were not consistently better at discriminating among their own generated alternatives than generating initial responses.The paper presents this pattern as evidence for SELF-[IN]CORRECT.Source: §1 · pp. 1–2
  • The reported pattern persists with additional in-context examples and chain-of-thought demonstrations, but does not appear in non-autoregressive models such as FLAN.The paper also discusses how its findings relate to prior self-improvement studies.Source: §1 · p. 2
  • The paper’s contributions are a framework for testing both capabilities across models and tasks, plus empirical evidence and further analyses of SELF-[IN]CORRECT.The study focuses on models using their own judgment rather than external feedback.Source: §1 · pp. 1–2

2 Related Work

Related work examines self-improvement through self-generated feedback, external feedback, and comparisons between language-model generation and discrimination. This paper focuses on intrinsic self-improvement and evaluates the generation–discrimination gap across a broader range of tasks with a unified metric.

Self-Improvement with LLMs

Discrepancy between LLM generation and discrimination

Using LLMs for self-evaluation

3 SELF-[IN]CORRECT

The section defines a framework that scores generation and discrimination on the same task using the same model and evaluation metric. It formalizes SELF-[IN]CORRECT as the hypothesis that discrimination is not reliably better than generation.

3.1 Establishing an Evaluation Criteria to Compare Generation vs. Discrimination

  • For each evaluation input, the framework samples multiple candidate generations from the same model, using a low temperature to keep outputs highly probable.The generation score uses a randomly chosen candidate, treated as equivalent to direct generation under the same sampling settings.Source: §3.1 · pp. 3–4
  • For discrimination, the model receives its generated responses in a task-specific prompt and selects a preferred answer; the framework averages the resulting per-sample scores.Candidate order is randomized, and reordering experiments produced very similar results.Source: §3.1 · p. 4
  • An alternative that scores each candidate separately before selecting the best produced little difference from the main discrimination setup.Source: §3.1 · p. 4

3.2 Hypothesis Formulation

  • SELF-[IN]CORRECT states that discrimination is not reliably better than generation, formalized as DG-DIFF = Sdisc − Sgen ≤ 0.Source: §3.2 · p. 4
  • The framework treats SELF-[IN]CORRECT as the null hypothesis and requires a sufficiently large positive DG-DIFF to reject it in favor of the conventional view.Source: §3.2 · p. 4
  • Sharing candidate generations across phases enables a fair comparison, while applying the task’s original metrics keeps assessments consistent; both scores remain proxies for distinct processes.The framework avoids human input and is described as more scalable and cost-effective than approaches relying on human annotation.Source: §3.2 · pp. 4–5

4 Empirical Support for SELF-[IN]CORRECT

The experiments compare generation and discrimination across diverse tasks and models; in most cases, discrimination does not reliably outperform generation. The results also show no expected discrimination advantage for instruction-tuned models and a tendency for stronger models to have larger DG-DIFF.

4.1 Experimental Setup

  • Evaluation covers GSM8K for math, TriviaQA for world knowledge using rc.nocontext, TruthfulQA for truthfulness in question answering using generation, and MT-Bench for instruction following, scored on a 0-10 scale.Models include Phi-3-mini-4k-Instruct, LLaMA-2 and LLaMA-3, Mixtral, GPT-3.5-turbo, and GPT-4-turbo. TruthfulQA uses separate GPT judges for truthfulness and informativeness, while MT-Bench uses GPT-4 scoring; GPT-4-turbo is excluded from MT-Bench.Source: §4.1 · pp. 5–6
  • The study tests the hypothesis with one-sided McNemar tests on binary-outcome tasks and Wilcoxon signed-rank tests on MT-Bench.The authors also report that invalid discrimination outputs were generally rare and used a lenient scoring rule for them.Source: §4.1 · p. 6

4.2 Main Findings

  • In 54 of 56 experiments, SELF-[IN]CORRECT was not rejected; DG-DIFF was generally small or negative across base and aligned models.Table 2 defines DG-DIFF as Sdisc − Sgen and reports the generation-to-discrimination performance pairs and p-values.Source: §4.2 · p. 6, Table 2 · p. 7
  • Instruction-tuned models did not show the expected discrimination advantage, despite instruction tuning being associated with better task-solving and classification-like tasks appearing in tuning datasets.The authors report that their empirical findings do not support the expectation.Source: §4.2 · p. 7
  • DG-DIFF tends to increase with model capacity, and several strongest tested models show positive DG-DIFF on nearly all tasks, though the gaps remain small.The reported pattern is especially pronounced within base, fine-tuned, and proprietary model categories.Source: §4.2 · p. 7

5 Further Analysis of SELF-[IN]CORRECT

The further analyses find that prompt changes do not consistently close the discrimination gap, while non-autoregressive Flan models show a different pattern. Replication and extension of self-refinement evaluations reveal that apparent score gains can occur without reliable preference for later outputs.

5.1 Better Discrimination via Prompt-Engineering

  • Doubling or tripling in-context examples tends to raise DG-DIFF but leaves it small or negative, while CoT rationales have minimal impact.The ICL improvement is not consistently monotonic; tripling examples was omitted for MT-Bench because of the 4096-token context limit.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 The Role of Objectives: Does Autoregressive Pre-training Explain our Results?

  • Flan-T5-XXL and Flan-UL2 have positive DG-DIFF on every tested task except Flan-T5-XXL on MT-Bench, with higher average DG-DIFF than tested autoregressive models.For TriviaQA and TruthfulQA, the SELF-[IN]CORRECT hypothesis is rejected; however, the Flan model classes also differ in pre-training processes and datasets, so the authors urge caution in attributing the pattern to objectives alone.Source: §5.2 · pp. 8–9, Table 4 · p. 9

5.3 Do Prior Findings in Self-Refinement Contradict SELF-[IN]CORRECT?

  • In self-refinement replications, constrained-generation scores can rise as outputs lengthen without incorporating new keywords or using feedback.The authors also report that models sometimes assign lower scores to refined acronym-generation and dialogue outputs.Source: §5.3 · pp. 9–10
  • Models prefer self-refined outputs over previous ones only around 54% of the time, extending the comparison to real-world tasks beyond reasoning.The authors interpret this as evidence that models are still not consistently better at discriminating among previously generated alternatives than generating initial responses.Source: §5.3 · p. 10

6 Further Discussion

The discussion distinguishes selecting ordered preference pairs from identifying the single best response, and proposes that this finer discrimination demand may relate to diminishing returns in self-rewarding. It also finds that simplifying distractors improves DG-DIFF.

6 Further Discussion

  • Self-rewarding methods select winning and losing responses from a larger generation set, which is simpler than identifying the single best response and does not have its effectiveness challenged here.Straightforward heuristics, such as selecting the highest- and lowest-scoring responses, can support the pair-ordering setup.Source: §6 · p. 10
  • The authors hypothesize that weaker overall discrimination may be linked to diminishing returns after a few self-rewarding iterations.They suggest it could make a virtuous cycle of improving instruction following and generating self-rewards challenging.Source: §6 · p. 10
  • The controlled experiments test whether replacing incorrect candidates with simpler distractors changes the outcome, using TriviaQA and GSM8K.TriviaQA distractors are correct answers from other questions; GSM8K distractors are incorrect generated answers multiplied or divided by 100.Source: §6 · p. 10
  • Simplifying incorrect candidates improves DG-DIFF; TriviaQA has Sdisc well above Sgen, and all tested GSM8K models improve DG-DIFF.Source: §6 · p. 10

7 Limitations

The study identifies limits in assessing pre-training influences and in the scope and design of its experiments. Its findings concern a simple discrimination setup and focus mainly on the first stage of self-improvement.

Challenges in controlled study of LLMs

  • Pre-training data and objectives are difficult to assess, leaving important aspects of their influence underexplored.Source: §7 · p. 10

Potential influence of lengthier discrimination prompt

  • The discrimination prompt is longer because it includes candidate answers, but its effect is difficult to isolate without introducing other prompt changes.Source: §7 · pp. 10–11

Limitations in experimental scope

  • The experiments cover multiple tasks and domains but a limited set of prominent LLMs, so more models and tasks could further validate SELF-[IN]CORRECT.The authors do not claim that LLMs can never outperform generation at discrimination; refinement-specific fine-tuning may improve discrimination, potentially at a cost to generality.Source: §7 · p. 11

Discrimination setup

  • The simple discrimination setup leaves more complex strategies, including problem decomposition and answer verification, outside the study’s scope.Source: §7 · p. 11

Challenges in determining model’s preference

  • LLM-as-a-Judge prompting may be affected by answer-option or format biases; ranking candidates by assigned probability could also be affected by text fluency.Source: §7 · p. 11

Limited focus on other stages of self-improvement

  • The analysis focuses mainly on self-discrimination, while other self-improvement stages receive less attention; it evaluates multiple generated answers followed by discrimination.Source: §7 · p. 11

8 Conclusion

Across the tested models and tasks, discrimination is not reliably better than generating responses directly and was often worse. The appendix also provides original and Chain-of-Thought prompts for several task settings.

8 Conclusion

  • Across the evaluated models and tasks, discrimination is not reliably better than direct generation and was often worse.Source: §8 · p. 11

A Prompts for TriviaQA and MT-Bench

  • The appendix provides discrimination prompts for GSM8K, TriviaQA, TruthfulQA, and MT-Bench; GSM8K has no Chain-of-Thought discrimination prompt because its default evaluation includes rationales.The TriviaQA example marks both answers 3 and 4 as correct; the TruthfulQA and MT-Bench examples mark answer 4 and answer 1 as correct, respectively.Source: A Prompts for TriviaQA and MT-Bench · p. 17, Figure 2 · p. 17, Figure 3 · p. 17

B Extra analysis with Self-Refine

The extra analysis examines self-refinement examples on tasks previously tested in Madaan et al. (2023). The examples show that iterative refinement does not guarantee steadily improving feedback scores or shorter outputs.

B Extra analysis with Self-Refine

C Predicting exact answers instead of answer options

The analyses test exact-answer generation and separate scoring of candidate answers as alternatives in the discrimination phase. Neither analysis reports a clear improvement over the original answer-choice setup.

C Predicting exact answers instead of answer options

D Evaluating discriminating ability on each candidate separately

E Ablation of prompts used for discrimination

The prompt ablation tests whether the observed SELF-[IN]CORRECT finding is an artifact of prompt wording. On GSM8K with LLaMA-2 13B, prompt variations do not significantly affect DG-DIFF.

E Ablation of prompts used for discrimination

F Results and experimental setup for simplified discrimination phase

The simplified discrimination comparison reports improved DG-DIFF with simplified negative candidates. The section also describes task-appropriate statistical tests for comparing generation and discrimination, including their test choices and decision procedures.

F Results and experimental setup for simplified discrimination phase

G Choice of Statistical Test and Detailed Formulation of Hypothesis Testing

F Results and experimental setup for simplified discrimination phase

G Choice of Statistical Test and Detailed Formulation of Hypothesis Testing

  • For MT-Bench, the Wilcoxon signed-rank test is chosen because the 160-sample dataset does not support a reliable normality assumption for paired differences.The passage describes it as a non-parametric alternative suited to paired ordinal or continuous outcomes.Source: G Choice of Statistical Test and… · p. 22
  • The Wilcoxon procedure ranks absolute paired differences, assigns signs, and sums positive and negative ranks; a sufficiently large W+ counts against DG-DIFF ≤ 0.Source: G Choice of Statistical Test and… · p. 23
  • The one-sided Wilcoxon test rejects its null when the upper-tail p-value for W+ falls below the chosen significance level α = 0.05.Source: G Choice of Statistical Test and… · p. 23
Loading 2404.04298v3…