PaperlayerView exact revision on arXiv

Source-linked AI summary

Shepherd: A Critic for Language Model Generation

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz

arXiv:2308.04592v1cs.CLcs.AI

TL;DR

LLMs still produce false, unreliable, and incoherent outputs, creating a need for methods that critique and refine their generations. Shepherd is tuned to provide domain-spanning, actionable critiques using community and expert-annotated feedback data. Across automatic and human evaluations, its critiques are generally preferred to competing models and reach performance on par with ChatGPT.

  • Problem

    LLMs can produce false, unreliable, and incoherent outputs, motivating methods that continuously critique and refine generated text.

    Source: §1 · p. 1

  • Method

    Shepherd is a language model tuned to critique generated outputs, trained with community feedback and expert human annotations collected across task types.

    Source: §1 · pp. 1–2

  • Results

    Shepherd effectively critiques answers across multiple datasets and evaluation settings, with performance on par with ChatGPT and critiques generally preferred over other models.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Shepherd provides an automatic mechanism for inspecting model generations that may improve generation quality and reduce hallucinations.

    Source: §7 · p. 9

    The evaluation would benefit from more annotators and a broader array of diverse tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As large language models improve, there is increasing interest in techniques that leverage these models' capabilities to refine their own outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT.

1 Introduction

LLMs can still produce false, unreliable, or incoherent outputs, motivating Shepherd, a model tuned to critique generated text across domains. It combines community and expert-annotated feedback data and is generally preferred over competing models in evaluations.

Motivation

  • LLMs still generate false, unreliable, and incoherent outputs despite improved fluency, motivating methods that continuously critique and refine generations.The paper presents Shepherd as a mechanism for improving generation quality and supporting more reliable outputs.Source: §1 · pp. 1–2

Shepherd

  • Shepherd is explicitly tuned to critique model-generated outputs across a wider range of domains than earlier self-feedback approaches.Prior work did not initially provide specific feedback for complex math and reasoning outputs.Source: §1 · p. 1
  • Shepherd critiques generated text by identifying factuality, logical, coherence, and alignment issues while suggesting actionable improvements.Its feedback can include deep domain knowledge rather than only an overall judgment or generic advice.Source: §1 · p. 2

Feedback data

  • The feedback dataset combines community critiques from online forums with expert human annotations of generations across task types.Community data is more diverse and informative but more informal, while high-quality human annotations improve finetuning performance.Source: §1 · p. 2

Evaluation

  • Shepherd critiques are generally preferred to those from Alpaca, SelFee, and ChatGPT under model-based and human evaluation.Alpaca often gives incorrect positive feedback, SelFee is vague or answers directly, and ChatGPT is more stable and better at correct judgments.Source: §1 · p. 2

2 Data Collection

Shepherd's feedback dataset combines community critiques with expert human annotations, using filtering and postprocessing to retain diverse, high-quality feedback across reasoning and summarization tasks.

2.1 Community Critique Data

  • Community data pairs questions, answers, and critiques from Stack Exchange and Pushshift Reddit, with vote scores attached to each component.The Reddit subset is restricted to 16 communities with more formal, often Q&A-structured discussions.Source: §2.1 · p. 2

2.1.1 Critique Postprocessing

  • Valid critiques either refine largely accurate answers or explicitly identify inaccuracies in incorrect answers.Keyword filtering and user edit histories help locate these two cases before further quality filtering.Source: §2.1.1 · p. 3
  • Community examples are filtered by answer and critique vote scores, profanity, non-text content, duplicate posts, and comments that question the prompt instead of the answer.These filters aim to preserve endorsed, diverse, text-only question-answer-critique triads.Source: §2.1.1 · pp. 3–4

2.2 Human Data Collection

  • Human annotation covers eight reasoning or entailment datasets and two summarization datasets, using contexts, correct outputs, and candidate outputs for feedback collection.Candidate outputs come from original datasets or prompted LLaMA-65B and LIMA-30B generations, followed by manual inspection.Source: §2.2 · p. 4
  • Expert reviewers annotate candidate errors with human-in-the-loop quality assessment, after which unsuitable examples and unhelpful error categories are removed.The resulting dataset contains 1,317 high-quality examples.Source: §2.2 · p. 4

3 The Shepherd Model

Shepherd is trained from LLaMA-7B with AdamW and selected through manual inspection followed by GPT-4 evaluation on held-out feedback examples.

3 The Shepherd Model

  • Shepherd uses LLaMA-7B as its base model and is trained with AdamW, a 1e-5 learning rate, 2,000 warmup steps, 64-example batches, and a maximum sequence length of 2,048.AdamW uses β1 = 0.9, β2 = 0.95, and weight decay of 0.1.Source: §3 · p. 5
  • Three best checkpoints are first selected by manually inspecting feedback on 20 held-out examples, then the final checkpoint is chosen with GPT-4 evaluation.Checkpoints are saved every 50 steps.Source: §3 · p. 5

4 Evaluating Feedback

This section evaluates Shepherd against Alpaca, SelFee, and ChatGPT using diverse task datasets, GPT-4 judgments, and human evaluation. It also describes pairwise and Likert-style assessment procedures and the challenges of evaluating critiques.

4 Evaluating Feedback

  • Shepherd is evaluated against Alpaca-7B, SelFee-7B, and ChatGPT using GPT-4 and human evaluation across multiple downstream tasks.The evaluation includes six public datasets covering varied topics and reasoning skills, plus the contamination-resistant CritiqueEval set, for 352 instances total.Source: §4 · p. 5, §4.1 · p. 5

4.2 Baseline Models

  • The baselines include ChatGPT, Alpaca-7B, and SelFee, whose training data and potential evaluation overlap are documented for comparison.SelFee uses 178K training instances, while the evaluation notes overlap between its training data and several benchmark subsets.Source: §4.2 · pp. 5–6

4.3 GPT4 Evaluation

  • The evaluation compares feedback quality through both independent 1–7 Likert ratings and pairwise selection of the better critique.GPT-4 rates whether feedback identifies errors or confirms correctness, while pairwise comparison emphasizes error identification and useful suggestions.Source: §4.3.1 · p. 6, §4.3.2 · p. 6

4.4 Human Evaluation

  • Human evaluators rate feedback on a 1–7 scale while comparing multiple critiques for the same question-answer pair.The procedure presents the question, answer, and model feedback together and asks annotators to rank competing feedback.Source: §4.4 · pp. 6–7
  • Human evaluation is difficult because some questions require specialized domain knowledge, even when expert annotators can consult external sources.The paper gives a medical or biology example as a case requiring deep expertise.Source: Table 2 · p. 6, §4.4 · p. 7

5 Results

Shepherd generally outperforms Alpaca and SelFee and is competitive with ChatGPT across critique evaluations. The results also reveal evaluator-specific discrepancies, score-distribution differences, and the importance of high-quality training data.

5.1 Pairwise Comparison

  • Shepherd significantly outperforms Alpaca and SelFee, slightly exceeds ChatGPT under GPT-4 pairwise evaluation, and performs on par with ChatGPT in human evaluation.On CritiqueEval, which ChatGPT and LLaMA models had not seen, Shepherd consistently generates better feedback.Source: §5.1 · p. 7, Table 3 · p. 8, Table 4 · p. 8

5.2 Score Distribution

  • Shepherd provides more score-7 feedback than ChatGPT but also more score-2 feedback, while Alpaca commonly receives score 2 and SelFee score 3.The score categories distinguish incorrect judgments, vague or direct-answer behavior, correct judgments, and helpful improvement suggestions.Source: §5.2 · p. 7

5.3 Likert Score

  • GPT-4 Likert evaluation can conflict with human judgments because it rates feedback independently, favors particular formats, and struggles when model differences are small.For example, GPT-4 gave Alpaca an average score of 4.7 compared with a human average of 2.91.Source: §5.3 · p. 7
  • Pairwise GPT-4 evaluation aligns better with human evaluation than independent GPT-4 Likert scoring.The paper therefore recommends pairwise comparison for GPT-4-based critique assessment.Source: §5.3 · pp. 7–8
  • GPT-4 evaluation can fail when it misjudges feedback correctness or lacks the knowledge needed to recognize an answer error.In one example, Shepherd identifies an error while other models and GPT-4 treat the answer as correct.Source: §5.3 · p. 8

5.4 Data Matters

  • Higher-quality community and human-annotated data makes additional fine-tuning data more useful for building critique models.Shepherd’s smaller, higher-quality data sources are reported as more useful than SelFee’s much larger training set.Source: §5.4 · p. 8

6 Related Work

Related work covers automatic and human evaluation, iterative critique-and-refinement methods, and evidence that LLM-based evaluation can be biased. Shepherd is positioned as a critique model evaluated against existing approaches.

Evaluation of LLMs

  • Human evaluation is reliable but expensive, domain-specific, and time-consuming, while automatic overlap or embedding metrics miss logical inconsistencies, factuality, and alignment.Source: §6 · p. 8
  • Human evaluation finds ChatGPT and Shepherd broadly similar, with ChatGPT generally better on public datasets and Shepherd significantly better on CritiqueEval.Source: Table 5 · p. 9
  • GPT-4 evaluation produces smaller model gaps and may not strictly follow evaluation instructions in this setting.Source: Table 6 · p. 9
  • LLM-based evaluations can be unfair because model biases may affect their judgments.Source: §6 · p. 9

Critique models

  • Recent critique systems use prompting, fine-tuning, single-model refinement, tool interaction, or separate correctors to improve generated text.Source: §6 · p. 9

7 Conclusion

The paper introduces Shepherd to critique language-model generations and reports effective critique performance across datasets and evaluation settings. Its authors argue that such critique mechanisms can improve generation quality and reduce hallucinations.

7 Conclusion

  • Shepherd effectively critiques answers across multiple datasets and evaluation settings, reaching performance on par with ChatGPT.Source: §7 · p. 9
  • Automatic critique mechanisms are presented as important for inspecting model generations as LLMs enter more real-life applications.Source: §7 · p. 9
  • The authors state that Shepherd can help improve generation quality and reduce hallucinations.Source: §7 · p. 9

Limitations

The evaluation uses automatic and human assessments, but the authors identify limited annotator coverage and task diversity as areas needing further analysis. A table provides qualitative feedback examples from different models.

Limitations

  • The evaluation would benefit from more meticulous analysis using a larger number of annotators and a broader range of diverse tasks.Source: Limitations · p. 9
  • The paper evaluates critique ability through both automatic evaluation and human evaluation.Source: Limitations · p. 9
  • Table 7 presents qualitative examples of feedback generated by different models.Source: Table 7 · p. 10

A Human Data Collection Guidelines

The human-data collection procedure gives annotators context, a correct output, and a candidate output so they can identify errors and provide natural-language feedback. Quality controls and an error taxonomy structure the collected examples.

A Human Data Collection Guidelines

B.1 Instruction Derivation

The annotation process collects natural-language critiques across diverse contexts, using guidelines and correct outputs to help annotators identify critical errors. GPT-4 can score examples with manual instruction tuning, but the paper documents cases where model-based evaluation fails.

  • GPT-4 gives reasonable Likert-scale scores on examples after manual instruction tuning, but the paper presents examples exposing limitations of model-based evaluation.Source: §B.1 · p. 13, Table 12 · p. 18
  • The annotation data includes prompts and identified error types from each dataset, with dataset-specific templates used to obtain or present candidate outputs.Source: Table 8 · p. 14, Table 10 · p. 15, Table 9 · p. 15
  • Human annotation collects natural-language feedback on candidate outputs across diverse contexts, with correct outputs provided to help identify errors.Source: Table 11 · p. 16, §B.1 · p. 17
  • Annotators are instructed to report whether candidate outputs contain critical errors, including when no errors exist, while ignoring minor mistakes.Source: §B.1 · p. 17
  • In one evaluation example, GPT-4 rates incorrect feedback highly, while only Shepherd identifies the answer’s error despite the provided ground truth.Source: Table 13 · p. 19, Table 14 · p. 19
Loading 2308.04592v1…