PaperlayerView exact revision on arXiv

Source-linked AI summary

PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, Yue Zhang

arXiv:2306.05087v2cs.CLcs.AI

TL;DR

Instruction tuning needs reliable, robust, and privacy-protected evaluation to select hyperparameters while capturing subjective response quality. PandaLM is a judge language model trained to compare candidate LLMs, supported by human-aligned evaluation data. Its use for hyperparameter selection improves tuned models over Alpaca’s default parameters, although the explored hyperparameter range and GPT-3.5-derived core data limit the conclusions.

  • Problem

    Instruction tuning requires effective hyperparameter selection and evaluation, but existing approaches face reliability, privacy, cost, and subjective-quality limitations.

  • Method

    PandaLM is a judge language model trained on paired responses and evaluation annotations to select superior candidate models and support instruction-tuning hyperparameter optimization.

  • Results

    PandaLM-selected hyperparameters yielded substantial performance enhancements over Alpaca’s hyperparameters across multiple open-sourced LLMs.

  • Takeaways & Limitations

    PandaLM provides an automatic, privacy-protected basis for evaluating and optimizing instruction-tuned LLMs without API-based evaluation.

  • Takeaways & Limitations

    The explored hyperparameter range may not contain the absolute optimum, while the core GPT-3.5-derived data may not fully reflect human preferences.

Abstract

from arXiv · show

Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, establishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage. All resources of PandaLM are released at https://github.com/WeOpenML/PandaLM.

1 INTRODUCTION

PandaLM addresses the need for reliable, privacy-protected evaluation to optimize LLM instruction-tuning hyperparameters. It evaluates subjective response qualities, is trained with model comparisons and annotations, and improves tuning outcomes over Alpaca’s defaults.

  • Motivation: Existing evaluation methods face reliability, privacy, cost, consistency, and reproducibility challenges, while often overlooking subjective response qualities.Relevant qualities include conciseness, clarity, instruction adherence, comprehensiveness, formality, and context relevance.
  • Approach: PandaLM is a judge language model that selects the superior model among candidates with different hyperparameters and provides evaluation rationales.It also targets logical fallacies, repetitions, grammatical inaccuracies, and context irrelevance.
  • Approach: PandaLM training uses paired responses from similarly sized foundation models fine-tuned with Alpaca’s data and hyperparameters, supplemented by GPT-3.5-generated annotations.The paired responses form training data for the judge model, reducing reliance on costly human annotation.
  • Evaluation: A diverse human-annotated test dataset covers varied tasks and contexts, with three independent evaluators per label and divergent samples excluded to maintain IAA above 0.85.The dataset is designed to align evaluation with human preferences and assess PandaLM’s reliability.
  • Results: 47.0 superior versus 26.2 inferior responses were achieved with PandaLM-selected hyperparameters under GPT-4 evaluation across 170 instructions.Human evaluation likewise found 79.8 superior versus 25.2 inferior responses, using the same training data and instructions.
  • Contributions: The work introduces PandaLM, a human-annotated validation dataset, and hyperparameter optimization experiments showing substantial gains over Alpaca’s hyperparameters.The approach is presented as privacy-protected and applicable to open-sourced LLMs.

2 RELATED WORK

Prior work establishes that hyperparameter choice strongly affects fine-tuned language models and that evaluation guides optimization. However, conventional metrics and human assessment do not fully capture subjective quality or provide an efficient evaluation process.

  • Hyperparameter Optimization: Hyperparameters such as learning rate, batch size, and training epochs can significantly influence fine-tuned language-model performance.Optimal hyperparameter sets may vary substantially across domains.
  • Evaluation of Language Models: Accurate language-model evaluation is important for selecting hyperparameters, but perplexity and accuracy may not adequately guide LLM improvement.LLM assessment may require subjective measures in addition to conventional objective metrics.
  • Evaluation of Language Models: Human evaluation can align assessments with human judgments but is costly and labor-intensive for iterative hyperparameter optimization.Model-based metrics such as BERTScore and MAUVE provide alternatives but do not replace the broader evaluation challenge described here.
  • Evaluation of Language Models: Subjective qualitative analysis considers whether outputs handle ambiguous instructions and provide contextually appropriate responses.Such qualitative measures include aspects beyond simple correctness.

3 METHODOLOGY

Instruction tuning evaluates many versions of a foundation model, each using different hyperparameters, to identify the best output. PandaLM is designed as an automated judge for comparing these candidates and is trained from structured response-comparison data.

  • Instruction-Tuning Pipeline: The instruction-tuning pipeline fine-tunes a foundation model with instructions, evaluates each tuned model, and searches across hyperparameters for the best output.The pipeline therefore requires repeated model assessment.
  • Instruction-Tuning Pipeline: PandaLM assesses LLMs fine-tuned with varied parameters and aims to identify the superior model from a candidate pool.Its role is to automate the evaluation stage of instruction tuning.
  • Training Data: Each training instance contains an instruction, input, two responses, an evaluation result, an evaluation reason, and a reference response.Instructions and inputs are sampled from Alpaca 52K, while response pairs come from various instruction-tuned models.
  • PandaLM Training: PandaLM uses LLaMA as its backbone and is fine-tuned with next-token cross-entropy in a sequence-to-sequence paradigm without a separate classification head.Training uses DeepSpeed with ZeRO Stage 2 on eight NVIDIA A100-SXM4-80GB GPUs.
  • PandaLM Training: Figure 4 compares instruction-tuned models using directed edges for significant superiority and dashed edges for similar performance, with response-order swapping used to resolve conflicting evaluations as ties.Displayed edge numbers represent win-minus-loss differences, while unlabeled dashed edges indicate differences smaller than 5.

4 RELIABILITY EVALUATION OF PANDALM

PandaLM is evaluated against human judgments across model comparisons, metrics, domains, and input conditions. Results indicate similar preferences to humans and strong performance, while reasons and references improve evaluation.

  • Human-annotated evaluation: The test dataset pairs instructions and inputs with responses from five instruction-tuned LLMs, annotated concurrently by three trained experts.The experts evaluate conciseness, clarity, comprehensiveness, formality, and instruction adherence beyond correctness.
  • Agreement with human judgments: GPT-3.5, GPT-4, PandaLM-7B, and humans exhibit similar partial-order preferences over model superiority.Conflicting results are revised to ties, and response order is swapped before final evaluation for the models.
  • Evaluation performance: PandaLM-70B surpasses GPT-4 in the reported accuracy, precision, recall, and F1-score comparison against human annotations.The analysis attributes the results to removing training-data noise and choosing suitable foundation architectures and instructions.
  • Cross-domain robustness: PandaLM is evaluated across LSAT, PubMedQA, and BioASQ to test robustness under legal and biological distribution shifts.GPT-4, rather than human annotators, supplies the gold-standard win/tie/lose labels for these evaluations.
  • Evaluation design: Comparisons test PandaLM with and without evaluation reasons and reference responses, with the full approach performing better than numerical comparisons alone.The reported analysis says reasons and references help LLMs understand evaluation tasks.
  • Practical advantages: PandaLM offers open-source availability, reproducibility, data-privacy protection, and unlimited access beyond the performance of proprietary evaluators.These advantages are presented as alternatives to GPT-3.5 and GPT-4’s API-based evaluation setting.

5 USING PANDALM TO INSTRUCTION TUNE LLMS

PandaLM selects instruction-tuning hyperparameters through repeated model comparisons. Across several base models and evaluators, its selected configurations outperform Alpaca’s defaults, although conventional task metrics show one exception.

  • Hyperparameter search: PandaLM searches 80 hyperparameter configurations using sequential block comparisons, repeated twice for robustness, totaling 800 comparisons.The configurations cover optimizer, learning rate, training epochs, and learning-rate scheduler choices.
  • Metric-specific behavior: Conventional lm-eval results improve for most models with PandaLM finetuning, but Cerebras shows a performance dip.Human, GPT-4, and GPT-3.5 evaluations instead indicate superior Cerebras performance with PandaLM tuning.

6 LIMITATIONS

The study’s hyperparameter search is bounded by common-practice choices and computational cost, while its GPT-3.5-derived data may not fully match human preferences.

  • Search scope: The explored hyperparameter range may exclude the absolute optimum because it follows common practice and prior literature.Expanding the search boundary would increase computational cost.
  • Data and evaluation scope: The core data are derived from GPT-3.5 and may not fully resonate with human preferences.The paper also notes that evaluation currently emphasizes resource-intensive outcome-based assessment.

7 CONCLUSION

The study finds that PandaLM provides an automatic and reliable basis for LLM hyperparameter tuning. Models tuned with PandaLM-selected parameters consistently outperform those using Alpaca’s defaults, motivating work on larger and more robust versions.

  • Conclusion: PandaLM is applied as an automatic and reliable judge model for LLM hyperparameter optimization.The conclusion frames the approach as feasible for tuning instruction-following models.
  • Conclusion: Models tuned with PandaLM-selected hyperparameters consistently outperform models tuned with Alpaca’s default parameters.The authors plan to expand PandaLM to larger models and analyze its intrinsic features.

B DIRECTED ACYCLIC GRAPH DEPICTING THE MIXTURE RANKING OF MODELS TRAINED USING BOTH ALPACA’S AND PANDALM’S HYPERPARAMETERS.

Figure 7 ranks models fine-tuned with Alpaca’s and PandaLM’s hyperparameters, with PandaLM-tuned models generally outperforming their Alpaca-tuned counterparts. PandaLM-LLaMA ranks highest overall, while Alpaca-LLaMA is second.

  • PandaLM-LLaMA ranks first, followed by Alpaca-LLaMA, PandaLM-Bloom, PandaLM-Pythia, and PandaLM-OPT.The complete descending order is given in Figure 7.
  • PandaLM-tuned models consistently rank above their corresponding Alpaca-tuned counterparts in the hybrid ranking.The ranking pattern supports the effectiveness of PandaLM’s hyperparameter selection.
  • The comparison uses downstream-task evaluations reported in Table 6 between foundation models fine-tuned with Alpaca’s or PandaLM’s hyperparameters.Table 6 uses lm-eval across several downstream tasks.

C COMPARISONS BETWEEN ORIGINAL MODELS AND MODELS TUNED USING PANDALM ON TRADITIONAL TASKS

Traditional-task evaluations show that PandaLM tuning improves most language models, although Cerebras declines on lm-eval. Pairwise evaluations and perplexity further indicate that instruction-tuned quality is not captured by objective predictive metrics alone.

  • Most language models improve after PandaLM fine-tuning on traditional tasks, but Cerebras exhibits a decline on lm-eval.Human, GPT-4, and GPT-3.5 evaluations instead indicate better Cerebras performance with PandaLM.
  • LLaMA-PandaLM outperforms LLaMA-Alpaca in pairwise comparisons and traditional tasks despite having higher perplexity.The reported comparisons include PandaLM, GPT, and human evaluations.
  • Lower perplexity does not always indicate better overall performance for instruction-tuned models.Perplexity may be unsuitable when lower values reflect overfitting and reduced generalizability.

D LAW / BIOMEDICAL DATASETS INTRODUCTION

The paper evaluates PandaLM across law and biomedical datasets and studies how training-data size, tuning method, evaluation setup, and supervision affect evaluation performance. These experiments compare full fine-tuning with LoRA and test whether tailored evaluation improves model selection.

  • Law / Biomedical datasets: The LSAT dataset contains 1,009 questions divided into AR, LR, and RC subsets.It is used to assess PandaLM on American law-school entrance-exam questions.
  • Law / Biomedical datasets: PubMedQA combines 1k expert annotations, 61.2k unlabeled entries, and 211.3k human-generated biomedical question-answer instances.The dataset supports evaluation in the biomedical domain.
  • Tuning comparisons: Full fine-tuning beats LoRA for Bloom in 66 instances versus 35, although outcomes vary across models.The broader analysis reports that full fine-tuning is favorable for most models despite many ties.
  • Evaluation setup: A precisely tuned smaller model outperforms a larger model in zero- and few-shot evaluation scenarios.The result comes from the ablation comparing pretrained and instruction-tuned models.
  • Supervision: Removing explanations and reference responses from supervision makes PandaLM less precise at identifying the preferable response.The ablation retains only categorical outcomes such as Tie, Win, and Lose.

J HYPERPARAMETER OPTIMIZATION ANALYSIS

Hyperparameter experiments show that optimal settings vary across models and training setups. Learning rates near 2e-5 and roughly four epochs often perform well, but early stopping can halt training before later improvements.

  • The search varies learning rates from 2e-6 to 2e-4 and compares checkpoints pairwise by counting win rounds.Checkpoints are saved at the end of each epoch.
  • Learning rates near 2e-5 and peak performance around the fourth or fifth epoch emerge as trends, not universal optima.The preferred settings vary across models and are influenced by data distribution, optimizer, and scheduler choices.
  • No universally optimal hyperparameter setting exists across different models and training setups.The paper therefore emphasizes model-specific hyperparameter searches.
  • Early stopping may prematurely halt large-model fine-tuning because later epochs can improve performance after an inferior epoch 3.In some LLaMA experiments, epoch 2 outperformed epoch 3, but subsequent epochs improved again.

K MODEL SHIFT ANALYSIS

The model-shift analysis compares PandaLM with human benchmarks across instruction-tuned LLaMA versions and examines performance variation across learning rates and epochs. PandaLM aligns closely with human preferences and consistently favors LLaMA-2.

  • Performance is examined across different learning rates and for variability across training epochs.
  • PandaLM aligns closely with human benchmarks when comparing different versions of instruction-tuned LLaMA models.
  • PandaLM consistently shows a preference for the LLaMA-2 model across the evaluated model shifts.The passage attributes this pattern to LLaMA-2 benefiting from more pre-training data.
Loading 2306.05087v2…