INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

1 Introduction

• The capabilities of instruction-tuned models remain incompletely understood amid models’ black-box nature and a lack of holistic evaluation studies.The paper highlights applications spanning language, mathematics, coding, medicine, and law.Source: §1 · p. 2
• INSTRUCTEVAL evaluates problem-solving, writing proficiency, and alignment with human values through a systematic, holistic assessment.The suite is designed to extend beyond earlier evaluation approaches.Source: §1 · p. 2
• The analysis considers pretrained foundations, instruction-tuning data and its quality, and training methods as factors affecting model performance.The authors aim to identify which factors shape performance.Source: §1 · p. 2
• Instruction-data quality has a critical influence on scaling performance, while open-source models still need improvement in problem-solving and alignment despite strong writing.The introduction also describes open-source writing abilities as impressive.Source: §1 · p. 2
• The authors report a leaderboard comparing over 60 open-source LLMs and select 10 models for their study based on foundation architecture, instruction set, and pretraining method.The leaderboard is publicly accessible.Source: §1 · p. 2

2 Overview of Open-Source Instructed LLMs

• Foundation models are pretrained LLMs that may be instruction-tuned to become instructed LLMs; the overview focuses mainly on open-source models because closed-source models lack transparency and reproducibility.Table 1 collects foundation-model details relevant to architecture, size, and data scale.Source: §2 · p. 2, Table 1 · p. 3
• Instruction datasets vary in quality, quantity, diversity, and format, and the overview collects details of several open-source datasets.The paper notes that synthetic instructions from closed-source models may bring black-box opacity and instability from noisy instructions.Source: §2 · p. 3, Table 2 · p. 3
• The open-source instructed-model overview collates each model’s foundation, size, instruction dataset, and training method.The authors report variety in model sizes and instruction data.Source: §2 · p. 3, Table 3 · p. 3

3 Challenges in Evaluating Instructed LLMs

• Closed-source models often withhold architecture, instruction-data, and training-method details, making their behavior and performance difficult to analyze rigorously.The authors describe this as a knowledge gap in the research community.Source: §3 · p. 4
• Open-source model development may outpace evaluation, and informal assessments can confuse comparisons or leave capability claims difficult to substantiate.The passage cautions that claims such as “90% ChatGPT Quality” lack meaning without rigorous evaluation.Source: §3 · p. 4
• A more complete understanding requires jointly considering pretraining, instruction data, and training methods rather than studying only one factor.The authors frame comparisons of factors’ impact and improvement needs as useful questions.Source: §3 · p. 4
• Benchmarks covering exams and other problem-solving tasks do not capture general writing ability or alignment with human values, motivating broader evaluation.The authors also note that instructed LLMs can solve problems across domains and use external tools.Source: §3 · p. 4

4 INSTRUCTEVAL Benchmark Suite

• INSTRUCTEVAL combines objective-scoring and qualitative evaluation methods to cover problem-solving, writing, and alignment to human values.Source: §4 · p. 4

4.1 Problem-Solving Evaluation

• The problem-solving suite spans world knowledge, complex instructions, passage-based arithmetic, programming, and counterfactual causal reasoning.Source: §4.1 · p. 5

4.2 Writing Evaluation

• IMPACT samples informative, professional, argumentative, and creative writing with 50 prompts per category; ChatGPT scores relevance and coherence from 1 to 5.Source: §4.2 · pp. 5–6, Table 4 · p. 5

4.3 Alignment to Human Values

• HHH assesses human-value understanding through dialogue response selection, covering helpfulness, honesty, harmlessness, and an “other” category with 61, 59, 58, and 43 samples, respectively.Source: §4.3 · p. 6

5.1 Problem Solving

• A solid pretraining foundation is necessary for problem-solving; models pretrained on under one trillion tokens underperform peers, larger foundations help, and architecture shows no clear link.The underperforming models cited are OPT-IML and Dolly V2.Source: §5.1 · p. 7
• Instruction data has greater impact than pretraining but is no panacea: high-quality human-annotated tasks accompany larger gains, while synthetic-model imitation yields modest or negative problem-solving gains.The authors suggest synthetic instruction samples may contain inaccurate answers that mislead training.Source: §5.1 · p. 7
• Human-feedback training modestly improves StableVicuna over Vicuna, while LoRA becomes more effective as parameter count grows and uses fewer compute resources.The authors describe parameter-efficient training as promising for more scalable instruction tuning.Source: §5.1 · p. 7

5.2 Writing Ability

• Writing performance is consistent across four categories and can diverge from problem-solving; Flan-Alpaca surpasses Flan-T5, while open models match ChatGPT in relevance but trail in coherence.The authors posit that diverse GPT-3 synthetic instructions aid generalization to real-world writing prompts despite potential noise.Source: §5.2 · pp. 7–8
• ΔAvg results are omitted for ChatGLM because its foundation model is unavailable and for Flan-UL2 because the public model did not yield reasonable results.ΔAvg denotes average improvement over the corresponding foundation model.Source: §5.2 · p. 7, Table 7 · p. 8

5.3 Alignment to Human Values

• Foundation models favor helpfulness and honesty over harmlessness; instruction data shifts alignment unevenly, with Tk-Instruct and Vicuna improving harmlessness, honesty, and “other,” but not helpfulness.StableVicuna shows the same lack of helpfulness improvement despite helpfulness- and honesty-targeted instructions; Flan-T5 and Flan-Alpaca lean more toward helpfulness than honesty after tuning.Source: §5.3 · pp. 8–9

5.4 Summary of INSTRUCTEVAL Results

• Instruction data has the highest impact, yet open-source instructed models still need improvement in problem-solving and human-value alignment, and capability strengths may specialize by task.Models with stronger problem-solving ability may not be better at writing or alignment.Source: §5.4 · p. 9

6.1 Towards More Scalable Language Models

• The T5 foundation model scales relatively flatly, while Flan-T5 scales better and is more parameter-efficient; even the smallest Flan-T5 outperforms the largest T5.The authors suggest resource-constrained developers focus on effective instruction datasets and training methods rather than model size.Source: §6.1 · p. 10

6.2 Are Few-Shot Demonstrations Always Better?

• Few-shot demonstrations have varying effects across tasks, with limited benefit on MMLU and a slight performance decrease for OPT-IML.The authors suggest multiple-choice questions may be easy to grasp without demonstrations, and note that OPT-IML was optimized for zero-shot settings.Source: §6.2 · p. 10
• In-context demonstrations may help more on complex tasks such as BBH, but Flan-UL2 and Flan-T5 show no marked few-shot effect.The authors note that these Flan models have instruction formats designed for in-context demonstrations.Source: §6.2 · p. 10

7 Conclusion

• INSTRUCTEVAL assesses instruction-tuned models across problem-solving, writing ability, and alignment to human values.The conclusion emphasizes high-quality instruction data for scaling performance and reports that open-source models excel in writing while needing improvement in problem-solving and alignment.Source: §7 · p. 10
• The authors envision extending instruction-tuning evaluation to multilingual and multimodal settings.They cite diverse languages as important for inclusivity and note adaptation of instructed models to vision and audio.Source: §7 · p. 10

A Appendix

• Table 9 presents statistics for the evaluation datasets and benchmarks used.Source: Table 9 · p. 11

A.1 Data Statistics

• Statistics for the evaluation datasets and benchmarks are reported in Table 9.Source: §A.1 · p. 11

• Models are evaluated as-is, without additional fine-tuning or training.Source: §A.2 · p. 11
• MMLU, BBH, DROP, CRASS, and HHH use greedy decoding; HumanEval uses one sample at temperature 0.1, and IMPACT uses temperature 1.0.Source: §A.2 · p. 11
• HHH is run seven times with the chosen and rejected options reordered, and results are averaged using greedy decoding.Source: §A.2 · p. 11

• The section details how IMPACT is evaluated and presents instances with outputs generated by various models.Source: §A.3 · p. 11

• ChatGPT evaluates generated text on a 1-to-5 scale using rubrics tailored to relevance and coherence.Source: §A.3.1 · p. 11
• The relevance rubric rates how well a response addresses the prompt, from inadequate coverage to comprehensive and insightful treatment.Source: §A.3.1 · p. 11
• The coherence rubric rates organization, clarity, and flow, from disjointed writing to fluent and engaging presentation.Source: §A.3.1 · pp. 11–12

A.3.2 Argumentative Samples

• ChatGPT’s scientific-research response presents arguments for relaxing ethical constraints and for safeguards against harm, concluding that research should balance progress with responsibility.The supporting case invokes urgent scientific needs and research limits; the opposing case emphasizes protection of people, animals, and the environment, and reliable results.Source: §A.3.2 · p. 13
• For responses about whether scientific research should be constrained by ethical concerns, Flan-Alpaca scored Coherence 3 and Relevance 3, Flan-T5 3 and 2, and StableVicuna 4 and 2.Source: §A.3.2 · pp. 13–14

A.3.3 Informative Samples

• In the new-parent sleep example, ChatGPT recommends naps, asking others for help, bedtime routines, a comfortable sleep environment, relaxation, and seeking professional help if needed.It also suggests limiting caffeine and alcohol.Source: §A.3.3 · p. 16
• Chronic-pain examples recommend medication, exercise, mind-body techniques, trigger identification, and comprehensive treatment planning.ChatGPT advises consulting a healthcare provider before taking new medication; Flan-Alpaca suggests identifying pain triggers and developing a management plan.Source: §A.3.3 · p. 18
• The informative excerpts recommend lifestyle changes, alternative therapies, and counseling or support for managing chronic pain.Source: §A.3.3 · p. 19

A.3.4 Professional

• The thank-you-note examples express gratitude for collaboration and connect partners’ contributions with project outcomes, while advising that the note be personal, warm, and clear.Source: §A.3.4 · p. 20
• In the product comparison, the iPhone is praised for design and integration, while Samsung’s Galaxy S is credited with a larger screen and customization options.Source: §A.3.4 · p. 22
• The comparison identifies limited iPhone customization and Samsung battery life and customer support as drawbacks, and proposes more iPhone customization and improved Samsung support.Source: §A.3.4 · p. 22

A.3.5 Creative Samples

• In one prehistoric-time story, John encounters a Tyrannosaurus Rex, escapes in his time machine, and hopes to return better prepared.Source: §A.3.5 · p. 24
• The tree-perspective story reflects on witnessing changing civilizations and seasons while remaining rooted and continuing to grow.Source: §A.3.5 · p. 26
• Other examples include a short first-person account of a sapling surviving a fierce storm and a story portraying an oak tree enduring rain and harsh weather.The oak-tree passage describes a rainstorm restoring the ground, while another response says the storm carried away all life on the tree.Source: §A.3.5 · p. 26
• Flan-T5 is rated Coherence: 3 and Relevance: 2, while StableVicuna is rated Coherence: 3 and Relevance: 4.The passages provide the ratings but do not define the scales or explain the basis for them.Source: §A.3.5 · p. 26
• One story follows a tree through a forest’s decline as people build homes and roads, animals flee, and the tree remains determined to protect the world.The tree narrates its changing surroundings, describing sadness at the forest’s destruction and its resolve to endure.Source: §A.3.5 · p. 27