PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models

Jiale Cheng, Yida Lu, Xiaotao Gu, Pei Ke, Xiao Liu, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang

arXiv:2406.16714v2cs.CLcs.AIcs.LG

TL;DR

Manual inspection is costly to scale, and conventional benchmarks are not designed to pinpoint individual model weaknesses. AUTODETECT uses three LLM agents to discover weaknesses across tasks; the authors report high identification success and improvements from weakness-derived training data.

  • Problem

    Manual examinations are costly to scale, while model-agnostic benchmarks are not designed to identify individualized weaknesses in LLMs.

    Source: §1 · p. 2

  • Method

    AUTODETECT uses Examiner, Questioner, and Assessor agents in an adaptive assessment process to identify weaknesses across tasks.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    AUTODETECT achieves over 50% identification success across multiple strong LLMs and over 30% on GPT-3.5-turbo and Claude-3-sonnet; weakness-derived data yields over 10% improvements across several benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fine-tuning with weakness-derived data improves performance, including on open-source Mistral and Llama models.

    Source: §1 · p. 2

    Detection is constrained by agent capability and becomes markedly harder when the target performs at or above the agents.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Although Large Language Models (LLMs) are becoming increasingly powerful, they still exhibit significant but subtle weaknesses, such as mistakes in instruction-following or coding tasks. As these unexpected errors could lead to severe consequences in practical deployments, it is crucial to investigate the limitations within LLMs systematically. Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies, while manual inspections are costly and not scalable. In this paper, we introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks. Inspired by the educational assessment process that measures students' learning outcomes, AutoDetect consists of three LLM-powered agents: Examiner, Questioner, and Assessor. The collaboration among these three agents is designed to realize comprehensive and in-depth weakness identification. Our framework demonstrates significant success in uncovering flaws, with an identification success rate exceeding 30% in prominent models such as ChatGPT and Claude. More importantly, these identified weaknesses can guide specific model improvements, proving more effective than untargeted data augmentation methods like Self-Instruct. Our approach has led to substantial enhancements in popular LLMs, including the Llama series and Mistral-7b, boosting their performance by over 10% across several benchmarks. Code and data are publicly available at https://github.com/thu-coai/AutoDetect.

1 Introduction

The paper presents AUTODETECT as a model-specific way to expose weaknesses that manual inspection and conventional benchmarks may miss. It reports successful weakness discovery across tasks and uses the resulting data to improve models.

1 Introduction

  • Manual examinations are costly to scale, while model-agnostic benchmarks are not designed to identify individualized weaknesses and also face update, leakage, and leaderboard limitations.Unexpected LLM mistakes can have consequences including system failures and safety issues.Source: §1 · pp. 1–2
  • AUTODETECT adapts an educational-assessment model to LLM testing through Examiner, Questioner, and Assessor agents that collaborate on tailored weakness identification.The Examiner develops a taxonomy, the Questioner creates challenging probes, and the Assessor analyzes responses.Source: §1 · p. 2
  • The authors present detected weaknesses as useful training data for improving models, rather than relying only on untargeted model enhancement.They report improvements from integrating weakness-derived data into fine-tuning.Source: §1 · p. 2
  • AUTODETECT exceeds 50% identification success across multiple strong LLMs, tops 30% on GPT-3.5-turbo and Claude-3-sonnet, and yields over 10% gains across several benchmarks.The reported improvements use data from the weakness-detection process.Source: §1 · p. 2

2 Related Work

The related work distinguishes model comparison from individualized weakness discovery, and manual red teaming from automated detection for general-purpose tasks.

2.1 Evaluation Benchmarks

  • Benchmarks rank models but are not designed to discover model-specific flaws; static benchmarks face leakage and leaderboard swamping, while dynamic ones can lack coverage and universality.The authors argue that relying solely on benchmarks limits thorough weakness discovery and practical improvement guidance.Source: §2.1 · pp. 2–3

2.2 Red Teaming

  • Manual red teaming is costly and lacks diversity, while automated weakness detection for general-purpose tasks remains underexplored despite automated red-team attacks being widely adopted.AUTODETECT is presented as a framework extending weakness identification beyond safety issues to instruction-following, mathematics, and coding.Source: §2.2 · p. 3

3 Method

AUTODETECT defines weaknesses as target-model failures on task-specific problems and uses three collaborating LLM agents to discover them. Its adaptive search probes underperforming areas, and the resulting questions and reference answers support fine-tuning.

3.1 Problem Definition

  • AUTODETECT maps a task and its description to the set of problems the target model fails to address accurately, evaluated by a strong LLM judge.The task and description are denoted (T, D), and the resulting weakness set is W.Source: §3.1 · p. 3

3.2 AUTODETECT Framework

  • The Examiner decomposes a task into categories and knowledge points; the Questioner probes those points, while the Assessor analyzes low-scoring responses and identifies potential weaknesses.The Examiner then dynamically refines the taxonomy based on those findings.Source: §3.2 · p. 4

3.3 Iterative Search

  • The iterative search starts with five questions per knowledge point, scores target-model responses against GPT-4 references, and uses low-scoring historical samples to generate further probes.The process adds the new responses and scores to its history to identify weaknesses at particular knowledge points.Source: §3.3 · p. 4

3.4 Model Enhancement

  • The authors fine-tune the target model using search-derived questions and GPT-4 reference answers to test whether identified weaknesses can contribute to model enhancement.The loss formulation uses the questions, reference answers, and reference-answer length.Source: §3.4 · p. 4

4 Experiments

Experiments assess weakness detection across instruction-following, mathematics, and coding, then test whether identified weaknesses can guide model improvement. They also compare AutoDetect with baselines and evaluate iterative recovery.

4.1 Weakness Detection

  • AutoDetect achieved high identification success rates across models and tasks, while average score and ISR rankings correlated with Chatbot Arena rankings at 0.95.Scores declined during iterative search, which the authors attribute to the method uncovering weaknesses.Source: §4.1 · pp. 5–6, Figure 3 · p. 5, Table 1 · p. 5

4.2 Model Enhancement

  • Improvements from AutoDetect data averaged over 6% across test sets for various Llama2 model sizes.The experiments also report that targeted assessment data is more effective than data from GPT-3.5-turbo for Llama2-7b-Chat.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Comparison with Baselines

  • Compared with baselines, AutoDetect performed better in identification success rate and diversity; OPRO and PAIR repeatedly exploited specific weaknesses, producing unbalanced problem distributions.PAIR was three times more costly than the other methods.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Iterative Weakness Recovery

  • AutoDetect consistently improved Llama2-13b-Chat across three assessment rounds, with a non-trivial improvement at each iteration.The iterative training reduced the frequency of similar error types.Source: §4.4 · p. 7, Table 5 · p. 7

5 Discussion

The discussion examines within-model weaknesses that benchmark-based evaluation may miss, alongside AUTODETECT’s ability to generate creative and compound challenges.

5.1 Limitations within LLMs

  • Over 90% accuracy on GSM8k coexists with struggles on simple geometry, including angle computation, for gpt-3.5-turbo, mistral-large, and claude-3-sonnet.The contrast within mathematics highlights gaps that benchmark-based weakness discovery may miss.Source: §5.1 · p. 7
  • Algorithm and data-structure competence does not ensure basic coding: models may import a module instead of all its functions or add duplicate set elements.These examples show models can falter on simpler operations despite handling complex tasks.Source: §5.1 · p. 7
  • Models often omit constraints or make mistakes in later steps when following complex instructions or performing multi-step reasoning.The passage identifies these shortcomings as relevant to complex scenarios essential for agent tasks.Source: §5.1 · pp. 7–8

5.2 Superiority of AUTODETECT

  • AUTODETECT generates complex, unique questions beyond typical human-written ones, especially for non-expert annotators; storytelling in JSON illustrates this creative challenge.The authors suggest such questions can assess advanced capabilities and provide training data for model enhancement.Source: §5.2 · p. 8
  • AUTODETECT sometimes generates compound instruction tasks without being asked to combine constraints, including requests to translate into multiple languages simultaneously.The passage notes that LLM effectiveness diminishes on the multiple-language translation example.Source: §5.2 · p. 9

6 Conclusion

The conclusion presents AUTODETECT as a unified framework for identifying weaknesses across models and generic tasks, and using assessment data to improve model performance.

6 Conclusion

  • AUTODETECT identifies weaknesses across instruction-following, mathematical reasoning, and coding, and uses assessment data to enhance model performance.The authors describe its results as indicating potential for automatic detection and addressing of weaknesses on general tasks.Source: §6 · p. 9

Limitations

The authors identify robustness and reliance on capable LLM agents as limitations requiring further work.

Limitations

  • A small number of generated questions may be illogical, and discovered weaknesses vary across runs; larger-scale weakness identifications are needed for further validation.Repeated experiments showed stable identification success rates, but the specific detected problems varied.Source: Limitations · p. 9
  • Detection effectiveness is limited by agent capabilities, making vulnerabilities harder to uncover when the target matches or surpasses the agents; self-evaluation bias can compound this in self-evolution.The authors leave approaches to address these constraints and bias for future work.Source: Limitations · p. 9

Ethical Considerations

The appendices describe ethical safeguards and provide details on AUTODETECT’s algorithm, task taxonomies, and prompts.

Ethical Considerations

  • Weakness cases are generated from scratch without existing datasets or license issues; the work excludes security tasks and used three paid Chinese annotators informed of the study purpose.Payment followed regional standards.Source: Ethical Considerations · p. 9

A AUTODETECT Algorithm

  • AUTODETECT uses two cycles and three roles—Examiner, Questioner, and Assessor—with the Questioner’s iterative search supporting weakness identification.The passage describes their interaction as a comprehensive and targeted assessment framework.Source: A AUTODETECT Algorithm · p. 12

B Task Taxonomy and Description

C Detailed Prompts

D Implementation Details

The framework uses GPT-4-1106-preview to identify weaknesses and sets limits on taxonomy coverage and iterative search. It also compares generated-case distributions with three baselines across three tasks.

D Implementation Details

  • GPT-4-1106-preview is used to identify weaknesses, with up to six knowledge points per subcategory, five seed questions, and a 10-step search.The limits balance coverage and cost.Source: D Implementation Details · p. 12
  • LoRA fine-tuning is used because full-parameter fine-tuning significantly impacts performance with limited data; models train for five epochs with batch size four.Training uses AdamW and a learning rate of 2e-5; inference uses vLLM with greedy decoding.Source: D Implementation Details · p. 12

E Baseline Category Classification

  • AUTODETECT maintains a balanced distribution and high diversity across instruction-following, mathematics, and coding cases compared with three baselines.GPT-3.5-turbo classifies the generated cases into the task taxonomy, merging uncategorized or infrequent cases into “others.”Source: E Baseline Category Classification · p. 12

F Robustness Experiment

Repeated experiments on Llama2-7b-Chat show slight identification-success-rate fluctuation across three tasks. The appendix also provides the human-evaluation annotation document.

F Robustness Experiment

  • Slight identification-success-rate fluctuation across instruction-following, mathematics, and coding tasks demonstrates the framework’s stability in repeated Llama2-7b-Chat experiments.Table 6 presents the repeated-experiment results.Source: F Robustness Experiment · p. 12

G Annotation Document

H Distribution Discrepancy in Identification Data

The identified categories differ across target models, including model-specific scenario weaknesses. Iterative training on Llama2-13b-Chat reduces the frequency of similar errors.

H Distribution Discrepancy in Identification Data

  • In instruction-following scenario simulation, the Assessor identifies new categories tied to individual models’ weaknesses, including distinct scenario types for Llama2-7B-Chat, Llama3-8B-Instruct, and GPT-3.5-turbo.Examples include business and financial scenarios for Llama2-7B-Chat, emotive and multilingual scenarios for Llama3-8B-Instruct, and historical and poetic scenarios for GPT-3.5-turbo.Source: H Distribution Discrepancy in… · pp. 12–13
  • The identification procedure breaks tasks into a taxonomy and iterates over knowledge points within each subcategory.Algorithm 1 initializes a set of identified weaknesses and processes knowledge points while any remain.Source: H Distribution Discrepancy in… · p. 13
  • Figures 7 and 8 provide task descriptions and the detailed taxonomy for instruction-following, mathematics, and coding.Source: Figure 7 · p. 13, Figure 8 · p. 14

I Iterative Improvement on Categories

Related papers

Loading 2406.16714v2…