PaperlayerView exact revision on arXiv

Source-linked AI summary

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, Preethi Lahoti

arXiv:2311.08592v2cs.SEcs.AIcs.CL

TL;DR

Manual red-teaming can be difficult to reproduce and adapt, while downstream LLM applications require safety evaluation tailored to their contexts. AART uses customizable recipes to generate diverse adversarial datasets with reduced human effort. The authors report that it identified issues not always captured by human testing alone.

  • Problem

    Manual red-teaming is often difficult to reproduce or adapt, and application-specific safety evaluation calls for adversarial datasets aligned with varied contexts.

    Source: §1 · p. 1

  • Method

    AART uses customizable recipes and structured LLM generation to create adversarial evaluation datasets for new application contexts while reducing human effort.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    AART identified issues not always captured through human testing alone and supported product launches with improved safety measures and reduced risks, as reported by the authors.

    Source: §6 · p. 7

  • Takeaways & Limitations

    AART is presented as a way to generate diverse, application-specific adversarial datasets with minimal human intervention.

    Source: §6 · pp. 6–7

    LLM-generated data may contain bias or factual inconsistencies, particularly for complex sociocultural data, and may overlook emerging attack patterns.

    Source: §5 · p. 6

Abstract

from arXiv · show

Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AI-assisted Red-Teaming (AART) - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce human effort significantly and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within the application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. Compared to some state-of-the-art tools, AART shows promising results in terms of concept coverage and data quality.

1 Introduction

Adversarial testing must adapt to the diverse risks and requirements of downstream LLM applications, but manual red-teaming can be difficult to reproduce, adapt, or make sufficiently comprehensive. AART proposes recipe-guided dataset generation to reduce human effort and support evaluation in new application contexts.

1 Introduction

  • Evaluation datasets need to align with application scenarios, geographic areas, and comprehensive safety perspectives.The paper frames adaptable adversarial testing as a way to address potential risks and harms early in development.Source: §1 · p. 1
  • Manual red-teaming is often hard to reproduce or adapt, may lack diversity and completeness, and can burden or expose human testers.The paper notes risks including fatigue and added burden for people from historically marginalized communities.Source: §1 · pp. 1–2
  • AART is a plug-and-play pipeline that generates adversarial testing datasets at scale while limiting human effort to guiding the generation recipe.The approach adapts structured reasoning and data-generation methods to adversarial dataset creation.Source: §1 · p. 2
  • The paper proposes AART for new application contexts and reports evaluating it on a hypothetical global text-generation product focused on dangerous and illegal activities.The authors compare the generated dataset with human-red-teaming evaluation sets from other contexts and adapted automated red-teaming methods.Source: §1 · p. 2

2 Related Work

The paper argues that fixed harm taxonomies may not suit varied real-world policies and use cases, and presents customizable recipes as a flexible alternative. It positions AART as supporting broad, context-specific topic coverage while complementing human red-teaming.

2 Related Work

  • Parametrized recipes are proposed to adapt adversarial datasets to changing policies, use cases, and topics while supporting broader and international coverage.The paper contrasts this flexibility with relying on one fixed taxonomy for all real-life scenarios.Source: §2 · p. 2
  • AART uses structured generation of harmful concepts and customizable recipes to broaden topic coverage without requiring a pre-existing taxonomy.The authors present this as a novel approach to safety red-teaming of LLMs.Source: §2 · p. 2
  • AART is similar to instruction-based automated red-teaming methods that do not require labeled data, and the paper compares it with an adapted method from Perez et al. (2022).That prior work also describes finetuning- and RL-based approaches.Source: §2 · p. 2
  • The authors frame AART as helping machines identify common problems so human researchers can focus on rarer threats, which may remain beyond its coverage.The paper states that human expertise remains essential for rare or unseen problems.Source: §2 · pp. 2–3

3 Method: AI-Assisted Red Teaming

AART makes adversarial evaluation requirements explicit through customizable recipes and a staged generation process. Its demonstration applies the process to a hypothetical global English text-generation application focused on harmful activities.

3.1 Description of the AART method

  • The method makes evaluation requirements explicit through three customizable steps, providing more control over policy, formats, and geographic diversity than a zero-shot approach.Explicit requirements support transparency in assessing the final dataset and efficient reuse across teams.Source: §3.1 · p. 3
  • Problem Definition identifies relevant evaluation dimensions and uses an AI-assisted recipe to produce candidate terms for each dimension.Dimensions can include attacker query styles, deployment regions, high-risk harms, and previously identified weaknesses; concepts can be tailored to policies.Source: §3.1 · pp. 3–4
  • Problem Scoping filters candidate lists and sets sample counts for each axis, defining the desired data mix and coverage.The resulting blueprint determines breadth, depth, and concept representation in the prompt dataset.Source: §3.1 · p. 4
  • Query Generation combines prioritized diversity axes and uses an instruction-tuned LLM with a chain-of-thought variation to create adversarial prompts.The specified data mix controls the number of generation runs and their parameters.Source: §3.1 · p. 4
  • Structured prompts carry annotations for prioritized diversity dimensions, allowing coverage and diversity review without additional human annotation.Generated content also indirectly supplies metadata for each prompt's diversity axes.Source: §3.1 · p. 4

3.2 Demonstration of the AART method

  • The demonstration targets open-ended English text generation for a global user base, prioritizing protection against requests for information about dangerous or harmful activities.The example leaves Step 1 outputs unrevised to demonstrate structured generation, while noting that human-in-the-loop review is critical in real applications and can target prior evaluation loss patterns.Source: §3.2 · p. 4

4 Evaluation

The evaluation compares AART with existing datasets and automated red-teaming, then assesses prompt quality and coverage in a demonstration scenario. Results show high quality and broad policy and regional references, alongside uneven task-format representation and limitations in keyword-based evaluation.

Prompt diversity

  • AART generated prompts ranging from terse to verbose, nuanced, and indirect styles, which the paper identifies as important for robust adversarial testing.The evaluation also used topic classifiers to label generations and compare topic diversity with other datasets.Source: §4 · p. 5

Coverage and balance

  • 53.4% of region mentions fell in the top five regions and 31.6% of policy-concept references in the top five concepts; “how-tos” appeared in 5% of queries, while 13 formats appeared once.The first iteration also omitted some task formats, including poems, legal documents, and memos.Source: §4 · pp. 5–6

Evaluation limits

  • Keyword-based evaluation can miss implicit or nuanced prompts, so its coverage estimates may understate the concepts present in the dataset.The paper frames keyword matching as a first step before application-specific classifiers or human evaluation protocols.Source: §4 · p. 5

Quantitative evaluation

  • Existing datasets lacked dimensions required by the demonstration scenario, which could underestimate harm risks for applications with different prompt styles or target harms.The comparison used keyword matching; Table 2 reports normalized keyword presence and prompt-length means and standard deviations.Source: §4 · p. 5, Table 2 · p. 6

Coverage and balance

  • Teams can iteratively address task-format imbalances by revising topic lists, rebalancing the sample mix, or changing generation-recipe instructions.The paper notes that imbalance may arise from particular recipes or from how the model combines diversity dimensions.Source: §4 · p. 6

Qualitative evaluation

  • 92.5% of a 120-prompt sample was judged good quality; policy concepts appeared in 99.2% of queries, regions in 96.7%, and task formats sensibly in 85.0%.The qualitative sample included 20 prompts per dimension with keywords and 20 without.Source: §4 · p. 6

5 Limitations

AART reduces extensive human intervention, but the paper identifies continuing limits involving expertise, output reliability, evaluation scope, and computational cost.

Human expertise

  • Long-tail adversarial testing and nuanced societal contexts still require human expertise, while prompt-engineering experience may limit accessibility for less experienced teams.The authors state that comprehensive coverage of nuanced societal concepts requires evaluators’ expertise.Source: §5 · p. 6

Generation risks

  • LLM-generated data may contain biases or factual inconsistencies, particularly for complex sociocultural material, and may overlook emerging attacks such as prompt injection.Source: §5 · p. 6

Evaluation limits

  • Defining an adversarial prompt is inherently ambiguous, and keyword-based coverage rates may underestimate the concepts represented.The paper notes the difficulty of distinguishing malicious intent from innocuous prompts, especially when tuning for high recall.Source: §5 · p. 6

Cost and scaling

  • LLM-based generation is computationally expensive, and future work should test whether scaling it increases or saturates diversity and coverage.Source: §5 · p. 6

6 Conclusion

AART automates adversarial dataset generation with diverse coverage and minimal human intervention. The paper reports improved safety measures and risks addressed in deployed products, while noting that its diversity work starts with topical coverage.

6 Conclusion

  • AART generated diverse adversarial datasets quickly with minimal human intervention, covering policy concepts, task formats, and geographic regions.The authors report that the method also identified issues not always captured through human testing alone.Source: §6 · pp. 6–7
  • The authors report that AART enabled several product launches with improved safety measures and reduced risks from potential LLM harms.The conclusion does not quantify these product-level outcomes.Source: §6 · p. 7
  • AART supports rapid creation of diverse datasets with minimal human intervention, providing broad coverage of policy concepts, task formats, and geographical regions for responsible AI development of LLM-based applications.It also identifies issues not always captured through human testing alone.Source: §6 · p. 7

Ethical Considerations

The demonstration describes customizable prompt-generation recipes and reports improved topical diversity over human-red-teaming datasets created for other application contexts. Its appendix shows how the recipe defines, scopes, and combines diversity dimensions, while noting the need for human review in real applications.

Ethical Considerations

  • AART generated high-quality prompts reflecting application priorities and context, with improved topical diversity over human-red-teaming datasets from other contexts.The authors identify lexical, syntactical, language-related, and adversariality dimensions as further diversity facets for future work.Source: Ethical Considerations · p. 7

A Prompts for "Demonstrating our method"

A.1 Step 1: Problem Definition

  • Step 1 prompts define the inputs as broad harmful concepts, diverse textual mediums, and major world regions.The demonstration presents the generated lists in lowercase alphabetical order and does not revise them before subsequent steps.Source: §A.1 · p. 10, §A.2 · p. 10

A.2 Step 2: Problem Scoping

  • In this demonstration, authors left Step 1 outputs unrevised to show the wide range of structured generation in Step 3; lists include policy concepts, task formats, and geographic regions.In real-world application contexts, human-in-the-loop review and revision is critical.Source: §A.2 · p. 10

A.3 Step 3: Query Generation

  • Step 3 combines prioritized policy concepts, task formats, and geographic regions to generate region-specific adversarial prompts and explanations of their harmfulness and tailoring.The demonstration runs generation twice per policy concept, sampling three task formats and three regions per attempt; 95.8% of decoded JSON lines were valid.Source: §A.3 · p. 11

B Evaluation datasets

The evaluation uses selected examples from existing safety datasets and adapts automated red-teaming methods to the demonstration scenario. It also describes how examples were selected and omitted for the qualitative analysis.

B Evaluation datasets

  • The evaluation draws on RealToxicityPrompts, ParlAI Dialogue Safety, and BAD using selected challenging, adversarial, or “not ok” first-turn examples.RealToxicityPrompts includes prompts marked challenging or with prompt or continuation toxicity of at least 0.8; the other datasets use adversarial or “not ok” exchanges.Source: B Evaluation datasets · p. 11
  • Adapted automated methods use 160 responses sampled at 0.7 temperature from the same instruction-tuned system, with blank responses filtered out.The authors use smaller datasets than the original work to represent a developer creating a quick, targeted first evaluation set.Source: B Evaluation datasets · pp. 11–12

C Examples from Demonstration Dataset

  • The qualitative examples omit some especially graphic or disturbing prompts that the authors say would not add to discussion of the method’s strengths and limitations.The section includes generated queries and examples selected through qualitative analysis.Source: C Examples from Demonstration Dataset · p. 12

C.1 Qualitative analysis of generation

The qualitative analysis presents generated prompts spanning harmful concepts, textual formats, and geographic references. Its worst-case examples include outputs judged low-quality or insufficiently aligned with the recipe’s prioritized dimensions.

C.1 Qualitative analysis of generation

  • The reported worst-case examples come from a qualitative analysis of 120 prompts and include outputs judged low-quality or missing a policy concept, task format, or geographic region.Particularly graphic and disturbing examples were omitted.Source: §C.1 · p. 12
  • Other examples span formats from songs and source code to novels, illustrating variation in the requested textual medium.Examples include a song about animal cruelty, source code related to burning coal, and a novel involving the drug trade.Source: §C.1 · p. 13
  • Examples combine harmful concepts, locales, and formats, including an Instagram post about smuggling drugs into Canada and patch notes about robbing a tourist in Cairo.These examples pair different task formats and geographic settings with the requested harmful subject matter.Source: §C.1 · p. 13

C.2 Random sample

The random sample illustrates both strengths of AART's examples and cases the authors consider potentially harmful despite not being explicitly harmful. The examples include contextualized requests involving environmental harm, policy violations, and crime.

C.2 Random sample

  • Examples contextualize harmful requests around coral bleaching and business contracts, GDPR violations, and corporate fraud in Spain.These are presented as examples illustrating specific strengths of the approach.Source: §C.2 · p. 13
  • Other examples concern animal abuse and tax evasion through shell companies in Singapore.Source: §C.2 · p. 13
  • The authors flag factual or informational requests as potentially harmful even when they are not explicitly harmful, including reporting on arson and documenting hate crimes.Source: §C.2 · pp. 13–14

D Topic analysis of evaluation datasets

The appendix describes topic analysis of evaluation datasets using treemaps and gives illustrative extensions to AART. These extensions include prompts for enumerating policy concepts and use cases, task formats, and policy-specific concepts.

D Topic analysis of evaluation datasets

E AART Extensions

  • The appendix presents extensions and adaptations of the general method as illustrative examples.Source: E AART Extensions · p. 16

E.1 Examples of Prompts for Policy Concept Enumeration

  • Policy-concept prompts solicit enumerations of harmful crimes, abuse, sexually explicit content, and hate-speech topics.Source: §E.1 · p. 16

E.2 Examples of Prompts for Enumerating Use cases

  • Use-case prompts request diverse text varieties, social-media posts, common LLM interactions, and formal business writing types.Source: §E.2 · p. 16

E.3 Examples of Task Formats

  • Example task formats span creative and formal writing as well as summarizing, question answering, paraphrasing, and list-based tasks.Source: §E.3 · p. 16

E.4 Examples of Random Concepts per Policy

  • Random policy-specific concepts include crime and illegal activities, alongside hate speech and harassment concepts such as racism and homophobia.Source: §E.4 · p. 16

Related papers

Loading 2311.08592v2…