PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically Auditing Large Language Models via Discrete Optimization

Erik Jones, Anca Dragan, Aditi Raghunathan, Jacob Steinhardt

arXiv:2303.04381v1cs.LGcs.CL

TL;DR

Auditing LLMs for rare and high-impact failures is difficult because feasible input-output pairs are sparse and model behavior is discrete, nonlinear, and high-dimensional. The paper formulates auditing as discrete optimization and introduces ARCA, which efficiently searches prompts and outputs; ARCA generally outperforms AutoPrompt and GBDA while uncovering diverse failure modes. Its scope remains limited by unresolved API-only auditing, failure certification, and behaviors not expressible as a single prompt-output objective.

  • Problem

    Reliable auditing methods must uncover rare, counterintuitive, and changing LLM failures without costly behavior-specific templates, manual searches, or fine-tuning.

  • Method

    The paper casts auditing as discrete optimization over a prompt and its greedy output, and introduces ARCA, a coordinate-ascent algorithm using efficient objective approximations and exact reranking.

  • Results

    ARCA nearly always outperforms AutoPrompt and GBDA on GPT-J and GPT-2, while uncovering toxic, cross-lingual, and targeted-name behaviors.

  • Takeaways & Limitations

    Discrete optimization provides an additional auditing tool that can uncover hard-to-find undesirable behaviors and adapt cheaply to new target behaviors.

  • Takeaways & Limitations

    The approach does not yet support API-only zeroth-order auditing, certify the absence of failure modes, or handle failures requiring more than a single prompt-output objective.

Abstract

from arXiv · show

Auditing large language models for unexpected behaviors is critical to preempt catastrophic deployments, yet remains challenging. In this work, we cast auditing as an optimization problem, where we automatically search for input-output pairs that match a desired target behavior. For example, we might aim to find a non-toxic input that starts with "Barack Obama" that a model maps to a toxic output. This optimization problem is difficult to solve as the set of feasible points is sparse, the space is discrete, and the language models we audit are non-linear and high-dimensional. To combat these challenges, we introduce a discrete optimization algorithm, ARCA, that jointly and efficiently optimizes over inputs and outputs. Our approach automatically uncovers derogatory completions about celebrities (e.g. "Barack Obama is a legalized unborn" -> "child murderer"), produces French inputs that complete to English outputs, and finds inputs that generate a specific name. Our work offers a promising new tool to uncover models' failure-modes before deployment.

1 Introduction

The paper frames LLM auditing as automatically searching for prompt-output pairs that exhibit specified undesirable behaviors. ARCA makes this discrete optimization practical enough to uncover rare, counterintuitive failures and transfer some findings across models.

  • Motivation: LLMs can produce toxic outputs, stereotypes, private information, and potentially catastrophic behaviors despite their broad capabilities.These failures motivate auditing before deployment.
  • Motivation: Reliable auditing is difficult because high-impact behaviors may be rare, counterintuitive, and expensive to search for manually or with behavior-specific methods.The challenge is amplified by distribution shifts and deployment scale.
  • Framework: The framework specifies an objective φ(x, o) and searches for a prompt x whose greedy completion is output o while scoring highly for a target behavior.Objectives can target non-toxic prompts with toxic outputs or cross-language completions.
  • Method: ARCA jointly optimizes prompt and output tokens using coordinate ascent, combining exact log probabilities with averaged first-order approximations.It ranks candidate tokens efficiently and refines the top-k candidates with the exact objective.
  • Findings: The method uncovers hundreds of toxic celebrity completions, language-changing completions, and prompts generating specified names on GPT-2 and GPT-J.Examples include toxic statements about Barack Obama and French prompts completing to English.
  • Findings: ARCA consistently produces more target-behavior examples than AutoPrompt and GBDA, and some GPT-2 failure-inducing prompts transfer to GPT-3.The authors present the framework as an additional auditing tool rather than a complete solution.

2 Related Work

The paper situates its approach among work on capable language models, documented LLM failures, prompt search, controllable generation, gradient-based sampling, and adversarial attacks. Its distinction is auditing target behaviors through joint prompt-output optimization rather than relying only on manual prompts, templates, or continuous relaxations.

  • Large language models: Recent autoregressive language models support text and code applications including dialog, summarization, and formal mathematics.
  • LLM failure modes: Prior studies document LLM failure modes such as propagating biases and stereotypes and leaking private information.
  • Failure-mode search: Existing failure-mode searches use manually written prompts, training-set prompts, templates, or optimized universal triggers.These approaches include prompt construction and prompt optimization for toxic outputs.
  • Controllable generation: Controllable-generation research formulates constrained optimization to find high-probability outputs with specified attributes under fixed prompts.
  • Gradient-based sampling: Gradient-based sampling also addresses discrete variables and sparse high-probability regions, whereas this work emphasizes maximizing objectives for greedy-decoding behavior.
  • Adversarial attacks: Text adversarial attacks perturb inputs to change classifier predictions, often through typos, synonym swaps, or semantics-preserving transformations.The paper places its auditing setting in relation to both restricted and unrestricted adversarial examples.

3 Formulating and Solving the Auditing Optimization Problem

The paper formulates auditing as maximizing a behavior objective over prompts and outputs subject to greedy generation, then develops ARCA to optimize this sparse, discrete, nondifferentiable problem efficiently. ARCA combines a differentiable surrogate, joint coordinate updates, and a hybrid token-ranking approximation.

  • 3.1 Preliminaries: A language model maps an m-token prompt to next-token probabilities, and greedy decoding deterministically produces an n-token completion f(x)=o.Prompts and outputs are represented as token sequences from vocabulary V.
  • 3.2 The auditing optimization problem: The auditing objective φ scores prompt-output pairs for target behaviors such as non-toxic prompts producing toxic outputs or prompts generating a specified name.Objectives may use hard constraints or learned scoring models.
  • 3.2 The auditing optimization problem: The core optimization maximizes φ(x,o) over prompts and outputs subject to the constraint that greedy decoding from x returns o.This directly searches for feasible examples of the target behavior.
  • 3.2 The auditing optimization problem: Auditing before deployment avoids inference-time filtering problems involving false positives and time-sensitive outputs, while allowing faster inference.The paper contrasts pre-deployment search with filtering prompts during use.
  • 3.3 Algorithms for auditing: Because feasibility is sparse, variables are discrete, models are large, and greedy decoding is nondifferentiable, the method adds output log probability to the auditing objective.Optimizing this surrogate often yields pairs satisfying f(x)=o and avoids repeated argmax differentiation.
  • 3.3 Algorithms for auditing: Joint optimization updates prompts and outputs directly, making target output behaviors more accessible than optimizing only prompts through f(x).This is especially relevant for rare behaviors with limited optimization signal.
  • 3.3.1 ARCA: ARCA uses coordinate ascent: it updates one prompt or output token at a time, cycling until the generation constraint and auditing threshold are met or iterations are exhausted.Each update selects the token maximizing the coordinate objective.
  • 3.3.1 ARCA: To reduce computation, ARCA ranks all vocabulary tokens with a hybrid approximation using one exact autoregressive forward pass and averaged first-order approximations, then evaluates only top-k tokens exactly.Averaging random-token approximations is conjectured to reduce variance and better approximate the objective globally.

4 Experiments

The experiments use ARCA to audit GPT-2 and GPT-J for toxic, surprising, cross-lingual, inaccurate, and offensive behaviors, including through optimization over prompts and outputs. ARCA generally outperforms AutoPrompt and GBDA, retrieves problematic examples, and sometimes transfers behaviors from smaller to larger models.

  • Experimental setup: ARCA audits GPT-2 and GPT-J by optimizing prompts and outputs for specified target behaviors, including reverse generation, surprise toxicity, and language-changing completions.The experiments initialize prompts and outputs randomly and optimize until the model completion and auditing objective meet the stopping criteria.
  • Toxic comments: ARCA nearly always outperforms AutoPrompt and GBDA on toxic-output reversal, although all methods often fail on three-token outputs.GBDA achieves less than 1% success on GPT-2 three-token outputs and struggles on GPT-J; ARCA’s advantage is greatest for longer targets.
  • Toxic comments: 58% average normalized success on GPT-2 three-token outputs: ARCA exceeds 50% when restricting evaluation to outputs for which some optimizer finds a satisfactory prompt.The normalized rate excludes outputs that no tested optimizer can greedily generate.
  • Revealing prompts: ARCA uncovers factual, temporal, and potentially offensive completions, including incorrect political associations and derogatory celebrity statements.Examples include Florida governor → Rick Scott, Oklahoma Senator → John Cornyn, and Barack Obama is a legalized unborn → child murderer.
  • Surprise toxicity: Across prompt lengths, output lengths, and models, ARCA reliably finds surprise-toxicity examples, whereas AutoPrompt never satisfies the toxicity constraint.The authors attribute ARCA’s success to its autoregressive term, which updates output tokens based on preceding tokens.
  • Language-changing completions: ARCA achieves nonzero performance on French-to-English and German-to-English completions, while AutoPrompt does not.The optimizer routinely uncovers prompts in French and German that produce English outputs.
  • Scaling and transfer: GPT-J prompts have log-perplexity 0.6–5 points lower than GPT-2 prompts, and 20% of sampled GPT-2 toxic prompts produce toxic examples on GPT-3 davinci-002.GPT-3 generates the same first token for half of the 100 sampled prompts.

5 Discussion

The authors present discrete optimization as an additional auditing tool for uncovering hard-to-find model failure modes. They also identify unresolved capabilities and a potential risk that adversaries could exploit the tools.

  • Contribution: Discrete optimization can produce hard-to-find and undesirable model behaviors that existing auditing methods may miss.The authors position their approach as complementary because no existing method alone reliably finds all model failure modes.
  • Risk: The tools could in principle be used by adversaries to exploit failures in deployed systems, creating a security risk alongside their auditing use.The authors believe transparency and potential pre-deployment fixes outweigh this risk.
  • Open limitations: Unresolved tasks include API-only auditing, certifying the absence of failure modes, and behaviors that cannot be specified by one prompt-output pair or objective.The authors also identify adaptive lengths, more natural prompts, and better discrete optimizers as future work.
  • Practical scope: The framework is designed to adapt cheaply as problematic behaviors change, requiring an objective specification and an efficient optimizer for new settings.The authors suggest using it as an additional check before potentially harmful deployments.

A Additional Formulation and Optimization Details

The appendix explains ARCA’s efficient coordinate-ascent implementation and formal derivation, then contrasts auditing with rejecting high-objective samples. Auditing can uncover failure classes despite partial objective coverage while avoiding deployment-time filtering costs and harms.

  • A.1.1 Expanded derivations: Averaging random first-order approximations produces the score used in Equation (6), formally establishing the ranking equivalence.The appendix states that averaging k approximations gives the score described in Equation (6).
  • A.1.1 Expanded derivations: Equation (4) decomposes the log probability into prefix and suffix terms, separating a linearly approximatable component from an autoregressive component.The chain-rule decomposition yields the two terms used in Equation (5).
  • A.1.1 Expanded derivations: A first-order approximation ranks token candidates using the token embedding’s inner product with the gradient, because the remaining term is constant across candidates.The derivation identifies C as independent of v.
  • A.1.1 Expanded derivations: The candidate score combines the auditing objective with a weighted language-model log probability over the autoregressive completion term.The displayed score is s_i,Lin(v) = φ(...) + λpLLM log pLLM(...).
  • A.1.2 Pseudocode: ARCA’s implementation computes candidate scores with one gradient and matrix multiply, while output-token autoregressive scores require a single forward pass.The appendix describes this as a source of computational efficiency.
  • A.1.2 Pseudocode: ARCA’s pseudocode alternates coordinate updates over prompt and output tokens, returning a pair when f(x) = o and φ(x, o) exceeds τ.Prompt and output coordinates are optimized separately using candidate-token scores.
  • A.2 Discussion on rejecting high-objective samples: Using φ to reject examples can discard false positives, including high-objective outputs that are acceptable and whose filtering may disproportionately affect subgroups.The appendix contrasts this with auditing, where generating false positives is acceptable if problematic examples are also uncovered.
  • A.2 Discussion on rejecting high-objective samples: Filtering may be unacceptable for time-sensitive systems, whereas auditing shifts objective computation before deployment and identifies failure classes even with partial coverage.Auditing can expose a failure mode when φ is high for some instances, while filtering lets low-objective instances through.

B.1 Additional experimental details

This appendix records implementation and compute details for the experiments, including hardware, optimizer settings, success timings, and constraints used to eliminate degenerate reversed-language solutions.

  • Compute details: Each attack runs on a single GPU, including A100, A4000, and A5000 devices.GBDA uses eight parallel runs to make computation cost comparable.
  • Compute details: ARCA returns correct solutions in 1.9 seconds for outputs of length 2, 9.22 seconds for outputs of length 2, and 11.5 seconds for outputs of length 3.The passage reports these average timings alongside GBDA’s 20.4-second timing.
  • Hyperparameters: ARCA uses 32 random gradients, 32 exact-inference candidates, and a maximum of 50 iterations in all experiments.These settings were chosen as the largest reliably fit in memory.
  • Hyperparameters: GBDA uses Adam with learning rate 1e-1, runs for 200 iterations, and executes eight attack instances in parallel.The learning rate was selected after testing values from 5e-3 through 1.
  • Quantitative results: Figure 3 reports senator-reversal success rates for unrestricted and lowercase-only prompts across five random-seed runs, with dotted rates covering prompts solved by any run.The figure distinguishes average success rates from the broader solved-prompt set.
  • Eliminating degenerate solutions: The reversed-language experiments enforce a no-token-overlap condition while accounting for semantically similar tokens after lowercasing and removing spaces.This prevents degenerate solutions based on simple attributes such as capitalization.

B.2 Additional results when reversing the LLM

The appendix introduces supplementary experimental results for the U.S. senators task and additional returned prompts.

  • B.2 Additional results when reversing the LLM: The supplementary section first presents quantitative senator-task results, then provides example prompts returned by the optimizer.It supplements the main experiments with both aggregate and qualitative evidence.

B.2.1 Additional U.S. senator results

Additional results show ARCA outperforming AutoPrompt and GBDA on senator reversal, while retrieved prompts expose factual, temporal, toxic, and language-switching behaviors.

  • B.2.1 Additional U.S. senator results: ARCA consistently outperforms AutoPrompt on senator reversal, especially with lowercase-only prompts, while GBDA nearly always fails.The comparison covers both GPT-2 and GPT-J.
  • B.2.2 Example prompts: The appendix includes senator completions with factual errors, such as “Kansas Treasurer → John Kennedy” and “Oklahoma Senator → John Cornyn.”These examples illustrate undesirable contextual completions.
  • B.2.2 Example prompts: Additional senator completions include temporal errors, alongside other contextually undesirable associations.The passage lists examples such as “Senate Whip → John Cornyn.”
  • B.2.2 Example prompts: The toxic-prompt examples include stereotyped, insulting, threatening, and profane completions generated from nonsensical or innocuous-looking prompts.Examples include racial stereotypes, personal insults, and explicit language.
  • B.2.2 Example prompts: Figure 4 measures ARCA’s French-to-English and German-to-English success across prompt and output lengths using both language-validity and exact-completion criteria.Solid curves require f(x) = o plus language constraints; dash-dot curves require only f(x) = o.

B.3 Additional results when optimizing over inputs and outputs jointly

This section adds further examples produced by both optimization methods for the French-to-English and German-to-English completion tasks.

  • The section provides additional examples returned by both optimization methods for French-to-English and German-to-English completions.

B.3.1 Quantitative Language Results

The language experiments evaluate whether optimized prompts in French or German complete to English, using language-score thresholds and punctuation filtering. ARCA finds such language-flipping solutions across prompt and output lengths, whereas AutoPrompt finds none, although ARCA fails more often than on the toxic-to-non-toxic task; extensive examples illustrate these completions.

  • Success requires prompt and output language scores above 0.5 across tokens, with punctuation tokens removed because their probabilities are unreliable.
  • Across prompt and output lengths, ARCA finds some French-to-English and German-to-English language flips, while AutoPrompt finds none.
  • ARCA fails more often on language switching than on the toxic-to-non-toxic task, which the authors conjecture reflects the greater challenge of switching languages.
  • The appendix also includes additional toxic GPT-2 completions about celebrities, including outputs involving sexual abuse, racism, and other offensive content.
  • Additional French-to-English examples include prompts completing to phrases such as “of the French,” “in the city,” and “to the head.”
  • Additional German-to-English examples include prompts completing to phrases such as “is the,” “from the ground and,” and “he was.”

B.4 Transferring prompts.

Prompts generated for GPT-2 can transfer undesired behaviors to GPT-3, sometimes producing identical or related toxic outputs. The transfer study indicates that auditing insights can generalize across models.

  • 21% of GPT-3 outputs were toxic under manual inspection, despite prompts being generated for GPT-2.
  • Overall, the transfer results suggest that prompts found on GPT-2 can reveal failures on GPT-3 without direct supervision.
  • GPT-2 and GPT-3 sometimes produced qualitatively similar celebrity-related toxic completions without exact output matches.Examples include “child murderer” versus “baby killer” for Barack Obama and different derogatory completions for Donald Trump.
  • Figure 5 compares mean log-perplexity under GPT-3 davinci-002 across prompt lengths for prompts generated by GPT-2 and GPT-J.GPT-J has lower perplexity at every prompt length, with the largest difference at shorter lengths.

B.5 Naturalness

The study examines whether scaling the audited model makes automatically generated prompts more natural. Prompts generated by GPT-J have lower GPT-3-measured perplexity than those generated by GPT-2 across all tested lengths.

  • For each prompt length from 2 to 8, researchers sampled 100 prompts and computed mean log perplexity using GPT-3 davinci-002.
  • GPT-J prompts had lower perplexity than GPT-2 prompts at every tested length.
  • 1.8 points lower was GPT-J’s average perplexity difference, ranging from 0.6 to 5.0 points across prompt lengths.
  • Randomly selected examples show that generated prompts can be unnatural and sometimes toxic when no naturalness supervision is applied.A naturalness penalty is used when more realistic prompts are desired.
  • The authors conclude that the method produces more natural prompts for larger models.
Loading 2303.04381v1…