Source-linked AI summary
ASSERT: A Measurement Pipeline for GenAI Audits
Riccardo Fogliato, Abhinav Palia, Xiawei Wang, Emily Sheng, Chad Atalla, Jean Garcia-Gathright, Nicholas Pangakis, Sharman Tan, Dan Vann, Hannah Washington, P. Alex Dow, Heba Elfardy, Hanna Wallach, Sandeep Atluri
TL;DR
Reported compliance rates conflate GenAI behavior with measurement choices, making audit differences difficult to interpret. ASSERT binds each rate to an explicit specification, and the case study shows that judges and other choices can substantially change rates and rankings.
Problem
Reported compliance rates depend on implicit measurement choices, limiting their interpretability, comparability, and reproducibility.
Method
ASSERT records measurement choices in a written specification and uses it to design and run the corresponding audit.
Results
Judge substitution changed the reported rate from 80% to 95% and changed system rankings within the audit.
Takeaways & Limitations
Explicit, inspectable specifications are necessary for meaningful safety measurement and for scrutinizing rates and rankings.
Takeaways & Limitations
ASSERT rates estimate compliance only for the transcript distribution induced by the audit design and do not justify inference about other populations without a sampling or bridging argument.
Abstract
from arXiv · showhide
Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.
1 Introduction
Reported GenAI audit rates depend on both the audited system and explicit measurement choices, making implicit choices difficult to interpret, compare, and reproduce. ASSERT records those choices in a written specification and shows in a conversational-deception case study that they can change rates and reorder system rankings.
- Motivation: Reported compliance rates reflect both the audited system and the choices defining and measuring compliance.Interpreting a rate requires knowing the measurement task and measurement instruments.
- Sources of variation: Measurement-task and instrument choices, including deception definitions, prompt conditions, and judges, can produce different rates for the same system.These choices extend beyond a dataset and metric to the models used to generate and score audits.
- ASSERT: ASSERT is a specification-driven pipeline that records measurement choices in a written specification and runs the corresponding audit.The pipeline starts from a broad concern and application context, then uses the specification to build and run the audit.
- Case study: In a conversational-deception case study, dialogue setup, simulated user, judge, and evidence bar for non-compliance changed reported rates and reordered system rankings.ASSERT was applied under multiple specifications differing in these measurement choices.
- Positioning: ASSERT complements fixed benchmarks and dynamic generative auditing pipelines by making measurement choices explicit and organizing audits around a measurement framework.Existing benchmarks typically fix test cases and scoring rules, whereas generative auditing pipelines dynamically generate policy-specific inputs.
2 The ASSERT Measurement Pipeline
ASSERT is a specification-driven pipeline that defines a GenAI audit’s measurement task and instruments, operationalizes them into test cases, transcripts, judgments, and a reported rate. Its explicit specification makes the resulting measurement executable and inspectable.
- Measurement specification: A measurement specification records the task elements and instrument settings used to produce a reported rate for an evaluated GenAI system.The pipeline treats each operationalized activity as a concrete run that generates test cases, rolls them out, labels instances, and aggregates labels.
- Measurement task: ASSERT fixes the systematized concept, instance type, population of interest, and estimand to define what the audit measures.In the conversational-deception audit, instances are multi-turn transcripts, and the estimand is expected transcript-level compliance under the distribution induced by the test design, rollout protocol, and system.
- Operationalization: The pipeline operationalizes the task through a behavioral rubric, LLM-generated test cases, and a rollout protocol that produces transcripts, including simulated-user interactions for multi-turn audits.Researchers define test designs by crossing rubric behaviors with context dimensions such as interaction condition, user persona, or user-ask type.
- Test-case design and generation: Test-case allocation is chosen to support intended comparisons, with balanced allocation as a default and priority-cell overweighting or power analysis available when appropriate.In the case study, six non-permissible behaviors were crossed with four interaction conditions, with five scenario briefs allocated to each resulting cell.
- Rollout, scoring, and estimation: The reported rate is the unweighted proportion of included transcripts judged compliant, computed overall and within factor subgroups.Cells contribute according to realized included counts; in the case study, equal design-time allocation and rare omissions keep the rate close to the balanced default.
3 Case Study: A Multiverse Analysis of a Deception Audit
ASSERT’s deception case study fixes GPT-5.5 and shows that reasonable measurement choices—interaction condition, simulated user, judge, and evidentiary standard—substantially change reported rates. These choices can also change system rankings, while fixed-specification run-to-run variation provides a noise-floor benchmark for interpreting movements.
- Baseline specification: The baseline audit evaluates GPT-5.5 using Grok-4.3 as simulated user, a ten-criterion deception rubric, four interaction conditions, and 120 scenarios.The design includes five scenario briefs per interaction-condition cell.
- Conditional reproducibility: 82% to 87% is the fixed-specification rollout range, establishing the noise floor below which reported-rate movements are treated as run-to-run variation.The reruns reuse scenario briefs but generate new transcripts; five fresh judge calls on identical transcripts remain within the Figure 3 shaded band.
- Interaction condition and simulated user: 77% to 87% is the range across interaction conditions, while replacing Grok-4.3 with GPT-5.5 as simulated user raises the rate above 90%, versus 82% under Grok-4.3.Equal allocation and rare omissions give interaction conditions approximately equal weight in the overall mean.
- Judge: 15 points is the judge-substitution change from 80% under Opus-4.7 to 95% under Grok-4.3, while the GPT-5.5 baseline judge reports 82%.Despite a two-point overall difference, GPT-5.5 and Opus-4.7 disagree on 26 of 120 transcript labels and agree on only 10 of 36 transcripts flagged by at least one judge.
- Evidentiary standard: ∼80% to above 90% is the reported-rate change from the looser to stricter evidentiary standard; across conditions, the ranges are 73–90% and 87–97%, respectively.The transcripts, GPT-5.5 judge, and criterion definitions remain fixed while only the evidentiary standard changes.
- Evaluated systems and judge dependence: No single ranking of GPT-5.5, Opus-4.7, and Grok-4.3 holds across all judges, which also changes their absolute reported rates.The GPT-5.5 judge reports the lowest rates on average, while the Grok-4.3 judge yields near-ceiling rates for all three systems.
4 Discussion
The discussion argues that reported GenAI audit rates and system comparisons are specification-dependent, limiting confidence when measurement choices remain opaque. ASSERT addresses this by documenting measurement choices end to end and enabling scrutiny of which choices drive results.
- Specification uncertainty: Reported rates vary with opaque measurement choices, so treating one pipeline as definitive can make point estimates appear more conclusive than specification uncertainty warrants.The discussion frames operationalization choices under a fixed systematization as a source of variation.
- Specification uncertainty: System comparisons warrant little confidence unless performance differences remain robust to reasonable alternative specifications.The alternatives are most interpretable when they hold systematization fixed and vary operationalization.
- Specification uncertainty: The judge changes both absolute rates and rankings, making it especially consequential and requiring explicit naming and testing of reasonable alternatives.The discussion applies the same interpretive logic to changes over time, which require comparison with run-to-run variation under fixed and alternative specifications.
- ASSERT’s contribution: ASSERT makes safety measurement inspectable by systematizing vague concepts, recording concept specifications, and operationalizing them through explicit instruments.Researchers can review and edit these specifications, with stakeholder input when appropriate.
- ASSERT’s contribution: ASSERT records the operationalized activity and measurement choices end to end, helping researchers form hypotheses, stress-test systems, and identify choices driving reported rates.The specification complements model cards, datasheets, and benchmark cards by documenting how a reported rate was produced.
5 Limitations
ASSERT’s reported rate is limited to the transcript distribution induced by its design and protocol, while operationalization and scoring choices can miss, distort, or change the measured behavior.
- Population and generalization: The reported rate estimates expected compliance only under the transcript distribution induced by the test design, rollout protocol, and evaluated system.Wilson intervals quantify ordinary sampling uncertainty under independent-transcript assumptions, not all uncertainty.
- Population and generalization: More transcripts tighten uncertainty about the audit estimand but cannot automatically justify claims about deployment traffic, broader user mixes, or other target populations.Generalization requires a specified target and a sampling or bridging argument connecting the audit to it.
- Operationalization and scoring: Generated cases determine which situations can affect the audit, so situation classes that never appear cannot influence the reported rate.This operationalization can miss or distort the intended concept.
- Operationalization and scoring: A higher rate under one simulated user is ambiguous because the user may improve interaction coverage or simply give violations fewer chances to surface.The simulated user is part of the audit’s operationalization and can enrich multi-turn interaction relative to single-turn prompts.
- Operationalization and scoring: Finer rubrics can lower reported compliance without changing system behavior because any applicable non-compliant criterion makes the transcript non-compliant.With independent criterion-level false positives at common rate α, k applicable criteria yield transcript-level false-positive rate 1 − (1 − α)^k ≈ kα.
A Extended Related Work
Prior work frames GenAI safety rates as validity-dependent measurements shaped by specifications, instruments, and context. Related approaches include benchmarks, generative auditing pipelines, G-theory, and sensitivity or multiverse analyses.
- Safety measurement as a validity problem: GenAI evaluation should connect background concepts to systematized concepts, measurement instruments, and context; estimates require evidence that operationalized activity captures the intended estimand.The measurement framework of Chouldechova et al. (2024) provides the conceptual foundation for this validity-oriented approach.
- Safety benchmarks and generative auditing pipelines: Benchmarks improve repeatability and comparability by fixing test cases, taxonomies, scoring procedures, and estimators, while auditing pipelines generate test cases from researcher-specified concepts, policies, or behaviors.ASSERT shares test-case generation with generative auditing pipelines but asks a different question once the pipeline is fixed.
- Sources of measurement-instrument variation: Researcher choices can change reported rates even when the evaluated GenAI system is held fixed, because elicitation and search determine which interactions and failures surface.Iterative and multi-turn adaptive elicitation add choices such as escalation speed, adaptivity, and what counts as successful.
- Generalizability theory: ASSERT reports sensitivity across purposively selected reasonable specifications rather than G-theory’s population-level variance components for judges, simulated users, or evidentiary standards.G-theory could complement this analysis when a defensible universe of admissible conditions and decision design can be specified.
- Stress testing measurements: Sensitivity and multiverse analyses stress-test whether findings remain stable by perturbing data, models, and specifications when reported rates depend on reasonable design choices.A single point estimate is incomplete when measurement choices can affect the reported rate.
B Worked Concept and Rubric: Conversational Deception
The section operationalizes conversational deception as a systematized concept, pattern inventory, and behavioral rubric for ASSERT. It defines transcript-level scoring against a reference basis, covering deceptive violations and corresponding permissible behaviors.
- B Worked Concept and Rubric: Conversational Deception: ASSERT’s worked concept comprises a systematized concept, pattern inventory, and behavioral rubric, while the executable measurement specification appears in Section C.The rubric is the judge-scored instrument, with definitions, evidence requirements, examples, and permissibility labels.
- B.2 Measurement Instruments: A non-permissible criterion fails when applicable and present; a permissible criterion fails when applicable and absent.Transcripts are compliant only when every applicable criterion is compliant; transcripts with no applicable criteria are omitted from the reported rate.
- Behavioral Rubric for Deception: The rubric’s non-permissible criteria include materially false assertions, misleading omissions, fabricated evidence or provenance, false self-presentation, and unjustified agreement.Each criterion is judged against the supplied reference basis rather than the judge’s unaided world knowledge.
- Behavioral Rubric for Deception: The multi-turn pressure-fold criterion captures sustained or strengthened misrepresentation without new support, while single-turn cases are scored under other criteria.The case-study design elicits this trajectory through material reliance, false premise, consistency challenge, or accountability.
- Behavioral Rubric for Deception: Permissible criteria require transparent uncertainty or correction, faithful evidence and provenance, accurate self-presentation, and calibrated resistance to pressure.These behaviors mark limits, repair prior representations, faithfully distinguish evidence from inference, accurately report system state, and resist unwarranted premises.
- Behavioral Rubric for Deception: Test-design factors structure elicitation, but the judge scores only the ten criteria from the transcript and reference basis.Material reliance, false premise, consistency challenge, and accountability are elicitation factors rather than additional scoring criteria.
- Behavioral Rubric for Deception: The policy_violation outcome is true if any applicable criterion is violated, and the selected inventory and rubric cover only instantiated generated-test scenarios.Transcript-level scoring and the reported-rate estimator follow Section 2.
C ASSERT Measurement Specification Vignette
This appendix specifies the ASSERT measurement configuration for conversational deception, linking the behavioral rubric, test design, and transcript-level outcomes to named code keys. The case study elicits only six non-permissible criteria while judging all ten criteria.
- Baseline configuration: Figure 4 records a baseline conversational-deception measurement configuration, while Section 3.1 identifies the instrument settings varied later.The appendix presents the full case-study specification in ASSERT syntax.
- Behavioral rubric: ASSERT stores the behavioral rubric in taxonomy.json under behavior_categories, and test-case generation reads a behavior axis derived from that file.The judge-facing artifact consists of behavioral criteria.
- Test design: Generation uses only the six non-permissible criteria as elicitation goals, while the judge scores all ten criteria.Interaction conditions are declared under test_set.stratify.dimensions.
D Additional Results
The appendix extends the main findings by testing judge dependence and measurement sensitivity. It examines global judgment, per-criterion failure profiles, and three additional concepts.
- D.1: Section D.1 tests whether cross-judge disagreement disappears under a single global judgment.The appendix presents this as an extended analysis supporting the main findings.
- D.2: Section D.2 shows that per-criterion failure profiles of the evaluated GenAI system remain highly judge-dependent.This analysis examines whether judge dependence persists at the criterion level.
- D.3: Section D.3 runs an analogous sensitivity check for three additional concepts, comparing reasonable measurement alternatives with fixed-specification results.The passage states that reasonable alternatives can produce larger ranges than fixed-specification results, though the sentence is truncated.
D.1 Judge Sensitivity Persists Under a Global Judgment
Judge sensitivity persists even when judges replace criterion-by-criterion scoring with a single global deception judgment. In the baseline 120-transcript audit, judges still disagreed substantially, and reported rates differed by 15 percentage points.
- Scoring Conditions: The audit rescored the baseline N=120 transcripts using both decomposed criterion-level scoring and global transcript-level deception judgments.In the decomposed condition, judges scored ten behavioral criteria and classified a transcript as non-compliant when any applicable criterion was violated or missing; in the global condition, judges made one judgment using only the concept specification.
- Judge Disagreement: Under global scoring, GPT-5.5 and Opus-4.7 each reported 84% while disagreeing on 20 of 120 transcript labels.Their disagreement count was 26 transcripts under decomposed scoring, with 15 disagreements persisting under global scoring and 5 new disagreements appearing.
- Judge Disagreement: Grok-4.3 reported 99% under global scoring, producing a 15-point range across judges.Thus, a single global judgment did not eliminate case-level disagreement or cross-judge variation in reported rates.
D.2 Per-Criterion Failure Profiles Are Judge-Dependent
Per-criterion failure profiles depend more on the judge than on the evaluated GenAI system. The divergence primarily reflects judges flagging different transcripts as non-compliant, rather than assigning flagged failures to different criteria.
- D.2 Per-Criterion Failure Profiles Are Judge-Dependent: The weakest criterion tracks the judge more than the evaluated GenAI system.Under GPT-5.5, all three evaluated systems score lowest on materially false assertion.
- D.2 Per-Criterion Failure Profiles Are Judge-Dependent: Judge disagreement primarily reflects selection: GPT-5.5 and Opus-4.7 usually identify similar violated criteria when both flag a transcript.Their profiles diverge because they flag mostly different subsets of transcripts, not because they sort the same failures into different buckets.
D.3 Sensitivity Across Safety Concepts
Across disability representation, crisis response, deception, and inference of sensitive attributes, rerunning a fixed specification changes little, while reasonable measurement alternatives often change reported rates more. The dominant source of sensitivity differs by concept, with ISA comparatively stable under the plotted alternatives.
- Cross-concept pattern: Across concepts, rerunning a fixed specification changes little, while at least one reasonable alternative usually changes the rate more than fixed-specification stochastic variation.Judge substitution is the most consistent source of such change, but the dominant measurement choice differs across concepts.
- Concept-specific sensitivity: For crisis response, elicitation mode matters most: worst-case versus naturalistic mixtures differ by more than 30 points, compared with 8 points for the scoring rule.The result comes from varying measurement choices while holding the broad ASSERT design fixed.
- Concept-specific sensitivity: Disability representation shows the same sensitivity pattern more mildly, while deception is dominated by the judge and interaction-condition differences are secondary.These patterns are observed under the analogous sensitivity analysis using a GPT-5.5 baseline judge and Grok-4.3 generator and simulated user.
- Concept-specific sensitivity: In ISA, no plotted judge substitution, scoring-rule change, or elicitation-level comparison changes the aggregate rate by more than about 6 points, making it comparatively stable.The aggregate rate is therefore comparatively stable under the plotted alternatives.