PaperlayerView exact revision on arXiv

Source-linked AI summary

What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

Ro Encarnación, Tina Behzad, Emma Lurie, Danaé Metaxa

arXiv:2608.06202v2cs.HCcs.AI

TL;DR

Benchmark evaluations often rely on one access modality, one run per prompt, and accuracy as the main outcome, despite the importance of deployed-system behavior. This paper audits ChatGPT’s chat UI and API with and without search, measuring response-level behavior across two benchmarks. Modality and search condition influenced responses across several dimensions, while the study’s scope is limited to two benchmarks, one model family, and one collection period.

  • Problem

    Benchmark evaluations commonly rely on a single modality, one response per prompt, and aggregate accuracy, leaving other dimensions of model behavior less systematically assessed.

    Source: §1 · pp. 1–2

  • Method

    The audit compares ChatGPT’s chat UI and API with search enabled or disabled, using 401 sampled prompts from BBQ and SafetyBench to examine accuracy and response-level behavior.

    Source: §3.1 · p. 3, §5 · p. 9

  • Results

    Modality and search condition influenced accuracy, response text, sources, consistency, and abstention, revealing variation that average accuracy alone does not capture.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The audit concludes that evaluations should account for interfaces, search conditions, repeated-run consistency, and response-level behaviors when assessing deployed-system variation relevant to AI safety.

    Source: §1 · p. 2, §6 · p. 10

    The findings reflect two benchmarks, one model family, and a single collection period, so their magnitude and direction may vary across providers, versions, benchmarks, and settings.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness. Yet most evaluations rely on a single access modality (model APIs), perform a single run per prompt, and report accuracy as the primary outcome metric, without accounting for conditions such as web search that may have effects on model behavior in deployment. We audit these assumptions for one of the most widely-used LLMs, comparing two modalities, ChatGPT's chat UI and OpenAI's API, with and without web search enabled. We use a stratified total sample of 401 prompts from two popular benchmarks, BBQ and SafetyBench, collecting 4,812 total responses across three repeated runs per prompt. Beyond standard performance measures, we evaluate model output dimensions including response consistency, response text similarity, citation grounding, and abstention behavior. For instance, chat UI responses were less accurate than API responses on both benchmarks with search disabled. Enabling web search reduced accuracy by up to 8 percentage points, and even reversed the direction of modality performance trends for one benchmark. Repeated runs of the same prompt produced inconsistent responses in up to 21\% of prompts. The two modalities also grounded answers in different citations, and abstention behavior was also inconsistent across both modalities. These results illustrate that, even within a model family, reporting only simple accuracy metrics can obscure important forms of model behavioral variation relevant to AI safety assessments. We argue that AI safety evaluations should systematically account for modality, multi-run consistency, search conditions, and response-level behaviors to better reflect how deployed AI systems behave in practice.

1 Introduction

The paper questions whether API-only, single-run evaluations and aggregate accuracy adequately represent deployed model behavior. It compares ChatGPT’s chat UI and API across search conditions, finding meaningful variation beyond accuracy that matters for safety assessment.

1 Introduction

  • Current evaluations often treat API results as representative, one response as sufficient, and similar accuracy as evidence of similar model behavior.Chat interfaces also include system prompts and moderation policies that can shape responses.Source: §1 · p. 1
  • The audit compares ChatGPT’s chat UI and API with web search enabled or disabled, measuring accuracy, consistency, response similarity, citations, and abstention on BBQ and SafetyBench.Source: §1 · p. 1
  • Accuracy differences across modalities were small, around 2–3 percentage points, statistically significant on SafetyBench, and reversed direction under search.Source: §1 · pp. 1–2
  • Search condition and benchmark affected consistency, while the interfaces often used different sources and varied in phrasing, semantic similarity, reasoning, and abstention.A prompt refused in one setting was often answered in another.Source: §1 · p. 2
  • These findings indicate that single-modality, single-search-condition evaluations focused mainly on accuracy may miss behavioral variation relevant to AI safety.Source: §1 · p. 2

2 Related Work

Prior benchmark research commonly uses controlled API-based evaluations and aggregate performance measures, while deployed systems and repeated interactions may behave differently. This paper compares interfaces and search settings while examining behavioral dimensions beyond accuracy.

2 Related Work

  • Benchmarks provide standardized comparisons across capabilities, but static API-based, single-turn evaluations are not well suited to understanding system behavior in deployment.The literature covers reasoning, retrieval, coding, safety, and social bias.Source: §2 · p. 2
  • Studies report behavioral differences across API and chat settings, but search-enabled LLM research has focused more on factual accuracy and retrieval than interaction behavior.Related work also finds that citation quality, grounding, and presentation shape perceived helpfulness beyond factual correctness.Source: Interaction Modality and Deployment… · p. 2
  • Aggregate scores leave response production, grounding, and variation across settings underexamined, while single evaluation runs limit assessment of repeated-prompt consistency.Prior consistency studies more often vary prompts, context, or system settings than repeat the exact same prompt.Source: Evaluation Metrics · pp. 2–3
  • The paper compares API and chat interfaces with and without search over multiple runs, assessing consistency, text similarity, abstention, citation, and grounding patterns.Source: Evaluation Metrics · p. 3

3 Method

The audit compares the same model family through ChatGPT’s chat interface and API, with search enabled or disabled, using stratified samples from two safety and bias benchmarks. It measures accuracy and multiple response-level behaviors, with repeated observations analyzed using mixed-effects models.

3.1 Audit Design

  • Modality refers to access through the consumer-facing ChatGPT interface or programmatic API, including each access method’s prompts, search, moderation, and other interface behavior.Search is tested both enabled and disabled.Source: §3.1 · p. 3
  • The audit samples 198 BBQ questions and 203 SafetyBench questions, uses zero-shot prompts, and repeats each prompt three times in each modality-search condition.The study compares logged-out ChatGPT with the corresponding API model and does not control system prompts to make the interfaces behave identically.Source: Benchmark sampling · p. 3, Model and environment selection · p. 3, Prompting technique · p. 3

3.2 Evaluation Measures & Primary Analysis

  • The analysis measures accuracy, consistency, response similarity, citation behavior, and abstention across repeated runs, modalities, and search conditions.Consistency is the proportion of prompts whose three runs select the same answer, regardless of correctness; similarity uses lexical overlap and semantic embeddings, compared within and between modalities.Source: §3.2 · p. 4

3.3 Supplementary Statistical Analysis

  • Binary outcomes are modeled with generalized linear mixed-effects models and continuous outcomes with linear mixed-effects models, with prompt-level random intercepts.Benjamini–Hochberg correction covers primary inferential tests; secondary analyses are exploratory and excluded from that correction family.Source: §3.3 · p. 4

4 Results

Across BBQ and SafetyBench, accuracy, consistency, response construction, citation grounding, and abstention varied across modality and search conditions. Search could reverse the accuracy gap between chat UI and API, while repeated-run and cross-modality differences extended beyond accuracy.

4.1 Search-enabled conditions shifted accuracy between modalities

  • With search enabled, SafetyBench accuracy reversed from API-leading to chat UI-leading, while chat UI remained slightly less accurate on BBQ; search reduced accuracy overall.On SafetyBench, chat UI accuracy was 84.4% versus API at 80.6%; on BBQ, chat UI was 75.9% versus API at 77.8%.Source: Enabling search further reduced… · p. 5, Figure 1 · p. 5

4.2 Response consistency varied by modality and search condition

  • Repeated-answer inconsistency depended on benchmark and search: BBQ increased by about six percentage points for both modalities, while SafetyBench chat UI inconsistency fell and API inconsistency rose with search.On SafetyBench, inconsistency was 12.3% for chat UI versus 6.4% for API without search, and 8.4% versus 12.8% with search.Source: The same prompt produced inconsistent… · p. 5
  • Between-modality disagreement exceeded within-modality run-to-run disagreement across both benchmarks and search conditions, ranging from 1.14 to 1.30 times the larger within-modality rate.The comparison was statistically significant (OR = 1.51, CI [1.19, 1.92], p = 0.002).Source: Modality introduced variation beyond… · pp. 5–6, SafetyBench · p. 6, Figure 2 · p. 6
  • Across benchmarks and search conditions, between-modality responses were less similar than within-modality responses in both wording and meaning, though chat UI was less consistent in wording but more consistent semantically across its runs.The Jaccard comparison showed lower within-chat than within-API similarity, whereas cosine similarity showed higher within-chat than within-API similarity.Source: API and chat UI responses phrased… · p. 6, API and chat UI responses were also… · p. 7, Figure 4 · p. 7

4.4 API and chat UI grounded responses in different information sources.

4.5 Abstention was inconsistent across modalities and independent runs

  • SafetyBench produced no abstentions, while BBQ had six rare abstentions across four prompts, usually not reproduced across all three runs.Only one prompt triggered abstention in both modalities; three prompts triggered it in one run and the fourth in two runs.Source: §4.5 · p. 8

5 Discussion

The audit finds that modality and search conditions shape accuracy and other response behaviors that single-run accuracy summaries miss. The authors argue that evaluations should better reflect deployed interfaces and account for practical limits on independent auditing.

5.1 Evaluation conditions determine what model response patterns are visible

  • Search-enabled conditions had lower accuracy across both benchmarks, while citation sources and response text also differed across interfaces.API and chat UI often relied on different sources for the same prompt, even when their answers agreed.Source: §5.1 · p. 9
  • A guardrail that produces different outputs after a user resubmits a question may be functionally useless as a safety mechanism.The authors note that systematic API–chat UI differences mean API results may not fully represent user experience.Source: §5.1 · p. 9

5.2 Implications for AI safety evaluation practices

  • Safety evaluations should measure consistency, phrasing and reasoning, sources, and whether safety mechanisms hold, alongside accuracy.The authors argue that one access layer and one accuracy statistic cannot establish whether users encounter the same behavior.Source: §5.2 · p. 9

5.3 The infrastructure gap is a validity gap that defines what can be evaluated

  • Independent API and chat UI audits are costly, and chat UI studies add technical complexity and platform constraints.The authors argue that stable testbed infrastructure would lower barriers to independent evaluation.Source: §5.3 · p. 10

5.4 Ethical considerations when auditing

  • The audit paced queries through staggered browser sessions and variable delays, limiting collection to what was necessary.The authors frame this as balancing audit value against potential risks to the systems under test.Source: §5.4 · p. 10

5.5 Limitations & Future Work

  • Findings cover two benchmarks, one model family, and one collection period; behavior and effect sizes may differ in other settings.The authors also note that cited-source quality was not systematically assessed and similarity results depend on one embedding model.Source: §5.5 · p. 10

6 Conclusion

The conclusion reports an audit of sampled prompts from two safety and bias benchmarks across API and chat UI, with and without search. It finds response patterns beyond aggregate accuracy and calls for evaluations that account for access interfaces and response variation.

6 Conclusion

  • The study compares API and chat UI outcomes, with and without search, using 401 prompts sampled from two widely used bias and safety benchmarks.It examines response consistency, textual similarity, and grounding behavior.Source: §6 · p. 10
  • The study finds distinct patterns across modalities, benchmarks, and search settings in response consistency, textual similarity, and grounding, even when aggregate accuracy is comparable.Source: §6 · p. 10
  • The observed differences support evaluation practices that account for user-facing interfaces and response characteristics beyond aggregate accuracy.Source: §6 · p. 10

A Appendix

The appendix provides methodological details and descriptive and inferential results tables that support the main text.

A Appendix

  • Appendix methodology details cover prompt formats, scoring, citation processing, and data collection.Source: A Appendix · p. 14
  • The appendix includes descriptive and inferential results tables supporting the main text.Source: A Appendix · p. 14

A.1 Prompt formats

The prompt formats differ across the two benchmarks: BBQ uses a context, question, and three options, while SafetyBench uses multiple-choice questions, mostly binary.

A.1 Prompt formats

  • BBQ prompts present a short context passage, a question, and three multiple-choice options.Source: §A.1 · p. 14
  • BBQ examples include ambiguous and disambiguated context variants, with “Can’t be determined” as the ambiguous-context answer.Source: §A.1 · p. 14
  • SafetyBench questions are multiple-choice, mostly Yes/No, though some have up to four options.Source: §A.1 · p. 14

A.2 Scoring

Accuracy is scored by matching each response’s selected answer option to the benchmark gold label, while abstentions are analyzed separately.

A.2 Scoring

  • Responses are correct when the selected option’s index matches the benchmark gold index; abstentions are excluded from accuracy and analyzed separately.Unparseable exact matches are manually reviewed and merged back where possible.Source: §A.2 · p. 14
  • For BBQ, 7 of 2,376 runs (0.29%) remained unresolved and were excluded, all from the no-search condition.Source: §A.2 · p. 14
  • For SafetyBench, 32 of 2,436 runs (1.3%) unresolved by exact match were assigned a uniformly random option using a fixed seed, following Zhang et al. 2024.Source: §A.2 · p. 14

A.3 Citation processing

Citation overlap is measured after URL normalization, using both exact URL matches and root-domain matches.

A.3 Citation processing

  • URLs are trimmed of tracking parameters before comparison, with overlap measured either by exact normalized URL or shared root domain.Domain-level matching treats different pages on the same website as equivalent.Source: §A.3 · p. 14

A.4 Data collection infrastructure

The collection pipelines gathered independent repeated responses through the API and chat UI, while the accompanying tables define accuracy, similarity, and abstention analyses across benchmark, modality, and search conditions.

A.4 Data collection infrastructure

  • The API collection script ran each query three times and stored each run separately, enabling web-search tool calling when search was enabled.Source: API pipeline · p. 14
  • Chat UI prompts ran in three parallel, isolated browser sessions; search was activated through interface settings when enabled, and surfaced sources were collected in that condition.No prior conversation context was carried between queries.Source: Chat interface pipeline · pp. 14–15
  • Enabling search reduced accuracy in every condition, with the change ranging from −1.5 pp for SafetyBench chat UI to −7.9 pp for SafetyBench API.Table 1 reports per-run accuracy by benchmark, modality, and search condition; abstentions are excluded.Source: Table 1 · p. 16
  • Table 2 reports pooled and benchmark-specific accuracy GLMMs using API/no-search as the reference, with prompt random intercepts and a modality-by-search interaction.The pooled modality main effect is a primary BH-corrected test; subset-fit p-values are raw follow-ups.Source: Table 2 · p. 16
  • Table 3 describes per-cell Jaccard similarity across benchmark, search condition, and between- versus within-modality response pairs; Table 4 models the per-prompt mean similarity.Between-modality pairs compare API with chat UI; within-modality pairs compare repeated runs of the same interface.Source: Table 3 · p. 16, Table 4 · p. 16
  • Table 5 describes cosine similarity of stripped response reasoning text, while Table 6 reports inferential models finding a significant pair-type-by-search interaction (p < .001, d = +0.42).The interaction indicates search moderately attenuates the cross-modality gap; the benchmark interaction was not significant (p = .27).Source: Table 6 · p. 17, Table 5 · p. 17
  • SafetyBench produced no abstentions; BBQ had six no-search abstention events across four prompts, with two API and four chat UI events.One BBQ prompt triggered abstentions in both modalities.Source: Table 7 · p. 18
Loading 2608.06202v2…