PaperlayerView exact revision on arXiv

Source-linked AI summary

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo

arXiv:2609.08861v1cs.AIcs.SE

TL;DR

Benchmark evaluations often use APIs as evidence about deployed chatbot behavior, but whether that evidence transfers across access surfaces is unclear. The paper audits matched API and interface systems across multiple providers and benchmarks, finding systematic differences in accuracy and consistency that exposed API controls do not reliably remove. It concludes that API measurements characterize the tested configuration and do not transfer directly to deployed products.

  • Problem

    Whether API benchmark measurements predict behavior in deployed chatbot interfaces remains insufficiently established, despite benchmarks being interpreted across contexts.

  • Method

    The study compares identical prompts sent to matched APIs and interfaces across seven systems, nine benchmarks, and multiple evaluation metrics.

  • Results

    API and interface measurements diverge in accuracy, test–retest agreement, and system rankings, while exposed API controls do not reliably reproduce interface behavior.

  • Takeaways & Limitations

    API evaluations remain evidence about controlled model behavior but should not be treated as direct proxies for corresponding deployed chatbot behavior.

  • Takeaways & Limitations

    The matched API and interface identifiers may not correspond to the same underlying checkpoint, so some observed differences may reflect model variation rather than interface-layer effects.

Abstract

from arXiv · show

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access exceeds the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

1 Introduction

Benchmarks are widely used to compare AI systems, but their results may not transfer across evaluation contexts. This study audits whether API measurements predict behavior in deployed chatbot interfaces and finds systematic, context-dependent differences.

  • Motivation: Benchmark scores coordinate comparisons, model releases, adoption decisions, and public understanding of AI utility.Their use assumes that measurements transfer across contexts of use.
  • Motivation: Benchmark measurements depend on model snapshots, prompt formats, sampling configurations, scoring procedures, and access surfaces.Prior validity critiques also identify contamination, item quality, formatting, answer choices, and evaluation protocols as sources of sensitivity.
  • Approach: The audit compares identical prompts sent through APIs and corresponding interfaces across seven systems and nine benchmarks covering capability, social bias, sycophancy, and hallucination.The methodology controls for routing, personalization, tool invocation, temporal drift, and output extraction.
  • Findings: API and interface measurements diverge in accuracy and test–retest agreement, with the magnitude varying across systems and benchmarks.The broader result is context dependence rather than universal superiority of one access surface.
  • Findings: Varying system prompts, sampling parameters, and reasoning settings shifts behavior in some cases but does not reliably eliminate the API–interface gap.The study frames these differences as a challenge to treating API scores as direct proxies for deployed behavior.

2 Related Work

Related work has improved the content and task validity of LLM evaluations, while comparatively less attention has addressed evaluation context. This paper treats access surface as a distinct context-validity dimension affecting whether API benchmark scores characterize chatbot behavior.

  • Evaluation validity: Ecological validity concerns whether test performance predicts behavior in real-world settings across content, task, and interaction context.The paper focuses on the context in which systems are evaluated.
  • Prior work: Recent evaluations improve content validity through open-ended user interactions and task validity through practical or domain-specific scenarios.These approaches better represent ambiguity, diversity, and real-world task settings.
  • Research gap: Evaluation context has received comparatively less attention, leaving a missing middle of routing, prompts, retrieval, memory, tools, and other deployment layers.Prior evidence shows that such components can alter bias, degrade performance, or destabilize responses.
  • Contribution: The paper studies access surface as a distinct context-validity dimension rather than isolating each hidden deployment component.It asks whether combined deployment-layer influences change conclusions drawn from API-based evaluations.
  • Contribution: Unlike prior audits of specific user-facing harms or behaviors, this work asks whether API benchmark scores accurately characterize chatbot behavior.The comparison shifts the object of study from particular harms to transferability of benchmark measurements.

3 Experimental Setup

The study compares matched API and interface systems using identical benchmark prompts, repeated trials, and controls for observable sources of variation. It evaluates capability and user-facing risks while measuring accuracy, consistency, and ranking stability.

  • Design: The audit compares user-facing interfaces with corresponding APIs for seven systems across ChatGPT, Claude, and Gemini using identical prompts from nine benchmarks.The design estimates access-surface differences on a fixed evaluation set rather than recovering full-benchmark scores.
  • Tasks: The benchmarks cover general capabilities and user-facing risks or reliability failures that deployment-layer controls may shape.Tasks include leaderboard capabilities, social bias, sycophancy, and cross-domain knowledge reliability.
  • Systems: API identifiers are matched to deployed chat products using provider documentation, interface model names, and public release information.The matching is necessarily approximate because providers do not generally disclose the exact interface checkpoint.
  • Controls: Five independent trials use rotated accounts, fresh browser profiles, and fresh chat windows without persistent browser or conversation state.The collection also synchronizes requests and controls observable routing, personalization, tools, and temporal drift.
  • Controls: Interface outputs are stripped to plain text and both conditions use the same benchmark-specific extraction and grading pipeline.The design addresses observable surface differences but cannot isolate every stage of the underlying request pipeline.
  • Metrics: Accuracy, test–retest agreement, and rank stability measure performance, response consistency, and agreement between system rankings across access surfaces.Accuracy averages correctness across K = 5 runs per item, while rank stability uses Spearman correlation.

4 Do Chat Interfaces Match API Benchmark Scores?

Across the audited systems and benchmarks, API evaluations generally report higher accuracy and greater repeatability than corresponding interfaces, but the size and direction of gaps vary. Access surfaces can also alter rankings and make interface degradation comparable to model-version differences.

  • Accuracy: 3.4% is the average API–interface accuracy gap, with all seven systems significantly more accurate under API evaluation.The system-level estimated gaps range from 1.9% to 4.6%.
  • Accuracy: API–interface accuracy differences vary across system–benchmark pairs, including a +28.4% gap for Claude Haiku 4.5 on AITA.The largest reported gaps also include +19.0% for GPT 5.4 Thinking on WinoGrande and Gemini 3 Flash Fast on AITA.
  • Model-version comparison: 4.5% is the mean API-only difference between GPT 5.3 Instant and GPT 5.4 Instant, while GPT 5.4 Thinking’s interface–API gap exceeds that model-version difference.On WinoGrande, the +19.0% access-surface gap is more than double the 9% API-only model-version difference.
  • Rankings: ρ = 0.52 is the mean correspondence between API and interface rankings, with weaker agreement when benchmark scores are tightly clustered.MMLU has ρ = 0.11 across scores spanning approximately 92–96%, whereas AA-Omniscience has ρ = 0.96 across approximately 12–63%.
  • Rankings: Access surfaces reversed some within-provider rankings, including GPT 5.4 Thinking versus GPT 5.3 Instant on BBQ and Claude Sonnet 4.6 versus Haiku 4.5 on WinoGrande.The reported reversals occur despite API leadership for the same model pairs.

5 Can API Controls Reproduce Interface Behavior?

The study tests whether exposed API settings can reproduce interface behavior. System prompts can move repeatability toward interface levels, but prompts, sampling settings, and reasoning budgets do not reliably eliminate the broader accuracy gap.

  • System prompts: System prompts reduce API test–retest agreement from 97.2% to 95.4%, close to the interface value of 94.8%.The remaining prompted-API versus interface difference is not significant.
  • System prompts: 0.1 pp is the nonsignificant reduction in mean absolute interface–API accuracy gaps after adding approximated system prompts.The mean gap changes from 6.5 pp to 6.4 pp, and the gap remains significant in 22 of 45 comparisons.
  • Sampling and reasoning: None of eight sampling sweeps significantly changes accuracy, while reasoning budget affects one sweep by 2.2 pp on GPT 5.4 HellaSwag.That shift exceeds the same cell’s 0.7 pp interface–API gap but does not account for the broader trend.
  • Sampling and reasoning: Test–retest agreement remains high across configurations, averaging 98.2% for sampling sweeps and 96.3% for reasoning settings.The results indicate consistent answers across the tested decoding and reasoning configurations.

6 Discussion

The findings establish access surface as part of benchmark context: identical items and scoring can produce different API and interface results. Consequently, API scores characterize tested API configurations rather than necessarily deployed chatbot behavior.

  • Context validity: API and interface evaluations can diverge in accuracy and consistency even with identical benchmark items and scoring procedures.The paper frames access surface as an additional context affecting benchmark measurements.
  • Scope of evaluation claims: API evaluations have limited value as direct proxies for deployed chatbot behavior because they do not reproduce active product layers.The paper identifies routing, retrieval, tool policies, personalization, and post-processing as intermediate layers between endpoint and response.
  • Scope of evaluation claims: Researchers should avoid generalizing API results to user-facing behavior without evidence that findings remain stable across access surfaces.Benchmark reports should document the access path and deployment context used to obtain each score.
  • Audit infrastructure: The access path shapes the object of measurement, but current audits cannot isolate which middle layer produced an observed behavioral shift.The relevant layers include routing, retrieval, tools, personalization, and post-processing.

7 Limitations

The audit's conclusions are qualified by limitations concerning matched model identity, subscription tier, interaction setting, provider coverage, and reproducibility constraints.

  • Matched API and interface identifiers may not correspond to the same underlying checkpoint, so some gaps may reflect model differences.
  • The audit covers one subscription tier per platform, so tier-based routing may produce different gaps for other users.
  • Standardized benchmarks may not capture how system components interact with longer, more conversational inputs.
  • Coverage of three providers and seven models limits generalization to other providers, future releases, and differently served open-source models.
  • Terms of service may create practical or legal barriers to reproducing or extending programmatic interface audits.

8 Ethical Considerations

The appendix documents the audit's statistical procedures, result tables, and interpretive framework for API–interface gaps. Its taxonomy distinguishes controlled, ablated, and unobservable pipeline layers while emphasizing that the evidence does not provide a full causal decomposition.

  • Ethical considerations: The audit collected interface outputs while minimizing provider burden and did not involve real users or sensitive or private data.
  • Ethical considerations: Throughput was capped at 150 queries per three-hour window, with a two-hour cooldown after rate-limit events.
  • Access-surface taxonomy: The taxonomy classifies each request-pipeline layer by whether it was controlled, tested through ablation, or unobservable under black-box access.
  • Interpretive scope: The taxonomy clarifies which explanations controls and ablations weaken, which remain plausible, and which require hidden deployment details, without providing a full causal decomposition.
  • Statistical procedures: The overall model estimates API–interface accuracy differences, with β1 representing the average gap across benchmarks and systems.
  • Accuracy results: +3.40 pp was the estimated average API–interface accuracy difference, while the paired bootstrap estimate was +3.19 pp.
  • Measurement and modeling: Accuracy was averaged across five runs, with cell-level models controlling for item-level repeated measures.
  • Cell-level results: Across 63 system–benchmark cells, 42 were significant at q < 0.05 and 36 at q < 0.01 after Benjamini–Hochberg correction.

B.7 Rank-Stability Results

API and interface system rankings only partially correspond, and the strength of correspondence varies substantially by benchmark. The comparison uses Spearman correlations and contextualizes access-surface differences against consecutive API-only model versions.

  • The mean API–interface ranking correlation was 0.52 across nine benchmarks, ranging from 0.07 for GSM8K to 0.96 for TruthfulQA and AA-Omniscience.
  • The ranking analysis averaged system scores across qualifying runs before computing Spearman correlations.
  • The study compares access-surface gaps with GPT 5.3 Instant versus GPT 5.4 Instant, both evaluated through the OpenAI API with reasoning disabled for GPT 5.4.
  • The model-version comparison uses the same benchmarks, item sets, and scoring pipeline for the only eligible consecutive pair in the dataset.

C.2 Results

The evaluation compares API and interface performance across model-version effects, sampling and reasoning controls, and benchmark settings. Accessible API controls produce limited variation and do not systematically reproduce the interface–API gap.

  • Model-version comparison: 4.5 pp is the mean absolute model-version accuracy difference, while the API–interface comparison shows a +5.0 pp accuracy difference.Both comparisons use the same benchmark items and scoring pipeline.
  • Sampling sweeps: Sampling sweeps produce no significant accuracy differences across eight model–benchmark cells, with a maximum range of 1.6 pp and mean range of 0.9 pp.All one-way ANOVAs have p > 0.30.
  • Reasoning sweeps: Reasoning sweeps reach significance in one of eight cells, with a maximum accuracy range of 4.2 pp, mean range of 1.4 pp, and mean test–retest agreement of 96.3%.The significant cell is GPT 5.4 on HellaSwag, with a 2.2 pp range and p = 0.001.
  • Comparison to the interface–API gap: 2.4 pp is the mean absolute interface–API gap for the swept models and benchmarks, and parameter changes do not systematically reproduce interface behavior.Accuracy can increase or decrease away from defaults, whereas the interface–API gap is directionally consistent in most capability-table cells.
  • Benchmark coverage: The evaluation uses six Open LLM Leaderboard benchmarks covering science reasoning, mathematics, commonsense reasoning, academic knowledge, misconceptions, and pronoun resolution.The supplied benchmark descriptions specify ARC, GSM8K, HellaSwag, MMLU, TruthfulQA, and WinoGrande.

E.2 Additional Benchmarks

The additional benchmarks extend evaluation to social bias, factual knowledge, and moral sycophancy, using benchmark-specific prompts, extraction, and grading procedures. The collection pipeline processed 102,048 item–run pairs across conditions.

  • AA-Omniscience: AA-Omniscience uses 200 expert-level questions across six domains, with free-form answers graded as CORRECT, INCORRECT, PARTIAL ANSWER, or NOT ATTEMPTED.CORRECT and PARTIAL ANSWER count as correct, while NOT ATTEMPTED is excluded from the denominator.
  • AITA: AITA uses 100 original–flipped post pairs to measure moral sycophancy, defined as validating the poster in both versions.The benchmark’s accuracy metric is the non-sycophancy rate.
  • Prompting and collection: All benchmarks are presented as plain text without a system prompt through both API and chat-interface conditions.API prompts are sent as one user message, while interface prompts are entered into the chat text box.
  • Pipeline scale: 102,048 item–run pairs were processed, with identical extraction and scoring code across conditions and raw response text as the only input difference.The total includes 60,049 metabench pairs and 41,999 pairs from the remaining benchmarks.

G.2 Metabench Benchmarks

The metabench suite combines multiple-choice and numeric-answer tasks with shared extraction procedures, while additional benchmarks require specialized extraction or grading. The released data contain 102,048 item–run pairs, with 101,977 receiving valid scores.

  • Metabench structure: Five of six Open LLM Leaderboard benchmarks are multiple-choice, while GSM8K requires extracting a final numeric answer.The six benchmarks are scored using a single-stage LLM extractor via the OpenAI Batch API.
  • Multiple-choice extraction: Multiple-choice answers pass through a regex cascade and then an LLM extractor, with the LLM result used when available and regex as fallback.Valid extracted letters are compared case-insensitively with gold answers; invalid or NONE outputs are excluded from accuracy.
  • Numeric extraction: GSM8K extraction asks a judge for the final numeric answer or NONE, then normalizes predicted and gold numbers before comparison.Normalization strips commas and trailing periods and converts whole-valued floats to integers.
  • Additional-benchmark extraction: BBQ uses regex extraction first and an LLM judge fallback restricted to answer letters A, B, and C.The judge returns NONE when the response does not select one option.
  • Additional-benchmark extraction: AITA extraction examines the first 150 characters of a cleaned response for verdicts including YTA, NTA, ESH, and YWTA/YWBTA.If both YTA and NTA appear, the first one is returned.
  • Scoring outcomes: 101,977 of 102,048 released item–run pairs receive a valid score, and annotator–extractor agreement is 98.7%.The agreement review found 17 disagreements among 1,280 reviewed items.

H Robustness Checks

Robustness checks find limited variation across accounts, request formats, and coarse collection-time measures, but they do not exclude all possible backend or deployment differences. The analyses therefore bound specific confounds without eliminating the broader limitation.

  • Account-level routing: Across-account accuracy differences are small and all provider-level equality tests fail to reject equal accuracy.Across-account standard deviations range from 0.26 pp for Gemini Fast to 1.83 pp for ChatGPT Instant.
  • Request-level routing: Fresh-request and main interface conditions do not differ significantly for Claude Haiku or ChatGPT Instant.For Claude Haiku, the fresh-request interface estimate remains below the main API estimate.
  • Time of collection: Collection-time terms add less than 0.01% incremental R2 for linear hour effects, below 0.11% for hour fixed effects, and below 0.04% for day-of-week effects.These models control for benchmark, surface, and question identity.
  • Scope of robustness checks: The robustness analyses are descriptive checks and should not be interpreted as excluding all possible backend or deployment differences.The supported checks cover account, fresh-request, and coarse collection-time dimensions.
Loading 2609.08861v1…