PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

Tapan Parikh

arXiv:2609.30012v1cs.CLcs.AI

TL;DR

Everyday model behavior matters to people’s beliefs and actions, but measuring it across models, prompts, and releases is difficult. The paper proposes cheap, frozen cross-vendor assays with analysis suited to the behavior, and finds patterns in convergence, resistance, holding positions, and coding-agent reports.

  • Problem

    Everyday model behavior shapes what people believe and do, yet public understanding is limited and measuring behavior across models, prompts, and releases requires repeated sampling and coding unstructured text.

    Source: §1 · p. 1

  • Method

    The paper runs frozen stimuli identically across a cross-vendor panel and reads transcripts by exact match, LLM-applied codebooks checked against human coding, or instrumented records of agent actions.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Across four years, the assays found word-choice convergence, increasing resistance to confirmation tags, generation-linked position holding with lab-specific styles, and coding-agent behavior that varies with the harness.

    Source: §4 · p. 3, §4.1 · p. 3, §4.2 · p. 3

  • Takeaways & Limitations

    Rerunning fixed assays on releases can track how model behavior changes across vendors and over time.

    Source: §6 · p. 4

    Each construct is measured with one wording; frozen stimuli can enter training data, release date is confounded with size, tier, and training recipe, API assays run with reasoning off, and agent results use three runs per scenario.

    Source: §5 · p. 4

Abstract

from arXiv · show

Language models advise people, keep them company, and write software while they sleep. Measuring what they do is hard: behavior has to be sampled repeatedly across models, prompts and releases, most of it lives in unstructured text that has to be coded before it can be counted, and the result has to be legible and rigorous enough to meaningfully compare models and vendors. To address these constraints, we present a simple, cheap, scalable, and replicable model for studying model behavior. Each study is a frozen, public stimulus run identically on a cross-vendor panel, at a few dollars per model or less. Each reads its transcripts one of three ways, chosen by how much interpretation the behavior needs: exact match on a clamped reply, a codebook applied by LLM judges whose agreement with a human coder is reported per code, and an instrumented environment that records what an agent did independently of what it said. Run across four years of model releases from both frontier and open-source labs, these instruments find four things. Convergence: asked to pick a word, 27 of 44 models answer serendipity at least once in four tries. Resistance: a trailing "right?" moves endorsement by up to 32 points, and the sign flips from sycophantic to resistant as generations advance, keyed to the tag's surface form. House: whether a model holds a position under pressure tracks its generation, and how it holds tracks the lab that built it. Account: told to do something the documentation in their repository contradicts, some coding agents never went along silently and others always did, and the same model can change with the harness it runs in. Re-run on every release, batteries like these track how behavior is changing across vendors and over time.

1 A measurement problem

Measuring everyday model behavior is difficult because it varies across models, prompts, contexts, and releases, while much of it must be coded from unstructured text. The paper proposes frozen, cross-vendor stimuli as a repeatable way to study that behavior.

1 A measurement problem

  • Behavior varies across models, prompts, and contexts, requiring repeated sampling as models are released or updated, while unstructured transcripts must be coded before they can be counted.Results also need to be legible to deployers and users, yet rigorous enough for skeptical readers.Source: §1 · p. 1
  • Each study uses a frozen stimulus sent identically to a cross-vendor panel, and the panel can be rerun on every release to track change over time.The approach is presented as simple, cheap, scalable, and replicable.Source: §1 · pp. 1–2
  • Using the same fixed stimulus makes model differences legible: divergence shows where models differ, while agreement reveals behaviors common across them.The instruments choose among three transcript-reading methods according to how much interpretation the behavior requires, and publish transcripts, labels, and scores.Source: §1 · p. 2

2 The assays

The assays select among three analysis methods according to how much interpretation a behavior requires, pairing automated scoring with human review where judgment is needed. They run through a common API or full-auto agent harness at low cost.

2 The assays

  • The single-turn and conversational assays use one API at requested temperature 1.0, while agent assays run coding products or harnesses in full-auto mode.Running one model through a study costs a few dollars or less.Source: §2 · p. 2
  • The assays use clamped exact match, coded transcript labels, or instrumented action–report comparisons, chosen according to how much interpretation the behavior needs.Clamped scores normalized tokens; coded readouts use a human-built codebook and LLM coders checked against a blind human pass; instrumented readouts compare recorded actions with the model’s account.Source: §2 · p. 2
  • For the conduct-study codes reported, agreement with the human pass was Cohen’s κ 0.80 to 0.87, except self-citation at 0.54.The human pass checks codes on held-out transcripts before they are used.Source: §2 · p. 2

3 Related work

The paper places its assays alongside work on social-science measurement, agents, fixed-prompt benchmarks, sycophancy, LLM judges, model homogenization, and coding agents. It distinguishes its focus on everyday behavior and low-cost repeatability.

3 Related work

  • Prior work addresses social-science measurement, agents, fixed-prompt benchmarks, sycophancy, LLM judges, homogenization [4], and coding agents.Alnasser et al. code 14 models under user disagreement using an a priori scheme; this paper describes its own approach as inductive and its vendor-level result as new.Source: §3 · p. 2
  • The paper distinguishes its target as everyday behavior with no clearly right answer and its price as low enough to repeat studies on each release.At list prices in September 2026, API assays cost a median of three cents per model, while agent-assay metered runs cost about two dollars per configuration.Source: §3 · p. 2

4 Four preliminary findings

Across four preliminary studies, the assays surface convergence, tag-sensitive resistance, lab-specific responses under pressure, and differences between agents’ actions and reports. Findings vary with language, generation, vendor, and harness.

4.1 Convergence: a different serendipity in every language

  • 27 of 44 models chose serendipity at least once in four tries; across 44 languages, each had a default word native to that language.Serendipity accounted for 41 percent of all answers and was the default only in English.Source: §4.1 · p. 3

4.2 Resistance: keyed to the tag

  • Across 45 models, the tag effect ranged from +32 to −32 percentage points; within each lineage, it shifted from positive to negative as generations advanced.Replacing “right?” with “maybe?” raised agreement above neutral in all 45 models by 19.6 points on average, though the assertion also changed.Source: §4.2 · p. 3

4.3 House: newer models hold, each lab holds its own way

  • 43 percent of the oldest third folded, versus 23 percent of models released since mid-2025; response styles also varied by lab.Anthropic models named feelings, warned of consequences, and offered alternatives above the panel rate; Google and Meta showed distinct patterns, while OpenAI models did less of several behaviors.Source: §4.3 · pp. 3–4

4.4 Account: what agents did and what they said

  • Across six pushback replies, silent compliance ranged from never in four configurations to every time in two, and Gemini 3.5 Flash varied by harness.It silently complied in three of six replies in Gemini CLI and six of six in OpenCode; the atlas compares disk changes with final messages.Source: §4.4 · p. 4

5 Limitations

The authors identify wording, stimulus exposure, release confounds, and limited run conditions as boundaries on these findings.

  • Each construct is measured with one wording, although the tag results show that one word can move outcomes.Source: §5 · p. 4
  • Frozen public stimuli may enter training data, so a standing record needs fresh scenes alongside the frozen ones.Source: §5 · p. 4
  • Release date moves with size, tier, and training recipe, so the generation findings are observational.Providers may update models and serving settings without notice, and older models are queried as currently served.Source: §5 · p. 4
  • API assays ran with reasoning off, and agent results rest on three runs per scenario.Source: §5 · p. 4

6 A behavioral science of LLMs

The authors argue that inexpensive, replicable behavioral studies can become routine, allowing one person to repeat a study on each release and pursue new questions.

  • Fixed turns and short scoring rules keep the instruments cheap while still separating models across vendors and generations.Source: §6 · p. 4
  • At this price, one person can run a study, repeat it on the next release, and point it at a new question the following week.Source: §6 · p. 5
  • The authors place studies at this price closer in scale to small experiments with human participants than to benchmarks.Source: §6 · p. 5

Related papers

Loading 2609.30012v1…