PaperlayerView exact revision on arXiv

Source-linked AI summary

CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments

Kianté Fernandez, Caitlin Chen, Jialu Zhou, Bartek Sadowski, Anthony C. Miceli, Ian Krajbich

arXiv:2610.36484v1cs.AIcs.CYcs.HCq-bio.NC

TL;DR

Text-prompt evaluations bypass the visual interfaces and timing constraints of online behavioral experiments, while browser-capable agents raise concerns for online data collection. CogArena tests agents in browser-based behavioral experiments and scores completion, accuracy, and human behavioral signatures. Across four multimodal models, many sessions did not finish, and completed sessions reproduced some signatures but not others.

  • Problem

    Behavioral-paradigm evaluations commonly use text prompts that omit the visual interfaces and time pressure of online experiments, while agents’ performance as participants in interactive experiments remains untested.

    Source: §1 · pp. 1–2

  • Method

    CogArena evaluates agents in ten browser-based behavioral experiments, scoring completion, accuracy, and alignment with task-specific human signatures.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    Many sessions did not finish, and completed sessions reproduced some signatures but missed others, with the pattern varying across tasks and sessions.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Completion and accuracy alone do not establish whether agents reproduce human behavioral signatures, so CogArena reports these dimensions separately.

    Source: §6 · pp. 9–10

    No human participants completed CogArena’s task versions, so the paper makes no claim comparing agents with people completing the platform experiments.

    Source: §7 · p. 10

Abstract

from arXiv · show

Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.

1 Introduction

Text-prompt evaluations bypass the visual interfaces and time pressure of online behavioral experiments, while browser-capable agents raise concerns for online data collection. CogArena evaluates agents in browser-based experiments and finds that finishing a task does not ensure reproduction of its human behavioral signatures.

1 Introduction

  • Text-only paradigms may test different abilities from visually presented experiments with response deadlines, such as perception, selective attention, and response inhibition.The paper’s Stroop example contrasts a text instruction with a colored word displayed on screen under a two-second deadline.Source: §1 · pp. 1–2
  • Prior work found that agents can complete web tasks and may produce human-like answers, while leaving their performance as participants in interactive behavioral experiments untested.The cited concern includes agents passing attention checks and submitting answers that look human, while current detection methods do not catch them.Source: §1 · p. 2
  • CogArena is intended to characterize which response modalities and behavioral signatures agents can reproduce for researchers studying cognition and online data collection.The paper frames these as two purposes: comparing agent behavior with human signatures and tracking threats to online experiment validity.Source: §1 · p. 2
  • CogArena’s ten-task benchmark scores task completion, accuracy against human baselines, and alignment with task-specific behavioral signatures using fixed statistical tests.Its preliminary evaluation covers four multimodal models, each run ten times per task.Source: §1 · p. 2
  • Many sessions did not finish, and completed sessions reproduced some human signatures but missed others, with patterns varying across tasks and models.The authors report that completion alone does not guarantee reproduction of signatures established in human research.Source: §1 · p. 2

2 Related Work

CogArena connects browser-based agent evaluation with cognitive-science assessments of machine intelligence. It extends text-based paradigm evaluations by testing agents in interactive online experiments and scoring their data against literature-derived behavioral signatures.

Interactive agent benchmarks

  • CogArena combines browser-based interaction with behavioral experiments scored against literature-derived signatures, distinguishing its benchmark from related work.The comparison is described as combining instrumented task-state scoring and separation of benchmark from harness with canonical experimental paradigms.Source: Table 1 · p. 3, §2 · p. 3
  • Browser-based agent benchmarks have expanded from simulated environments to live websites and graphical tasks, with CogArena applying that interface to experimental paradigms.The paper relates this approach to TurkingBench’s use of crowdworker webpages as a browser-based testbed.Source: §2 · p. 3

The application of cognitive science to artificial systems

  • Earlier cognitive-science evaluations applied paradigms through text prompting or text APIs, while CogArena uses interactive experiments with human-like stimulus formats, timing, and response requirements.The related work also describes Psych-101 and Psych-201 as aggregating trial-by-trial human responses across more than 160 experiments.Source: §2 · p. 3

3 The CogArena Benchmark

CogArena v1 adapts ten behavioral experiments for browser interaction and scores completion, accuracy, and human behavioral signatures. Its scoring distinguishes untestable signatures from absent effects and combines levels using a stated weighting scheme.

3 The CogArena Benchmark

  • CogArena v1 contains ten browser-based experiments spanning learning, decision-making, memory, perception, judgment, foraging, compositionality, and moral decision-making.Tasks have 16 to 200 trials and use keypresses, button clicks, or typed responses.Source: §3 · p. 3

3.1 Task Design Principles

  • Tasks require sustained interaction with 16 to 200 sequential stimuli under configurable per-trial deadlines, reproducing demands placed on online participants.Randomized stimuli use reproducible session-specific sequences that differ across sessions.Source: §3.1 · pp. 3–4
  • Removing screenshots lowered composite scores by 31–53% across four tasks, significantly on three.In the check, agents received underlying page text and interactive-element lists instead of rendered screenshots.Source: §3.1 · p. 4

3.2 Three-Level Scoring Pipeline

  • The benchmark scores task completion, performance accuracy, and whether trial data show task-specific human cognitive signatures.Accuracy is compared with participants from source studies; signature tests assess expected directions and statistical significance.Source: §3.2 · pp. 4–5
  • Level 3 grades each signature as 1.0 when significant in the expected direction, 0.5 when directionally expected but not significant, and 0.0 otherwise.The overall Level 3 score is a weighted mean over testable signatures, with weights reflecting each signature’s centrality.Source: §3.2 · p. 5
  • Untestable signatures are excluded from Level 3 because early termination, insufficient response variation, or unseen information can prevent a valid test.A session with no testable signatures receives a Level 3 score of 0.Source: §3.2 · p. 5
  • The headline composite weights completion, accuracy, and behavioral alignment at 0.15, 0.35, and 0.50, respectively, and scales the weighted sum to 0–100.When a task lacks Level 2, its weight is redistributed proportionally between Level 1 and Level 3.Source: §3.2 · pp. 4, 6
  • The Random Agent supplies a chance-floor reference by pressing a random valid key on each trial and receiving the same scoring.The figure depicts trial data being scored at three levels and notes that partial sessions still produce usable data.Source: §3.2 · p. 6, Figure 1 · p. 6

4 Methods

CogArena hosts browser-based experiments and scoring while allowing submitters to choose the agent scaffold. The study tested four multimodal models repeatedly with a shared scaffold, alongside a separate single-session evaluation of a state-of-the-art model.

4.1 Platform

  • The platform hosts ten jsPsych experiments, accepts sessions and trial data, and scores submissions while leaving browser control to the agent’s scaffold.Sessions use the same experiments, and the platform records scaffold and model-visibility metadata for comparison.Source: §4.1 · p. 6

4.2 Reference Agents and Public Submissions

  • The platform provides a Random Agent for end-to-end checks and chance-floor estimates, and a Browser-Use Agent scaffold that can drive models.Researchers may also submit sessions from their own agents.Source: §4.2 · p. 7

4.3 Study Design

  • Four economy-tier multimodal models ran through Browser-Use for ten sessions on each of ten tasks, producing 400 sessions with new stimulus seeds.A separate state-of-the-art model was evaluated once per task; the authors report it separately because one session is too few for comparison with ten.Source: §4.3 · p. 7
  • All 400 sessions used screenshots, and each session was limited to 1800 seconds and 500 steps per task.Response deadlines were switched off so a slow model response alone would not count as a missed trial.Source: §4.3 · p. 7

5 Results

Across the ten-task set, finishing experiments and reproducing human behavioral signatures diverged: many sessions failed completion checks, and signature reproduction varied sharply by paradigm. Aggregate rankings were stable, but individual model-task results varied across repeats.

5 Results

  • 58.6 was gemini-3-flash-preview’s mean composite across all 100 sessions, and it ranked first in all 2,000 bootstrap resamples.Its mean composite over the 96 sessions with measurable Level 3 scores was 58.7.Source: §5 · p. 7, Figure 2 · p. 8

5.1 The Completion-vs-Signature Gap

  • 167 of 400 sessions passed all five Level 1 checks; 191 of the 233 failures ended when agents declared unfinished tasks complete.Only 30 failures ran out of time, one ran out of steps, and 11 followed ten consecutive failed actions.Source: §5.1 · p. 8
  • Mean Level 2 was 0.345 against a 0.159 chance floor, and mean Level 3 was 0.490 against 0.184, but signature reproduction ranged from universal to none.Phishing discrimination appeared in 90% of testable sessions; saving more people and cooperating after cooperation each appeared in 100%, while image recognition, the travel-cost effect, and faster responses at higher motion coherence appeared in 0 of 25, 0 of 32, and 0 of 21 sessions, respectively.Source: §5.1 · p. 8
  • 28 of 40 model-task cells had both zero and above-zero Level 3 scores across repeats, and 60 of 139 model-signature combinations passed in some sessions but failed in others.The median within-cell standard deviation of Level 3 was 0.20; the authors therefore limit claims to the panel as a whole.Source: §5.1 · p. 8
  • 0.960 vs 0.750 on Level 1 and 0.656 vs 0.490 on Level 3: once-run gpt-5.6-sol exceeded panel means but beat the best panel model on only four tasks.With one session per task, the authors cannot estimate how much of this advantage would persist across repeats.Source: §5.1 · p. 9

5.2 Robustness and Open Leaderboard

  • The all-session composite ranking stayed unchanged under three alternative weightings, while alternative Level 3 grades shifted agent–floor gaps by at most 0.021.New submissions appear on a shared leaderboard with per-level results and the chance-floor reference.Source: §5.2 · p. 9

6 Discussion

The discussion distinguishes human-like behavioral signatures that reflect normative task performance from preferences or biases, which need not indicate better performance. Agents reproduced these categories at different rates, and response-time comparisons with people require particular care.

6 Discussion

  • CogArena separately measures experiment completion and human-signature reproduction, with open tasks, scoring procedures, and a leaderboard for updating its picture of current agents.The results also identify signatures agents reproduce and those they do not, and suggest which measures may still distinguish people from current agents in online studies.Source: §6 · pp. 9–10
  • A capacity is a pattern predicted by a normative task model, whereas a bias departs from that benchmark or is a preference no benchmark prescribes.In the repeated prisoner’s dilemma, cooperating after the opponent cooperates is a capacity, while tit-for-tat reciprocity is a bias.Source: §6 · p. 9
  • Agents reproduced bias signatures in 66% of testable cases, versus 31% for capacity signatures (36% when response-time signatures are excluded).The authors caution that overall Level 3 scores are raised by patterns that good performance does not require; claims about agent capabilities should rest on capacity signatures.Source: §6 · p. 9
  • Agents took up to ten times longer than people to respond, mostly because each response waits for screenshot processing and action generation.Cross-human response times therefore reflect model-serving speed more than decision-making; agents passed relative response-time signatures in 30 of 217 testable cases, though slow responding may still help flag AI-generated data.Source: §6 · p. 10

7 Limitations

CogArena’s limitations center on missing on-platform human baselines, a narrow model sample, and a task set that does not cover every phenomenon. The accompanying specifications define the benchmark’s scope and how its behavioral signatures are tested and scored.

7 Limitations

  • No humans have completed CogArena’s task versions, so agent–human differences may reflect implementation differences, especially for Level 2 magnitude comparisons.The authors make no claim about comparisons with people completing CogArena itself; they identify collecting on-platform human data as the first priority for the next release.Source: §7 · p. 10
  • The four-model evaluation is a narrow sample, and shared Browser-Use scaffolding means model differences are differences between model-and-scaffold combinations.The state-of-the-art model was run only once; the screenshot-removal check tests Browser-Use’s use of screenshots, not models’ capabilities without them.Source: §7 · p. 10
  • The ten-task set is an initial release, not an exhaustive test of known phenomena; grid_bandit, for example, lacks a horizon manipulation for testing horizon-dependent exploration.The authors note that signatures and manipulations can be added to each task.Source: §7 · p. 10

A Task Implementation Details

  • The v1 specifications cover ten curated tasks, with each task implemented as a standalone browser app and session-seeded randomization producing unique, reproducible sequences.All reported numbers use task and scoring-code version v1.3.0; configurable trial and timing parameters and separate Level 2 and Level 3 scoring specifications are documented per task.Source: A Task Implementation Details · p. 14

B Behavioral Signatures Evaluated in v1

  • The 39 Level 3 signatures include 33 capacity and six bias signatures; agents reproduced them in 31% and 66% of measurable test cases, respectively.The set omits horizon-dependent exploration in grid_bandit and scores only primacy, not recency, in serial_recall; the authors caution that overall Level 3 means overstate ability because bias signatures need not be shared.Source: B Behavioral Signatures Evaluated in v1 · pp. 14–15

C Statistical Test Descriptions

  • Level 3 tests use proportions, paired or conditional contrasts, and correlations; signatures receive 1.0 for significant expected-direction results, 0.5 for nonsignificant expected-direction results, or 0.0 otherwise.Tests account for within-session dependence where specified; a constant predictor can make a signature untestable, while a constant outcome scores 0.Source: C Statistical Test Descriptions · p. 15

D Per-Task Signature Descriptions

The per-task specifications define the human-pattern signature each task tests, with selected response-time signatures measuring relative speed within a session. Table 3 organizes these signatures by task, test type, and whether they are capacity or bias patterns.

Per-task signature overview

  • Table 3 lists behavioral signatures by task, their statistical tests, and importance weights, while selected tasks also test within-session response-time patterns.The response-time signatures test relative speed, which the paper says survives uniform model-induced slowing.Source: Table 3 · p. 16, D Per-Task Signature Descriptions · p. 16

Random-dot-motion and grid_bandit

  • Random-dot-motion signatures test whether accuracy rises with motion coherence and whether response times fall, while grid_bandit tests risky-tile avoidance, value-directed clicks, reward, and within-block learning.Random-dot-motion also includes above-chance accuracy tests at the lowest nonzero coherence and overall; the grid_bandit set includes above-chance reward.Source: D Per-Task Signature Descriptions · pp. 16–17

Marbles_risk and repeated_games

  • Marbles_risk tests expected-value sensitivity, optimal choice, and payoff magnitude, while repeated_games tests tit-for-tat reciprocity, coordination, and conditional cooperation.The repeated-games tests cover prisoner’s-dilemma and battle-of-the-sexes behavior.Source: D Per-Task Signature Descriptions · p. 17

Serial_recall and visual_recognition

  • Serial_recall tests recall accuracy, similarity effects, and primacy, whereas visual_recognition tests discrimination, overall accuracy, and avoidance of an old-response bias.Serial_recall does not score recency; its position bands use the configured 16-pair list length so truncated sessions cannot relabel positions.Source: D Per-Task Signature Descriptions · p. 17

Effort_foraging and tiny_alchemy

  • Effort_foraging tests travel-cost residence, MVT-optimal choices, and reward; tiny_alchemy tests discovery and non-redundant element-pair attempts.Foraging residence is measured by stay rates because agents leave patches too rarely for residence times to be testable; an empowerment-preference signature was dropped after firing mechanically under random play.Source: D Per-Task Signature Descriptions · p. 17

Moral_machine and phishing_detection

  • Moral_machine tests four preferences, while phishing_detection tests phishing discrimination, post-training learning, and accuracy across all trials.Moral_machine’s intervention-aversion signature was removed after a power audit found it untestable at n = 4.Source: D Per-Task Signature Descriptions · p. 18

E Level 2 Accuracy Metrics in v1

CogArena v1 scores 31 Level 2 accuracy metrics across eight tasks, with exclusions where a higher value is not uniformly better. Most baselines come from released participant data, but they remain source-study anchors rather than on-platform human comparisons.

E Level 2 Accuracy Metrics in v1

  • CogArena v1 scores 31 Level 2 accuracy metrics across eight tasks, excluding absolute response time and withholding Level 2 scores where preferences or risk attitudes lack a uniformly better direction.Moral_machine and marbles_risk have no Level 2 score; effort_foraging’s stay and residence-time metrics were removed because they rewarded over-harvesting, and five metrics reverse the scoring sign because lower is better.Source: E Level 2 Accuracy Metrics in v1 · pp. 18–19

F Human-Baseline Provenance

  • Twenty-four of 31 Level 2 baselines come directly from released participant data, but source-study anchors may not match CogArena implementations, making visual_recognition agent scores likely optimistic.Level 3 effect directions transfer across implementations better than Level 2 magnitudes; on-platform human data are the planned remedy.Source: F Human-Baseline Provenance · p. 19

G Run Settings and What the Model Saw

The study used a consistent Browser-Use scaffold with screenshots enabled, while the platform itself does not prescribe how agents drive the browser. Removing screenshots reduced composite scores on all four tested tasks, and the motion task’s frame-based stimulus poses a specific visibility limit.

Run settings

  • Each model ran through Browser-Use with screenshots enabled and common limits of 5 actions per step, 10 consecutive failed actions, 500 steps, and 1800 seconds per task.The platform does not constrain browser-driving methods, so these settings describe the study rather than the benchmark.Source: G Run Settings and What the Model Saw · p. 19

Baseline provenance

  • Table 4 distinguishes human-baseline provenance: six tasks use released participant data throughout, while visual_recognition uses paper-derived values.The table also marks tasks without Level 2 metrics.Source: Table 4 · p. 20

What the model saw

  • At each step, Browser-Use supplied a screenshot and a text list of clickable and typeable elements; the random-agent floor read page text only.Agent response deadlines were switched off, while jsPsych still recorded actual response times.Source: G Run Settings and What the Model Saw · p. 20

Screenshot-removal check

  • Removing screenshots reduced composite scores by 31–53% across four tasks, significantly on three; the result supports screenshot dependence but not necessarily stimulus reading.The text tasks began two to three times higher than the image tasks, so proportional changes—not raw drops—are the appropriate comparison.Source: G Run Settings and What the Model Saw · pp. 20–21, Figure 3 · p. 21

Temporal stimuli

  • Random-dot-motion coherence exists across a 2000-ms sequence, so one screenshot per step cannot convey the manipulation; the study did not use video or frequent screenshots.Recovering this stimulus requires video capture or several screenshots per second.Source: G Run Settings and What the Model Saw · p. 21

H Signature Trial Counts and Statistical Power

Level 3 signatures draw on widely varying trial budgets, and short budgets constrain what the tests can detect. Dependence corrections alter some grades but leave the overall model ranking unchanged; many signature observations are untestable, limiting per-cell interpretation.

Trial counts

  • Per-signature trial budgets range widely: random_dot_motion’s coherence psychometric uses 100 trials, while several moral_machine and phishing_detection signatures use a few dozen or fewer.The task’s trial budget is divided across dimensions or blocks that a signature contrasts.Source: H Signature Trial Counts and… · p. 21, Table 5 · p. 22

Power

  • A legality preference of 0.68 against chance 0.5 needs about 50 trials for 80% power, but v1 allocates six; a null result is therefore weak evidence that the pattern is absent.Intervention aversion in moral_machine used four trials, making it untestable at p < 0.05 and leading to its removal from the specification.Source: H Signature Trial Counts and… · pp. 21–22

Dependence

  • Moving-block bootstrap regrading changed 5.5% of grades while leaving the model ranking unchanged; ten-trial blocks changed 12.2% and also preserved the all-session composite ranking.Neither correction separates serial dependence from a genuine within-session trend, so a learning curve is penalized like a sticky response.Source: H Signature Trial Counts and… · p. 22

Untestable observations

  • Across 1,560 signature observations, 497 (32%) were untestable, mostly because sessions ended early or agents responded identically throughout.Mean Level 3 was 0.365 over all sessions, versus 0.490 over measurable sessions.Source: H Signature Trial Counts and… · p. 22

Consequence for reading Level 3

  • Per-cell Level 3 values remain noisy under discrete {0, 0.5, 1} grades; increasing short-signature trial budgets requires task revision rather than a scoring change.Each cell contains ten sessions, and the grade can shift a full step at the significance boundary.Source: H Signature Trial Counts and… · p. 23

I Chance-Floor Analysis

The random agent provides a directly comparable chance floor under the same scoring code. It completes every session, but still earns a nonzero Level 3 because some directional tests reach significance on noise.

Chance-floor scores

  • Across 100 sessions, the random agent’s mean Level 3 was 0.184 and its mean composite was 31.31.It ran ten times per task, and all sessions were scored.Source: I Chance-Floor Analysis · p. 23

Scoring comparison

  • The random-agent floor uses the same scoring code as the agents, making their scores directly comparable.All Level 1 values were 1.0 because the driver always reaches the end of the experiment.Source: I Chance-Floor Analysis · p. 23, Table 6 · p. 23

Interpreting the floor

  • The floor’s Level 2 was 0.159 over eight tasks, while its Level 3 of 0.184 reflects directional tests reaching significance on noise across 100 sessions.No signature was passed by the random agent in at least half its sessions, supporting the benchmark’s discriminative property.Source: I Chance-Floor Analysis · p. 23

J Per-Task Results

Per-task results show large differences in difficulty and model ordering, while overlapping dispersion limits task-level model comparisons. The single-session state-of-the-art check and sensitivity analyses provide qualified comparisons: the panel leader remains stable, but individual per-task claims remain uncertain.

J Per-Task Results

  • Every model clears the random floor by 30–50 points on phishing_detection, while on visual_recognition the best is 5.4 points above the floor and the weakest measurable model is 2.2 points above.On effort_foraging, every model scores below the random floor because agents almost never leave a patch.Source: J Per-Task Results · p. 23, Table 7 · p. 24
  • The gemini-3-flash-preview model leads overall but loses to every other panel member on tiny_alchemy; its marbles_risk margin over the runner-up is 23.5 points.Per-signature pass rates vary across repeats, making the instability visible at the signature level.Source: J Per-Task Results · pp. 23–25
  • Within-cell standard deviations reach 24.8 on marbles_risk, and overlapping intervals mean per-task, per-model comparisons are not identified at this sample size.The table reports means with standard deviations and measurable-session counts; per-task means alone do not rank models.Source: J Per-Task Results · pp. 23–24, Table 7 · p. 24
  • Two pilot models were excluded from panel aggregates because each supplied only one session per task; one had also been delisted and could not be rerun.The paper notes that reproducibility depends partly on continued model availability.Source: J Per-Task Results · p. 25
  • Two attempted models could not be scored because Browser-Use rejected their action formats at the first response point; these failures reflect scaffold incompatibility, not measured ability.Their sessions were excluded.Source: J Per-Task Results · p. 25

K State-of-the-Art Model

  • In one session per task, gpt-5.6-sol beat the panel’s per-task mean Level 3 on eight of ten tasks but led the panel’s best member on only four.It achieved L1 = 1.000 on nine of ten tasks, compared with the panel mean of 0.750 over all sessions; repeats are needed for task-specific claims.Source: K State-of-the-Art Model · p. 25, Table 9 · p. 27

L Composite-Weight and Grading-Rule Sensitivity

Related papers

Loading 2610.36484v1…