PaperlayerView exact revision on arXiv

Source-linked AI summary

System1-Arena: Which Decisions Can Be Left to System 1?

Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou

arXiv:2610.21244v1cs.CLcs.AIcs.LG

TL;DR

Many high-volume or latency-sensitive applications need discrete decisions with probability estimates, yet evidence on when fast System-1 models suffice is limited. System1-Arena compares these models with generative LLMs on shared requests and closed-loop tasks; the strongest System-1 models match fast closed LLMs on non-agentic decisions, while reasoning LLMs achieve the highest accuracy and large models succeed most often in closed loops.

  • Problem

    Existing evidence on which discrete decisions System-1 models can handle is limited, despite applications that need high-volume decisions with probability estimates and settings where inference latency matters.

    Source: §1 · p. 1

  • Method

    System1-Arena evaluates System-1 models and generative LLMs on shared decision requests, scoring their probabilities across a broad question set and closed-loop environments.

    Source: §1 · p. 2

  • Results

    The strongest System-1 models match fast closed LLMs on non-agentic decisions, while reasoning LLMs achieve the highest accuracy at about 70x the cost of a fast LLM.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results support deciding case by case where fast System-1 decisions suffice and where deliberation may be worth its price; large models succeed most often in closed loops.

    Source: §6 · pp. 8–9

    A high benchmark score does not establish fitness for legal, medical, financial, or safety decisions, where human review remains necessary.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

1 Introduction

System1-Arena asks which discrete decisions can be handled by fast probability-producing models rather than generative LLMs, addressing limited benchmark coverage of these choices. Across its broad evaluation, the strongest System-1 models match fast closed LLMs on non-agentic tasks, while reasoning LLMs lead accuracy and small System-1 models struggle in long-horizon loops.

1 Introduction

  • High-volume decisions often need labels and probabilities, while generative reasoning can add computation and latency in settings where the world changes during inference.Source: §1 · p. 1
  • System1-Arena brings together 92,125 typed questions across 105 tasks from 268 public sources and 138 closed-loop environments, with real-time environments evolving during model inference.Source: §1 · p. 2
  • The benchmark gives models shared decision requests, scores their probability distributions, evaluates closed-loop decisions, and obtains probabilities from open LLM answer tokens.Source: §1 · p. 2
  • 73.0% accuracy for Decision-2.0-Vega-27B and 72.6% for Jev compare with 71.2% for GPT-6 Luna; the strongest System-1 models also offer better-calibrated probabilities at lower latency.Figure 1 summarizes the comparison.Source: Figure 1 · p. 1, §1 · p. 2
  • System-1 models remain less accurate on knowledge and forecasting tasks, even as they perform well on other decision settings.Source: §1 · p. 2
  • 81.1%: Claude Opus 5.5 achieves the highest accuracy, at about 70× the cost of a fast LLM; Qwen3.8-27B reaches 75.6% with reasoning but is less calibrated than Jev.Qwen3.8-27B uses a few hundred reasoning tokens per question.Source: §1 · p. 2
  • On long-horizon agentic tasks, Gemma-4-31B has the highest success rate at 32.9%, while four small System-1 models reach at most 3.1%.Decision-2.0-Vega-27B, Jev, and Qwen3.5-9B each reach 67.8% accuracy on single-step tasks.Source: §1 · p. 2

2 Related Work

Prior work studies short versus extended reasoning within generative models, while System1-Arena compares generative LLMs with models that return decision probabilities directly. The paper situates this comparison among broad benchmarks, calibration and cost methods, and interactive evaluations that account for latency.

System 1 and System 2 in language models

  • System1-Arena distinguishes its focus from studies of System 1 and System 2 behavior within generative models: it compares those LLMs with models that return decision probabilities without generating text.Source: §2 · pp. 2–3

Holistic benchmarks, calibration, and cost

  • Broad benchmarks such as MMLU, BIG-bench, and HELM provide common evaluation protocols, while System1-Arena asks which decisions can be handled by System-1 models to inform routing to generative LLMs.The related work also describes calibration, efficiency, and cost-aware routing approaches.Source: §2 · p. 3

Interactive and latency-sensitive evaluation

  • Unlike many interactive benchmarks that pause the environment during model decisions, System1-Arena incorporates measured model latency into environment dynamics.Its real-time mode advances the environment while the model decides.Source: §2 · p. 3

3 System1-Arena

System1-Arena represents static examples and environment steps as typed decision requests, then evaluates both standalone decisions and action sequences in shared tasks. Its metrics assess decision accuracy and probability quality, with task-equal weighting for overall scores.

3.1 Decision requests

  • A decision request pairs a text-field state with shared questions, each answered by a probability distribution over options.Questions are boolean, multiple-choice, or ordered-score items, with instructions and option descriptions.Source: §3.1 · p. 3

3.2 Non-agentic decisions

  • The non-agentic component contains 78,362 questions across 13 domains and 87 tasks, using source labels and preserving or adapting source instructions to the request format.Source: §3.2 · p. 4

3.3 Agentic tasks and environments

  • The agentic component includes 138 environments from 18 families; a shared harness turns each state and interaction history into typed action questions and executes the highest-probability options.Individual decisions are also evaluated on fixed states and recorded trajectories; real-time mode advances the world by measured model latency, and flagged easy environments receive reduced assistance or decision budgets.Source: §3.3 · p. 4

3.4 Metrics

  • Accuracy scores the highest-probability option against the reference label, while the Brier score measures probability quality from 0 to 2, with lower scores better.Overall scores average across tasks, giving each task equal weight.Source: §3.4 · pp. 4–5

4 Experimental Setup

The evaluation compares three model families using probabilities over the same discrete options, with different ways of producing those probabilities. It covers non-agentic and agentic tasks and reports accuracy, calibration, latency, cost, and output tokens.

4 Experimental Setup

  • System-1 models directly return option-probability distributions for typed questions.The evaluated APIs include Jev, Microsoft-Decision-1, and the OpenAI Decisions API; other System-1 models use official code.Source: §4 · p. 5
  • Closed LLMs provide nonnegative option values in JSON, which are normalized into probabilities; missing options receive zero.The evaluation does not use token probabilities from closed LLMs.Source: §4 · p. 5
  • Open LLMs produce answer-label probabilities in one forward pass, which are normalized into option probabilities.For models that require reasoning, label probabilities are read at the start of the final answer; DiffusionGemma uses a read-only denoising step.Source: §4 · p. 5
  • Thirty models are evaluated: seven closed LLMs, 13 open LLMs, and 10 System-1 models.Reasoning is disabled where possible; five models require reasoning and use their lowest available effort.Source: §4 · p. 5
  • The benchmark spans four groups, with 78,362 questions across 87 non-agentic tasks and 138 agentic environments evaluated on 3,481 steps.The recorded agent steps contain 6,943 questions across seven tasks.Source: §4 · p. 5
  • The protocol reports accuracy, Brier score, median request latency, cost per 1,000 requests, and mean output tokens per request.Hosted-model costs use US dollars, while local-model costs use H100-equivalent GPU-seconds; the units are not converted.Source: §4 · p. 5, Table 1 · p. 6

5 Results

System-1 models are competitive with low-reasoning-effort LLMs on non-agentic accuracy, but reasoning LLMs lead overall and forecasting remains difficult. In agentic tasks, similar single-step accuracy can correspond to very different closed-loop success rates.

Non-agentic results

  • Claude Opus 5.5 leads overall accuracy at 81.1%, while Decision-2.0-Vega-27B (73.0%) and Jev (72.6%) outperform GPT-6 Luna (71.2%).Both System-1 models remain within 1.4 percentage points of Gemma-4-31B.Source: §5 · pp. 6–7
  • Decision-2.0-Vega-27B and Jev achieve mean Brier scores of 0.343 and 0.349 across 87 tasks, while Gemma-4-31B scores 0.474.Gemma-4-31B assigns at least 0.999 probability to its selected option on 87% of questions.Source: §5 · p. 7
  • At 2,048 tokens, Qwen3.8-27B rises from 70.2% to 75.6% accuracy while its Brier score worsens from 0.391 to 0.430.Larger reasoning budgets improve accuracy for all four tested models, but the gains do not necessarily improve calibration.Source: §5 · p. 7, Figure 4 · p. 7
  • The best System-1 forecasting result is 48.3% accuracy for the OpenAI Decisions API, versus 61.6% for Claude Opus 5.5.The paper says using System-1 models as forecasting components requires careful design and validation.Source: §5 · pp. 7–8

Agentic results

  • At the same 67.8% step accuracy, Decision-2.0-Vega-27B, Jev, and Qwen3.5-9B reach 26.5%, 17.4%, and 5.7% closed-loop success, respectively.Gemma-4-31B achieves the highest closed-loop success rate at 32.9%; the four open LLMs under 10B parameters succeed in fewer than 6% of episodes.Source: Table 2 · p. 7, §5 · p. 8

Paired comparisons, latency, and cost

  • System-1 models improve non-agentic accuracy in three of five paired comparisons but reduce agentic step accuracy in all five.Closed LLMs take 1.3–6.7 seconds, versus 21–105 ms for open LLMs without reasoning; Jev achieves 72.6% accuracy at $0.052 per 1,000 requests, compared with GPT-6 Luna's 71.2% at $0.14.Source: §5 · p. 8, Figure 5 · p. 8

6 Conclusion

System1-Arena compares fast probability-based decisions with generative reasoning across static and closed-loop tasks. System-1 models are encouraging on many static decisions, but the strongest accuracy comes from a reasoning LLM, and large models succeed most often in closed loops.

6 Conclusion

  • On most static decisions, the largest System-1 models match fast closed LLMs with better-calibrated probabilities at one-quarter of the latency.When the best accuracy is needed, a reasoning LLM delivers it at about 70 times the cost of a fast LLM.Source: §6 · p. 8
  • Large models succeed most often in closed loops, while small System-1 models lack the planning the loop needs.Step accuracy does not predict loop success; the paper suggests small System-1 models lack the planning that a loop needs.Source: §6 · p. 9
  • Step accuracy does not predict closed-loop success, and incorrect decisions can alter all subsequent states.The paper concludes that deliberation may be worth its price depending on the decision.Source: §6 · p. 9

7 Limitations

The evaluation has limitations involving inherited labels, possible training exposure, online-service measurements, and single-draw sampling.

7 Limitations

  • Labels inherit source noise, tier-C instructions were written by the authors, and public, several-years-old sources may have appeared in training.Online-service latency includes service and network time, and costs reflect prices on run dates; sampled models received one draw per request, leaving run-to-run variation unmeasured.Source: §7 · p. 9

8 Ethical Considerations

System1-Arena reuses public datasets and environments under their licenses and includes intentionally offensive or unsafe text in some task inputs. The authors caution that benchmark scores do not establish suitability for consequential decisions.

8 Ethical Considerations

  • A high benchmark score does not establish fitness for legal, medical, financial, or safety decisions; human review remains necessary where errors can harm people.Toxicity, hate-speech, and safety tasks retain offensive or unsafe source text and add no new personal data.Source: §8 · p. 9
  • Sources that prohibit redistribution are rebuilt locally by a loader and are not redistributed.Other reused public datasets and environments remain subject to their licenses.Source: §8 · p. 9

A LLM Prompts

The appendix describes distinct prompting and probability-reading procedures for closed LLMs, open LLMs, and System-1 models. Each model family uses a fixed prompt, and the closed and open LLM prompts differ because their probabilities are obtained differently.

A.1 Closed LLMs: verbalized probabilities

  • Closed LLMs express probabilities as JSON values rather than token probabilities, and all receive the same two messages.One request containing all questions is sent in one call.Source: §A.1 · p. 24
  • The closed-LLM system prompt requires probabilities over allowed options that sum to 1 for each question.The model is instructed to return only a JSON object.Source: §A.1 · p. 24
  • The closed-LLM request gives questions and answer options, with omitted options assigned probability 0.Boolean questions use true/false; score questions list levels and their descriptions.Source: §A.1 · p. 24

A.2 Open LLMs: probabilities of label tokens

  • Open LLMs receive one question per prompt, and the evaluation reads each label token’s probability where the answer begins.The prompt ends at the assistant reply start without reasoning; DiffusionGemma receives the same two messages.Source: §A.2 · p. 24
  • Open-LLM prompts ask for only the answer label, with the user message supplying the state and one question.Choice options are labeled, and the evaluation reads the log probabilities of those label tokens; boolean and score questions use their respective answer labels.Source: §A.2 · p. 24
  • GPT-OSS uses its own chat format, and its labels are read after its reasoning message.Source: §A.2 · p. 24

A.3 System-1 models

  • Jev receives JSON state and typed questions with instructions and option or level descriptions through its native interface.The other System-1 models use code released by their authors.Source: §A.3 · p. 24

B Evaluation Details

Evaluation uses fixed samples and task-specific scoring, with model calls and episodes handled under explicit rules. The setup also varies thinking budgets and records the services, hardware, and input limits used.

Static tasks

  • Each static task has about 500 requests, usually stratified by label and split evenly across sources; rare-event tasks are balanced instead.Soft labels are included for 4,425 requests, and Tier C instructions cover 33% of requests.Source: B Evaluation Details · p. 24

Metrics

  • Accuracy uses avg@1, while Brier score, calibration error, NDCG, and pair accuracy assess probability quality or task-specific ranking and paired outcomes.A missing answer is treated as a uniform distribution for Brier score, expected calibration error, and NDCG.Source: B Evaluation Details · p. 25

Reading open models

  • Open-model probabilities are read from answer-token probabilities, with multi-token option labels scored through a token prefix tree.GPT-OSS may generate a reasoning message before probabilities are read at the first token of its final message; DiffusionGemma reads one denoising step.Source: B Evaluation Details · p. 25

Thinking budgets

  • Budget 0 is less accurate than thinking off by 2.1 points for Qwen3.8-27B, 3.6 for Gemma-4-26B-A4B, and 13.5 for DiffusionGemma.The four models used 58.8 H100 GPU-hours across all budgets; budget 0 uses an empty-thought prompt that can prompt models to keep thinking rather than answer.Source: B Evaluation Details · p. 25

Inputs and latency

  • Inputs are generally not shortened, except Laya inputs are cut to 512 tokens and CLM inputs to their last 2,048 tokens.Latency is measured as median single-request time, locally on idle GPU servers or through the network for closed models and specified services.Source: B Evaluation Details · pp. 25–26

Calls and episodes

  • Across 138 environments, 128 report success or failure and 10 report only a score; the loop success mean covers 127 environments.One environment was played only by Jev, and static games and embodied tasks are excluded from this evaluation.Source: B Evaluation Details · p. 25

Services and hardware

  • Closed LLMs use different services across task types, while local System-1 models use author-released code and local hardware.The stated service pricing includes $0.10 per million input tokens for the OpenAI Decisions API and $0.042 per million input tokens for Microsoft-Decision-1.Source: B Evaluation Details · p. 25

Static benchmark scope

  • Table 3 covers static benchmark tasks, including single agent steps, while games and embodied domains also have static tasks outside the full evaluation.Requests that cannot be redistributed are rebuilt locally by loaders.Source: B Evaluation Details · p. 25, Table 3 · p. 26

C Domains and Sources of Static Tasks

The static-task sources are organized by task, request count, instruction tier, and license, with most inputs and instructions in English.

Sources

  • Table 4 lists each source alongside its task, request count, instruction tier, and license.A BibTeX entry per source provides the dataset page and license, while task cards contain construction notes and exact instructions.Source: C Domains and Sources of Static Tasks · p. 26

Language coverage

  • All instructions and questions are in English, and nearly all source inputs are English, with stated exceptions.The exceptions are FanLang-9 and DuLeMon in Chinese, WiLI-2018 in multiple languages for language identification, and CORD receipts in Indonesian.Source: C Domains and Sources of Static Tasks · p. 26

Source attribution

D Closed-Loop Environments

This section documents the closed-loop environments used to evaluate agents, defining discrete-choice procedures and environment-specific settings, limits, and reproducibility fixes. It catalogs tasks across coding, browser and web use, navigation, games, control, and other domains, including paused and real-time settings, task-native outcomes, and scores where provided.

D Closed-Loop Environments

  • The suite spans web, text-game, support, coding, long-horizon game, and control benchmarks; failed answers use a fixed default, and outcomes use simulator-defined success, return, score, or progression.Examples include WebShop, SWE-bench, BALROG, MuJoCo, and highway-env.Source: D Closed-Loop Environments · p. 26
  • In SWE-bench Verified, a fixed proposer supplies candidate actions for the model to choose; one proposer includes the reference patch, while GPT-5.6 Luna proposes five actions without it.The chosen action runs in the task’s official container, or the model submits the changes so far for SWE-bench’s tests. The setup tests recognizing and stopping at a reference fix or steering a proposer toward a fix (Jimenez et al., 2023; OpenAI, 2024).Source: D Closed-Loop Environments · p. 26
  • In real-time mode, each decision incurs fixed median request latency, during which the environment advances with the previous action held; two-car intersections use independent calls of the same model.Latency is measured on an idle A100 GPU, rounded to 10 ms, and fixed to keep episodes reproducible.Source: D Closed-Loop Environments · p. 26

Forecasting

  • Forecasting sources include resolved ForecastBench questions, platform questions, Metaculus and Polymarket events, and FRED-series prediction tasks.The listed collections include binary and multi-outcome questions and numeric-range forecasts.Source: D Closed-Loop Environments · p. 30

Dialogue and interaction

  • Dialogue tasks cover clarification, turn endings, handoffs, state tracking, and satisfaction using AmbigQA, ShARC, Switchboard, Taskmaster-1, ABCD, and MultiWOZ.The entries also include HINT3 and NatCS handoff cases, Schema-Guided Dialogue, and user-satisfaction ratings from EmoWOZ and USS.Source: D Closed-Loop Environments · p. 30

Evaluators and assessment

Safety and moderation

  • Safety and moderation sources cover agent-action risk, custom policies, harmfulness, and trajectory safety, including ASSEBench and ATBench (Li et al., 2026).Other entries include CoSApien policies and test prompts, DynaBench, OpenAI moderation policy variants, Aegis 2.0, and BeaverTails-330k.Source: D Closed-Loop Environments · p. 31
  • Injection, privacy, and refusal tasks use AgentDojo, BIPIA, ToxicChat, TAB, generated personal-information examples, CoCoNot, and XSTest v2.The entries include tool-use trajectories with and without injected content, test attacks, annotated anonymization examples, and safe/unsafe contrast prompts.Source: D Closed-Loop Environments · p. 32

Routing and orchestration

  • Routing tasks draw on context-pruning, difficulty, escalation, memory, correctness, and model-selection datasets, including QASPER, LiveBench, RouterArena, and RouterBench.The entries include generated multi-hop fact-chain ablations, cached model replies, and test or validation splits.Source: D Closed-Loop Environments · p. 32

D Closed-Loop Environments

  • Each environment step is a discrete choice, such as selecting one of 43 tools or “done,” a legal chess move, a command or link, or a simulator-listed action.WebShop offers clickable actions or two to four instruction-derived queries; ScienceWorld actions can involve up to three questions; BALROG games use their text observations and listed key-press actions.Source: D Closed-Loop Environments · p. 35
  • Environment-specific changes are limited to interface needs; the quadrotor uses position mode with sparse reward and a 30-second limit, while Atari reports game score because success is undefined.The two-car intersection uses crossing or shared-exit routes, and the τ2 telecom environment runs without a user with two construction fixes.Source: D Closed-Loop Environments · p. 35
  • The newer suites retain benchmark settings where possible, with bounded evaluations for WebShop, ScienceWorld, Jericho, BALROG games, and a 40-instance SWE-bench Verified coding controller.WebShop uses its first 100 test goals; ScienceWorld runs one easy test variation per task type for up to 100 actions; Jericho allows 100 commands. SWE-bench instances are balanced across repositories, validated before use, and run in official evaluation images with variable command-output fields masked.Source: D Closed-Loop Environments · p. 35

Closed-Loop Environments

  • 0.33 / 33.3 was reported for Baba Is AI’s goto_win-distr_obj_rule task, while make_win-distr_obj_rule scored 0.00 / 0.0 (Cloos et al., 2024).Both rows specify a 3-episode run and a progression score where 100% is awarded if solved.Source: D Closed-Loop Environments · p. 37
  • The listed environments include SWE-bench Verified coding control, τ2-bench telecom customer service, ScienceWorld experiments, Jericho text games, and MiniWoB++ web tasks.The entries specify official SWE-bench evaluation images and grading, a no-user telecom mode, and 30 ScienceWorld task types with test variations and easy simplifications.Source: D Closed-Loop Environments · pp. 38–39
  • MiniWoB++ buy-ticket, choose-list, and click-button each show a reward of 1.00 / 1.0 (Liu et al., 2018).The listed runs use five episodes; the displayed maximum-decision counts are five for buy-ticket and click-button and six for choose-list.Source: D Closed-Loop Environments · p. 39

MiniWoB++ environments

  • Four MiniWoB++ examples—click-dialog-2, search-engine, social-media, and tic-tac-toe—are paused with five episodes; listed limits are 5, 10, 6, and 10, and reward fields are 0.80/0.6, 0.20/-0.6, 1.00/1.0, and 0.40/0.1, respectively.Source: D Closed-Loop Environments · pp. 39–40

Web and navigation environments

  • The suite also lists text-mode WebShop over 1.18M products with human test-split instructions and Wikispeedia using a Wikipedia link graph and completed human games.WebShop is attributed to Yao et al. (2022), and Wikispeedia to West and Leskovec (2012).Source: D Closed-Loop Environments · p. 40

Real-time games

  • Real-time game environments include Atari Boxing, Breakout, Freeway, and Pong, plus ViZDoom scenarios with per-task episode and decision limits.The listed ViZDoom scenarios are basic, deadly_corridor, defend_the_center, and health_gathering.Source: D Closed-Loop Environments · p. 40

Real-time control

  • Real-time control entries include QuadX waypoint flight and highway driving scenarios, including a multi-agent intersection with conflicting paths.The listed tasks include 10 episodes each, with decision limits varying by environment.Source: D Closed-Loop Environments · p. 40

Related papers

Loading 2610.21244v1…