PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Intelligence -- Evaluating Agents on What Users Don't Say

Ved Sirdeshmukh, Marc Wetter

arXiv:2602.20424v1cs.AI

TL;DR

Real-world agent requests are underspecified, while existing benchmarks largely evaluate fully specified instructions rather than implicit requirements. The paper introduces Implicit Intelligence and Agent-as-a-World to test contextual goal fulfillment in simulated interactive worlds. Across the benchmark, even the best-performing model achieves only 48.3% scenario pass rate, indicating substantial room for improvement.

  • Problem

    Existing agent benchmarks primarily test explicit instruction-following with fully specified success criteria, leaving implicit contextual requirements insufficiently evaluated.

  • Method

    The paper evaluates four categories of implicit requirements using Agent-as-a-World, where YAML-defined interactive environments are simulated by constrained language-model World Models.

  • Results

    48.3% scenario pass rate is achieved by GPT-5.2-pro, the best-performing model, which fails to satisfy all implicit requirements in more than half of scenarios.

  • Takeaways & Limitations

    Implicit intelligence remains a distinct frontier challenge from general reasoning capabilities, with substantial room for improvement in current agents.

  • Takeaways & Limitations

    Scenario expectations may not represent diverse users, validity may degrade as iOS features change, and the action space covers approximately 300 native iOS actions rather than all real-world applications.

Abstract

from arXiv · show

Real-world requests to AI agents are fundamentally underspecified. Natural human communication relies on shared context and unstated constraints that speakers expect listeners to infer. Current agentic benchmarks test explicit instruction-following but fail to evaluate whether agents can reason about implicit requirements spanning accessibility needs, privacy boundaries, catastrophic risks, and contextual constraints. We present Implicit Intelligence, an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers, paired with Agent-as-a-World (AaW), a harness where interactive worlds are defined in human-readable YAML files and simulated by language models. Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration. Evaluating 16 frontier and open-weight models across 205 scenarios, we find that even the best-performing model achieves only 48.3% scenario pass rate, revealing substantial room for improvement in bridging the gap between literal instruction-following and human-like contextual reasoning.

1 Introduction

Real-world requests contain unstated constraints that explicit-instruction benchmarks largely omit. The paper introduces Implicit Intelligence and Agent-as-a-World to evaluate whether agents infer and satisfy those requirements through environmental context.

  • Natural communication relies on shared context, implicit assumptions, and unstated constraints that can make technically correct solutions useless or harmful when violated.
  • Existing benchmarks test explicit instruction-following and tool use with fully specified ground truth, creating misalignment with real-world communication.
  • Implicit Intelligence measures whether agents identify, reason about, and satisfy requirements users expect but never explicitly state.
  • The framework evaluates four categories: Implicit Reasoning, Catastrophic Risk Avoidance, Privacy & Security, and Accessibility.
  • Agent-as-a-World defines interactive worlds in single YAML files, with contextual information, entities, execution rules, and evaluation rubrics for implicit requirements.
  • Scenarios combine simple-looking requests, hidden complexity, and constraints discoverable through proactive environmental exploration.

2 Related Work

Prior agent benchmarks cover domain-specific and general-purpose capabilities, but many assume fully specified success criteria. This paper positions implicit-requirement evaluation as a complement to existing work on simulation, reasoning, and alignment.

  • Early agent benchmarks target specific domains, including software engineering, web interaction, and large-scale API usage.
  • General-purpose benchmarks span operating systems, databases, web tasks, multimodal reasoning, complex tool use, and strict constraint adherence.
  • Frontier-model performance is increasingly saturated on established benchmarks, motivating evaluations beyond explicit task completion.
  • LLM-based environment simulation has been explored for plausible feedback, planning, and faithful user-agent interactions.
  • Existing implicit-reasoning benchmarks do not systematically test whether agents infer and satisfy unstated requirements in agentic settings.
  • This work complements safety and alignment research by making catastrophic-risk and privacy violations directly measurable in realistic agentic scenarios.

3 The Implicit Intelligence Framework

The Implicit Intelligence framework defines a capability distinct from following explicit instructions and organizes implicit requirements into four failure-oriented categories. Each category tests contextual adaptation beyond literal compliance.

  • Implicit intelligence is the capacity to identify, reason about, and satisfy requirements users expect but never explicitly state.
  • The framework contrasts implicit intelligence with explicit intelligence, which existing benchmarks primarily measure through well-specified instructions.
  • The four categories represent failure modes arising when agents optimize for literal compliance instead of genuine goal fulfillment.
  • Implicit Reasoning: Implicit Reasoning requires contextual factors such as time, location, and recent activity to modify an otherwise straightforward response.
  • Catastrophic Risk Avoidance: Catastrophic Risk Avoidance requires verification before irreversible actions, such as deleting files that may be active or unbacked up.
  • Privacy & Security: Privacy & Security requires sanitizing internal comments or sensitive attachments before forwarding communications externally.
  • Accessibility: Accessibility requires adapting actions to discoverable user characteristics, such as enabling high contrast, larger fonts, and voice guidance for vision impairment.

4 Agent-as-a-World

Agent-as-a-World addresses the tension between realistic but costly simulators and simple but context-poor task descriptions. It uses declarative YAML and a constrained language-model World Model to simulate interactive environments for reproducible evaluation.

  • Hand-crafted simulators provide realism but require substantial domain engineering, while toy environments lack the contextual richness needed for implicit reasoning.
  • Agent-as-a-World specifies environments declaratively in human-readable YAML and delegates state interpretation, action execution, and rule enforcement to a language model.
  • Existing engineered environments impose domain-specific overhead, whereas synthetic task descriptions sacrifice interactive exploration needed to discover hidden constraints.
  • Each scenario YAML file includes metadata, world context, entities, and explicit state and action descriptions.
  • World Model: The World Model validates action availability, checks preconditions, determines results from current state and world logic, and updates entity state.
  • World Model: Subsequent state queries reflect prior action effects, preserving consistency across interactions.
  • World Model: The World Model is restricted to deterministic execution of YAML-specified action semantics and has no access to user intent or the evaluation rubric.
  • Evaluation uses a turn-based protocol in which the agent selects actions, the World Model returns feedback and state changes, and interaction continues until termination.

5 Dataset Construction

The dataset combines synthetic scenario generation with expert human refinement to create challenging, fair tests of implicit requirements. It contains 205 scenarios grounded in realistic iOS actions and screened for difficulty, consistency, balance, and diversity.

  • Scenario Generation and Validation: A hybrid pipeline combines synthetic generation with expert human refinement to make scenarios challenging without relying on arbitrary tricks.Authors refine scenarios, obtain two independent expert reviews, require unanimous consensus, and apply a difficulty gate.
  • Seed Sources: 303 executable iOS actions across seven domains provide realistic seeds for scenarios spanning apps, documents, location, media, scripting, sharing, and web.PersonaHub supplies demographic and contextual diversity through varied ages, occupations, and health conditions.
  • Seed Sources: iOS Shortcuts provides ecological validity and action diversity, while the evaluation targets remain domain-agnostic and expose implicit requirements through settings, relationships, and conflicts.The grounding environment includes opportunities involving accessibility, privacy, communication, and system control.
  • Scenario Generation and Validation: Iterative Plan–Attempt–Refine cycles add timing constraints, order dependencies, and verification steps after observing attempter-model trajectories.The rotating attempter pool includes GPT-5-series, Claude Opus/Sonnet, and Gemini 3 Pro models.
  • Dataset Statistics: 205 scenarios are distributed across four categories, with each scenario containing a user prompt, 3–5 entities, 2–4 actions per entity, at least three rubric criteria, and at least three hidden rules.Dataset-level checks enforce rough category balance and remove near-duplicate scenarios.

6 Evaluation Methodology

The evaluation measures whether agents satisfy implicit requirements through objective, trajectory-based rubrics. It combines strict complete-task success with criterion-level partial-credit scoring and human-validated evaluator judgments.

  • Rubrics: Each scenario rubric contains natural-language criteria and specific pass conditions tied to actions, states, or their absence.For example, a criterion may require checking backup status before deleting a file.
  • Rubrics: Rubrics are binary and objective, covering positive actions, negative constraints, and final-world-state requirements across typically 3–5 criteria.Examples include enabling accessibility features, avoiding deletion of favorite files, and enabling suitable parental controls.
  • Evaluator: Evaluator judgments show high agreement with independent human annotations on a randomly sampled subset of scenarios.Annotators used the same rubric and had access to trajectories and final world states.
  • Evaluator: GPT-5.2-high evaluates each criterion using the prompt, rubric, complete trajectory, execution feedback, and final world state.The evaluator returns a boolean pass/fail judgment with supporting reasoning for every criterion.
  • Metrics: Scenario Pass Rate (SPR) is the proportion of scenarios in which an agent satisfies all rubric criteria.SPR is the strict metric for complete task success.
  • Metrics: Normalized Scenario Score (NSS) measures partial completion as the fraction of satisfied criteria per scenario, averaged across scenarios.For scenario i, k_i denotes its number of criteria.

7 Experiments

The benchmark evaluates 16 models across 205 scenarios and finds that implicit intelligence remains difficult despite strong performance on established reasoning benchmarks. Results vary by requirement category, model family, and use of extended thinking.

  • Experimental Setup: 16 models are benchmarked across all 205 scenarios, with aggregate performance, category-specific results, and failure-pattern analyses reported.The models span OpenAI, Anthropic, Google, and leading open-weight alternatives.
  • Main Results: 48.3% SPR is achieved by the best-performing model, GPT-5.2-pro, which fails to satisfy all implicit requirements in more than half of scenarios.The result contrasts with near-ceiling performance on established reasoning benchmarks.
  • Category Results: Catastrophic-risk avoidance and privacy protection favor different models: Claude Opus 4.5 excels at harmful-action recognition, while GPT-5.2-pro leads privacy-sensitive scenarios.Open-weight models particularly struggle with catastrophic-risk scenarios and often proceed with dangerous operations that frontier models refuse.
  • Model Comparisons: GPT-5 outperforms GPT-4.1, GPT-5.1, and GPT-5.2, showing non-monotonic progression within the model family.The results indicate that implicit intelligence does not automatically improve with model iterations.
  • Additional Analyses: Extended thinking produces mixed results: GPT-5.2 gains +1.4 pp SPR and Claude Opus 4.5 gains +1.5 pp SPR, while other models show no improvement or slight degradation.The reported gains are modest and model-dependent.
  • Failure Analysis: Partial-failure traces reveal insufficient environmental exploration, incomplete feature configuration, and inadequate state preservation as recurring failure patterns.The analysis examines 156 trajectories where models satisfied some but not all rubric criteria.

8 Discussion

The discussion frames the framework as a reproducible way to evaluate implicit requirements while acknowledging limits in representativeness, temporal validity, and action-space coverage. The authors conclude that implicit intelligence remains distinct from general reasoning capabilities.

  • Evaluation Validity: The World Model is intended as a deterministic rule executor, not an intelligent agent, because action behavior is fully specified in YAML returns fields.This design addresses concerns about using Claude Opus 4.5 both as World Model and as an evaluated agent.
  • Evaluation Validity: Strategic clarification is a complementary capability that the current evaluation does not measure alongside implicit inference.The authors identify separate measurement of inference and question-asking as a natural extension.
  • Limitations: The scenarios may not represent the full diversity of implicit user expectations across cultures, ages, and technical backgrounds.The authors explicitly identify author perspectives as a source of limitation.
  • Limitations: Scenario validity may degrade as iOS features evolve, and the approximately 300-action native iOS space excludes third-party applications and their implicit conventions.The YAML architecture makes extending coverage to new applications straightforward.
  • Conclusion: The gap between strong established-benchmark performance and modest results here indicates that implicit intelligence is a distinct frontier challenge.The conclusion presents Implicit Intelligence and Agent-as-a-World as a scalable evaluation framework for this capability.

A.1.1 Scenario YAML

The scenario YAML specifies an accessibility task through device and application entities, their state, available actions, and explicit evaluation criteria for shared AirPods listening.

  • Scenario setup: The scenario asks the agent to connect a colleague’s AirPods for shared listening.Its initial state includes separate user and colleague AirPods, with only the user’s device paired and connected.
  • Entities and actions: Bluetooth, accessibility-audio, and podcast entities expose actions for pairing, connecting, inspecting settings, controlling playback, and adjusting volume or balance.The world tracks connection, routing, mono-audio, balance, playback, and media-volume state.
  • Evaluation criteria: The rubric requires connected colleague AirPods, enabled Mono Audio, centered balance between 0.45 and 0.55, and active playback.Connecting a device updates the podcast output route, while toggling Mono Audio pauses playback and requires resumption.

A.1.2 Model Trajectory

The demonstrated trajectory solves the accessibility scenario by exploring device state, configuring the colleague’s AirPods and audio settings, and resuming playback on the correct output.

  • Model Trajectory: The agent first lists Bluetooth devices to inspect pairing and connection status.The read-only result identifies the user’s AirPods as paired and connected and the colleague’s AirPods as unpaired and disconnected.
  • Model Trajectory: The agent resumes podcast playback after changing Mono Audio, completing the shared-listening configuration.The trajectory’s stated approach includes starting playback on the correct output after configuring the audio settings.
  • Model Trajectory: The agent pairs the colleague’s AirPods, verifies the current connection, and then connects the target device.The connection changes the active Bluetooth device and podcast output route to AirPods (Colleague).
  • Model Trajectory: The trajectory satisfies all four evaluation criteria and receives a pass rate of 1.The reported result describes the final world as configured for the requested shared listening experience.

A.2 System and User Prompts

The evaluation uses realistic iOS actions to expose recurring implicit-intelligence failures: agents often act without exploring context, configure features incompletely, and make changes without preserving state. Additional failures involve safety-critical verification, negative constraints, privacy minimization, and accessibility modality selection.

  • A.3 iOS Actions Dataset: 303 executable iOS actions ground scenarios in realistic interactions across apps, documents, location, media, scripting, sharing, and web.
  • A.5.2 Pattern 1: Insufficient Environmental Exploration: Agents frequently act on initial state without checking contextual factors such as playback routes, calendar conflicts, or accessibility baselines.
  • A.5.2 Pattern 1: Insufficient Environmental Exploration: 73% of AirPlay-related scenarios across 8 models involved failing to check the playback route before enabling captions on the correct device.
  • A.5.2 Pattern 1: Insufficient Environmental Exploration: 58% of temporary-change scenarios across 7 models missed restoring the user’s baseline settings after completing the requested task.
  • A.5.3 Pattern 2: Incomplete Feature Configuration: 71% of media accessibility scenarios skipped preview or verification actions, reflecting incomplete configuration beyond merely enabling a primary feature.
  • A.5.3 Pattern 2: Incomplete Feature Configuration: Across 11 models, 89% enabled Mono Audio but only 11% also centered balance, showing that dependent settings are often treated independently.
  • A.5.4 Pattern 3: Inadequate State Preservation: 64% of temporally scoped requests across 10 models lacked revert planning, while 43% of applicable scenarios used broad global settings instead of scoped tools.
  • A.5.5 Category-Specific Observations: Failures also included skipping safety-critical checks, overlooking negative constraints, oversharing sensitive data, and selecting accessibility modes that distort preserved content.
Loading 2602.20424v1…