PaperlayerView exact revision on arXiv

Source-linked AI summary

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

Hang Yan, Fangzhi Xu, Qiushi Sun, Jinyang Wu, Zixian Huang, Muye Huang, Jingyang Gong, Zichen Ding, Kanzhi Cheng, Yian Wang, Xinyu Che, Zeyi Sun, Jian Zhang, Zhangyue Yin, Haoran Luo, Ben Kao, Qika Lin

arXiv:2602.05843v2cs.CL

TL;DR

Existing benchmarks largely test deductive instruction-following, leaving active discovery of hidden transition rules over long horizons underassessed. OdysseyArena introduces interactive inductive benchmarks, and evaluations show that even leading LLMs achieve low success rates, with Gemini 3 Pro Preview reaching 44.17%.

  • Problem

    Existing interactive benchmarks predominantly assess deductive intelligence, overlooking agents’ active induction of hidden environmental rules.

  • Method

    OdysseyArena formalizes latent transition dynamics into four structural primitives and provides Lite and Challenge suites for long-horizon, active, inductive evaluation.

  • Results

    44.17% was the highest success rate, achieved by Gemini 3 Pro Preview, while leading commercial models remained far below human-level performance across four environments.

  • Takeaways & Limitations

    The observed low-performance plateau across leading models identifies inductive discovery as a fundamental bottleneck for autonomous discovery.

  • Takeaways & Limitations

    Fixed stochastic sequences ensure deterministic evaluation but remove the non-stationary dynamics of real interactive systems.

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g., > 200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Our code and data are available at https://github.com/xufangzhi/Odyssey-Arena

1 Introduction

OdysseyArena reframes agent evaluation around long-horizon, active interaction and inductive discovery of latent transition dynamics, addressing benchmarks’ emphasis on deductive compliance. It introduces four interactive environments, a 120-task standardized suite, and evaluations showing substantial inductive limitations in leading LLMs.

  • Motivation: Current benchmarks predominantly assess deductive intelligence using explicit instructions or success criteria rather than inferring latent rules from interactive experience.This evaluation gap overlooks agents’ need to probe environments, react to feedback, and iteratively adjust actions.
  • Motivation: Inductive evaluation requires agents to actively explore environments and induce hidden rules that are not pre-specified.The omission restricts assessment of agent proficiency in complex environments with latent rules.
  • OdysseyArena: OdysseyArena centers evaluation on long-horizon, active, and inductive reasoning by modeling environments as generative state transition functions, (s_t+1, r_t) = T(s_t, a_t).The transition function T implicitly encodes environmental rules and regularities that agents must infer from empirical interactions.
  • OdysseyArena: 4 diverse environments instantiate OdysseyArena’s abstract interaction primitives as computationally efficient and lightweight testbeds for inductive evaluation.The environments are designed to remain functionally representative of real-world systems while providing a tractable and scalable evaluation setting.
  • OdysseyArena-Lite: 120 curated tasks comprise OdysseyArena-Lite, a standardized benchmark preserving active and inductive discovery challenges while supporting high evaluation throughput.Tasks use computationally tractable but non-trivial horizons that require long-horizon planning and induction of latent rules.
  • Results: 44.17 was the highest success rate achieved by Gemini 3 Pro Preview among over 15 evaluated LLMs, while leading commercial models remained far below human-level performance.The results identify substantial gaps in long-horizon reasoning, active exploration, and inductive generalization across four OdysseyArena-Lite environments.

2 Related work

Prior interactive benchmarks have progressed from simplified grid-worlds to sophisticated digital and real-world systems, but many still emphasize short horizons and deductive reasoning. This leaves error accumulation, long-term planning consistency, and active discovery of latent rules insufficiently captured.

  • Interactive Benchmarks: Interactive benchmarks have advanced from grounded language understanding in simplified grid-worlds to sophisticated digital and real-world systems.This progression spans both digital environments and real-world systems.
  • Interactive Benchmarks: Most existing environments favor short horizons or trajectories, limiting their ability to capture error accumulation and decay in long-term planning consistency.The passage identifies temporal depth as a critical remaining bottleneck.
  • Interactive Benchmarks: Representative multi-turn benchmarks differ in inductive-reasoning requirements, task horizon, and deployment setup.Horizon categories are short (<50), long (50–100), and X-Long (>100), while API-based deployment is the most lightweight.
  • Inductive Reasoning: Frameworks such as ReAct and Reflexion primarily rely on deductive reasoning or test-time interactions to apply internal knowledge or provided rules.These approaches contrast with inferring latent rules and transition dynamics from raw observations.
  • Inductive Reasoning: Intelligent agents need inductive reasoning to infer latent rules and transition dynamics from raw observations, whereas static benchmarks remain passive.ARC and Zebra-Logic evaluate rule synthesis but do not capture active discovery.

3 ODYSSEYARENA

OdysseyArena evaluates agents on inductive world-structure discovery: they must infer latent transition functions through strategic interaction rather than optimize policies in known or fixed environments. Its taxonomy decomposes environment dynamics into four orthogonal primitives, instantiated in environments requiring logical, statistical, and temporal reasoning.

  • Core Framework: Agents must autonomously discover the latent transition function governing an environment’s evolution to support long-horizon planning and decision-making.The transition function encodes unobservable regularities and constraints, distinguishing OdysseyArena from paradigms centered on known or fixed MDPs.
  • Taxonomy: Four orthogonal structural primitives organize OdysseyArena’s world-modeling challenges, each inducing a distinct cognitive requirement that cannot be reduced to the others.The resulting environments assess fundamental capacity for world-structure induction across requirements ranging from logical deduction to relational abstraction.
  • Turn On Lights: Turn On Lights requires agents to infer hidden Boolean couplings by intervening on individual lights and observing deterministic cascades across an N-light network.Rules remain fixed within an episode but vary across episodes; success requires turning all lights on before the interaction budget is exhausted.
  • AI Trading: AI Trading requires agents to disentangle latent market factors from stochastic noise and exploit persistent factor relationships in multi-step portfolio decisions.Price transitions follow st+1 = Wz_t + ϵ, with W invariant within an episode but varying across tasks; performance is measured by cumulative return adjusted for transaction costs and risk constraints.
  • Energy Dispatch: Energy Dispatch models long-horizon grid management in which agents allocate four resource types under budget and demand constraints while accounting for latent, time-varying efficiency.Realized generation approximately follows Preal ≈ a_t ⊙ E_t, and repeated budget or demand violations trigger immediate termination.

4 ODYSSEYARENA-LITE and ODYSSEYARENA-CHALLENGE

Section 4 defines two deterministic benchmarking protocols: ODYSSEYARENA-LITE enables efficient, reproducible evaluation, while ODYSSEYARENA-CHALLENGE stress-tests agent stability and inductive resilience over extreme horizons. Lite samples 30 tasks per environment with environment-specific step limits, whereas Challenge often exceeds 1,000 interaction steps to expose failures absent under standard budgets.

  • Protocol Design: ODYSSEYARENA-LITE provides efficient, reproducible assessment, while ODYSSEYARENA-CHALLENGE stress-tests stability and inductive resilience over extreme horizons.Lite is the primary evaluation setting; Challenge is designed for more demanding stress tests.
  • Benchmark Construction: Tasks are deterministic instances sampled from bounded parameter distributions and decomposed into structural configurations and temporal trajectories.Latent rules remain invariant, while time-varying factors are fixed in task metadata to enable verifiable and fair comparisons.
  • Task Sampling Strategy: Sampling calibrates difficulty by assigning Lite tractable parameter ranges and Challenge parameters near empirical limits.The strategy modulates task difficulty across the two protocols and supports stress-testing over extreme horizons.
  • ODYSSEYARENA-LITE: 30 tasks are sampled for each environment in the fixed ODYSSEYARENA-LITE evaluation suite.The suite uses environment-specific maximum interaction budgets.
  • ODYSSEYARENA-LITE: 200 steps are allowed for TurnOnLights, compared with 120 steps for AI Trading, Energy Dispatch, and Repo Management.These limits reflect the inherent complexity of each environment and support balanced comparison.
  • ODYSSEYARENA-CHALLENGE: Challenge tasks often exceed 1,000 interaction steps, extending the reasoning horizon beyond standard evaluation budgets.The variant targets failures such as long-term credit assignment breakdown and compounding planning errors.

5 Experiments

Experiments across more than 15 LLMs show a substantial gap between current agents and humans in long-horizon autonomous rule induction. Frontier models lead overall and perform near-perfectly when rules are explicit, but struggle to infer latent dynamics—especially extended periodic patterns—without them.

  • Experimental Setup: More than 15 proprietary and open-source LLMs are evaluated on OdysseyArena-Lite using four executions per test case to report Avg.@4 and Pass@4.The evaluation includes models such as Gemini 3 Pro Preview, GPT-5, Grok 4 Fast, DeepSeek-V3.2, gpt-oss-120b, Qwen3, Llama 3, and GLM-4.
  • The General Performance Gap: Human participants outperform LLMs by isolating causal variables and distilling latent rules through trial-and-error, whereas LLMs degrade substantially as interaction horizons increase.The comparison identifies autonomous rule induction as a central weakness of current agents.
  • Proprietary Model and the Scaling Limit: Gemini 3 Pro Preview establishes the current performance ceiling and substantially outperforms other models across most environments.This advantage does not prevent ubiquitous failure in Energy Dispatch.
  • Proprietary Model and the Scaling Limit: Failure in Energy Dispatch reflects an inability shared across models to synthesize periodic patterns over extended observation windows of approximately 20 steps.The result suggests that scaling alone does not resolve this architectural limitation.
  • Deductive Proficiency vs. Inductive Deficiency: Frontier models achieve near-perfect success when latent transition rules are provided but falter significantly without them, revealing deductive proficiency alongside inductive deficiency.Providing the underlying logic isolates rule discovery, rather than deductive reasoning, as the fundamental bottleneck.

6 Analysis

The analysis identifies exploration, memory, action adaptation, and related behavioral failures as major obstacles for LLMs in long-horizon inductive interaction. Success often plateaus after initial exploration, while action loops and weak latent-rule extraction further limit performance.

  • 6 Analysis: Manual analysis of 47 Gemini 3 Pro Preview failure trajectories reveals four error categories stemming primarily from exploration, memory, and action-adaptation limitations.These categories are summarized in Table 3.
  • 6 Analysis: 38/47 failures involve exploration limitations, with agents prematurely converging to local optima and adopting suboptimal strategies without systematic hypothesis testing.Memory constraints affect 27/47 failures, as agents gather relevant information but fail to retrieve or prioritize key evidence across long histories.
  • 6 Analysis: Success-rate curves show negligible marginal gains beyond the initial exploratory phase for most models, indicating a bottleneck in forming coherent internal world models.Weaker models frequently underperform the random baseline, suggesting difficulty extracting latent regularities from the environment.
  • 6 Analysis: Higher action-loop ratios correlate directly with lower success rates, as models repeatedly perform invalid operations despite negative environmental feedback.This behavior reflects failure to synthesize hidden rules from unsuccessful trials and refine long-term strategies.

7 Conclusion … A.5 Action space for ODYSSEYARENA

OdysseyArena shifts agent evaluation toward long-horizon, active, inductive modeling by instantiating four environment primitives and exposing a persistent inductive bottleneck. Its curated environments provide guaranteed or deterministic challenges spanning symbolic logic, stochastic trading, constrained energy dispatch, and dependency resolution, with explicit action-space rules.

  • 7 Conclusion: OdysseyArena formalizes four structural primitives and establishes a reproducible framework for evaluating long-horizon, active, inductive agent modeling.The conclusion describes a transition from deductive instruction-following and reports a low-performance plateau across multiple flagship models.
  • A.1 Turn On Lights curation: Turn On Lights hides a latent Boolean dependency graph that agents must discover through strategic interaction, with each light depending only on lower-index predecessors.A random mapping obscures numerical ordering, while the goal is to illuminate all lights from the all-off state.
  • A.1.2 Resolvability and diversity: Turn On Lights instances are exhaustively verified as solvable and vary in state-space size, predecessor density, and logical combinations of ∧, ∨, and ¬.Accepted instances require a solution path and a minimum completion length above a predefined threshold; N lights yield 2^N configurations.
  • A.2 AI Trading curation: AI Trading requires agents to infer an invariant latent dependency matrix W from stochastic market signals while selecting buy/sell operations and quantities.The task maximizes cumulative reward over a fixed trading horizon, with stock prices updated from returns governed by latent market factors.
  • A.2.2 Temporal trajectory: AI Trading and Energy Dispatch use pre-determined temporal factors, making each task instance deterministic and enabling reproducible comparisons across agents.Trading stores factor and noise timelines, while Energy Dispatch fixes demand, budget, and efficiency sequences in task metadata.
  • A.3 Energy Dispatch curation: Energy Dispatch allocates thermal, wind, solar, and battery power while satisfying demand, budget, carbon, and stability constraints under hidden renewable efficiencies.Three consecutive demand or budget violations in ODYSSEYARENA-LITE trigger early termination, and feasible instances are designed with adequate capacity and budget.
  • A.3.2 Temporal trajectory: Energy Dispatch difficulty varies through hidden five-level renewable-efficiency dynamics, period lengths T, and constraint tightness, requiring inference of stable periodic signals from transient fluctuations.Wind and solar include piecewise-linear patterns, 2–5-day segments, 5% stochastic spikes, and random cycle offsets; T ∈ [15, 25].
  • A.4 Repo System curation: Repo System requires discovering package-version configurations satisfying latent graph constraints, while installation transitions may automatically upgrade, downgrade, or uninstall conflicting packages.Instances are generated around a ground-truth solution, and action execution is restricted to one action in Turn On Lights and Repo System; trading sells before buying, while Energy Dispatch forbids simultaneous charging and discharging.

A.7 Limitation … B.3 Reasoning boosts inductive reasoning

The analysis identifies deterministic evaluation as a limitation, confirms that frontier models remain stronger deductive than inductive reasoners, and shows that exploration stability and reasoning budget affect inductive performance.

  • A.7 Limitation: Fixed stochastic sequences ensure reproducible, deterministic evaluation but remove the non-stationary dynamics of real interactive systems.Environment responses therefore cannot evolve or co-adapt with agents over time.
  • B.1 Detailed comparison between w/ and w/o rules: Detailed comparisons support the conclusion that SOTA models are strong deductive reasoners rather than good inductive reasoners.
  • B Details of analysis: Manual analysis of 16 Turn On Lights problems found only two failures attributable to random guesses, with most variance arising from unstable exploration.The analysis focused on Gemini 3 Pro Preview problems showing partial success rather than absolute success or failure.
  • B.2 Comparison between successful and unsuccessful trajectories: An agent may wrongly infer that only light2 is necessary when it activates light3 immediately after activating light2, ignoring light1.This example illustrates how exploration limitations can produce incomplete transition-law discovery.
  • B.3 Reasoning boosts inductive reasoning: 27.50% average success rate was achieved by gpt-oss-120b with high reasoning budget in Turn On Lights, versus 16.67% medium and 7.50% low.The comparison indicates better inductive performance with more reasoning budget.
  • B.3 Reasoning boosts inductive reasoning: Table 5 compares performance under settings with and without rules across proprietary models and three reasoning efforts for gpt-oss-120b.For AI Trading, the table reports profit rate and pass@4; for the other environments, it reports success rate.

B.4 More results and implementation details of evolution progress

Evolution results across 15 models and a random baseline show that long-horizon interaction and hidden-rule discovery remain difficult, with performance varying across steps. The evaluation therefore uses multi-step evolution curves, while excluding Energy Dispatch because it lacks a well-defined step-wise success metric.

  • More Results of Evolution Progress: 15 models and a random baseline are evaluated in detailed evolution-progress results.The results are presented in Figure 7.
  • More Results of Evolution Progress: Most models saturate as interaction steps increase in Turn On Lights and Repo System, indicating that long horizons remain a bottleneck.The saturation pattern is reported for these two environments.
  • More Results of Evolution Progress: GLM-4-9B-Chat performs equivalently to or worse than the random baseline, indicating difficulty discovering hidden environmental rules.This comparison is explicitly reported among the evolution-progress results.
  • Implementation Details: Evolution curves plot interaction step against cumulative task success rate for Turn On Lights and Repo Management, measuring completed tasks from the beginning through the current step.For AI Trading, the y-axis instead reports profit rate relative to initial capital, averaged across tasks.
  • Implementation Details: Because model performance varies across interaction steps, single-step evaluation is insufficient and multi-step assessment is needed for reliable inductive-reasoning measurement.Energy Dispatch has no evolution curve because its complex multi-constraint success conditions do not yield a well-defined step-wise success metric.

B.5 More results and implementation details of task success status … B.9 Eliciting inductive reasoning is NOT trivial

The results show a substantial gap between human and leading LLM performance on long-horizon inductive tasks, with most models failing Energy Dispatch and the best model solving only 25 of 30 Repo System tasks. Additional analyses indicate that inductive reasoning remains constrained by memory length, repetitive actions, and insufficient prompting alone.

  • B.5 More results and implementation details of task success status: Most LLMs fail all Energy Dispatch tasks, revealing an inability to perform long-horizon inductive reasoning.The study reports a significant performance gap between humans and state-of-the-art LLMs across Energy Dispatch and Repo System.
  • B.6 Analyze of task success status: 25 tasks: Gemini 3 Pro Preview solves 25 of 30 Repo System tasks, whereas humans solve all 30.Repo System tasks are divided into easy, medium, and hard sets using the number of required packages and lights.
  • B.7 Memory constrain: the impact of memory usage: About 100 turns of context yield the best Repo System performance, while about 50 turns are insufficient and about 200 turns exceed current agents’ inductive reasoning capacity.The comparison varies the length of interaction history provided to agents.
  • B.8 Loop implementation details: Interaction volume is measured by aggregating the total executed actions across trajectories, with Loop Ratio additionally analyzed for Turn On Lights and Repo System.The analysis targets repetitive actions that produce no effective task progress.
  • B.8 Loop implementation details: Loop Ratio is the normalized proportion of immediately repeated state-action pairs that make no task progress.States are defined by light on/off configurations or package versions, and repeated action counts are divided by total actions.
  • B.8 Loop implementation details: A smaller Loop Ratio indicates greater capability to discover hidden rules through inductive reasoning.The metric is interpreted as an indicator of inductive competence.
  • B.9 Eliciting inductive reasoning is NOT trivial: Merely providing in-context examples does not strengthen agents’ inductive reasoning ability in Repo System.The experiments include a one-shot example containing 12 actions for completion.
  • B.9 Eliciting inductive reasoning is NOT trivial: Future agents may require memory management or systematic encouragement of exploration to improve inductive reasoning.The proposed techniques go beyond simply supplying in-context examples.

B.10 Step distribution analysis

Step distributions use total trajectory steps to assess inductive reasoning efficiency in Turn On Lights and Repo System, while Energy Dispatch steps measure constraint-satisfying days. Most models fail to sustain inductive performance over the evaluated horizons, whereas Gemini 3 Pro Preview, Gemini 2.5 Pro, and GPT-5 reach the 120-step concentration associated with satisfying Energy Dispatch constraints.

  • Step distribution analysis: Total trajectory steps measure inductive reasoning efficiency in Turn On Lights and Repo System, with fewer steps indicating stronger capability.In Energy Dispatch, steps instead represent days of energy supply without violating the three-day consistency constraint.
  • Step distribution analysis: 200 steps for Turn On Lights and 120 steps for AI Trading mark sharp distribution concentrations, showing most SOTA models cannot solve these tasks within their limits.The concentration around the maximum step limits indicates limited inductive reasoning capability in these environments.
  • Step distribution analysis: 120 steps for Gemini 3 Pro Preview, Gemini 2.5 Pro, and GPT-5 indicate they can satisfy Energy Dispatch demand and budget constraints across the evaluated horizon.Most other models concentrate at relatively small step values, indicating inability to satisfy the constraints for three consecutive days.

B.11 Token efficiency analysis · B.12 Different strategies for AI trading · B.13 ODYSSEYARENA-CHALLENGE results

The analysis compares token efficiency across model families, rule-based trading strategies, and LLMs, then shows that extremely long-horizon Repo System interactions remain a bottleneck for inductive reasoning.

  • B.11 Token efficiency analysis: Token efficiency is defined as success rate or trading profit divided by total token usage.The analysis covers GLM, Llama, and Qwen series in Figure 10.
  • B.11 Token efficiency analysis: GLM-4-32B-0414 is most token-efficient in Turn On Lights and AI Trading, while Llama 3.3 70B Instruct leads in Repo System.
  • B.11 Token efficiency analysis: GLM-4-9B-Chat has negative token efficiency because it generates negative profit in AI Trading, while all three model series fail every Energy Dispatch task.
  • B.11 Token efficiency analysis: Qwen achieves the best success rate or trading profit among the three series, but its consistently high token usage makes token efficiency extremely low.
  • B.12 Different strategies for AI trading: The AI Trading comparison includes LLM-free Python strategies alongside human annotation and two SOTA models.The optimal strategy uses full future price trajectories and greedily allocates the portfolio to the highest single-day return, while the conservative strategy waits at least num_factors + 2 days.
  • B.12 Different strategies for AI trading: Progressive, correlation, rolling-window, and ridge-regression strategies differ in how they estimate or regularize the dependency matrix.Progressive learning updates from the full history, correlation estimates elementwise under factor independence, rolling windows use 15 days, and ridge regression shrinks coefficients to suppress noise.
  • B.12 Different strategies for AI trading: SOTA models show a significant performance gap against human annotation and the proposed strategies, indicating substantial room to optimize inductive reasoning.
  • B.13 ODYSSEYARENA-CHALLENGE results: ODYSSEYARENA-CHALLENGE results on Repo System indicate that extremely long-horizon scenarios remain a bottleneck for current LLM inductive reasoning.The results are reported in Table 9 for the proposed complex dataset.

C Main results details … D.1 Instructions given to participants

The paper specifies reproducible baseline settings, structured evaluation prompts, and human-annotation procedures across four interactive environments. Participant instructions emphasize discovering hidden dynamics through active experimentation while satisfying environment-specific goals and constraints.

  • C.1 Baseline settings: Models use specified recent checkpoints and reasoning settings, including medium reasoning effort for GPT-5 and low, medium, and high settings for gpt-oss-120b.Qwen3 models use 2507 versions; Llama checkpoints are from February 2025, GLM-4-9B-Chat from January 2025, and DeepSeek-V3.2 uses its non-thinking version.
  • C.1 Baseline settings: Experiments use NVIDIA H200 GPUs, 56-core Intel Platinum 8480+ CPUs, 1 TB memory, temperature 0.6, and vLLM 0.8.5.post1.These settings are reported for open-source model experiments and all experiments, respectively.
  • C.2 Evaluation prompts: Each environment supplies a system prompt, interaction history, and current state observation, with formats tailored to lights, trading, energy dispatch, and repository management.The prompts specify goals, rules or dynamics, action formats, historical feedback, and current-state information appropriate to each environment.
  • D Human annotation details: Human annotators receive complete instructions, a comprehensive guide, an intuitive interface, and user instructions for all environments.The annotation interface is illustrated in Figure 11, and the paper provides detailed system prompts for the four environments.
  • D.1 Instructions given to participants: Lights participants must infer hidden bulb-access rules, toggle indexed bulbs, experiment actively, and light all bulbs within 200 steps.The supplied example uses three bulbs, while the rules remain hidden in actual tasks.
  • D.1 Instructions given to participants: Trading participants maximize portfolio value over 120 days by using market-factor news to predict stock-price changes and choose buy or sell actions.The example ends with a total value of about 110.42 and a return rate of about 10.42%.
  • D.1 Instructions given to participants: Energy instructions require at least 120 days of generation control while balancing demand, budget, stability, carbon emissions, efficiency patterns, and battery operation.Three consecutive demand or budget violations terminate the task, whereas stability and carbon affect final success conditions.
  • D.1 Instructions given to participants: Repository participants configure package versions through one-command actions, diagnose feedback such as missing modules or API mismatches, and make python run.py execute successfully.The environment includes partial dependency information, non-monotonic installation effects, hidden rules, incremental sub-script debugging, and a final global rerun.

D.2 Recruitment and Compensation · D.4 Annotation Process · D.5 Inter-Annotator Agreement

OdysseyArena used AI-background student annotators compensated at about $15 per hour, with four-person, tutorial-based annotation and environment-specific labeling. Agreement analysis applied Fleiss’ Kappa to discrete tasks and ICC to continuous AI Trading outcomes, finding moderate consensus in discrete environments but divergent inductive strategies in AI Trading.

  • D.2 Recruitment and Compensation: Student annotators with Artificial Intelligence backgrounds were recruited to assess model performance against OdysseyArena’s defined criteria.The background was intended to provide the domain knowledge needed for accurate assessment.
  • D.2 Recruitment and Compensation: Annotators were compensated at a rate of about $15 per hour.
  • D.4 Annotation Process: Each dataset example was annotated independently by four annotators to mitigate individual bias.
  • D.4 Annotation Process: The annotation workflow began with one-shot demonstrations using simplified, non-test examples as tutorials.
  • D.4 Annotation Process: During main annotation, annotators labeled generated outputs according to the specific rules defined for each environment.
  • D.5 Inter-Annotator Agreement: Fleiss’ Kappa was used for binary-success tasks in Turn On Lights, Energy Dispatch, and Repo System, whose coefficients indicated moderate consensus.
  • D.5 Inter-Annotator Agreement: ICC was used for AI Trading’s continuous profit-rate metrics, quantitatively confirming significantly divergent inductive reasoning strategies among annotators.
Loading 2602.05843v2…