PaperlayerView exact revision on arXiv

Source-linked AI summary

OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows

Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, Saravan Rajmohan

arXiv:2508.09124v1cs.CL

TL;DR

Existing benchmarks mostly test atomic tasks rather than the contextual, multi-interaction workflows expected in real-world applications. OdysseyBench evaluates long-horizon office workflows, and HOMERAGENTS automates their creation. Evaluations find that the benchmark challenges current agents and assesses their real-world capabilities more accurately than atomic-task benchmarks.

  • Problem

    Existing agent benchmarks predominantly focus on self-contained atomic tasks, missing the contextual dependencies and multi-interaction coordination of realistic workflows.

    Source: §1 · p. 2

  • Method

    OdysseyBench provides two long-horizon office-workflow splits, while HOMERAGENTS generates tasks by enriching existing benchmarks or creating new tasks from scratch.

    Source: §1 · p. 2

  • Results

    OdysseyBench challenges state-of-the-art agents and provides a more accurate assessment of their capabilities in real-world contexts than atomic-task benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors report that semantic compression and coherent aggregation of dialogue storage are essential for effective multi-step reasoning and agent performance.

    Source: §1 · p. 3

Abstract

from arXiv · show

Autonomous agents powered by large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon workflows. However, existing benchmarks predominantly focus on atomic tasks that are self-contained and independent, failing to capture the long-term contextual dependencies and multi-interaction coordination required in realistic scenarios. To address this gap, we introduce OdysseyBench, a comprehensive benchmark for evaluating LLM agents on long-horizon workflows across diverse office applications including Word, Excel, PDF, Email, and Calendar. Our benchmark comprises two complementary splits: OdysseyBench+ with 300 tasks derived from real-world use cases, and OdysseyBench-Neo with 302 newly synthesized complex tasks. Each task requires agent to identify essential information from long-horizon interaction histories and perform multi-step reasoning across various applications. To enable scalable benchmark creation, we propose HomerAgents, a multi-agent framework that automates the generation of long-horizon workflow benchmarks through systematic environment exploration, task generation, and dialogue synthesis. Our extensive evaluation demonstrates that OdysseyBench effectively challenges state-of-the-art LLM agents, providing more accurate assessment of their capabilities in complex, real-world contexts compared to existing atomic task benchmarks. We believe that OdysseyBench will serve as a valuable resource for advancing the development and evaluation of LLM agents in real-world productivity scenarios. In addition, we release OdysseyBench and HomerAgents to foster research along this line.

1 INTRODUCTION

Existing agent benchmarks largely test self-contained tasks, while real-world office workflows require sustained context across interactions. OdysseyBench evaluates those workflows, and HOMERAGENTS automates creation of its benchmark tasks.

1 INTRODUCTION

  • Existing benchmarks mostly test atomic, self-contained tasks, unlike real workflows that require agents to integrate information accumulated across interactions.Such agents may struggle with contextual dependencies, information persistence, and collaborative workflow management.Source: §1 · pp. 1–2
  • OdysseyBench evaluates long-horizon workflows across office applications through OdysseyBench+ and OdysseyBench-Neo, with tasks requiring dialogue-history reasoning and coordination across tools.OdysseyBench+ contains 300 tasks derived from OfficeBench use cases; OdysseyBench-Neo contains 302 newly generated tasks.Source: §1 · p. 2
  • Existing benchmarks often depend on costly human annotation, while LLM-generated benchmarks have typically remained focused on atomic tasks.The authors identify automated, systematic benchmark generation as a need for evaluating realistic long-horizon work.Source: §1 · p. 2
  • HOMERAGENTS+ iteratively enriches OfficeBench tasks, while HOMERAGENTS-NEO generates new tasks from scratch in realistic application environments.Both components automate generation of long-horizon tasks and dialogues.Source: §1 · p. 2
  • OdysseyBench evaluations challenge state-of-the-art agents and provide a more accurate assessment of their capabilities in real-world contexts.The authors also analyze dialogue storage formats, reporting that semantic compression and coherent aggregation are essential for effective multi-step reasoning and agent performance.Source: §1 · p. 3

2 RELATED WORK

Prior agent benchmarks and synthetic-task approaches leave gaps in sustained, context-rich workflows. OdysseyBench targets realistic office scenarios, while HOMERAGENTS provides an automated way to construct them.

2 RELATED WORK

  • Existing web and office benchmarks primarily assess atomic performance, lacking mechanisms to evaluate extended interaction with complex environments.The related work identifies planning, long-term information retrieval, and execution as capabilities that robust assessment must address.Source: §2 · p. 3
  • Prior synthetic benchmark efforts use LLMs for task and trajectory generation, but often focus on web activities or simple interactions rather than extensive tool use and multi-step reasoning.Human-annotated benchmark creation is also described as costly and limiting diversity.Source: §2 · p. 3
  • OdysseyBench uses HOMERAGENTS to assess context curation in realistic office tasks requiring agents to infer intent, extract dialogue-history information, and assemble workflows.HOMERAGENTS+ builds on OfficeBench task descriptions, whereas HOMERAGENTS-NEO creates tasks and dialogues from scratch in realistic environments.Source: §2 · p. 3, Figure 2 · p. 4

3 METHODOLOGY

The methodology section introduces HOMERAGENTS and its two components, then describes OdysseyBench and its dataset analysis, quality controls, and human evaluation.

3 METHODOLOGY

  • The methodology is organized around HOMERAGENTS generation components followed by OdysseyBench dataset analysis, quality control, and human evaluation.It covers HOMERAGENTS+ and HOMERAGENTS-NEO as separate components.Source: §3 · p. 4

3.1 HOMERAGENTS: AUTOMATING BENCHMARK CREATION

HOMERAGENTS automates benchmark creation through two complementary approaches: iterative enrichment of existing tasks and generation of new tasks in application environments. Its pipeline produces contextualized tasks and multi-day dialogues, with simulated execution supporting generation at scale.

3.1 HOMERAGENTS: AUTOMATING BENCHMARK CREATION

  • HOMERAGENTS automates benchmark creation with HOMERAGENTS+ for enriching existing tasks and HOMERAGENTS-NEO for generating new tasks.The framework aims to produce scalable, diverse, contextually grounded tasks.Source: §3.1 · p. 4, §3.1.2 · p. 6

3.1.1 HOMERAGENTS+: STANDING ON THE SHOULDERS OF OfficeBench

  • HOMERAGENTS+ transforms OfficeBench task descriptions into long-horizon scenarios through a generator-verifier loop that refines intent and dialogue using feedback.The verifier assesses realism, task alignment, and contextual coherence; iterations stop upon approval or at the Nmax limit.Source: §3.1.1 · p. 5

3.1.2 HOMERAGENTS-NEO: SCALING UP THE BENCHMARK CREATION

  • HOMERAGENTS-NEO plans generation, explores application environments, creates task specifications, and then generates corresponding dialogues.Specialized surfers gather contextual information from applications to ground generated tasks in realistic capabilities and workflows.Source: §3.1.2 · pp. 5–6
  • NEO task specifications include descriptions, intent, subtask instructions, and evaluation criteria; its dialogues simulate multi-day interactions for each subtask and may include irrelevant conversation.The phases are designed to maintain coherence between tasks and dialogues while enabling diverse, contextually grounded benchmark creation.Source: §3.1.2 · p. 6

3.1.3 IMPLEMENTATION DETAILS

  • The implementation uses GPT-4.1, sets the maximum HOMERAGENTS+ iterations to 5, and generates at least five days of dialogue for HOMERAGENTS-NEO.NEO is based on Magentic-One; during dialogue generation, assistant task execution is simulated rather than actually performed.Source: §3.1.3 · p. 6

3.2 OdysseyBench: LONG-HORIZON WORKFLOW BENCHMARK

OdysseyBench evaluates long-horizon, multi-application workflows in a Docker environment and documents tasks through extended dialogues. Its dataset analysis and quality checks report task complexity, generated-data validation, and human task performance.

3.2.1 EVALUATION

  • Agents work in a Docker environment with pre-installed applications and task files; customized exact-match, fuzzy-match, and execution-based checks evaluate task completion.The evaluation reports pass rate as the percentage of tasks completed successfully.Source: §3.2.1 · pp. 6–7

3.2.2 DATASET ANALYSIS

  • OdysseyBench contains 602 tasks across one-, two-, and three-app categories (153, 166, and 283), with most tasks requiring 3–15 execution turns.OdysseyBench-Neo averages 5,169.9 tokens per conversation versus 3,468.9 for OdysseyBench+; Figure 3 also maps actions, objects, and applications.Source: §3.2.2 · pp. 7–8, Table 1 · p. 7, Figure 3 · p. 7

3.2.3 QUALITY CONTROL

  • Automated checks filter for measurable criteria and task-dialogue consistency; five GPT-4.1 judges assess completeness and soundness, with a 70% minimum pass rate reflecting data quality.Three native English-speaking annotators also review completeness, appropriateness, and coherence, retaining quality-compliant tasks that the LLM cannot solve.Source: §3.2.3 · p. 8

3.2.4 HUMAN EVALUATION

  • Human annotators achieved over 90% performance on a sampled subset of tasks using the same productivity applications as agents, indicating solvability and coherence.Source: Table 3 · p. 9, §3.2.4 · p. 9

4 EXPERIMENTAL SETUP

The experiments evaluate agents with full dialogue histories or retrieval-augmented context, using pass rate to measure task completion. The setup includes proprietary and open-weight language models, with GPT-4o RAG results organized by context configuration.

  • Long-context evaluation provides the entire dialogue history, while RAG retrieves relevant dialogue context using embeddings in raw or summarized formats.Raw context is stored by session or utterance; summarized context is stored by session or coherent chunk.Source: §4 · p. 9
  • The experiments use pass rate—the percentage of successful task completions—and evaluate proprietary and open-weight LLMs.The listed models include o3, GPT-4o, GPT-4.1, GPT-5, DeepSeek-R1, and Qwen3-32b.Source: §4 · p. 9
  • GPT-4o RAG results cover OdysseyBench+ and OdysseyBench-Neo, with tables describing storage format, retrieval granularity, top-k documents, and retrieved-token counts.Source: Table 5 · p. 10, Table 6 · p. 10

5 EXPERIMENTAL RESULTS

Evaluations examine how application count, context storage, and retrieval choices affect agent performance. Summarized context performs strongly with lower token use, while added complexity and excessive retrieval can hurt results.

  • As tasks involve more applications, performance drops across all models; on OdysseyBench+, o3 falls from 72.83 on single-app to 30.36 on three-app tasks.On OdysseyBench-Neo, o3 declines from 68.33 to 59.06, while GPT-4o declines from 75.00 to 45.61.Source: §5 · p. 10
  • The long-context baseline scores 33.67 on OdysseyBench+ using 8000 tokens and 51.99 on OdysseyBench-Neo using 6700 tokens; utterance-level retrieval peaks at 29.41 and 730 tokens, and 38.74 and 915 tokens, respectively.Utterance-level retrieval beats the baseline on single-app and three-app OdysseyBench+ tasks but underperforms it on OdysseyBench-Neo.Source: §5 · p. 11
  • 53.64 on OdysseyBench-Neo with session-level summaries and one third of the token usage; chunk-level summaries reach 56.29 with less than 20% of tokens.Chunk-level summaries also introduce negligible execution-step overhead, and can reduce the number of steps required.Source: §5 · p. 11, Table 7 · p. 11
  • Increasing top-k from 25 to 50 lowers OdysseyBench-Neo performance from 56.29 to 54.97; summaries maintain 2-3x higher performance than raw utterances on three-app tasks.The authors suggest that excessive retrieval adds noise and that fragmentation can undermine context integrity.Source: §5 · p. 12

6 CASE STUDY

Trace analysis identifies failures in retrieving dialogue-referenced files, completing requested actions, choosing tools, and planning task sequences. File creation and writing errors are especially common for docx and xlsx.

  • Agents fail by overlooking required files or actions, including missing a referenced training manual or omitting a requested relationship analysis.The missed action left no content for the save operation.Source: §6 · p. 12
  • Incorrect tool calls and incoherent plans also cause failures, such as creating a PDF with the PDF tool instead of Word or writing before reading source PDFs.Source: §6 · p. 12
  • Most errors involve file creation or writing, particularly for docx and xlsx formats.Source: §6 · p. 12

7 CONCLUSION

OdysseyBench evaluates agents on long-horizon workflows across office applications, while HOMERAGENTS automates benchmark creation through two complementary approaches. Evaluations reveal performance gaps relative to atomic task benchmarks and highlight challenges in realistic productivity settings.

  • OdysseyBench addresses the limitation of atomic task benchmarks by evaluating language agents on long-horizon workflows across diverse office applications.Source: §7 · p. 12
  • HOMERAGENTS+ transforms existing atomic tasks into contextually rich scenarios for OdysseyBench+, while HOMERAGENTS-NEO generates new complex tasks for OdysseyBench-Neo.Source: §7 · p. 12
  • Evaluations reveal substantial performance gaps between state-of-the-art agents on OdysseyBench compared with atomic tasks, underscoring contextual dependencies and multi-interaction coordination challenges.Source: §7 · p. 12

A CRITERIA OF VERIFIER AGENT

The verifier criteria specify dialogue length, information preservation, task splitting, conversational protocol, and content constraints for HOMERAGENTS+ generated dialogues.

B PROMPTS FOR AGENTS

The appendix presents prompts that specify how HomerAgents generates tasks, evaluation criteria, memories, and long dialogue histories. The instructions emphasize precise, distributed task information and evaluations aligned with the requested office actions.

B PROMPTS FOR AGENTS

B.1 PROMPTS FOR HOMERAGENTS+

  • The HOMERAGENTS+ generator prompt requires over 100 turns across at least five days, with each subtask appearing once and the final subtask reserved for the final user turn.It also instructs the generator to avoid altering artifacts unless required and to keep agent replies under 180 words.Source: §B.1 · p. 17

B.2.1 TASK GENERATOR PROMPT

  • Task-generation guidance says descriptions should cover 1–5 subtasks, and evaluation criteria should follow the supplied examples rather than introduce new criteria.The task description, evaluation criteria, and ground-truth memory are logged using TaskTool with specified argument structures.Source: §B.2 · p. 18, §B.2.1 · p. 19

B.2 PROMPTS FOR HOMERAGENTS-NEO

  • Office-specific evaluation rules favor selected Excel cell checks over exact-file matching and restrict Calendar checks to supported event fields.The prompts also specify handling for Word keyword checks, question-answering output, and Email evaluation formats.Source: §B.2 · p. 18
  • Task facts and action items must be precise, while ground-truth memory adds necessary details without revealing the task solution.The query sentence omits some details so the agent must retrieve them from memory.Source: §B.2 · p. 18
  • Dialogue-generation instructions distribute exact memory pieces across five days, avoid duplication, and allow unrelated chitchat only when it adds no task-solving noise.The prompt specifies around 100–120 turns per day and permits chitchat after all memories are covered.Source: §B.2 · p. 19

B.2.2 ORCHESTRATOR PROMPT

  • The orchestrator reads available files, proposes task specifications and memory, logs them, then expands the memory into dialogue histories.The task description, evaluation criteria, memory, and query sentence are hidden from the evaluated agent.Source: §B.2.2 · p. 20

B.2.3 CHAT GENERATOR PROMPT

  • The orchestrator prompt directs a separate chat-generator assistant to expand ground-truth memory into histories where the information is scattered.The dedicated dialogue-generator prompt supplies the task, subtask instruction, and orchestrator instruction as inputs.Source: §B.2.3 · p. 21

Related papers

Loading 2508.09124v1…