Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models

1 Introduction

• Many multi-hop benchmarks require only a few retrieval or inference steps and do not test following long procedures across external manuals.Real-world coding and sentencing can involve dozens of interdependent steps, with earlier choices affecting what follows.Source: §1 · pp. 1–2
• TAM is a benchmark for long-horizon procedural reasoning over authoritative manuals with tens of thousands of rules and interdependent decisions.Correctness depends on executing the entire procedure, rather than retrieving or combining a small set of facts.Source: §1 · p. 2
• The benchmark covers ICD-10-CM coding and federal sentencing, using 1,000 coding cases and 200 human-reviewed sentencing cases.The coding cases are sampled from real encounters; the legal benchmark uses annual Title 18 archives and USSG manuals spanning 2021–2025.Source: §1 · p. 2
• ICD-10-CM exact match stays at 1% or below, while federal-sentencing exact match reaches at most 15.5% across evaluated GPT-5 baselines.The baselines include single-pass RAG, agentic RAG, ReAct-style tool use, and an agent-harness baseline.Source: §1 · p. 2

2.1 Task Definition

• A TAM instance is a case description, a manual, and the exact output obtained by correctly applying that manual.A system iteratively updates its accumulated context and partial output, and succeeds only when it reaches the exact output through an admissible trajectory.Source: §2.1 · p. 2
• TAM is a constrained search problem because locally plausible actions can foreclose a globally valid answer, requiring systems to track obligations and revise or backtrack.Retrieval exposes relevant rules but does not ensure that an evolving output is complete, correctly ordered, and globally valid.Source: §2.1 · p. 2

2.2 Illustrative Example: ICD-10-CM Clinical Coding

• ICD-10-CM coding maps a discharge-summary narrative through term lookup, code hierarchy, and guidelines to an exact reportable code set with sequencing information.The manual supplies inclusion and exclusion criteria, sequencing rules, and requirements for additional codes.Source: §2.2 · p. 2
• Across the two domains, tasks can repeat lookup loops before aggregating results, with sentencing additionally involving count-level subtotals and grouping adjustments.ICD-10-CM coding aggregates, de-duplicates, and orders its final code set; sentencing aggregates across counts, cross-references, and later adjustments.Source: Figure 1 · p. 3
• A coding example shows why diagnosis-like phrases cannot be coded independently: the system must distinguish active billable conditions from background and order required codes.Coding only the cancer can miss a required encounter-for-chemotherapy code, while coding every diagnosis-like phrase creates an overinclusive list.Source: §2.2 · p. 3

3.1 Retrieval and short-horizon reasoning benchmarks

• Retrieval and multi-hop benchmarks such as HotpotQA [35] and MuSiQue typically combine a small number of facts rather than evaluate long decision trajectories.Their emphasis on retrieval relevance or final-answer correctness offers limited visibility into failures that emerge over longer, stateful reasoning.Source: §3.1 · pp. 3–4

3.2 Procedural and constraint-based reasoning benchmarks

• Interactive and constraint-based benchmarks study environment interaction, abstract feasibility, or local transitions, not faithful execution of a fixed cross-referenced manual.TAM instead requires dependent decisions whose admissible actions and outputs are specified by distributed source documents.Source: §3.2 · p. 4

3.3 Domain-specific rule-governed tasks

• Domain-specific coding and legal benchmarks often predict outputs directly, abstracting away the procedural steps practitioners follow.For ICD coding, those steps include navigating manuals, applying exclusions, checking specificity, and enforcing coding rules.Source: §3.3 · p. 4

3.4 Retrieval-augmented and agentic approaches

• RAG, agentic retrieval, ReAct-style reasoning, and agent harnesses provide retrieval or tool interaction, but are not designed to preserve consistency over long manual-governed trajectories.The paper evaluates these paradigms as prompting-based reference baselines.Source: §3.4 · p. 4

3.5 Positioning of TAM

• TAM makes manual-governed procedural execution explicit, distinguishing it from retrieval, short-horizon reasoning, constraint satisfaction, and domain-specific prediction.Its tasks use case inputs and manuals directly rather than relying on large task-specific training sets.Source: §3.5 · p. 4

4.1 ICD-10-CM Clinical Coding

• ICD-10-CM coding maps discharge summaries to exact diagnosis-code sets through repeated Index lookup, Tabular validation, and guideline checks, rather than a single-pass span-labeling task.The Index proposes candidate paths, the Tabular List determines validity and specificity, and guidelines impose coding rules.Source: §4.1 · pp. 4–5
• The 2019 ICD resources span 120 guideline pages, a 1,304-page Index with 7,995 main terms, and a 1,942-page Tabular List with 34,502 leaf codes.An Index term can trigger a Tabular specificity decision that activates a guideline rule and changes the final code set.Source: §4.1 · p. 5
• From MIMIC-IV, the authors filtered discharge cases for 2019 code consistency and usable notes, then sampled 1,000 cases from 38,332 eligible admissions.Coder-assigned codes serve as evaluation targets; prior model exposure to some public records cannot be ruled out.Source: §4.1 · p. 5
• The ICD evaluation sample covers 2,555 unique output codes and averages 12.85 codes per case, with median 12, p90 23, and p99 36.It includes 620 cases with 10 or more codes and tracks the full cohort's code-count distribution.Source: §4.1 · p. 5

4.2 Federal Sentencing Guideline Calculation

• Federal sentencing targets the exact total offense level using year-aligned Title-18 and USSG sources, with guideline selection, adjustments, and multi-count grouping creating a long reasoning path.The benchmark spans five annual source sets from 2021 through 2025; errors can arise at statute selection, Appendix A mapping, characteristic application, or recomputation after grouping.Source: §4.2 · pp. 5–6
• The legal benchmark retains 200 human-verified cases; each averages 65 extracted facts across 3.07 documents, and offense levels range from 2 to 43.Curation prioritized source sufficiency and label verification over raw dataset size.Source: §4.2 · p. 6

5.1 Retrieval-Augmented Generation Baselines

• Single-pass RAG retrieves manual chunks once for a one-shot answer, while agentic RAG can retrieve again over several rounds before answering.Both use domain-aware chunking and source metadata to make retrieval workable.Source: §5.1 · p. 6
• RAG chunks preserve manual structure: ICD sections, code families, and Index terms, while legal chunks group statute sections or USSG rules with related notes.Legal retrieval can be filtered by manual year.Source: §5.1 · p. 6

5.2 ReAct (Tool-Using Baseline)

• ReAct replaces fixed retrieval with a restricted manual-browsing API, requiring the model to choose tools, inspect results, connect components, and decide when evidence suffices.The ICD and sentencing agents receive eight and seven domain-specific browsing tools, respectively.Source: §5.2 · pp. 6–7

5.3 Agent Harness Baseline

• The agent-harness baseline divides each task among specialized workers that pass concise reports, while reusing the ReAct baseline's manual-browsing tools.Workers handle staged coding or sentencing subtasks with local context and workflow instructions.Source: §5.3 · p. 7

6.1 Evaluation Protocol

• Each baseline receives the same fixed ICD and sentencing cases, case inputs, and underlying manuals; only the manual-access approach differs.Targets are the final diagnosis-code output or the final offense level, with one prediction per case.Source: §6.1 · p. 7

6.2 Metrics

• Exact match counts a case as successful only when its complete manual-governed final output exactly equals the reference.ICD code-set order is ignored, while the sentencing offense level must match exactly.Source: §6.2 · p. 7
• Partial-credit metrics report ICD micro-averaged precision and recall, primary-diagnosis accuracy, and sentencing mean absolute error to quantify locally correct progress.These metrics complement exact match by distinguishing final-answer success from recovered structure along the way.Source: §6.2 · p. 7

7.1 Main results

• Agentic RAG paired 37% recall and 45.7% primary diagnosis accuracy on ICD-10-CM with only 1% exact match.Coder disagreement can lower the exact-match ceiling, but the authors judge label noise alone unlikely to explain the gap.Source: §7.1 · p. 7
• ICD-10-CM exact match stayed at 1% or below across baselines, while federal-sentencing exact match reached at most 15.5%.Source: §7.1 · p. 7, Table 2 · p. 8

7.2 Failure Modes

• Global inconsistency dominated sampled ReAct failures, appearing in 38 of 50 trajectories, including 22 of 25 ICD and 16 of 25 sentencing cases.The analysis uses 50 saved trajectories as a descriptive view, not a population-frequency estimate; errors often involved a wrong anchor or an unchecked aggregation step.Source: §7.2 · p. 8
• Incomplete execution appeared in 7 of 50 trajectories, including 3 of 25 ICD and 4 of 25 sentencing cases, with procedures stopping before completion.Source: §7.2 · p. 8
• Missing required elements appeared in 5 of 50 trajectories, all in sentencing, often through missed reductions or late-step constraints.Source: §7.2 · p. 8

• Potential improvements include stronger state tracking, explicit verification before finalization, and revision or backtracking after early mistakes.Source: §8 · p. 8
• Future evaluation could include domain-specialized systems, fine-tuned models, and future agentic methods as TAM is adopted.Source: §8 · p. 8
• The proposed direction is to treat reasoning as a structured, stateful process rather than a sequence of loosely connected steps.Source: §8 · p. 8

• TAM covers two domains with well-defined manuals and exact outputs, enabling precise evaluation within that scope.Source: §9 · p. 8
• Tasks with looser procedures or heavier reliance on human judgment are outside the benchmark's stated scope.Source: §9 · p. 8
• The evaluation uses representative prompting-based baselines, not an exhaustive set of specialized or trained systems.Source: §9 · p. 8

• TAM identifies a gap between short-horizon benchmarks and real-world tasks requiring step-by-step use of large application manuals.Source: §10 · p. 8
• Across ICD-10-CM coding and federal sentencing, retrieval-based and tool-using approaches often make plausible local decisions but fail at full-procedure correctness.Source: §10 · p. 8
• The conclusion emphasizes maintaining consistency, satisfying constraints, and completing the manual-governed process end to end, not merely accessing relevant information.Source: §10 · p. 8

A Appendix Overview

• The appendix overview identifies prompt excerpts, tool inventories, and condensed case traces as supplementary materials.The traces illustrate end-to-end manual-following in one ICD-10-CM case and one federal sentencing case.Source: A Appendix Overview · p. 8

B Prompt Templates and Agent Instructions

• Prompt excerpts show the task framing, workflow instructions, grounding rules, and expected outputs, with omitted boilerplate marked explicitly.The excerpts are clipped stretches of the prompts actually seen by each method.Source: B Prompt Templates and Agent… · p. 8

B.1 RAG and Agentic-RAG Prompts

• RAG prompts constrain predictions to retrieved context, while agentic variants use broad first-pass and focused searches and return supported outputs with rationale.ICD prompts request supported diagnosis codes; legal prompts prioritize offense level and permit additional fields when context supports them.Source: §B.1 · p. 9

B.2 ReAct/Manual-Navigation Prompts

• ReAct instructions require manual-specific navigation, including Index-to-Tabular confirmation for coding and statute-to-guideline mapping before sentencing adjustments.Both workflows direct the agent to return null or omit unsupported conclusions rather than guess.Source: §B.2 · p. 9

B.3 Agent-Harness Overview

• The agent harness divides work among staged workers that pass structured reports, unlike ReAct’s single growing conversation.Workers receive only the case materials, manual text, and prior notes needed for their step.Source: §B.3 · p. 9

B.4 Construction-Time Review Prompts

• Construction-time review separates offense-level verification from high-recall extraction of grounded case facts.The verification artifact keeps calculation steps, docket facts, guideline support, and justification separately legible; fact extraction excludes rules and arithmetic.Source: §B.4 · p. 9

B.5 ReAct Tool Summary

• The ReAct baseline has eight ICD-10-CM tools and seven federal-sentencing tools for browsing each domain’s manuals.ICD tools cover the Alphabetic Index, Tabular List, and Official Guidelines; sentencing tools cover Title 18, Appendix A, and the USSG.Source: §B.5 · pp. 9–10

C Additional Examples and Case Studies

• The appendix includes two condensed traces from real benchmark cases that retain key procedural decisions and governing manual paths.Repetitive lookup steps are omitted.Source: C Additional Examples and Case Studies · p. 10

C.1 ICD-10-CM Case Study

• The ICD-10-CM case’s reference code set contains 17 diagnosis codes spanning cancer, complications, comorbidities, and status-history coding.The case includes opioid-induced constipation alongside metastatic prostate cancer and other conditions.Source: §C.1 · p. 10
• ICD coding follows separate Index and Tabular paths; opioid-related constipation opens an adverse-effect code path, and neoplasm notes can impose sequencing or dependency constraints.The example emphasizes that identifying diagnoses alone may not complete the coding path.Source: §C.1 · p. 10
• The coding workflow resolves status/history items and assembles the retained codes only after candidate paths are checked against the manual.The final output is checked for exactly 17 reportable codes.Source: §C.1 · p. 10

C.2 Federal Sentencing Case Study

• The sentencing trace fixes one manual edition, calculates separate bank-fraud and marijuana group subtotals, and maps two units to a two-level increase.The aggravated identity-theft count is excluded from the grouped offense-level calculation.Source: §C.2 · p. 10
• The sentencing case reports a final grouped offense level of 29 after the subtotal rises from 30 to 32 and a three-level acceptance reduction applies.The two substantive count groups are combined under grouping rules; the aggravated identity-theft count remains outside that arithmetic.Source: §C.2 · p. 10