Source-linked AI summary
$τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge
Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres
TL;DR
Existing benchmarks rarely evaluate retrieval, policy reasoning, and tool use together over private, unstructured knowledge during long-horizon conversations. τ-Knowledge introduces τ-Banking, a fintech benchmark with roughly 700 documents and discoverable tools, and finds that frontier agents achieve only 25.52% pass^1 at best, with reliability degrading over repeated trials. The benchmark exposes bottlenecks in retrieval, reasoning, and interaction efficiency while supporting controlled study of knowledge-grounded agents.
Problem
Existing evaluations usually isolate retrieval or tool use instead of testing agents over private, unstructured knowledge in ambiguous, evolving user interactions.
Method
τ-Knowledge extends τ-Bench with τ-Banking, where agents retrieve natural-language knowledge, discover documented tools, and produce verifiable database state changes.
Results
25.52% pass^1 is the best observed result, and reliability falls to at most 13.40% pass^4 across tested frontier models and retrieval configurations.
Takeaways & Limitations
τ-Knowledge exposes bottlenecks requiring reasoning over complex policies, cross-document dependencies, and evolving database state, while motivating solution-efficiency evaluation.
Takeaways & Limitations
User simulations omit variation in expertise, colloquial or localized language, and grammatically imperfect or ambiguous inputs.
Abstract
from arXiv · showhide
Conversational agents are increasingly deployed in knowledge-intensive settings, where correct behavior depends on retrieving and applying domain-specific knowledge from large, proprietary, and unstructured corpora during live interactions with users. Yet most existing benchmarks evaluate retrieval or tool use independently of each other, creating a gap in realistic, fully agentic evaluation over unstructured data in long-horizon interactions. We introduce $τ$-Knowledge, an extension of $τ$-Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs to produce verifiable, policy-compliant state changes. Our new domain, $τ$-Banking, models realistic fintech customer support workflows in which agents must navigate roughly 700 interconnected knowledge documents while executing tool-mediated account updates. Across embedding-based retrieval and terminal-based search, even frontier models with high reasoning budgets achieve only $\sim$25.5% pass^1, with reliability degrading sharply over repeated trials. Agents struggle to retrieve the correct documents from densely interlinked knowledge bases and to reason accurately over complex internal policies. Overall, $τ$-Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments.
1. Introduction
τ-Knowledge addresses the gap between isolated retrieval or tool-use benchmarks and realistic agentic interactions over private, unstructured knowledge. It introduces τ-Banking to evaluate knowledge retrieval, policy reasoning, tool discovery, and state-changing actions in long-horizon support tasks.
- Existing benchmarks usually evaluate retrieval or tool use separately, without requiring reasoning over private knowledge bases and ambiguous user interactions.
- τ-Banking contains roughly 700 natural-language documents covering product details, procedural policies, and tool documentation for fintech support workflows.Tasks require agents to find and apply this information while coordinating knowledge-base evidence with tool outputs.
- 25.52% pass^1 is the best observed result across tested frontier models and retrieval configurations, while reliability falls to at most 13.40% pass^4.In a golden-retriever setting, the strongest model reaches only 39.69% pass^1, showing that retrieval is not the sole bottleneck.
- GPT-5.2 with terminal-based search achieves performance comparable to Claude-4.5-Opus but uses approximately 1.7× more tokens, 2.3× more shell commands, and 9× more time.Brittle, assumption-driven searches can cause excessive interaction turns and tool calls when user intent is underspecified.
- Freeform terminal search can improve performance for strong reasoning models, whereas dense retrieval generally produces faster turns and fewer searches.Weaker or noisier retrieval is often compensated for by more search and tool usage, increasing latency and reducing interaction efficiency.
- τ-Knowledge motivates evaluating solution efficiency alongside task success, including time, tool calls, and conversational backtracking in human-facing deployments.The benchmark provides a controlled testbed for studying how search, reasoning, and efficiency interact.
2. Related Work
Prior benchmarks separately study tool use, retrieval relevance, or simulated interaction. τ-Knowledge combines these concerns by requiring agents to acquire procedural knowledge during partially observable conversations.
- Tool-use benchmarks typically provide fully specified interfaces and evaluate agents without explicit conversational dynamics.
- τ-Bench adds extended, goal-oriented conversations with partially observable environments, but generally provides tools and procedures to agents in advance.
- Embedding-based retrieval benchmarks measure query–document relevance but do not capture effects on decision-making, tool use, or long-horizon task success.
- Human–agent interaction research uses behavioral simulators and user tools to evaluate interactive instruction following without exposing privileged future-state information.
3. τ-Knowledge
τ-Banking models customer-support tasks as partially observable interactions where agents retrieve knowledge, reason over constraints, discover tools, and modify a banking database through documented operations.
- Each task is a multi-turn agent–user interaction requiring relevant knowledge retrieval, policy reasoning, and tool invocations that modify an underlying banking database.
- The shared state combines banking records and conversation history, while agents infer database state only from tool outputs and user messages.
- Task rewards depend on whether retrieved knowledge, tool invocations, and interactions produce the specified final database state.
- The knowledge base contains 698 documents spanning 71 topics across 21 product categories, including product specifications and internal agent protocols.Its rules govern actions such as card replacement, account closure, referrals, identity verification, permissions, and blocking.
- Discoverable tools are initially unavailable and must be found in the knowledge base before agents can invoke state-changing operations.Discovery expands the agent’s action space based on its knowledge state.
- Flow-based user simulation uses conditional rules to steer conversations toward evaluation-critical junctures and edge cases.
4. Benchmark Construction
The benchmark is constructed by expanding structured fintech specifications into linked natural-language documents, then co-constructing and auditing tasks with human and LLM assistance.
- Stage 1: Structured Database Generation: LLMs first generate categories, features, and typed variables to form a structured database representing feasible product configurations under domain constraints.
- Stage 2: Conversion from Structured to Unstructured Documents: The structured records are converted into natural-language documents by assigning variables to plausible titles and generating FAQs or policy articles.
- Stage 3: Task and Database Creation: Tasks and databases are co-constructed with LLM assistance around fintech workflows such as replacement cards, transaction disputes, and account recommendations.
- Stage 4: Human-in-the-Loop Refinement: Human-in-the-loop refinement produces roughly 700 documents and 200,000 tokens across dozens of topics and 21 product categories.
- Stage 5: Review: Two independent reviewers audit each task’s final database state, complete and minimal gold documents, and successful executability using documented tools.
- Advantages of This Protocol: The pipeline scales knowledge-base creation with LLM automation while using targeted human refinement and structured representations to reduce unintended collisions.
5. Experiments
The experiments evaluate frontier agents across multiple retrieval and search strategies, including dense, sparse, terminal-based, and gold-document settings. They measure task reliability with pass^k while tracking model configurations, context management, and interaction costs.
- Retrieval and Search: Agents are evaluated with dense retrieval, sparse BM25 retrieval, terminal-based exploration, and a golden-retriever condition that supplies task-critical documents directly.The benchmark also supports arbitrary search strategies and uses gold documents to isolate knowledge utilization from retrieval.
- Context Management: A lightweight truncation policy evicts the oldest retrieval outputs when conversations exceed the model context limit.The policy removes one-quarter of accumulated retrieval outputs to limit context overflow.
- Models: The evaluation includes flagship and lower-latency variants of frontier models selected to compare reasoning performance with faster interaction.Lower-latency variants use smaller sibling models or reduced reasoning effort when available.
- Metrics: Performance is measured with pass^k, the probability that a task succeeds in all k independent trials, with k ≤4.This metric captures reliability across repeated task executions rather than a single completion.
6. Results
τ-Knowledge is difficult for current agents: the best standard configuration reaches only 25.52% pass^1, and reliability declines sharply across repeated trials. Search strategy creates a performance–efficiency tradeoff, while low gold-document performance shows that failures also arise from reasoning over retrieved information.
- Overall Performance: 25.52% pass^1 is the best standard result, achieved by GPT-5.2 (high) with terminal use; the best pass^4 reaches only 13.40%.With gold documents supplied directly, Claude-4.5-Opus reaches 39.69% pass^1 and 26.80% pass^4.
- Efficiency: Claude-4.5-Opus averages 8.7 KB_search calls per task versus 18.5 for GPT-5.2 with high reasoning, despite comparable performance.The results show substantial efficiency differences across models and retrieval configurations.
- Retrieval Comparisons: Terminal use outperforms dense and sparse retrieval on average, but its gains are concentrated in recent high-reasoning models rather than appearing uniformly across model classes.The improvement is statistically significant when averaged across models and is not accompanied by uniformly higher document recall.
- Efficiency: 14.5 searches per task for terminal grep versus 9.9–10.1 for dense retrieval increases median turn time by 6.6 seconds.At the 90th percentile, terminal use is roughly 10% to 85% slower than the next-slowest strategy depending on the model.
- Efficiency: Weaker or noisier retrieval is often compensated for by more searches and tool interactions, preserving success rates at the cost of longer latency.This tradeoff makes solution efficiency relevant alongside final task success in human-facing deployments.
- Failure Sources: Golden-retriever results remain low, indicating that success depends on both retrieving knowledge and correctly interpreting and applying it.Document recall also varies substantially across model–retriever pairings, such as 57% for text-embedding-3-large with Opus versus 28% with GPT-5.2 without reasoning.
- Ablations: ∼2% average pass^1 in the no-knowledge setup confirms that the tasks require retrieved information, while full-context performance peaks at only ∼12%.The nonzero no-knowledge score comes from two grounding-check tasks, and the full-context result indicates that targeted retrieval remains beneficial amid realistic document noise.
7. Qualitative Analysis
The qualitative analysis clusters unsuccessful trajectories into recurring reasoning, planning, verification, and search failures. The most common highlighted patterns involve assumption-driven retrieval and errors caused by interdependent products and policies.
- Analysis Approach: The analysis uses unsuccessful trajectories to distinguish failures arising from agent reasoning, tool use, or user simulation.Failure explanations are generated from task specifications and trajectories, then clustered into common patterns.
- Complex Interdependencies between Financial Offerings: ∼14.5% of failures arise from complex interdependencies between financial offerings that require multi-hop reasoning across documents.Agents can select an inferior product pairing when advertised bonuses must be compared with higher benefits elsewhere.
- Failure to Respect Implicit Subtask Ordering: ∼5% of failures involve violating implicit subtask ordering when one requested action can block or invalidate another.For disputes and credit-limit increases, policy requires resolving the dispute before submitting the increase request.
- Overtrusting User Assertions: ∼4% of failures result from overtrusting user assertions instead of verifying them against the system state.Agents may apply credits after a user claims disputes were approved even though the system still marks them under review.
- Search Inefficiency and Making Assumptions: ∼23% of failures involve search inefficiency and unwarranted assumptions when user requests are underspecified.Agents often commit to an interpretation instead of clarifying the request or performing targeted retrieval.
8. Conclusion
τ-Knowledge evaluates conversational agents on retrieving, reasoning over, and applying unstructured knowledge, finding that even top models struggle with reliable performance. It also identifies solution efficiency as important for human-facing deployment and notes several scope limitations.
- τ-Knowledge evaluates tasks requiring retrieval, reasoning, and application of unstructured knowledge in conversational settings.
- Even top LLMs with state-of-the-art retrieval configurations struggle to perform reliably on these tasks.
- Simplified user simulations omit variation in expertise, colloquial or localized language, and grammatically imperfect or ambiguous inputs.
- Under one- or few-shot search constraints, retrieval configuration is expected to play a more decisive role in performance.
10. Impact Statement
The impact statement presents τ-Banking as a realistic environment where agents must retrieve policy and product information, reason over constraints, and execute tool-mediated actions. Its examples show that documentation access governs available operations and that tasks require coordinated, multi-step knowledge use.
- The τ-Banking knowledge base contains 698 documents covering product information, policies, and tool documentation that agents must retrieve during tasks.
- Policy documents encode step-by-step procedures, conditional branches, and references to the tools required for customer actions.
- Discoverable tools must first be located in the knowledge base and unlocked before agents can invoke the corresponding state-changing operations.
- Product documents supply features, fees, rates, eligibility requirements, and other details needed to answer customer questions and perform workflows.
- τ-Banking tasks jointly require knowledge acquisition, constraint reasoning, and multi-step execution rather than retrieval or tool use in isolation.
B.2. Sample 2: Procedural Policy Execution
The sample procedural-policy task requires an agent to follow a documented credit-card retention protocol and a separate account-reorganization workflow with ordering dependencies. Success depends on discovering the relevant tools and executing policy-constrained actions in the correct sequence.
- Credit Card Retention Protocol: The retention protocol requires checking disputes, replacement cards, account age, and balance before proceeding with closure.
- Credit Card Retention Protocol: A prior closure-reason record requires skipping retention offers and proceeding directly to closure processing.
- Credit Card Retention Protocol: The sample agent first pays a $75 balance, then checks prior retention attempts, logs the annual-fee reason, and applies a fee waiver.
- Operation Sequencing with Dependencies: The reorganization task contains four hidden order dependencies that require opening accounts and transferring funds before closing existing accounts.
- Operation Sequencing with Dependencies: Closing an account first can block business-checking eligibility, while closing Evergreen checking leaves only a 12-day-old account against a 14-day tenure requirement.
C. Sample Conversation Trace
The trace illustrates an agent interacting with a simulated user, retrieval system, and discoverable tools across a lost-wallet workflow. The prompts require searching documentation, verifying identity when accessing customer data, and unlocking tools before use.
- Conversation Trace: The trace follows a customer who lost a wallet containing two debit cards and a credit card, with omitted tool outputs shown through searches and calls.
- Conversation Trace: The agent searches the knowledge base and unlocks or calls tools for transactions, card unfreezing, card closure, and replacement ordering.
- Retrieval Configurations: Dense and sparse retrieval configurations provide a knowledge-base search tool, while terminal use enables filesystem exploration with standard Unix commands.
- Agent Discoverable Tools: Discoverable agent tools must be found in the knowledge base, unlocked with exact names, and then called with required arguments.
- Identity Verification: Customer-information access or modification requires identity verification, followed by verification logging, before disclosure or action.
F. Retrieval Hyperparameter Study
The study tests retrieval and terminal-use choices through ablations of reranking, grep access, retrieval-set size, and write commands. Most changes do not significantly improve task outcomes, motivating simpler configurations.
- Retrieval configurations: Four retrieval configurations combine reranking and grep access across dense and sparse retrievers.The configurations cover retrieval-only, reranked retrieval, retrieval with grep, and retrieval with both grep and reranking.
- Reranking: Reranking does not significantly change pass^1 for any retriever, although some Qwen configurations improve Action Recall.The main evaluation omits reranking to reduce runtime and avoid additional task-duration confounders.
- Grep access: Grep does not significantly change pass^1 for any retriever and improves Action Recall significantly only for Qwen3-embedding-8B with reranking.Grep used alone also does not outperform dense or sparse retrieval, so the main configurations omit it.
- Retrieval-set size: Retrieving 20 rather than 10 documents produces no statistically significant performance difference across configurations.The pilot balances retrieval-set size against latency when selecting an intermediate value.
- Terminal write commands: Allowing terminal write commands changes pass^1 and Action Recall by −0.052 in the head-to-head test, with no significant behavioral use of writing commands.The ablation concerns permission to take notes and reorganize the knowledge base.
G. Full Results for Tested Configurations
The full-results analysis reports task success, document recall, efficiency, tool usage, and pairwise significance across models and retrieval configurations. Additional comparisons remove knowledge access or place the entire knowledge base in context.
- Additional configurations: Additional ablations compare agents with no knowledge-base access against agents given the entire knowledge base in context.The long-context condition is unavailable for Opus because its context window is insufficient.
- Statistical comparisons: Pairwise bootstrap tests compare retrieval configurations and agent models on pass^1 and Action Recall.Cells report whether the row outperforms the column, with significance marked at p < 0.05.
- Reported outcomes: The full results report pass^k, document recall, cost, and latency across model and retrieval configurations.These results include task success and efficiency measures for the tested combinations.
- Tool usage: The analysis reports average knowledge-tool calls per task by model and configuration.Terminal conditions count shell commands, while dense and sparse retrieval conditions count KB_search calls.
- Model comparisons: Older GPT models receive a separate pass^1 comparison across dense, sparse, terminal, and Gold retrieval settings.The Gold setting places ground-truth documents directly in context, and parenthetical values show differences from Gold.