Source-linked AI summary
$τ$-Rec: A Verifiable Benchmark for Agentic Recommender Systems
Bharath Sivaram Narasimhan, Karthik R Narasimhan
TL;DR
Agentic recommender evaluation lacks reliable alternatives to static metrics and subjective judging. τ-Rec introduces deterministic predicate-based rewards, reveal-tagged elicitation, policy checks, and pass^k evaluation; across models, performance falls from about 57% pass^1 to about 35% pass^4. The benchmark also exposes strong difficulty effects from hidden constraints, while its current scope and statistical power remain limited.
Problem
Existing agentic-recommender benchmarks rely on inadequate static metrics or subjective judging, leaving reliable multi-turn evaluation insufficiently addressed.
Method
τ-Rec evaluates tool-agent-user recommendation episodes with deterministic catalog predicates, reveal-tagged constraints, policy enforcement, and pass^k reliability.
Results
∼57% pass^1 and ∼35% pass^4 mark the strongest agent’s reliability cliff, while hidden constraints sharply reduce pass^1 across models.
Takeaways & Limitations
τ-Rec provides a systematic test of consistent conversational reasoning, revealing that capability does not reliably persist across repeated trials or hidden-intent tasks.
Takeaways & Limitations
The benchmark covers movies with a 153-title catalog and only four trials per task, limiting domain breadth, failure-mode diversity, and statistical power.
Abstract
from arXiv · showhide
As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace. Current benchmarks often rely on "LLM-as-a-judge" evaluations, which introduce subjectivity, high costs and inconsistency. We present $τ$-Rec, a benchmark for agentic recommender systems that replaces subjective evaluation with verifiable rewards and a reveal-tagged elicitation (RTE) mechanism that controls how task constraints surface during dialogue. By testing agents against structured catalog predicates and employing a pass^k reliability metric, $τ$-Rec provides a systematic test for consistent reasoning. Our evaluation of nine configurations across five model families -- GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B and GPT-5 mini -- reveals a steep reliability cliff, where even the best model achieves only ~57% at pass^1 and ~35% at pass^4, highlighting a critical gap in current conversational agent deployment. All code and data are publicly available at https://github.com/nbharaths/tau-rec.
1 Introduction
Agentic recommender systems require evaluation beyond static dialogues and subjective judging. τ-Rec addresses this gap with verifiable, reliability-focused evaluation of elicitation, constraint reasoning, and policy compliance.
- Agentic recommenders use multi-turn dialogue, tool calls, planning, catalog reasoning, and progressive preference elicitation.
- Existing benchmarks rely either on fixed-dialogue surface metrics vulnerable to memorization or expensive, subjective, inconsistent LLM judging.
- τ-Rec frames recommendation as constrained optimization in a POMDP-based tool-agent-user loop.
- The benchmark tests preference elicitation and constrained reasoning as simultaneous agent competencies.
- pass^k measures success across k independent trials, exposing reliability that capability metrics such as Recall@N and Hit Rate@N miss.
- Policy compliance evaluates whether recommendations are appropriately gated, disclosed, filtered, or withheld for impossible requests.
- ∼57% pass^1 and ∼35% pass^4 mark the strongest agent’s reliability cliff across the evaluated configurations.
2 Related Work
Prior conversational-recommendation benchmarks emphasize static dialogue similarity, ranking, or subjective judgments, while newer tool-use approaches still lack a unified agentic evaluation framework. τ-Rec combines verifiable interaction, policies, fresh catalogs, and hidden-intent simulation.
- Early CRS resources evaluate fixed reference dialogues with BLEU and Recall@k, measuring surface similarity rather than task completion and remaining vulnerable to contamination.
- AgentRecBench uses single-turn Hit Rate@N ranking over Amazon, Goodreads, and Yelp, without multi-turn simulation, pass^k, policies, or fresh catalogs.
- Table 1 compares CRS resources by support for verifiable rewards, multi-turn dialogue, tool use, policy checks, fresh catalogs, and hidden-intent simulation.
- τ-bench supplies deterministic tool-agent-user verification, policies, user simulation, and pass^k, but recommendation adds elicitation, catalog exploration, and multi-criteria optimization challenges.
- Tool-augmented CRS systems commonly report Recall@K or NDCG@K on existing datasets without standardized tasks, policies, or pass^k.
- User-simulation studies validate simulators or structured needs but do not combine role-play with reveal-tagged constraints and verifiable downstream tasks.
3 𝜏-Rec Benchmark Design
τ-Rec models recommendation as a partially observed tool-agent-user interaction and evaluates it with deterministic predicates, controlled constraint revelation, reliability, and policy checks. Its movie tasks vary constraint complexity and reveal difficulty.
- The TAU interaction exposes dialogue, prior tool results, and catalog metadata while keeping hidden user constraints unobservable.
- The agent can converse, search or filter catalogs, inspect metadata or availability, and submit one candidate through a terminal recommendation tool.
- Verifiable rewards evaluate typed catalog predicates deterministically, combining constraint_score and policy_score and supporting partial-credit failure attribution.
- RTE tags constraints as volunteer, on_ask, or hidden, forcing agents to elicit requirements rather than receive complete preferences initially.
- pass^k reports the probability of solving a task across all k independent trials instead of averaging trial success.
- Policy enforcement audits recommendation-tool use, watch history, availability, and age restrictions among seven domain-specific policies.
- The catalog pipeline discovers, enriches, normalizes, validates, and filters movie records so task-relevant attributes support clean predicate evaluation.
- Tasks specify personas, typed constraints, RTE tags, optional soft preferences, and active policies, then span simple, medium, and complex constraint counts.
4 Benchmark Validation
Across nine configurations, τ-Rec reveals substantial reliability and difficulty gradients in agentic recommendation, alongside trade-offs among capability, latency, policy compliance, and tool use.
- Main results: pass^1 scores range from 0.537–0.571 among the top tier, while every model records lower pass^2 and pass^4 scores.GPT-5.4, Sonnet 4.6, and DeepSeek V4 Flash variants form the top tier with overlapping confidence intervals.
- Policy Compliance: >0.92 policy compliance distinguishes the strongest tier, whereas Qwen3-32B reaches 0.756 with a 0.21 availability-violation rate.The availability violations occur when titles are recommended without verifying streaming-service eligibility.
- Efficiency: 28 median tool calls per trial for DeepSeek V4 Flash contrasts with 5 for Qwen3-32B and 8 for Gemini 2.5 Flash.Higher-volume models take more turns to recommend but achieve higher constraint scores.
- Failure Analysis: Balanced models invest 15–28 median tool calls and achieve the highest constraint scores of 0.57–0.58, while abstainers exit without issuing recommend().Qwen3-32B, GPT-5 mini, and Gemini 2.5 Flash have No-rec rates of 0.43, 0.45, and 0.67, respectively.
- Difficulty gradient: pass^1 for DeepSeek V4 Flash falls from 0.846 on volunteer tasks to 0.586 on mixed tasks and 0.200 on hidden tasks.The same reveal-difficulty gradient appears for GPT-5 mini and Qwen3-32B, distinguishing elicitation from retrieval.
- Capability-latency frontier: GPT-5.4 without thinking anchors the fast frontier at pass^1 = 0.47 and approximately 7 seconds per step, while DeepSeek V4 Flash reaches pass^1 = 0.57.DeepSeek variants occupy approximately 15–24 seconds per step on the remaining frontier.
5 Limitations and Future Work
The benchmark’s current scope is limited by a small, movie-only catalog and uncertainty from only four trials per task. Future work targets broader domains, larger catalogs, and stronger statistical power.
- Catalog scale: The 153-title catalog isolates reasoning from retrieval scale but limits failure-mode diversity.Agents access the catalog through API tools, so its size is opaque to them; scaling while preserving post-cutoff freshness is future work.
- Single domain: The released benchmark covers movies only, although its typed-predicate scoring supports extension to other domains.Cross-domain task suites are identified as a natural next step.
- Statistical power: Four trials per task yield non-trivial 95% bootstrap confidence intervals of ±0.10–0.13 for pass^4.Distinguishing models with overlapping confidence intervals would require more trials per task or more tasks per cell.