PaperlayerView exact revision on arXiv

Source-linked AI summary

ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents

Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Zhongni Hou, Yufei Zhang, Derong Xu, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu

arXiv:2608.30685v1cs.AI

TL;DR

Industrial tool-use agents need evaluation that identifies scenario-dependent deficiencies, guides priorities, and assesses interventions beyond aggregate performance. ATLAS combines request-trajectory and ongoing-interaction diagnostics; online, its optimized policy improved engagement and downstream business outcomes, with session follow-up rate rising 6.9%.

  • Problem

    Industrial tool-use agent evaluation must move beyond aggregate performance to identify scenario-dependent deficiencies, prioritize improvements, and assess interventions.

    Source: §1 · pp. 1–2

  • Method

    ATLAS combines trajectory-wise within-request diagnostics with user-wise across-interaction signals to localize capability gaps and assess responsiveness as needs develop.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    +6.9% session follow-up rate; the policy optimized with ATLAS diagnostic feedback improved engagement and downstream business outcomes online.

    Source: §4.5 · p. 12

  • Takeaways & Limitations

    ATLAS provides production settings with behavioral evidence for capability analysis and targeted iteration across execution and continued interaction.

    Source: §5 · pp. 12–13

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly deployed in user-facing services that require iterative tool use under dynamic business conditions. Reliable evaluation is essential for sustained improvement: it must reveal capability deficiencies, inform priorities, and assess interventions. Yet industrial agent service unfolds both through the iterative trajectory of a current request and through continued user interaction. Final-outcome assessment can therefore obscure where deficiencies arise and whether later service remains aligned with context from earlier exchanges. We propose ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. At the request horizon, trajectory-wise diagnostic signals relate deficiencies to execution locations and capability concerns. At the interaction horizon, user-wise signals assess whether service remains responsive across continued interaction. Together, these views provide structured diagnostic evidence for analyzing execution deficiencies and sustained service behavior. ATLAS instantiates them as executable signals with explicit evidence scopes and decision boundaries. LLM judge interfaces are calibrated against high-confidence references from real business logs; when needed, their decision behavior is distilled into efficient diagnostic models for lower-latency, lower-cost evaluation. The resulting feedback supports policy optimization. We evaluate ATLAS on Meituan Xiaotuan production traffic. Offline experiments assess diagnostic-signal fidelity and replay-based policy improvement, while online A/B experiments show concurrent gains in user engagement, downstream business outcomes, and sampled human-audit quality.

1 Introduction

ATLAS addresses the need for industrial tool-use agent evaluation to guide sustained improvement by diagnosing both execution within a request and responsiveness across continued interaction. It turns those diagnostics into evidence-bound feedback for optimization and validates the framework on Meituan Xiaotuan production traffic.

1 Introduction

  • Industrial evaluation should identify deficiencies, prioritize them in the current user-task and product context, and establish whether interventions worked.Priorities vary by scenario; local-services decision support may emphasize accurate, timely, actionable business information.Source: §1 · pp. 1–2
  • ATLAS combines trajectory-wise within-turn signals that localize execution deficiencies with user-wise across-turn signals that assess responsiveness as user needs develop.The two horizons provide complementary views of execution-level gaps and sustained service quality.Source: §1 · pp. 2–3
  • Within-turn diagnosis locates deficiencies across Thinking & Reflection, Tool & Skill Execution, and Response Generation, and assesses Relevance, Factuality, Timeliness, Reliability, and Intent & Planning.Norms & Compliance acts as a guardrail across execution for safety, privacy, compliance, and structural validity, outside ordinary quality trade-offs.Source: §1 · p. 3
  • ATLAS makes diagnostics actionable through calibrated, evidence-bound signals with explicit decision boundaries, using real-business-log references and distilled models for lower-cost, lower-latency evaluation.The same diagnostic semantics carry into optimization as multidimensional feedback and improvement targets that can be reassessed under the framework.Source: §1 · p. 3
  • On Meituan Xiaotuan production traffic, offline evaluation validates diagnostic signals and diagnosis-driven optimization, while online A/B experiments show gains in engagement, downstream business outcomes, and sampled human-audit quality.The online experiments cover real local-services interactions.Source: §1 · p. 3

2 Related Works

Prior work develops end-to-end, process-sensitive, and deployment-grounded agent evaluation, alongside richer optimization signals. ATLAS connects request-level diagnosis and continued user service with policy optimization and reassessment.

2.1 Tool-Use Agent Evaluation

  • Prior evaluations span end-to-end benchmarks [2] [42] [44], process-level analysis [6], and deployment-grounded settings [33] [21] [46] [24] [49]; ATLAS distinguishes request trajectories from continued user service.Request-level evaluation localizes capability deficiencies within an execution, while user-level evaluation assesses responsiveness as needs develop.Source: §2.1 · pp. 3–4

2.2 Evaluation Signals for Agent Policy Optimization

  • Agent-optimization signals range from outcome rewards to process, rubric, and multidimensional feedback [23] [14] [30] [34] [3] [5] [18] [40] [17] [28] [20]; ATLAS carries calibrated diagnostic signals through optimization and reassessment.This continuity preserves the signals’ diagnostic meaning across evaluation, policy improvement, and subsequent evaluation.Source: §2.2 · p. 4

3 Methodology

ATLAS evaluates industrial tool-use agents at request and interaction horizons using executable diagnostic signals with explicit evidence scopes and decision boundaries. It calibrates semantic judges, distills selected signals for efficient evaluation, and reuses diagnostic feedback in policy optimization.

3.1 Dual-Horizon Diagnostic Structure

  • ATLAS separates request-level trajectory diagnostics, locating execution deficiencies by dimension and capability, from user-wise checks that prior context is resolved, maintained, or corrected across lifecycles.The within-turn matrix leaves cells without stable signal families uninstantiated and keeps Norms & Compliance as an independent safety, privacy, compliance, and structural-validity guardrail.Source: §3.1 · pp. 5–6, Figure 3 · p. 6

3.2 Diagnostic Signal Construction

  • Each executable signal defines its target, admissible evidence, decision boundary, and output; rules handle programmatic checks, while LLM judges assess open semantic evidence.Binary verdicts are the default; continuous scores aggregate local judgments, counts, or rankings, keeping results grounded in assessable objects rather than holistic scalar judgments.Source: §3.2 · pp. 6–7

3.3 Signal Calibration and Usability

  • Calibration tests whether each LLM judge reproduces its signal’s intended boundary on high-confidence, real-traffic references, refining prompts when F1 falls below the target range.Response-generation references use manual annotation and expert adjudication; complex trajectory references use unanimous agreement across three teacher models, with perturbations covering rare safety and compliance boundaries.Source: §3.3 · pp. 7–8

3.4 Efficient Diagnostic Model Distillation

  • For selected LLM-based signals, ATLAS distills lower-latency, lower-cost diagnostic models, grouping signals only when their execution dimension, capability concern, and input-output form are compatible.Training uses real-log traces and teacher-voted supervision, response perturbations, and format/confidence filtering; when present, <think> receives lower loss weight to emphasize verdicts and structured fields.Source: §3.4 · pp. 8–9

3.5 Policy Optimization with Diagnostic Feedback

  • ATLAS combines calibrated diagnostic signals as a weighted reward for GRPO [31], aligning policy optimization with evaluation semantics while preserving capability-specific feedback.Each reward term remains associated with a capability location in the diagnostic structure.Source: §3.5 · p. 9

4 Experiments

ATLAS evaluates diagnostic-signal fidelity, efficient-model distillation, offline policy improvement, and online transfer. Results support calibrated judge interfaces and diagnostic-feedback optimization, with online gains in engagement, business outcomes, and sampled audit quality.

4 Experiments

  • The experiments assess judge-interface fidelity, efficient-model fidelity, replay-based policy improvement, and transfer to online traffic.RQ1 and RQ2 establish the empirical basis for signal construction, calibration, and distillation; RQ3 and RQ4 evaluate policy changes offline and live.Source: §4 · p. 9

4.1.1 Datasets

  • Three query-disjoint datasets use real Xiaotuan production traffic: DJudge supplies high-confidence references for 41 signals, while DPolicy-Eval and DRL support replay and optimization.DPolicy-Eval contains approximately 2,000 queries and DRL approximately 10,000; DJudge associates each calibrated signal with approximately 500–1,000 instances.Source: §4.1.1 · p. 9

4.1.2 Evaluation Environment

  • Offline replay executes real queries through production-facing tool interfaces and resolves merchants, POIs, and supply states from live business data.This preserves experimental controllability while keeping offline evidence conditions close to online service.Source: §4.1.2 · p. 10

4.1.3 Compared Methods

  • Comparisons use shared-backbone judge interfaces and Qwen3.5-9B distilled models; reported measures include F1, replay ATLAS scores, online product metrics, and sampled human-audit metrics.The judge variants are Direct Judge, Static Rubric, Curated Rubric, and ATLAS Judge; distilled models are compared with untuned and larger model baselines.Source: §4.1.3 · p. 10, §4.1.4 · p. 10

4.2 Effectiveness of LLM Judge Interfaces (RQ1)

  • ATLAS Judge is strongest across all reported signal groups, with evidence binding, stepwise review, and boundary control supporting faithful signal-specific decisions.Direct Judge remains competitive for local, portable decisions, while rubric-based interfaces help with structured criteria, business evidence, and exception-sensitive boundaries.Source: §4.2 · pp. 10–11, Table 1 · p. 11

4.3 Efficient Diagnostic Model Distillation (RQ2)

  • Distilled 9B diagnostic models preserve most high-capacity reference performance and outperform the untuned 9B backbone and larger open baselines on the 17 selected signals.Gains concentrate on business-grounded factuality and reliability-oriented signals; ATLAS Judge remains the reference for semantically delicate cases, while distilled models offer lower-latency, lower-cost evaluation at scale.Source: §4.3 · p. 11, Table 2 · p. 11

4.4 Offline Policy Evaluation (RQ3)

  • In offline replay, the ATLAS-feedback policy achieves the highest score in every reported group, with the strongest improvement in Tool & Skill Execution.It improves trajectory-wise execution, response generation, and user-wise evaluation while maintaining the already high Norms & Compliance score.Source: §4.4 · p. 12, Figure 6 · p. 12

4.5 Online Evaluation (RQ4)

  • Online A/B results show session follow-up rate +6.9%, paid GTV +7.32%, response–supply relevance +6.2 pp, P0 hallucination rate −6.9 pp, and ID hallucination rate −2.0 pp.The test used a fixed 10% of live traffic from June 27 to June 29 and audited a sample of 1,000 responses; remaining product metrics also moved favorably.Source: §4.5 · p. 12

5 Conclusion · Appendix

ATLAS evaluates industrial tool-use agents through complementary request-trajectory and continued-interaction evidence, turning calibrated diagnostics into support for targeted iteration. Experiments on Meituan Xiaotuan report offline policy improvements and concurrent online gains in product outcomes and sampled human-audit quality.

5 Conclusion

  • Online A/B tests in real local-services traffic show concurrent gains in product outcomes and sampled human-audit quality.Table 3 reports relative changes in online A/B results and percentage-point changes in sampled human audits; the supplied passages do not give the values.Source: Table 3 · p. 12, §5 · p. 13
  • ATLAS provides an evidence-based foundation for analyzing capability deficiencies and supporting targeted iteration of industrial tool-use services.Source: §5 · pp. 12–13
  • ATLAS jointly assesses trajectory-wise evidence from a current request and user-wise evidence from continued interaction.These views preserve complementary perspectives on how service is produced and sustained.Source: §5 · p. 13
  • Fine-grained diagnostic signals make both evaluation horizons executable through explicit evidence scopes and decision boundaries.Source: §5 · p. 13
  • Selected calibrated signals are distilled into efficient diagnostic models for lower-latency, lower-cost evaluation at scale.Source: §5 · p. 13
  • Calibrated diagnostic signals provide multidimensional feedback for policy optimization.Source: §5 · p. 13
  • Experiments on Meituan Xiaotuan validate LLM judge interfaces, efficient diagnostic models, and offline-replay policy improvements across both evaluation horizons.Source: §5 · p. 13

A Additional LLM Judge-Interface Experiments … B ATLAS Diagnostic Signal Inventory

The appendices specify how LLM judge interfaces are compared and configured, then catalog ATLAS diagnostic signals for within-turn trajectories and across-turn user interactions.

A.1 Fine-Grained LLM Judge-Interface Results

  • Tables 4–5 compare Direct Judge, Static Rubric, Curated Rubric, and ATLAS Judge on all LLM-based diagnostic signals against corresponding high-confidence references; rule-based signals are excluded.The variants share the same backbone, and the tables report fine-grained F1 comparisons.Source: §A.1 · p. 18, Table 4 · p. 18, §A.2 · p. 19

A.2 Judge Interface Configurations

  • All RQ1 judge variants use the same DeepSeek-V4-Pro backbone and signal-specific reference sets, differing in how explicitly each interface defines the signal’s decision boundary.The configurations range from implicit interpretation to explicit criteria and procedures.Source: §A.2 · p. 19
  • Direct Judge leaves signal-specific criteria, evidence selection, and exemptions to implicit model interpretation, while Static Rubric adds generic checklist guidance without complete decision rules.Both interfaces retain the task definition, input description, and output schema as a base specification.Source: §A.2 · p. 19
  • Curated Rubric gives compact signal-specific criteria and key exemptions with partial evidence-review procedure, whereas ATLAS Judge adds relevant evidence binding, stepwise review, and reference-calibrated decision boundaries.ATLAS Judge also specifies decision criteria and exemptions and constrains the final output format.Source: §A.2 · p. 19

B ATLAS Diagnostic Signal Inventory

C Metric Definitions

This section defines the automatic metrics used in the main experiments.

C Metric Definitions

C.1 LLM Judge-Interface and Diagnostic Model Evaluation

This section evaluates LLM judge interfaces and efficient diagnostic models using F1 against high-confidence reference labels, with signal-specific handling for score-valued signals and aggregate summaries.

C.1 LLM Judge-Interface and Diagnostic Model Evaluation

  • Both LLM judge-interface validation and efficient diagnostic model evaluation report F1 against high-confidence reference labels.For a signal, its target event is treated as the positive class.Source: §C.1 · p. 21
  • Precision is TP/(TP + FP), and recall is TP/(TP + FN).Source: §C.1 · p. 21
  • Score-valued signals are mapped to comparable signal-specific decision units using discretization and tolerance rules before F1 is computed.Group-level and overall summaries in Tables 1 and 2 use the unweighted mean of signal-level F1 values; rule-based diagnostic signals are excluded from these comparisons.Source: §C.1 · p. 21

C.2 Offline Policy Evaluation · C.3 Online Product Metrics

C.2 defines replay-based scoring of policies with ATLAS diagnostic signals, while C.3 specifies automatic product metrics measured from live traffic. Together, they describe offline evaluation and the online measures used to assess product behavior.

C.2 Offline Policy Evaluation

  • Under replay, each tested policy generates trajectories scored by ATLAS signals, with group-level scores averaged across the group’s signals and replayed queries.The score is based on the scalar output of each signal for each query’s generated trajectory.Source: §C.2 · p. 21
  • The overall ATLAS score is computed analogously, preserving signal-level diagnostic detail while enabling group and overall comparisons.The supplied description does not specify the full overall-score formula.Source: §C.2 · p. 21

C.3 Online Product Metrics

  • C.3 reports automatic product metrics measured from live traffic to evaluate AI-search interactions and outcomes.Source: §C.3 · p. 21
  • Read-through rate measures sessions reaching response completion, per-user dwell time averages time on AI-search responses, and 5-second exit rate counts sessions ending within five seconds.Source: §C.3 · p. 21
  • AI search query volume counts issued queries, while Effective QV counts valid AI-search visits meeting the platform’s effectiveness criterion.Source: §C.3 · p. 22
  • The metric set also includes session follow-up rate for within-session repeat queries and Paid GTV for paid gross transaction volume attributed to AI-search traffic.Source: §C.3 · p. 22

D Additional Implementation Details

The appendix specifies implementation setups for judge-interface validation, diagnostic-model distillation, and reinforcement learning, including hardware, models, and training configurations.

D Additional Implementation Details

  • Experiments used NVIDIA H20 GPUs, with all compared judge interfaces sharing a DeepSeek-V4-Pro backbone [37].The calibration pipeline also used Claude Opus-4.6, GPT-5.4, and Gemini 3.1-Pro to construct high-confidence supervision.Source: D Additional Implementation Details · p. 22
  • Diagnostic-model distillation trained Qwen3.5-9B with Swift [50] on 8 GPUs using group-wise construction.Training used AdamW, batch size 128, learning rate 1×10−5, maximum output length 2148, and maximum total sequence length 16384; supervision included DeepSeek-V4-Pro, DeepSeek-V4-Flash [37], and GLM-5.1 [45]. Data construction retained multi-model voting, response-level perturbation, format-consistency and confidence-based filtering, and reduced weighting on the <think> field when present.Source: D Additional Implementation Details · p. 22
  • RL optimized Qwen3.6-35B-A3B with GRPO on 32 GPUs, using Verl [32], Megatron [22], and vLLM [11].The setup used batch size 128, rollout group size 16, sampling temperature 1.0, maximum output length 8192, maximum total sequence length 32768, and AdamW with learning rate 1 × 10−6.Source: D Additional Implementation Details · p. 22

E Case Study

Four representative cases from Xiaotuan’s real online traffic show that policies optimized with ATLAS feedback produced more supply-grounded, structured, and appropriately qualified responses than the deployed baseline. Human-audit scores favored the optimized policy on response–supply relevance, P0 hallucination, and ID hallucination in every selected case.

E Case Study

  • Across all four selected cases, human auditors favored the ATLAS-optimized policy on response–supply relevance, P0 hallucination, and ID hallucination.The examples illustrate complementary failure modes and complement the aggregate online results in RQ4.Source: E Case Study · p. 22
  • For a family-dinner request, the optimized policy grounds restaurant recommendations in verified merchants and bookable deals; for qualification-sensitive service search, it separates regulated medical TCM clinics from traditional blind-massage shops.The baselines respectively offered unverifiable descriptions and mixed entertainment-oriented SPA venues into the recommendation.Source: E Case Study · p. 22, Figure 7 · p. 23, Figure 8 · p. 24
  • For a new-shoe query, the optimized policy returns store-level product cards with verifiable prices, monthly sales, and instant-delivery information, unlike the baseline’s product names without purchasable supply.The recommendations are organized by usage scenario and followed by purchase suggestions.Source: E Case Study · p. 22, Figure 9 · p. 24
  • For a hotel sound-insulation query, the optimized policy retrieves the target hotel and provides review-grounded evidence, practical booking information, and a caveat about room-type differences instead of generic advice.The baseline fails to retrieve the hotel.Source: E Case Study · p. 22, Figure 10 · p. 25

Related papers

Loading 2608.30685v1…