Source-linked AI summary
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Zhongni Hou, Yufei Zhang, Derong Xu, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu
TL;DR
Industrial tool-use agents need evaluation that identifies deficiencies and priorities across both current-request execution and continued user interaction. ATLAS provides dual-horizon, evidence-bound diagnostics calibrated on business logs and used for optimization; on Meituan Xiaotuan production traffic, online A/B experiments show gains in engagement, downstream business outcomes, and sampled human-audit quality.
Problem
Industrial agent evaluation must reveal capability deficiencies, guide scenario-dependent priorities, and assess interventions across iterative execution and continued interaction, rather than relying only on final outcomes.
Method
ATLAS combines trajectory-wise within-turn signals and user-wise across-turn signals, implementing them as executable, evidence-bound diagnostics with calibrated LLM judges and distilled efficient models.
Results
Online A/B experiments on Meituan Xiaotuan show that ATLAS-optimized policies improve user engagement, downstream business outcomes, and sampled human-audit quality.
Takeaways & Limitations
ATLAS provides structured diagnostic evidence for analyzing execution deficiencies and sustained service behavior while supporting policy optimization and scalable evaluation.
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly deployed in user-facing services that require iterative tool use under dynamic business conditions. Reliable evaluation is essential for sustained improvement: it must reveal capability deficiencies, inform priorities, and assess interventions. Yet industrial agent service unfolds both through the iterative trajectory of a current request and through continued user interaction. Final-outcome assessment can therefore obscure where deficiencies arise and whether later service remains aligned with context from earlier exchanges. We propose ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. At the request horizon, trajectory-wise diagnostic signals relate deficiencies to execution locations and capability concerns. At the interaction horizon, user-wise signals assess whether service remains responsive across continued interaction. Together, these views provide structured diagnostic evidence for analyzing execution deficiencies and sustained service behavior. ATLAS instantiates them as executable signals with explicit evidence scopes and decision boundaries. LLM judge interfaces are calibrated against high-confidence references from real business logs; when needed, their decision behavior is distilled into efficient diagnostic models for lower-latency, lower-cost evaluation. The resulting feedback supports policy optimization. We evaluate ATLAS on Meituan Xiaotuan production traffic. Offline experiments assess diagnostic-signal fidelity and replay-based policy improvement, while online A/B experiments show concurrent gains in user engagement, downstream business outcomes, and sampled human-audit quality.
1 Introduction
ATLAS frames industrial agent evaluation as a dual-horizon diagnostic problem: it analyzes both current-request execution and service across continued interaction. Its evidence-bound signals support prioritization, optimization, and reassessment on production traffic.
- Motivation: Industrial evaluation must reveal deficiencies, prioritize improvements within product context, and determine whether interventions worked.Priorities depend on the scenario, such as accuracy, timeliness, and actionability in local-services decision support.
- Framework: Within-turn evaluation organizes deficiencies by execution location and behavioral shortfall, with Thinking & Reflection, Tool & Skill Execution, and Response Generation as key locations.Norms & Compliance acts as a guardrail for safety, privacy, compliance, and structural validity.
- Diagnosis-Driven Iteration: ATLAS makes diagnostics actionable through executable signals with explicit evidence scopes and decision boundaries, calibrated using high-confidence references from real business logs.Selected signals are distilled into lower-latency, lower-cost diagnostic models, while calibrated signals provide multidimensional optimization feedback.
- Framework: ATLAS evaluates both an individual request’s iterative trajectory and the agent’s service across continued user interaction.Trajectory-wise signals localize execution deficiencies, while user-wise signals assess responsiveness as user needs develop.
- Validation: Offline evaluation and online A/B experiments validate ATLAS on real Meituan Xiaotuan production traffic.Online experiments report improvements in user engagement, downstream business outcomes, and sampled human-audit quality.
2 Related Works
Related work extends agent evaluation from end-to-end success toward process-level, deployment-grounded, and multidimensional feedback. These directions motivate richer evaluation and optimization signals for agents operating in realistic environments.
- Agent Evaluation: Agent evaluation complements end-to-end task success with process-level analysis of intermediate reasoning and actions.Process supervision assigns credit to intermediate steps, while tool-use reinforcement learning constructs local feedback around invocation and execution behavior.
- Evaluation Challenges: Coarse-grained judgments can miss defects in deployed multi-turn transactional agents, making evaluation-signal design consequential.LLM-based evaluators may also vary substantially across agent capabilities.
- Deployment-Grounded Evaluation: Deployment-oriented benchmarks ground evaluation in realistic requests, interaction logs, and dynamic external environments across mobility, research, recommendation, and finance.These settings move evaluation closer to real application conditions rather than isolated task completion.
- Multidimensional Feedback: Recent work develops multidimensional rubrics and reward signals for open-ended, safety-oriented, financial, and embodied agents.These signals provide richer supervision for evaluation-guided optimization in concrete application domains.
3 Methodology
ATLAS evaluates industrial tool-use agents across current-request trajectories and continued user interactions, using structured signals to localize deficiencies and assess sustained service. It operationalizes this framework through evidence-bound diagnostics, calibration, efficient implementations, and policy-optimization feedback.
- Dual-Horizon Diagnostic Structure: ATLAS organizes evaluation across within-turn trajectories and across-turn user interactions to capture execution deficiencies and continued service quality.Within-turn analysis examines complete request trajectories, while across-turn analysis considers successive lifecycles and evolving user context.
- Dual-Horizon Diagnostic Structure: Within-turn diagnostics localize deficiencies by execution dimension and capability concern, while Norms & Compliance remains an independent guardrail.Execution dimensions include Thinking & Reflection, Tool & Skill Execution, and Response Generation; capability concerns include Relevance, Factuality, Timeliness, Reliability, and Intent & Planning.
- Evidence-Based Diagnosis: ATLAS links signal outcomes to behavioral evidence, execution locations, capability concerns, and user-wise context for structured diagnosis.The framework uses trajectory evidence and aggregated signal profiles to surface recurring patterns across dimensions and concerns.
- Diagnostic Signal Construction: Signals specify diagnostic targets, admissible evidence, decision boundaries, and output semantics, using rule-based or LLM-based mechanisms as appropriate.Binary verdicts are the default, while continuous signals aggregate local judgments or statistics into interpretable measures such as repetition rate, stale-item ratio, correction rate, or ranking quality.
- Implementation and Optimization: Diagnostic signals provide multi-dimensional feedback for policy optimization, preserving interpretable capability-specific semantics through the improvement loop.The implementation pathway connects signal construction and calibration with trajectory evaluation and policy optimization.
- Implementation and Optimization: Calibrated LLM-based signals can be distilled into efficient diagnostic models that retain signal-specific decision behavior while reducing latency and inference cost.Distillation uses the same evidence inputs and decision outputs as the calibrated signals.
4 Experiments
ATLAS is evaluated through judge-interface fidelity, efficient-model distillation, offline replay policy optimization, and online traffic outcomes. Across these studies, calibrated diagnostic interfaces and feedback support improved diagnostic quality, offline ATLAS scores, and live engagement, business, and audit metrics.
- RQ1: LLM Judge Interfaces: RQ1 compares four LLM judge interfaces against high-confidence signal-specific references using average F1 over comparable diagnostic signals.The interfaces are Direct Judge, Static Rubric, Curated Rubric, and ATLAS Judge; rule-based signals are excluded from this comparison.
- RQ1: LLM Judge Interfaces: ATLAS Judge is the strongest interface across all reported signal groups.Its advantage is attributed to explicit evidence binding, stepwise review, and stable decision-boundary control rather than prompt length alone.
- RQ2: Efficient Diagnostic Models: Distilled 9B diagnostic models preserve most high-capacity reference performance while outperforming the untuned 9B backbone and larger open baselines.The largest gains concentrate on business-grounded factuality and reliability-oriented signals; ATLAS Judge remains the reference interface for semantically delicate cases.
- RQ3: Offline Policy Evaluation: The policy optimized with ATLAS diagnostic feedback achieves the highest offline replay score in every reported within-turn, guardrail, and user-wise group.The strongest improvement appears in Tool & Skill Execution, while Norms & Compliance remains high.
5 Conclusion
ATLAS frames industrial tool-use-agent evaluation around complementary trajectory-wise and user-wise evidence. Experiments on Meituan Xiaotuan support its diagnostic signals, efficient models, policy optimization, and concurrent online product and audit gains.
- Framework: ATLAS is a dual-horizon diagnostic evaluation framework for industrial tool-use agents.It jointly considers trajectory-wise evidence from current-request execution and user-wise evidence from continued interaction.
- Diagnostic Evidence: Fine-grained signals make both evaluation horizons executable through explicit evidence scopes and decision boundaries.Selected calibrated signals are distilled for lower-latency, lower-cost evaluation, while calibrated signals provide multidimensional policy-optimization feedback.
- Validation: Online A/B results show concurrent gains in product outcomes and sampled human-audit quality in real local-services traffic.The conclusion reports this alongside offline validation across both evaluation horizons.
Contributions
The paper lists its authors, their institutional affiliations, and contribution-status markers.
- Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Zhongni Hou, Yufei Zhang, Derong Xu, Guojun Yin, Wei Lin, Zhi Zheng, and Tong Xu are listed as authors.
- Wei Chen and Peilun Zhou are marked equal contributors, Peilun Zhou is the project leader, and several authors are corresponding authors.The corresponding-author marker applies to Guojun Yin, Wei Lin, Zhi Zheng, and Tong Xu.
- The authors are affiliated with the State Key Laboratory of Cognitive Intelligence at the University of Science and Technology of China and Meituan.
A.1 Fine-Grained LLM Judge-Interface Results
The appendix defines ATLAS’s fine-grained diagnostic-signal organization and compares calibrated judge interfaces and signal groups. It also separates within-turn trajectory-wise signals from across-turn user-wise signals.
- LLM Judge Interfaces: Tables 4 and 5 compare Direct Judge, Static Rubric, Curated Rubric, and ATLAS Judge on LLM-based diagnostic signals.All variants use the same backbone and corresponding high-confidence references; rule-based signals are excluded.
- LLM Judge Interfaces: ATLAS Judge binds judgment to behavioral evidence, specifies criteria and exemptions, applies stepwise review, and constrains output formatting.Its decision boundary is refined against the corresponding high-confidence reference set.
- Within-Turn Evaluation: Tables 6–9 list within-turn trajectory-wise signals for Response Generation, Tool & Skill Execution, Thinking & Reflection, and Norms & Compliance.Norms & Compliance is treated as an independent guardrail.
- Across-Turn Evaluation: Table 10 separately lists across-turn user-wise signals that assess service behavior over multiple complete agent lifecycles.Each signal includes its concern where applicable, type, and a one-line definition.
C Metric Definitions
This section defines the automatic metrics used in the main experiments.
- The main experiments use automatically defined metrics to evaluate the system.
C.1 LLM Judge-Interface and Diagnostic Model Evaluation
The evaluation reports F1-based diagnostic fidelity, replay-based ATLAS scores for policy evaluation, and automatic product metrics from live traffic. These measures preserve signal-level detail while supporting group-level comparisons and tracking user engagement, effectiveness, and business outcomes.
- LLM Judge-Interface and Diagnostic Model Evaluation: F1 is reported against high-confidence reference labels for LLM judge-interface validation and efficient diagnostic model evaluation.
- LLM Judge-Interface and Diagnostic Model Evaluation: Precision and recall define the positive-event evaluation components used for F1 computation.
- LLM Judge-Interface and Diagnostic Model Evaluation: Score-valued signals are discretized using signal-specific decision units, while group and overall summaries use unweighted means of signal-level F1 values; rule-based signals are excluded.
- Offline Policy Evaluation: Replay evaluation scores each tested policy’s generated trajectories with ATLAS diagnostic signals and averages over signals within each reported group and replayed queries.
- Offline Policy Evaluation: The formulation retains signal-level diagnostic detail while enabling group-level and applicable overall comparisons.
- Online Evaluation: Live-traffic metrics measure AI-response read-through, per-user dwell time, five-second exits, query volume, effective QV, follow-up interaction, and paid GTV.
- Online Evaluation: Session follow-up rate measures the fraction of AI-search sessions in which users issue a follow-up query within the same session.
- Online Evaluation: Paid GTV measures total paid gross transaction volume attributed to AI-search traffic.
D Additional Implementation Details
The experiments use specified hardware, model backbones, distillation settings, and reinforcement-learning infrastructure for judge validation, diagnostic-model distillation, and policy optimization.
- LLM Judge-Interface Validation: All experiments run on NVIDIA H20 GPUs, with LLM judge interfaces sharing a DeepSeek-V4-Pro backbone.Calibration supervision additionally uses Claude Opus-4.6, GPT-5.4, and Gemini 3.1-Pro.
- Diagnostic Model Distillation: Efficient diagnostic models are trained with Swift on eight GPUs using Qwen3.5-9B and supervision from multiple strong models.The training setup uses AdamW with batch size 128 and learning rate 1×10^-5, among other stated settings.
- Diagnostic Model Distillation: Distillation data construction includes multi-model voting, response perturbation, format-consistency filtering, confidence filtering, and reduced weighting of the <think> field when present.
- Policy Optimization: RL optimizes Qwen3.6-35B-A3B with a GRPO-based pipeline across 32 GPUs using Verl, Megatron, and vLLM.The setup uses rollout group size 16, sampling temperature 1.0, maximum total sequence length 32768, and AdamW learning rate 1 × 10^-6.
E Case Study
The case study compares a deployed baseline with an ATLAS-feedback-optimized policy on four representative Xiaotuan production-traffic queries. Across complementary failure modes, the optimized policy provides more supply- and evidence-grounded responses, with sampled human-audit scores shown for comparison.
- Case Study Setup: Four representative real-traffic cases compare the deployed baseline and ATLAS-optimized policy on the same queries using human-audit scores.Auditors score response–supply relevance, P0 hallucination, and ID hallucination from 0 to 3, with higher scores better.
- Supply-Grounded Recommendation: For family-dinner recommendations, ATLAS organizes verified merchants and bookable deals into intent-aligned groups with traceable attributes and concrete selection advice.
- Qualification-Sensitive Search: For qualification-sensitive service search, ATLAS separates regulated medical TCM clinics from traditional blind-massage shops and maps needs to recommendations using evidence.
- Human Audit: The displayed audit scores include baseline zeroes and optimized-policy threes for response–supply relevance, P0 hallucination, and ID hallucination in the reported cases.
- Product Availability: For new-product availability, ATLAS returns store-level product cards with verifiable prices, monthly sales, and instant-delivery information organized by usage scenario.
- Long-Tail Attribute Question: For a long-tail hotel attribute query, ATLAS retrieves the hotel and answers with review-grounded evidence, booking information, and a caveat about room-type differences.