Source-linked AI summary
Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotation
Shiyu Zhang, Leisheng Cheng, Huifu Li
TL;DR
Industrial query-to-agent matching can confuse semantic relatedness with executable capability, creating a supervision gap for long-tail and boundary-sensitive requests. The paper formulates annotation as capability-bound process supervision through Debate-to-Skill, which structures capability evidence and verdict extraction. Its evaluation tests this target against direct labels, generic reasoning, and structural controls, while the conclusion reports improved annotation when the learned object matches capability-qualified executability.
Problem
Query-to-agent supervision often treats matching as semantic relevance even though production errors arise when related agents lack the capability to execute requests.
Method
Debate-to-Skill uses reusable capability principles, structured deliberation, verifier-based verdict extraction, and disagreement-driven refinement as process supervision.
Results
The paper reports that industrial annotation improves when the learned object matches capability-qualified executability.
Takeaways & Limitations
Capability-bound decision traces are presented as a more appropriate learning object than final relevance labels for industrial query-to-agent annotation.
Takeaways & Limitations
The implementation uses lexical rather than embedding-based skill retrieval, rule-based disagreement updates, and no calibrated abstention thresholds, potentially underusing semantic retrieval, learned memory updates, and uncertainty calibration.
Abstract
from arXiv · showhide
Industrial query-to-agent matching fails when topical relevance is mistaken for executable capability, especially on long-tail and boundary-sensitive requests. We formulate annotation as \emph{capability-bound process supervision} and instantiate it with Debate-to-Skill, which uses reusable decision principles, structured deliberation, verifier-based verdict extraction, and disagreement-driven refinement. On an industrial Query2Agent benchmark, we compare Debate-to-Skill with direct-label supervision, reasoning-SFT, and structural ablations. The results test whether gains come from supervising the capability-critical decision process itself, especially on grey-zone cases where semantic relatedness and executable capability diverge.
1 Introduction
Industrial query-to-agent annotation fails when topical relevance is mistaken for executable capability, especially in long-tail and boundary-sensitive cases. Debate-to-Skill addresses this by supervising a capability-bound decision process rather than only final labels.
- The paper reframes query-to-agent annotation as capability verification under asymmetric risk because topical similarity can mask missing tools, authority, real-time access, or service boundaries.
- Debate-to-Skill supervises reusable principles, support-versus-failure evidence, structured verdict extraction, and disagreement-driven refinement for capability-bound decisions.
- The evaluation compares direct-label supervision, reasoning-SFT, and structural ablations to test whether gains arise from supervising the capability-critical process, especially on grey-zone cases.
- The industrial workflow routes requests to candidate agents online while converting traffic and review signals into offline Debate-to-Skill supervision for routing-policy updates.
- The framework is instantiated through reasoning-SFT and verifier-based GRPO, treating capability-bound traces as a stronger learning target than direct labels or generic reasoning.
2 Problem Setting
The paper represents each example as a query-agent pair with a ternary raw annotation distinguishing executable matches, capability-insufficient related pairs, and irrelevant pairs. Training and standard evaluation collapse this into a binary target, while grey-zone diagnostics retain the ternary distinction.
- Each training example is a query-agent pair consisting of a user request and a candidate agent profile.
- The raw annotation is ternary: 2 means the agent can satisfy the request, 1 means semantic relatedness with insufficient or risky capability, and 0 means irrelevance.
- Only raw label 2 becomes positive in the binary deployment target, while grey-zone diagnostics use the original ternary verdict before collapse.
- A travel assistant without ticketing or transaction APIs is topically related to flight rebooking but fails the executable-capability requirement.
3 Method
Debate-to-Skill turns query–agent annotation into a capability-bound process: retrieve reusable principles, deliberate over support and capability objections, extract a verdict, and refine principles from recurring conflicts.
- 3.2 Principle-Grounded Deliberation: Debate-to-Skill retrieves reusable decision principles, separates semantic support from capability objections, guides debate, extracts a structured verdict, and refines skills from recurring disagreements.The disagreement buffer promotes recurring high-priority conflicts into new or refined principles rather than retaining raw traces.
- 3.2 Principle-Grounded Deliberation: The skill bank represents each query–agent example with principle titles, capability principles, application conditions, and tags before deliberation.The bank maps instance ambiguity into a compact principle space.
- 3.2 Principle-Grounded Deliberation: Lexical retrieval selects three skills for each example, falls back to seed skills when scores are zero, and prioritizes transparency, bounded latency, and auditability.Dynamic-skill comparisons use the same lexical retriever, while the ablation uses fixed seed skills.
- 3.2 Principle-Grounded Deliberation: The debate state conditions each round on the query–agent input, retrieved skills, prior dialogue history, and prior judge guidance, factorizing relevance from executable capability.The scaffold preserves a process distinction between semantic support and capability-critical objections.
- 3.3 Process-Supervised Learning: Reasoning-SFT trains on complete structured traces, while verifier-based optimization checks verdict correctness, required sections, valid skill identifiers, and capability consistency.The verifier does not assess arbitrary natural-language faithfulness; full support-vs.-veto traces provide that training signal.
- 3.4 Risk-Aware Verdicts and Adaptation: Risk-aware adaptation updates principles only for recurring, informative conflicts, including repeated overacceptance of consulting-only agents for execution requests.A conflict enters refinement when its priority reaches the threshold, and recurring patterns are revised through principle text, conditions, or tags.
4 Experiments
The experiments train routing models from offline synthetic supervision and evaluate them on fixed industrial Query2Agent splits, comparing direct labels, reasoning, Debate-to-Skill, and structural variants across standard, grey-zone, domain, long-tail, process, and online metrics.
- 4.1 Experimental Setup: The offline pipeline trains Qwen2.5-14B-Instruct routing models from direct labels, reasoning traces, or Debate-to-Skill process traces using SFT and GRPO.The teacher is the offline Debate-to-Skill pipeline, while the student is the trained routing model.
- 4.1 Experimental Setup: The dataset contains 170K anonymized production query–agent pairs, 3,000 validation examples, 3,000 human-labeled test examples, and 312 grey-zone examples.The raw-label distribution is {2, 1, 0} = 5:2:3, and the evaluation data use an 80/20 temporal composition.
- 4.1 Experimental Setup: Human evaluation uses ternary labels for executable matches, capability-insufficient or risky related pairs, and irrelevant pairs, with two annotators, adjudication, and 86.4% raw agreement.The offline teacher produces the 170K training labels.
- 4.2 Offline Evaluation: The comparison chain is SFT Label → SFT Reasoning → w/o Debate → Debate-to-Skill, with ablations isolating the support-vs.-veto scaffold, dynamic skills, and judge aggregation.Table 1 covers MAIN TEST and the grey-zone slice as both the primary comparison and core structural ablation.
- 4.2 Offline Evaluation: Tables 2–4 diagnose grey-zone over-acceptance, missed true positives, robustness across professional and long-tail traffic, structured validity, skill grounding, and capability-conflict resolution.These metrics distinguish capability-bound decision quality from generic output length or uniform conservatism.
- 4.3 Online Evaluation: The online A/B test ran for four weeks with 10% treatment traffic, reporting significant relative lifts in card impression rate, card CTR, and downstream redirect CTR without launch-blocking degradation.Absolute traffic volumes, business rates, and interval estimates are undisclosed; Table 5 reports relative lifts over the deployed baseline.
5 Related Work
Prior work improves label quality, reasoning trace checkability, retrieval, routing, or skill reuse, but Debate-to-Skill targets capability-consistent executability as the supervised object.
- Related methods optimize label quality, trace checkability, selection, execution efficiency, or task reuse, whereas Debate-to-Skill optimizes capability-consistent executability.
- Weak supervision and reasoning methods do not directly resolve cases where topical relevance masks missing tools, authority, real-time access, or service boundaries.
6 Conclusion
The paper concludes that industrial query-to-agent supervision should be capability-bound rather than based on topical relatedness. Debate-to-Skill operationalizes this through reusable principles, structured deliberation, verifier optimization, and disagreement-driven refinement.
- Debate-to-Skill frames the key decision as whether a candidate agent remains executable after explicit capability-bound inspection.
- The framework combines reusable decision principles, structured deliberation traces, verifier-based optimization, and disagreement-driven refinement.
7 Limitations
The implementation favors industrial auditability but remains limited by conservative retrieval and update mechanisms, incomplete verification of natural-language evidence, and evaluation tied to one industrial setting.
- Lexical skill retrieval, rule-based disagreement updates, and uncalibrated risk-aware routing improve inspectability but may underuse semantic retrieval, learned memory, and uncertainty calibration.
- The verifier checks verdict correctness, section validity, and skill legality but does not fully verify natural-language capability-evidence faithfulness.
- Evaluation remains tied to an industrial Query2Agent setting, while public benchmarks lack the semantically plausible but capability-invalid category targeted by the work.