Source-linked AI summary
From Monolithic Blending to Agentic Orchestration: Dynamic Response for Conversational Assistants at Scale
Cen, Zhao, Peng Wang, Chuan Shi, Yufeng Zhang, Ying Lyu, Wanmeng Ren, Robert Xue, Claire Na Cheng, Yashar Mehdad
TL;DR
Customer-support assistants must jointly make high-consequence decisions and produce grounded wording, but a blended model path entangles these roles. The paper reports a production migration to Dynamic Response, which uses a typed-tool ReAct orchestrator and a smaller contract-bound generator. Replay and low-ramp online evidence show safer structured decisions and lower escalation rates, while the authors scope architecture claims to identical replayed turns and report operational trade-offs.
Problem
Support turns combine retrieval, action selection, escalation, and wording, while requiring cross-source reasoning, traceability, and human handoff.
Method
Dynamic Response separates routing and wording through a bounded ReAct orchestrator over typed tools and a smaller generator constrained by a backend-validated context contract.
Results
Replay and low-ramp A/B results report zero observed structured-action hallucinations, reduced escalation responses, and directional self-solve improvement, with architecture effects isolated on identical replayed turns.
Takeaways & Limitations
Typed, backend-validated decisions and downstream constrained wording make orchestration behavior more inspectable and failures more localizable in this production support setting.
Takeaways & Limitations
The evidence comes from one support domain with mature tools, stable policy sources, and human fallback, and the migration changes architecture, prompts, alignment, and serving together.
Abstract
from arXiv · showhide
Conversational assistants can blend retrieval, action selection, escalation, and wording in a single model path, or separate those roles. We report a production migration of a customer-support assistant at a large accommodation marketplace (millions of conversations per month, 11 languages, 10-second P90). Dynamic Response (DR) replaces a single Qwen3-235B-A22B blended responder with a bounded ReAct orchestrator over typed tools plus a smaller generator that writes from a backend-validated context contract. Because the migration also changed prompts, alignment, and serving, we attribute each effect to its cause and claim as architecture effects only those measured on identical replayed turns: typed entity selection moves the reservation selector to a precision-first operating point (precision 8.3% to 89.1%, recall 75.2% to 67.3%), and typed action IDs with a membership check remove observed structured-action hallucination (2.14% to 0.0%). A low-ramp A/B test reproduces the replay escalation reductions: hard-escalation responses fall from 5.60% to 3.08% and soft-escalation responses from 9.56% to 2.49%, while production handoff volume holds roughly steady; self-solve is directional (+5.1 points, 95% CI [-2, +12]). Serving optimizations cut orchestrator P90 latency from 3.87s to 2.24s on a GPU footprint reduced by roughly one-third, and self-hosting reduces estimated annual model-serving cost by more than an order of magnitude.
1 Introduction
Customer-support turns require both decision making and wording under a strict latency and multilingual production setting. Dynamic Response separates high-consequence decisions from user-facing prose so failures can be localized and decisions constrained.
- Support assistants must answer policy questions, select cancellation and refund flows, clarify requests, and route users to humans within a 10-second P90 budget across 11 languages.
- A blended responder combines retrieval, entity and action selection, escalation decisions, and reply writing, but these responsibilities fail differently.
- Dynamic Response assigns routing, action, and escalation decisions to a large orchestrator while a smaller generator writes from a backend-validated context contract.
- The paper claims architecture effects only for identical replayed turns because the migration also changed prompts, alignment, and serving.
2 Background and Problem Setting
The support setting combines cross-source reasoning, structured actions, clarification, information delivery, and human routing. The predecessor’s single blended path simplified serving but entangled deciding what to do with deciding what to say.
- Support turns may require reservation, listing, payment, account, policy, help-center, translation, structured-action, clarification, and human-routing decisions.
- A cancellation decision can depend on reservation dates, listing policy, payment state, and regional rules, making traceability and unsupported-action prevention important.
- The predecessor used one Qwen3-235B-A22B-class model path to choose response strategy, structured actions, escalation, and user-facing wording from preassembled context.
- RAG grounds generation in retrieved evidence, while ReAct and tool-augmented LLMs provide related patterns for interleaving reasoning, actions, observations, and external-tool use.
3 Dynamic Response Architecture
Dynamic Response uses a bounded, observable ReAct runtime that separates routing from wording and retrieves context on demand. A large model orchestrates typed tools, while a smaller model generates from the resulting contract.
- Design Principles and Runtime: DR separates routing from wording, retrieves only needed context, and logs each orchestration step as a structured event.
- Design Principles and Runtime: The orchestrator runs a bounded ReAct loop capped at three steps and may call retrieval, selector, escalation, or generation tools before post-generation validation.
- Model Roles: The large Qwen3-235B-A22B orchestrator makes structured tool-call, terminal-action, and fallback decisions, while Qwen3-Next-80B-A3B writes the final response.
- Tool Registry: Typed registry entries define schemas, eligibility, logging fields, and guardrail hooks, enabling backend validation of IDs, reason codes, and source membership.
- Migration Stages: Stage 2 introduced the full tool-mediated loop, letting the orchestrator choose retrieval, entity selection, structured action, escalation, and generation.
4 Context Contract, Guardrails, and Fallback
The context contract makes the orchestrator–generator boundary explicit and inspectable. Guardrails validate inputs, tool use, state consistency, references, actions, formatting, and risk-sensitive phrasing, with conservative fallback for invalid execution.
- Context Contract: The forward contract carries task state, eligible tools, retrieved sources, user-state variables, policy constraints, locale, and prior observations.
- Context Contract: The generator may reference only tools, sources, and user-state keys present in the contract, with post-generation checks enforcing this closure.
- Context Contract: Including orchestration state preserves coherence because cited-only handoff lost context such as why an action was unavailable.
- Guardrails and Fallback: Pre-, mid-, and post-generation checks validate safety, tool schemas, eligibility, prior-state consistency, references, action-card IDs, format, and risk-sensitive phrasing.
- Guardrails and Fallback: DR falls back to conservative generation over available context when the orchestrator exceeds its iteration limit, emits an invalid tool call, or receives inconsistent observations.
5 Model and Serving Choices
DR uses self-hosted serving and latency-focused optimizations to meet production constraints while reducing infrastructure cost. Orchestrator P90 latency falls from 3.87s to 2.24s, with GPU footprint reduced by roughly one-third.
- Cost: More than one order of magnitude lower estimated annual model-serving cost follows self-hosting relative to a proprietary-API baseline.The estimate covers amortized hardware and power for the production footprint, excluding engineering, model-development, and shared-platform costs.
- Latency: 3.87s to 2.24s: tensor-parallel-only serving, asynchronous scheduling, and FlashInfer reduce orchestrator P90 latency.The GPU footprint also shrinks by roughly one-third.
- Serving footprint: Roughly one-third lower GPU footprint accompanies the reduction in orchestrator P90 latency.Latency, rather than throughput, was the binding constraint on H100 GPUs.
6 Controlled Comparison and Launch Process
DR is evaluated through replay, review, risk monitoring, and a low-ramp A/B test, with claims separated by their evidence and causal scope. The comparison reports architecture-local gains in typed decisions, reduced escalation-framed responses online, and production release within latency and quality constraints, while retaining important limitations and monitoring boundaries.
- Launch process: 6.539s end-to-end P90 remains within the 10-second release gate on live traffic, with orchestration the dominant stage at about 58% of median turn time.The latency table reports 4,155 turns and stage sums computed per turn.
- Attribution: Identical replayed turns localize architecture effects, while model, alignment, and serving effects are treated separately because they could also benefit a single-call system.The paper explicitly does not claim a pure architecture-only ablation.
- Evaluation design: Replay, human and risk review, and online A/B testing map evidence to architecture-local behavior, release-gated quality, online outcomes, and serving claims.The evaluation uses fixed production release checks and identical eligibility for the online experiment.
- Shadow replay: 0.0% delivered structured-action hallucination follows typed action references, contract closure, and backend blocking of spurious action IDs.The claim is scoped to structured-action references; blocked attempts are logged separately.
- Shadow replay: On the miss slice, DR gives entity-free answers judged correct in 47.75% of turns, while wrong-entity references occur in 1.01% of turns.This slice is identified as an ongoing weakness in resolution quality.
- Shadow replay: The deployed structured-action configuration sacrifices a small amount of F1 to satisfy false-positive and recall gates, while Monolithic-level F1 remains achievable on the frontier.The shipped claim is gate compliance; parity refers only to the achievable frontier.
- Shadow replay: Escalation as a typed tool reduces avoidable soft escalation, while replay indicates nearly all Monolithic hard escalations remain hard-escalated.Risk prevalence on the same replay traffic provides a check against interpreting the reduction without accounting for exposure.
7 Production Rollout and Operations
DR rolled out through staged replay, canary, regional expansion, and retained rollback, with separate logs enabling component-level diagnosis and rollback.
- DR rolled out through zero-exposure replay, a low-traffic canary with immediate fallback, regional and language expansion, and retained rollback.
- Separate logging for orchestration, generation, tool calls, and post-generation checks lets operators localize failures and roll back affected components.
8 Lessons Learned
DR treats the orchestrator–generator handoff as an explicit typed boundary and assigns consequential decisions to tools rather than final prose. This separation lets the generator focus on grounded wording while observability and self-hosting support operational control.
- The typed contract makes the orchestrator–generator boundary inspectable and gives guardrails concrete objects to validate.
- DR assigns escalation, entity selection, and action-card rendering to orchestrator tools, preventing final wording from creating new actions or escalations.
- The generator can be smaller because it no longer performs global planning, provided the contract carries grounding evidence and prevents action changes.
- Self-hosting makes the split practical by enabling serving optimization while retaining the previous system as a large-scale fallback.
9 Conclusion
Dynamic Response is a two-model agentic architecture in which a large orchestrator plans through tools and a smaller generator writes the final reply. Self-hosting and serving optimizations support multilingual production latency, while the architecture’s benefits depend on explicit contracts and guardrails.
- Dynamic Response uses a large orchestrator for planning and tool calls and a smaller generator for final responses.
- Self-hosted open-weight MoE models and serving optimizations fit a 10-second P90 multilingual production budget with observable, modular control flow.
- The added complexity pays off when the handoff contract, guardrails, and serving stack are treated as first-class architectural components.
Limitations
The paper’s evidence is bounded by its production setting, limited reproducibility materials, and a comparison that is not a pure architecture-only causal ablation.
- DR was evaluated in one support domain with mature tools, stable policy sources, and a human-support fallback.
- Decomposition is most likely to pay off when typed decisions require logging, validation, and rollback, latency can absorb a second model call, and tools and policy are mature.
- The paper releases interface schemas and a release-evidence framework, but proprietary shadow sets, judge prompts, and labeled data cannot be released.
- The paper omits the companion methodology for constructing evaluator roles, certifying LLM judges, calibrating offline scores to online outcomes, and selecting evaluation materials.
- The comparison should not be read as a pure causal ablation of architecture alone because DR was evaluated as a production migration.
Ethical Considerations
The system uses layered safeguards, typed escalation, and restricted context contracts to reduce harmful or unsupported outputs while keeping human handoff available and monitorable.
- Safeguards: Human escalation remains available, and safety-sensitive topics can be routed automatically through typed escalation reason codes.The system monitors under-escalation by risk category and language in production.
- Safeguards: Pre-, mid-, and post-generation guardrails reduce harmful, discriminatory, privacy-violating, and policy-violating outputs.Dedicated controls are applied at multiple stages of the response pipeline.
- Contract controls: Reference closure requires every returned article ID, structured action, and user-state key to come from the forwarded context.This constrains what the generator can reference and return to the backend.
- Contract controls: Role separation prevents the generator from choosing a different action path while allowing it to explain the selected action.The return contract also supports replayability and keeps localization in backend services.
B Additional Error and Component Analysis
This appendix documents DR’s operational checks, typed interfaces, routing analyses, and serving-cost estimates, while separating architecture evidence from evaluation-methodology contributions.
- Structured selection: DR reaches F1 parity for structured-action selection while making the routing operating point explicit.Table 10 reports precision, recall, F1, and false-positive proportion across routing-prompt iterations.
- Scope and evidence: DR’s appendix reports component checks and design choices, while scoring-instrument construction and production-outcome validation remain out of scope.The documented decomposition covers planner choices, context, routing prompts, and model placement.
- Architectural decomposition: DR separates routing from wording: the orchestrator chooses tools, actions, and escalation, while the generator writes from a curated typed contract.The bounded ReAct loop logs structured events and post-generation checks validate references, unsupported actions, and safety-sensitive output.
- Structured selection: The selector can return a concrete entity ID, null when no entity is needed, or a clarification request when multiple entities remain plausible.Its input includes candidate IDs, activity status, provenance, and mention spans.
- Routing optimization: Routing-prompt iteration first increased recall and over-firing, then reduced over-firing while preserving most recall.Adding selector context raised recall further before later instructions adjusted the operating point.
- Retrieval routing: Stage-aware routing sends “why” and “can I” queries toward policy or overview articles and “how” or “what do I do” queries toward procedural articles.The generator explains the selected source rather than searching again, preventing late-stage source drift.
- Model placement: The smaller generator S was selected because it met release thresholds at lower P90 and about half the serving footprint of L.The comparison included the larger model and frontier APIs on a production-shaped synthetic set.
- Serving-cost estimate: Self-hosting is estimated at roughly $200K annually versus roughly $6M for the proprietary-API baseline at the stated volume and decoding settings.The authors characterize these as internal, order-of-magnitude estimates rather than audited financials.