PaperlayerView exact revision on arXiv

Source-linked AI summary

TaoLive Digital Avatar Agent Technical Report: Training Agents to Evolve with Their Harness

TaoLive AIGC LLM Team, Yuhan Sun, Wenhao Lin, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weihang Pan, Jiaxin Zhao, Zulong Chen

arXiv:2608.15763v1cs.CL

TL;DR

When an e-commerce avatar’s Harness changes while its model remains fixed, a compact agent trained on one configuration may become stale. The paper introduces Harness-Aware Training to expose the model to Harness-state variation; the trained model reaches 94.8 Live-Stream QA AVG without the general-set drop observed after Fixed-Harness SFT.

  • Problem

    As runtime Harness states evolve while model weights remain fixed, models trained on one state may become stale, creating a training gap around adapting to the current execution environment.

    Source: §2.3 · p. 4

  • Method

    HAT incorporates Harness-state variation into training through HSA-SFT, general OPD, and HSA-RL.

    Source: §8 · p. 20

  • Results

    94.8 Live-Stream QA AVG, with no general-set drop observed after Fixed-Harness SFT and retained adaptability to evolving production Harnesses.

    Source: §8 · p. 20

  • Takeaways & Limitations

    HAT can produce a latency-feasible compact agent while preserving adaptability needed for evolving production Harnesses.

    Source: §8 · p. 20

    The production A/B test is ongoing, and its first-seven-day point estimates lack confidence intervals and significance tests.

    Source: §6.4 · p. 18

Abstract

from arXiv · show

AI-powered digital-avatar streamers in live e-commerce must answer product questions, engage viewers, and execute changing business strategies in real time. This requires low latency, factual and effective replies, and rapid adaptation to updated campaign, compliance, and style requirements. We develop an evolvable Harness that decouples Skills, Hooks, system prompts, and tools from model weights, allowing runtime behavior to change without retraining. However, Harness evolution creates a moving execution environment: compact models fine-tuned on one configuration may memorize names, schemas, and prompt templates rather than follow the Harness currently provided, while stronger zero-shot models are too slow for real-time use. We address this tension with Harness-Aware Training (HAT), which makes Harness states part of the training distribution. HAT applies task-preserving Harness-State Augmentation (HSA) to Skills, tool schemas, prompt structures, and interaction constraints, and comprises three stages: HSA-based supervised fine-tuning, general on-policy distillation to recover general capabilities, and HSA-based agentic reinforcement learning in a production-informed live-room simulator. Across four evaluation sets with more than 4,500 cases, our compact 35B model scores 94.8 on real-world Live-Stream QA, versus 80.3 for the base model and 93.0 for the strongest evaluated general LLM, while scoring 94.6 on Harness-Variant QA and retaining 83.5 on IFEval. By contrast, fixed-Harness SFT reduces IFEval by 7.7 points. In a controlled complete-agent replay on one NVIDIA H20 GPU with MTP enabled, the system achieves 3.407 s P50 and 8.114 s P95 latency. These results show that HAT produces a latency-feasible compact agent that remains effective under evaluated Harness changes without sacrificing general instruction following.

1 Introduction

The paper presents an evolvable Harness for live-commerce agents and Harness-Aware Training (HAT), which trains compact policies across Harness states so runtime rules can change without retraining while the model learns to adapt to those changes.

Harness Evolution

  • The evolvable Harness separates Skills, Hooks, system-prompt assembly, and commerce tools for developer-reviewed updates without retraining, while changing the execution environment beneath fixed model weights.Before this design, updates to failure cases or merchant strategies could require changes to central prompts or control code followed by service deployment or retraining.Source: §1 · pp. 1–2

Motivation

  • DeepSeek-V4 models had median end-to-end latencies exceeding 11 seconds, while Fixed-Harness SFT degraded IFEval by 7.7 points as models overfit to names and prompt templates.Test-time adaptation was ruled out by latency budgets, and continual learning would require retraining after each Harness edit.Source: §1 · p. 2

Harness-Aware Training

  • HAT trains over Harness states using task-preserving perturbations to Skills, tool schemas, prompts, and interaction constraints, in HSA-SFT, General OPD, and HSA-RL stages.HSA-RL takes place in a production-informed live-room simulator, where the model learns from consequences under shifting conditions.Source: §1 · pp. 2–3

Results

  • 94.8 average score on real-world Live-Stream QA surpassed the untuned base model (80.3) and strong zero-shot models; the selected checkpoint remained robust to evaluated Harness changes and avoided Fixed-Harness SFT’s IFEval loss.The introduction does not give scores for the strong zero-shot models or the size of the IFEval loss avoided.Source: §1 · p. 2
  • 3.407s P50 and 8.114s P95 wall-clock latency were achieved by the complete agent in controlled deployment tests on one NVIDIA-H20 GPU with MTP enabled.The Harness-based judge was calibrated against human annotations.Source: §1 · p. 2

Evaluation and Findings

  • Four complementary evaluation sets covered more than 4,500 cases, and experiments found a larger observed impact from HSA during SFT than during RL, with General OPD recovering capabilities lost during domain adaptation.The Harness-based Agent-as-a-Judge was validated against human labels for scalable evaluation.Source: §1 · p. 3

2 Digital-Avatar Harness Agent Architecture

The architecture separates a slowly updated policy model from a rapidly evolving Harness, letting operators change business behavior without retraining while requiring the policy to remain effective as its environment changes. It supports a live-commerce avatar host through modular runtime components and a human-in-the-loop process for evolving those components.

2.1 Live-Streaming Digital-Avatar Scenario

  • The digital-avatar host answers product questions, handles open-ended conversation, and invokes external tools in live commerce, where business rules evolve weekly and low latency is required.Public spoken responses are synthesized through TTS and avatar driving for the whole room.Source: §2.1 · p. 3

2.2 The Harness Agent Runtime

  • The runtime combines dynamically loaded Skills, a System Prompt Pipeline, validation Hooks, and orchestration for external product, inventory, and commerce services.These modules are versioned independently of model weights, so operators can deploy a new Skill or tool as a Harness change rather than a training cycle.Source: §2.2 · p. 3

2.3 Harness Evolution without Weight Updates

  • Harness Evolution changes Skills, prompts, Hooks, or tools while holding the model fixed, with AI proposing edits and a developer evaluating and deciding whether to promote, revise, or stop them.The process is human-in-the-loop, not fully autonomous.Source: §2.3 · p. 3
  • Evolution 2 reaches 92.55 Accuracy and 92.75 Effectiveness on the 482-case calibration set and is selected after later long-tail edits regress one or both dimensions.Moving from ReAct to the modular harness raises Effectiveness from 84.58 to 87.16; Evolution 1 raises Accuracy from 82.40 to 92.13 but reduces Effectiveness to 84.16.Source: Figure 2 · p. 4, §2.3 · p. 4
  • Each promoted Harness version creates a new state while the model remains fixed, so a model that memorizes an earlier state can become stale as the system evolves.Source: §2.3 · p. 4

Task Interface

  • Each request combines the viewer comment with anchor utterances, live-room and product context, FAQ entries, and recent dialogue, then returns a reply and the structured tool-call trajectory.The reply may be public speech routed to TTS and avatar driving or a private text response.Source: Task Interface · p. 5

Skill Routing

  • The Harness offers retrieval, commerce, and flow-control interfaces, can discover marketing interfaces through MCP, and requires at least one Reply Skill while allowing optional Strategy Skills.The release inventory and Skill versions are reconciled against a manifest before promotion.Source: Skill Routing · p. 5, Runtime Interfaces · p. 5

Execution and Trajectory Record

  • The request path loads and normalizes context, assembles the prompt, runs a bounded policy–tool loop, applies lifecycle checks, then dispatches the reply while retaining the structured trajectory.Hooks validate inputs, tool parameters, Skill-loading requirements, response format, factuality risks, and retry control; per-call fallback and trajectory-level recovery address different failure scopes.Source: Execution and Trajectory Record · pp. 5–6, Figure 3 · p. 6

3 Harness-Aware Training (HAT)

HAT trains the agent to operate across changing Harness configurations while retaining general instruction-following capability. It combines Harness-State Augmentation (HSA), general on-policy distillation, and agentic reinforcement learning in a live-room simulator.

3.1 Problem Formulation

  • The policy conditions on user input and business context plus a Harness comprising Skills, tools, system prompts, and Hooks, and is trained to handle deployment Harness changes while retaining general capabilities.The objective covers effectiveness under a deployment change envelope broader than a single training snapshot and preservation of capabilities such as instruction following.Source: §3.1 · pp. 6–7

3.2 Harness-State Augmentation (HSA)

  • HSA varies five Harness dimensions—Skill identifiers and content, tool definitions, system prompts, and Hooks—to reduce dependence on fixed names, wording, order, and templates.It renames or rewrites Skills and tools, masks or reorders Skill rules, reorders prompt instructions and varies valid numeric constraints, and changes Hook behavior and messages.Source: §3.2 · p. 7

3.3 Simulated On-Policy Environment

  • The production-informed simulator trains agents through multi-step interactions and their own failures, using tool, reply, and Skill-selection rewards with GRPO optimization.Sandboxed tools return catalog information and inject timeout, malformed-response, and permission failures; the reward also measures accuracy and effectiveness, while CoT length is discouraged with thresholds of 100 and 200 tokens. GRPO [Shao et al., 2024] uses grouped complete-trajectory rollouts.Source: §3.3 · pp. 8–9

3.4 Training Pipeline

  • HSA-SFT uses teacher-generated trajectories under original and augmented Harnesses, filtered for accuracy and effectiveness, to teach shared task semantics across Harness variants.For most Hook-triggered retries, failed intermediate outputs and Hook interactions are removed to avoid teaching a default-error strategy; a small fraction of complete retries remains to teach recovery.Source: §3.4 · pp. 9–10
  • After HSA-SFT, General OPD distills the pre-SFT base model on Tulu3 instructions, then HSA-RL trains complete simulated trajectories under original or augmented Harnesses.General OPD minimizes KL divergence between student and base-model distributions; HSA-RL applies augmentation before each rollout, exposing some trajectories to the original production configuration and others to augmented states.Source: §3.4 · p. 10

4 Evaluation Design

The evaluation combines real live-room QA, Harness and prompt/tool robustness, and general instruction following with a fixed complete-Agent deployment replay. It defines task-specific scoring and a controlled protocol for measuring serving performance.

4.1 Evaluation Sets and Sources

  • Live-Stream QA and Harness-Variant QA assess real-room reply quality and robustness to Harness changes, alongside synthetic robustness subsets, IFEval, and a separate judge-calibration cohort.Tool Robustness and Prompt Robustness expand real-industrial scenario seeds using LLM generation and consistency validation; the fixed complete-Agent replay measures deployment latency and serving performance.Source: §4.1 · p. 10

4.2 Metrics and Scoring

  • Accuracy, Effectiveness, and their per-sample mean AVG score Live-Stream QA and Harness-Variant QA responses using a Harness-based Agent-as-a-Judge.Accuracy is binary and penalizes factual errors, unsupported product claims, false promises, and capability overreach; Effectiveness scores intent satisfaction and useful information as 0, 0.5, or 1, with AVG computed before rounding.Source: §4.2 · p. 10
  • Tool Robustness uses ADR, RC, and TC, while Prompt Robustness uses ADR, LLM, and TCO to score query response, tool use, and instruction constraints.Tool Robustness measures core-query coverage, use of required tool-returned information, and expected tool calls; Prompt Robustness measures core-query coverage, active instruction constraints, and required tool-call order.Source: §4.2 · pp. 10–11
  • IFEval uses its official evaluator to report prompt-level IFE-P and instruction-level IFE-I accuracy.Deployment replay reports wall-clock latency P50 and P95, TTFT, decoding throughput, execution success, and 15-second attainment; the attainment denominator includes all measured requests.Source: §4.2 · p. 11

4.3 Deployment Inference Protocol

  • The replay evaluates 100 measured requests after 10 warm-up cases, running the complete Agent with one Harness commit, real MCP services, and at most four agent loops.Tool calls time out after 5 seconds; Qwen3.6-35B-A3B and the task-trained checkpoint run in BF16 on one H20 with TP=PP=1, using NextN MTP and a shared EAGLE-style speculative-generation path.Source: §4.3 · p. 11

5 Experiments

Experiments train Qwen3.6-35B-A3B on 10K SFT examples and 4K RL interaction tasks, then evaluate offline quality, Harness robustness, instruction following, and deployment behavior. The results compare HAT with fixed-Harness training and assess optimization choices, deployment performance, Harness edits, and human preferences.

5.2 Main Results

  • 94.8 on T1: HAT exceeds the strongest Top-model score of 93.0 on the same business dataset.The full pipeline is HSA-SFT + General OPD + HSA-RL; it avoids the robustness and generalization drops observed after Fixed-Harness SFT.Source: Table 3 · p. 12, §5.2 · p. 12
  • −4.6 Prompt Robustness and −7.7/−5.3 IFEval accompany Fixed-Harness SFT’s in-family score gains, consistent with reduced robustness and generalization.The reported pattern may reflect shortcut learning or memorization of tool-use patterns rather than robust interpretation of the current Harness context.Source: §5.2 · p. 12, Controlled On-Policy Optimization… · p. 13

Controlled On-Policy Optimization Comparison

  • OPD estimates trail paired RL; HSA-RL improves over Agentic RL in the original Harness, while RL reaches a better final operating point from HSA-SFT + General OPD than from Fixed-Harness SFT + General OPD.In the original Harness, OPD is lower than RL by 0.1 points on T1, 0.2 on IFE-P, and 4.8 on Prompt Robustness; in the HSA environment, the corresponding gaps are 0.9, 1.8, and 2.1 points.Source: Controlled On-Policy Optimization… · pp. 12–13

HSA Stage Isolation

  • +9.4 IFE-P is associated with HSA-SFT, compared with +1.0 for HSA-RL, in runs without General OPD.The differences are descriptive and exclude retraining variance; reward trajectories separately diagnose Accuracy, Effectiveness, Skill Selection, and Tool Rationality across four SFT/RL configurations.Source: HSA Stage Isolation · p. 13, Table 6 · p. 14, §5.4 · p. 14

5.4 Controlled Deployment Inference

  • 8.114 s and 9.047 s are selected-MTP P95 latencies at concurrency 1 and 2; all measured requests meet the 15-second bound.On the same H20 checkpoint, MTP raises decoding throughput from 160.30 to 271.40 tokens/s at concurrency 1 and from 129.22 to 196.15 at concurrency 2, with no aggregate quality degradation observed in this run.Source: §5.4 · pp. 13–14, Table 7 · p. 15

5.5 Held-Out Evaluation of Harness Edits

  • The same Harness edits reduce held-out error counts for both Fixed-Harness SFT and HAT, with the largest absolute reduction observed for HAT.The edits were developed on a separate development split, kept model weights fixed, and applied unchanged to both checkpoints.Source: §5.5 · pp. 14–15, Table 8 · p. 15

5.6 Human Blind-Test Protocol and Results

  • 35 of 36 clear-preference cases favor Harness, a 97.2% non-tie win rate (two-sided binomial p<0.001) against the ReAct-style baseline.Input understanding, output reliability, and scenario-appropriate behavior account for 80.0% of Harness-preferred cases in the blind test.Source: Table 9 · p. 16, §5.6 · p. 16, Table 10 · p. 16

5.7 Agentic Optimization Diagnostics

  • HSA-SFT is associated with higher reply-quality rewards, HSA-RL with stronger agentic-behavior rewards, and their combination with the most balanced late-stage profile.The agentic-behavior dimensions highlighted are Skill Selection and Tool Rationality.Source: §5.7 · p. 16, Figure 5 · p. 17

6 Deployment and Production Lessons

Deployment separates policy and Harness evolution, applies release gates and runtime safeguards, and provisions for low concurrency. A preliminary production A/B snapshot reports uplifts for the complete Harness system relative to ReAct.

6.1 Evolution-Aware Release Process

  • Policy checkpoints and Harness components evolve independently, with a release manifest and three gates required before promotion.Proposed changes must pass the 15-second latency bound, business regression suite, and general-capability gate; operators retain the last passing version when long-tail fixes cause cross-category regressions.Source: §6.1 · pp. 17–18

6.2 Failure Containment and Recovery

  • Defense-in-depth Hooks, bounded retries, timeouts, and a compact fallback route contain tool and response failures.After four model rounds or a 10 s trajectory budget, recovery failure leads to a conservative response or suppression; release checks target unsupported product claims, false human-driver claims, and repeatedly failing or incomplete tool calls.Source: §6.2 · p. 18

6.3 Low-Concurrency Serving Policy

  • At concurrency 1–2, MTP gives the selected checkpoint 1.69×/1.52× higher client-observed decoding throughput, with 8.114/9.047 s P95 and 100% 15-second attainment.At concurrency 4–8, the decoding benefit falls to 1.27×/1.19× and MTP no longer improves P95, motivating additional replicas rather than treating MTP as a universal high-concurrency optimization.Source: §6.3 · p. 18

6.4 Preliminary Online A/B Test on Taobao Live

  • 5.54% higher confirmed-receipt GMV per bucketed user and 0.80% higher completed orders per bucketed user were observed for Harness versus ReAct.These are unadjusted seven-day point estimates from an ongoing user-level A/B test, with no confidence intervals or significance tests available. Metrics are relative changes; the control and treatment had 678,598 and 75,754 bucketed users, respectively.Source: §6.4 · pp. 18–19, Table 11 · p. 18

7 Related Work

The section frames Harness Evolution as making editable runtime configuration an explicit release process and training-distribution object. It distinguishes this approach from prior work on agent architectures, evaluation robustness, post-training, and distribution shift.

Agent architectures and Harness Evolution

  • Harness Evolution makes editable runtime configuration an explicit release process and training-distribution object, complementing agent architectures such as ReAct [Yao et al., 2023] and DSPy [Khattab et al., 2023].Unlike infrastructure configuration-drift detection in RIVA [Abuzakuk et al., 2026], the changing configuration here is the agent’s own execution environment.Source: §7 · p. 19

Prompt, schema, and agent-evaluation robustness

  • Prompt and tool-description edits can change model behavior [Zhuo et al., 2024; Faghih et al., 2025], motivating interface variation and evaluation beyond a single frozen surface form.T3 targets broad live-stream avatar interactions and complements T2’s controlled Harness variants; prior benchmarks include multi-turn stateful settings and dynamic e-commerce simulation.Source: §7 · p. 19

Agent post-training

  • This work’s contribution is treating the execution environment as a changing training-distribution object and combining established mechanisms, rather than introducing a new reward, credit-assignment rule, or optimizer.Its environment preserves the real business ontology while randomizing the editable Harness; on-policy distillation anchors general behavior after domain SFT [Agarwal et al., 2024].Source: §7 · p. 19

Training under distribution shift

  • Harness-State Augmentation is domain randomization [Tobin et al., 2017] over coupled Skills, prompts, tools, Hooks, and message logic, adapting agents to declared Harness changes.T2 tests in-family changes and T3 broader in-domain scenario transfer; neither validates unseen Harness families.Source: §7 · p. 19

8 Conclusion … B.1 Evolution Loop

The paper presents Harness-Aware Training (HAT) as a way to train compact live-streaming agents for changing Harness configurations, combining Harness-State Augmentation with a three-stage training pipeline. It reports benchmark and controlled-replay results, and describes inventories and a human-reviewed Harness evolution loop.

8 Conclusion

  • HAT treats Harness-state variation as part of the training distribution, using HSA-SFT, General OPD, and HSA-RL.It frames this approach within an explicitly defined change envelope for versioned runtime changes and policy stability.Source: §8 · p. 20
  • Scoped offline sets, a human-aligned evaluation panel, and controlled complete-Agent replay distinguish checkpoint-quality evidence from serving-route behavior.The replay was conducted on one H20 with MTP enabled.Source: §8 · p. 20
  • General OPD mainly recovers instruction-following ability after Fixed-Harness SFT, while HSA is most effective when introduced during SFT.The HAT-trained compact model does not exhibit the general-set drop observed after Fixed-Harness SFT.Source: §8 · p. 20
  • 94.8 Live-Stream QA AVG; the controlled replay reached 8.114 s P95 at concurrency 1 and 9.047 s at concurrency 2, with 100% 15-second attainment.With MTP enabled, client-observed decoding throughput increased by 1.69×/1.52× over the same checkpoint without MTP.Source: §8 · p. 20

A Runtime Interface and Skill Inventories

  • The runtime-interface inventory lists built-in and MCP-discovered interfaces, while the Skill inventory lists Skills and their routing responsibilities.Source: Table 12 · p. 24, Table 13 · p. 25

B.1 Evolution Loop

  • Harness evolution holds the model fixed while changing runtime modules through AI diagnosis, human confirmation, AI-assisted editing, human-triggered evaluation, and regression review.The operational stopping rule is to stop when errors are sparse long-tail cases and local rule additions are more likely to cause cross-category regression than systematic improvement.Source: §B.1 · p. 25

B.2 What Changed across Evolution Stages

The report attributes the largest gains to specific Accuracy and Effectiveness categories, while later evolution stages show that targeted fixes can interact across the Harness. It records remaining failures after Evolution 2 and motivates configuration diversity and regression-aware stopping.

B.2 What Changed across Evolution Stages

  • Accuracy gains peaked at 60.7 points for negative-feedback chat, 53.3 for after-sales and returns, and 33.3 for purchase/payment cases.These category values are engineering diagnostics, not pre-specified inferential analyses.Source: §B.2 · p. 25
  • Effectiveness gains peaked at 27.3 points for emotion expression, 18.8 for membership or gift benefits, and 14.3 for positive-feedback or already-purchased cases.These category values are engineering diagnostics, not pre-specified inferential analyses.Source: §B.2 · p. 25
  • After Evolution 2, 36 Accuracy failures remained across 16 categories and 35 Effectiveness failures had no dominant systematic cluster.Evolutions 3–4 showed that targeted long-tail fixes can interact across Skills, Hooks, and global instructions, motivating configuration diversity during training and regression-aware early stopping during production evolution. Table 14 records six direct evaluation summaries on the same 482-item dev-set; “Selected” denotes an engineering early-stop decision.Source: §B.2 · p. 25, Table 14 · p. 26

C HSA-RL Training Details

HSA-RL combines Harness-State Augmentation with multi-dimensional advantage estimation and sequence-level importance sampling. In a single training run, diagnostic windows show shorter tool-call and final-reply reasoning traces.

C HSA-RL Training Details

  • Algorithm 1 combines HSA with GDPO multi-dimensional advantage estimation, an auxiliary CoT-length dimension, GSPO sequence-level importance sampling, and zero-advantage sample filtering.The dimensions cover Tool Rationality and Skill Selection for tool actions and Accuracy and Effectiveness for replies; the post-normalization CoT weight is λ=0.1.Source: C HSA-RL Training Details · p. 26
  • 290.6→171.2 extracted tokens for tool-call CoT and 93.6→37.0 for final-reply CoT; mean q(L) fell from 0.466 to 0.272.The strongest reduction occurred early, followed by a noisy plateau; the raw reward contribution before GDPO normalization moved from −0.0466 to −0.0272.Source: C HSA-RL Training Details · p. 26

D Judge as a Harness Evolution Case · E Evaluation and Reproducibility Details · E.1 Judge Roles and Independence

The paper treats judging as an evolvable Harness component, using disagreement-driven revisions to improve alignment on human-labeled interactions. It also distinguishes the judges’ training and evaluation roles and documents where their independence is limited.

D Judge as a Harness Evolution Case

  • Skills, prompts, and tool-verification procedures are revised from human-label disagreements, targeting recurring clusters such as missed multi-comment intents and unsupported capability claims.Other clusters include incorrect product linkage and inconsistent grading of partial answers.Source: D Judge as a Harness Evolution Case · p. 26

E.1 Judge Roles and Independence

  • The workflow separates five roles: DeepSeek-V4-pro generates SFT candidates and votes in final evaluation; rejection sampling filters candidates; DeepSeek-V4-flash scores HSA-RL; humans calibrate.The Final Evaluation Judge produces offline scores through majority voting over DeepSeek-V4-pro, GLM-5.2, and Qwen3.7-max.Source: §E.1 · p. 26
  • The Reward and Final Evaluation Judges have zero model overlap because DeepSeek-V4-flash is not in the final three-model panel.DeepSeek-V4-flash scores HSA-RL trajectories; the final panel comprises DeepSeek-V4-pro, GLM-5.2, and Qwen3.7-max.Source: §E.1 · p. 26
  • DeepSeek-V4-pro is both the SFT teacher and a final voter, but a two-of-three majority prevents its vote from deciding an item alone.GLM-5.2 and Qwen3.7-max provide two non-teacher votes.Source: §E.1 · p. 26

D Judge as a Harness Evolution Case

  • Accuracy agreement rises from 81.54% at Evolution 1 to 90.46% at Evolution 6, while Effectiveness rises from a 63.70% uncalibrated baseline to an 83.40% peak; selected voting reaches 82.2% Effectiveness agreement.The calibration cohort contains 482 human-labeled interactions and evaluates the scoring instrument rather than the policy.Source: D Judge as a Harness Evolution Case · p. 26, Figure 7 · p. 27

E.1 Judge Roles and Independence

  • All three final voters can call evidence tools for product-fact checks, and human alignment is measured before the panel is used for model comparison.Source: §E.1 · p. 27

E.2 Offline Scoring and Implementation … F Deployment Inference and Operating Details

These sections specify offline scoring and HSA-RL implementation, describe how synthetic T3 robustness subsets are constructed and scored, and explain an independent bootstrap stability check. The supplied passages do not provide deployment-inference or operating details.

E.2 Offline Scoring and Implementation

  • HSA-RL applies GRPO to the Qwen3.6-35B-A3B policy, using Accuracy, Effectiveness, Tool Rationality, and Skill Selection rewards under five types of Harness perturbations.The perturbations cover Skills, tools, prompts, Hooks, and message processing.Source: §E.2 · p. 28
  • Accuracy is binary, Effectiveness takes 0, 0.5, or 1, and AVG is their per-sample mean before rounding; model comparisons use identical set versions and the same Final Evaluation Judge.T3 reports Tool and Prompt robustness separately, while IFEval uses its official prompt- and instruction-level evaluators.Source: §E.2 · p. 28

E.3 T3 Robustness Subset Construction

  • T3 comprises synthetic Tool Robustness (n=1,532) and Prompt Robustness (n=491) subsets seeded from live-streaming e-commerce scenarios.The seed library records product context, viewer intent, possible tool needs, and reply constraints for each scenario family.Source: §E.3 · p. 28
  • A shared seed-driven pipeline expands real-business data into viewer comments, contexts, and expected behaviors, then formats and validates each item against its Harness configuration and rubrics.The source data include product context, FAQ-style knowledge, dialogue history, and historical tool-use traces; inconsistent items are discarded and regenerated.Source: §E.3 · p. 28
  • Tool Robustness tests tool selection amid distractors, while Prompt Robustness tests valid instruction constraints, using separate rubric metrics and per-item pass fractions for AVG.Tool metrics are TC, RC, and ADR; Prompt metrics are ADR, LLM, and TCO. T3 targets robustness to unseen in-domain tools and instructions in synthetic live-streaming scenarios.Source: §E.3 · pp. 28–29

E.4 Independent Bootstrap Stability Check

  • Table 15 reports 95% percentile bootstrap intervals from 10,000 paired resamples over evaluation items in an independent fixed run with cached model outputs.The run regenerates answers rather than reusing outputs underlying the main tables.Source: §E.4 · p. 29, Table 15 · p. 29
  • Small differences from the main-table point estimates reflect generation stochasticity, so the independent bootstrap results serve as a qualitative stability check, not a replacement for the main results.Pboot(Δ ≤0) is the fraction of bootstrap replicates whose contrast is non-positive.Source: §E.4 · p. 29

F.1 Deployment Inference Benchmark … H.3 Deployment Measurement Contract

The deployment sections characterize MTP as a low-concurrency latency optimization and document how its draft head, serving diagnostics, workload, and measurements were evaluated. They also qualify cross-checkpoint speed comparisons and describe the evidence as run-specific rather than universal.

Protocol and Workload Shape

  • The deployment benchmark retained 100 requests after excluding 10 warm-ups and used a one-H20 BF16 setup with real RAG/MCP services and a four-round Agent limit.The NextN module provides the draft head, while inference uses an EAGLE-style speculative generation and verification path; a separate FP8 trial showed no material additional serving gain.Source: Protocol and Workload Shape · pp. 29–30

MTP Draft-Head Adaptation

  • A task-trained policy can adapt a transplanted base-model NextN head through standalone multi-token cross-entropy training on policy-rollout labels, without reward or advantage in the MTP objective.The reported grafted-head run moved from higher early loss toward the native-head trajectory’s late-training range; the authors describe this as a run-specific engineering observation, not universal or multi-seed convergence.Source: MTP Draft-Head Adaptation · p. 30

MTP across Concurrency

  • At C=1–2, MTP raises the selected checkpoint’s Decode TPS by 1.69×/1.52× while retaining 100% 15-second attainment; at C=4–8, gains fall to 1.27×/1.19× and P95 does not improve.The authors therefore treat MTP as a low-concurrency latency optimization, not a universal capacity multiplier.Source: MTP across Concurrency · pp. 30–31, Table 17 · p. 32
  • At C=4, Ours MTP Off→On Decode ranged 97.40–102.55 to 121.61–130.18, while attainment ranged 99–100% to 96–100%.These are run ranges; Table 17 reports C=4 values as means of three run-level statistics, not pooled percentiles.Source: MTP across Concurrency · p. 32, Table 17 · p. 32

G Deployment Workload Shape

  • Table 18 summarizes C=1 workload means over successful complete-Agent requests and their constituent model calls.Calls and tokens vary by model because each model chooses its own Agent trajectory; token counts across different tokenizers are useful workload descriptors but are not identical measures of Chinese text length.Source: Protocol and Workload Shape · p. 30, Table 18 · p. 32

H.1 MTP Serving Configuration

  • The MTP-On quality point estimates are not lower on either dimension, but the available results do not establish equivalence.Table 16 is not an equivalence test because sample-level paired outputs and repeated decoding runs are unavailable; Table 19 records the serving configuration.Source: Table 16 · p. 31, §H.1 · p. 32, Table 19 · p. 32

H.2 SGLang MTP Diagnostics

  • From Base to HAT, mean acceptance length rises from 2.94 to 3.12 and mean acceptance rate from 0.735 to 0.780, providing a mechanism-level explanation for MTP-On TPS differences beyond output length.Acceptance measures speculative-token verification, not semantic answer quality; the trace’s mean and median running requests were 2.03 and 1, with P95 of 7.Source: §H.2 · p. 33, Table 20 · p. 33

H.3 Deployment Measurement Contract

  • Each benchmark run is bound to its run ID, checkpoint, Harness commit, case manifest, serving route, concurrency, generation settings, timeout rules, client region, and run times.A future production-traffic claim would additionally require a frozen service version, explicit traffic window, request-weighted concurrency and queue distributions, exclusion rules, and raw telemetry.Source: §H.3 · p. 33

Related papers

Loading 2608.15763v1…