PaperlayerView exact revision on arXiv

Source-linked AI summary

Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

Haotian Yan, Jiaqi Liu, Bingyi Jing, Tianjun Yao

arXiv:2610.23239v2cs.LG

TL;DR

Multi-turn routing must choose a model before seeing its response, even though task success is revealed only after the interaction and intermediate turns lack their own labels. RIVET trains a pre-call router alongside a response-informed auxiliary estimator using a shared representation. On MINT, it improves task accuracy over MTRouter while reducing model-call cost by 74.7%.

  • Problem

    Multi-turn routers make decisions before each response, but only the final task outcome is graded, leaving intermediate choices without individual labels.

    Source: §1 · p. 1

  • Method

    RIVET jointly trains a pre-call Routing Value Estimator and a training-only response-informed estimator over a shared state–model representation.

    Source: §1 · p. 2

  • Results

    On MINT, RIVET improves task accuracy over MTRouter while reducing model-call cost by 74.7%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Logged responses can provide auxiliary training information while routing remains strictly pre-call at deployment.

    Source: §1 · p. 2

    The estimation-error result is established under local shared-signal and regularity conditions stated in Appendix A.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Multi-turn LLM routing selects a model at every turn before its response is available, yet task success is revealed only when the interaction ends, leaving each intermediate choice without a label of its own. Logged trajectories, however, retain the response each call produced, which exposes what each turn required and how well the chosen model served it, evidence that outcomes alone hide. We introduce RIVET (Response-Informed Value Estimation for Turn-Level Routing), which exploits this post-call evidence during training while keeping routing strictly pre-call at deployment. RIVET jointly trains two value estimators over a shared state-model representation: a deployable Routing Value Estimator that scores candidate models from pre-call information, and a training-only Response-Informed Value Estimator that additionally observes the realized response. Conditioning on the response turns response-induced return variation from residual noise into an observed covariate, and we show that this reduces the leading asymptotic estimation error of the shared representation. On the MINT benchmark, RIVET solves more held-out tasks than the strongest multi-turn baseline while using 74.7% less total call cost.

1 INTRODUCTION

Multi-turn routing must choose a model before each response, while task success is graded only at the end and logged responses reveal turn-level requirements hidden by outcomes. RIVET uses those responses during training to improve a pre-call router, and on MINT it improves task accuracy over MTRouter while reducing model-call cost by 74.7%.

1 INTRODUCTION

  • Final-outcome-only supervision leaves intermediate decisions unlabeled, while changing one turn’s model changes later states and each logged trajectory costs multiple model calls.These challenges make propagating delayed consequences and extracting more from logged experience important for multi-turn routing.Source: §1 · p. 1
  • Logged responses expose what each turn required and whether its selected model delivered it, information the final outcome does not show.The needed ability and failure mode can vary by turn, so a fixed strong/weak model ranking does not capture turn-level value.Source: §1 · pp. 1–2
  • RIVET formulates turn-level selection as cost-aware sequential value learning with trajectory-level supervision and reuses logged transitions through Bellman updates.The framework uses a small online adaptation budget instead of large-scale on-policy rollout collection.Source: §1 · p. 2
  • RIVET trains a deployable pre-call Routing Value Estimator and a training-only Response-Informed Value Estimator over a shared state–model representation.The response-informed estimator observes the executed response, helping the shared representation by treating response-dependent return variation as observed rather than noise.Source: §1 · p. 2
  • 74.7% lower model-call cost than MTRouter, with improved task accuracy, on the MINT multi-turn benchmark.The paper also reports competitive task performance alongside cost advantages across three out-of-distribution benchmarks.Source: §1 · p. 2
  • Conditioning the auxiliary estimator on the response reduces the leading asymptotic estimation error of the shared representation.This response-informed training signal does not change the strictly pre-call deployment path.Source: §1 · pp. 2–3

2 RELATED WORK

Prior routing work largely scores models within a fixed query context, while agentic workloads require sequential decisions and remain sparsely studied. RIVET combines off-policy value learning with response-informed auxiliary supervision for turn-level model selection.

2 RELATED WORK

  • Existing multi-turn approaches use on-policy policy gradients, workflow assembly, one-shot escalation, search distillation, or supervised outcome estimation; none learns turn-level action values off-policy from logged trajectories.MTRouter selects a model at every turn, but estimates outcomes from history–model embeddings.Source: §2 · p. 3
  • Most conventional routers, cascades, and bandits evaluate choices in a fixed query context, whereas the paper identifies multi-turn routing as sequential and sparsely studied.The cited methods include predictors, cascades, bandits, and RouterBench’s accuracy–cost evaluation.Source: §2 · p. 3
  • RIVET applies Double DQN targets and CQL regularization to model selection, with a training-time critic that observes responses unavailable to the deployed router.Unlike cascades and SWE-Router, it uses responses during training rather than paying for a call before deciding; the paper analyzes when this auxiliary supervision improves the shared representation.Source: §2 · p. 3

3 PROBLEM FORMULATION

The paper models multi-turn LLM routing as a cost-aware Markov decision process in which each model choice affects the response and later states. Its objective maximizes expected discounted return, balancing task outcome against per-call cost.

3 PROBLEM FORMULATION

  • The state combines the task query with interaction history before the current model call, and the selected model generates the response that helps determine the next state.The router chooses a model from its policy, after which the environment produces feedback and transitions the interaction.Source: §3 · p. 3
  • The per-turn reward incorporates model-call cost, with a reference cost and λ controlling the trade-off between task success and cost.The objective maximizes expected discounted return, with discount γ and trajectories induced by the policy, model responses, and environment.Source: §3 · pp. 3–4
  • Model choices are sequentially coupled because each choice changes the response and therefore the states encountered at later turns.The formulation treats routing as a sequential decision problem rather than independent turn-level predictions.Source: §3 · p. 4

4 METHOD: RIVET

RIVET trains a pre-call router and a response-informed auxiliary estimator over a shared state–model representation, then refines the value estimates using logged experience. The response-conditioned signal makes one component of return variation observable and reduces its leading contribution to representation-estimation error.

4.1 DUAL VALUE ESTIMATORS WITH A SHARED STATE–MODEL REPRESENTATION

  • RIVET jointly trains a deployable pre-call Routing Value Estimator (RVE) and a training-only Response-Informed Value Estimator (RIVE) over a shared state–model representation.RIVE additionally observes the executed response; both estimators predict the same return, and RVE alone routes at deployment.Source: §4 · p. 4, §4.1 · p. 4

4.2 RESPONSE-INFORMED OFF-POLICY LEARNING

  • Completed-trajectory returns first supervise both estimators; RIVET then applies off-policy Bellman refinement and limited online adaptation using logged and newly collected transitions.RVE selects the continuation action for Bellman targets, while logged replay anchors online learning and fresh trajectories expose router-induced states.Source: §4.2 · pp. 4–6
  • The joint Huber objective weights the response-informed regression by β, and both losses back-propagate through the shared representation used by RVE.The target is the same scalar return for both estimators.Source: §4.2 · p. 4

4.3 WHY RESPONSE-INFORMED AUXILIARY LEARNING HELPS

  • Under local shared-signal and regularity conditions, response-informed learning reduces the leading response-induced representation-error contribution by 1/(1 + β)^2, strictly when Σ∆ ≠ 0.Observing the response explains ∆, which RVE otherwise treats as residual variation; the contribution of ϵ is unchanged.Source: §4.3 · p. 6

5 EXPERIMENTS

RIVET is evaluated for accuracy–cost performance, transfer, cost control, component contributions, and the value of turn-level adaptation. Results cover MINT and out-of-distribution benchmarks, with deployment using RVE alone.

5 EXPERIMENTS

  • The experiments examine in-distribution and OOD performance, cost-coefficient effects, online adaptation, response-input ablations, and turn-level routing interventions.The evaluation trains and tests on MINT Reasoning and uses MuSiQue, DROP, and MMLU-Pro for transfer; deployment uses RVE alone.Source: §5 · p. 7, §5.1 · p. 7

5.2 ACCURACY–COST PERFORMANCE AND OOD GENERALIZATION

  • RIVET improves accuracy over MTRouter on MINT while reducing model-call cost by 74.7%.It also reduces cost by 80.1% versus the strongest fixed model.Source: §5.2 · p. 7, Table 1 · p. 7
  • Across three OOD benchmarks, RIVET Online has the lowest cost among routers and matches or exceeds MTRouter’s task scores.On MMLU-Pro, it matches MTRouter’s accuracy at 92.4% lower cost; on MuSiQue, online adaptation gives up some exact match.Source: Table 1 · p. 7, §5.2 · p. 8

5.3 COST CONTROL AND ONLINE ADAPTATION

  • Removing the cost penalty raises model-call cost to 3.2× the default and lowers accuracy; at λ = 0.50, cost falls 39.2% versus default while accuracy returns to MTRouter’s level.The default λ = 0.05 gives the best operating point; the figure identifies the highest-accuracy and lowest-cost settings among those evaluated.Source: §5.3 · p. 8, Figure 3 · p. 9
  • With the full online budget, accuracy rises from 85.82% to 87.94% while test-time cost falls 29.2% relative to the offline router.The single-run figure tracks checkpoints after cumulative batches of 35, 70, 105, and 140 trajectories.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 WHAT DRIVES THE IMPROVEMENT?

  • Removing RIVE lowers accuracy by 7.80 points; keeping RIVE but zeroing its response input causes the same drop and a larger cost increase.All ablation configurations deploy RVE alone, and the zero-response condition retains RIVE while replacing its response input with a zero vector.Source: §5.4 · p. 8, Table 2 · p. 8
  • Removing CQL raises model-call cost by more than 100% and lowers accuracy by 2.12 points, even with RIVE intact.The cost increase indicates that CQL stabilizes RVE rankings for actions weakly supported by logged replay.Source: §5.4 · p. 8

5.5 WHY TURN-LEVEL ADAPTATION MATTERS

  • On episodes where RIVET switches, staying with the first-turn model lowers cost by 17.5% but reduces successful episodes from 50 to 43; forcing a switch adds one success at 42.9% higher cost.The interventions cover 59 episodes where RIVET switches and 46 where it stays with one model.Source: §5.5 · p. 9, Table 3 · p. 9

6 CONCLUSION

RIVET separates pre-call deployment decisions from response-informed training, using a shared representation to improve the deployed estimator. The conclusion reports a high-accuracy, lower-cost operating point while noting that narrower causal attribution to response content remains unresolved.

6 CONCLUSION

  • RVE makes deployment decisions from pre-call state, while training-only RIVE conditions on the executed response and improves RVE through their shared representation.The response-informed signal reduces the leading asymptotic estimation error of the deployed representation without adding an inference-time path.Source: §6 · p. 9
  • Cost-controlled response probes and matched transfer ablations remain necessary for narrower causal attribution to response content.The current evaluation establishes an end-to-end difference between the full and RVE-only configurations.Source: §6 · p. 9

ETHICS STATEMENT

The work studies routing among existing models in multi-turn interactions to reduce call cost while preserving task performance. Its experiments use public benchmarks and models, without human subjects or personal data.

ETHICS STATEMENT

  • The study targets lower model-call cost while preserving task performance in multi-turn routing among existing language models.The stated aim is to reduce compute and monetary costs and make agentic systems more accessible.Source: ETHICS STATEMENT · p. 10
  • Experiments use publicly available benchmarks and models and involve no human subjects or personal data.Source: ETHICS STATEMENT · p. 10
  • The method selects which existing model to call and does not generate new content or alter model behavior.The authors report no foreseeable negative societal impact beyond risks already present in underlying models.Source: ETHICS STATEMENT · p. 10

A THEORETICAL ANALYSIS: PROOFS AND EXTENSIONS

The theoretical analysis decomposes return variation into response-resolved and residual components, then shows how response-informed training can reduce shared-representation estimation error. It extends the result beyond idealized assumptions, identifies when response variation is substantial, and applies the analysis to bootstrapped targets.

A.1 PROOF OF PROPOSITION 4.1

  • The Proposition 4.1 decomposition gives RVE a residual containing response-resolved variation plus noise, while RIVE’s residual contains only the latter.On terminal turns with deterministic grading from state and response, the remaining conditional variance is zero.Source: §A.1 · p. 12

A.2 PROOF OF THEOREM 4.2

  • Under the shared-signal and centering assumptions, adding RIVE reduces the shared representation’s asymptotic covariance by a positive-semidefinite amount.The reduction is positive definite on the range of the response-induced variation covariance.Source: §A.2 · p. 13

A.3 GENERAL FORM WITHOUT THE SHARED-SIGNAL AND CENTERING CONDITIONS

  • Theorem 4.2’s broader form shows RIVE never increases response-induced estimation error, with attenuation between 1/(1 + β)^2 and 1.The clean factor is recovered when the estimators’ shared-signal gradients match and their cross block is zero.Source: §A.3 · p. 14

A.4 STRUCTURE OF THE RESPONSE-RESOLVED COMPONENT

  • Under identity-blind feedback, response-conditioned progress and realized token cost determine the response-resolved component of return variation.The conditions include response-only environment feedback and success labels, price-per-token costs, and continuation based only on the next state.Source: §A.4 · pp. 14–15
  • When normalized costs are small, response-induced variation is dominated by differences in downstream progress across responses from the same model.The analysis relates this variation to ambiguous outcomes: RIVE can assign failures to response content, while the shared representation captures typical model behavior.Source: §A.4 · p. 15

A.5 BOOTSTRAPPED TARGETS

  • With bootstrapped targets, response-resolved variation can account for almost all target variance, but the analysis does not bound error from an imperfect target network.The reduction applies to estimating the shared representation for a fixed target; it is largest when tool feedback is deterministic or target variance is mostly response-resolved.Source: §A.5 · p. 15

B.1 DATASETS, TRAJECTORY COLLECTION, AND MANIFESTS

The appendix documents the evaluation datasets and trajectory collection, training protocol, and state, response, and model-card representations used by the routers.

B.1 DATASETS, TRAJECTORY COLLECTION, AND MANIFESTS

  • Training uses MINT Reasoning and held-out MuSiQue, DROP, and MMLU-Pro data; the MINT manifests contain 200 training, 35 validation, and 141 test tasks.Six fixed-model policies and a Random Router generate 1,400 offline trajectories and 7,492 transitions; online adaptation collects at most 140 additional trajectories.Source: §B.1 · pp. 15–16

B.2 OPTIMIZATION AND STOPPING PROTOCOL

  • Offline optimization runs five Monte Carlo epochs followed by 20 temporal-difference epochs, with online replay mixing offline and newly collected data equally.Online adaptation validates every 35 trajectories and uses early stopping after 140 trajectories, with patience two and at most three passes through the task pool.Source: §B.2 · p. 16

B.3 STATE, RESPONSE, AND MODEL-CARD ENCODINGS

  • RVE scores candidate models from a shared state–model representation, while RIVE also encodes the selected model’s executed response.The state serialization retains the task and recent turns, omits earlier model identities, and is fused with model-card features; the two value heads have separate parameters.Source: §B.3 · p. 16

B.4 ROBUSTNESS TO CANDIDATE-MODEL UNAVAILABILITY

RIVET retains a favorable accuracy–cost operating point on MINT Test-141 when either a strong, high-cost or weaker, low-cost candidate model is removed.

B.4 ROBUSTNESS TO CANDIDATE-MODEL UNAVAILABILITY

  • Without either tested candidate model, RIVET preserves a favorable accuracy–cost operating point relative to MTRouter.The tested removals are DeepSeek V4-Pro and Qwen3.5-9B.Source: §B.4 · p. 17, Table 5 · p. 17
  • Without Qwen3.5-9B, RIVET matches MTRouter’s 85.82% accuracy while reducing cost by 74.4%.Qwen3.5-9B is the weaker, low-cost option in this test.Source: §B.4 · p. 17, Table 5 · p. 17
  • Without DeepSeek V4-Pro, RIVET achieves 83.69% accuracy at 72.3% lower cost than MTRouter, with a 0.71-point accuracy gap.DeepSeek V4-Pro is the strong, high-cost option in this test.Source: §B.4 · p. 17, Table 5 · p. 17

B.5 ROUTING AFTER EXECUTION ERRORS

The post-error analysis reports fewer recorded execution errors and more frequent model switching by RIVET than MTRouter. The stage-wise analysis and case studies describe how routing choices vary across turns and show selected task-level comparisons.

B.5 ROUTING AFTER EXECUTION ERRORS

  • RIVET’s pooled error-event rate is 3.96% versus MTRouter’s 8.48%; after errors, it switches models in 26.3% versus 4.2% of cases and spends 23.9% less on the switched call.Next-turn recovery is 93.3% for RIVET and 100.0% for MTRouter; the authors separately test the effect of preventing model changes.Source: §B.5 · p. 18, Table 7 · p. 18

B.6 STAGE-WISE MODEL-SELECTION PREFERENCES

  • RIVET’s two variants select Qwen3.5-9B most often in Analyze and Solve turns, then DeepSeek V4-Flash most often in Submit turns.Figure 5 shows conditional model-selection distributions, with each stage ring normalized independently.Source: §B.6 · p. 18, Figure 5 · p. 19

C.1 COMPARISON WITH OTHER ROUTERS

  • In Example 6, RIVET switches after a rejected tool import and answers correctly in two turns; MTRouter answers correctly in eight turns without switching models.Both resolve the execution error on the next turn, but RIVET completes the task earlier at lower episode cost.Source: §C.1 · pp. 19, 23
  • In Example 2, RIVET and EmbedLLM both answer correctly in two turns, while RIVET uses a lower-cost model.RIVET selects DeepSeek V4-Flash instead of DeepSeek V4-Pro.Source: §C.1 · pp. 19–20
  • In Example 3, RIVET returns the correct 144 in two turns at $0.000179, while RouterDC returns an incorrect 14400 in four turns at $0.000403.The case attributes RouterDC’s error to an unnecessary unit conversion.Source: §C.1 · pp. 19, 21

C.2 COMPARISON WITH RESPONSE-ABLATED TRAINING

Examples 9 and 10 compare RIVET with response-ablated training, illustrating differences in both episode cost and accuracy.

C.2 COMPARISON WITH RESPONSE-ABLATED TRAINING

  • 95.9% lower episode cost in Example 9 and 15.6× lower cost in Example 10 illustrate RIVET’s cost advantage; in Example 10, only RIVET answers correctly.Both variants answer correctly in Example 9, while the response-ablated variant is incorrect in Example 10.Source: §C.2 · p. 23
  • In Example 7, both variants answer correctly, while RIVET uses 3 turns and costs $0.000230 versus 8 turns and $0.001073 for the ablated variant.The ablated variant repeats and revises its divisor-sum computation before submitting the same answer.Source: §C.2 · p. 24
  • For task mmlu:7, RIVET and RIVE w/o response both produce correct final answers in 3 turns, with respective costs of $0.000343 and $0.000453.Source: §C.2 · p. 25
  • In Example 9, both variants answer correctly in 1 turn, but RIVET costs $0.000087 versus $0.002121 for the ablated variant.RIVET selects DeepSeek V4-Flash; the ablated variant selects DeepSeek V4-Pro.Source: §C.2 · pp. 23, 25
  • In Example 10, RIVET’s Qwen3.5-9B answer is correct in 1 turn at $0.000084; the ablated variant’s GPT-4o-mini answer is incorrect after 10 turns at $0.001307.The ablated variant repeats its differentiation plan through turn nine before submitting −240.Source: §C.2 · p. 26
Loading 2610.23239v2…