Learning What to Skip: Counterfactual Credit Assignment for Efficient Multi-Agent LLM Workflows

1 INTRODUCTION

• A component’s utility depends on the current state: extra workflow steps cost tokens and can overwrite a correct intermediate answer.The same verifier, summarizer, or planner can help, be redundant, or harm the result across tasks.Source: §1 · pp. 1–2
• Existing routing and graph methods choose models, roles, or structures, but do not directly decide which future component remains worthwhile after a workflow prefix.Source: §1 · p. 1
• LW2S uses full-workflow and single-component skip traces to train action-specific safety models, accepting skips only with calibration and domain-native guard support.When an early skip is rejected, it can abstain to the full workflow or reconsider a later skip.Source: §1 · p. 2
• Skip evidence must fit the domain: numerical agreement may support math skips, while shared wrong options or buggy code make agreement alone insufficient.Source: §1 · p. 2
• Across the main evaluated settings, LW2S reduces component cost while matching or improving aggregate full-workflow accuracy.Evaluation covers mathematical reasoning, multiple-choice QA, and code generation with two instruction-model families, plus scale-up and second-topology analyses.Source: §1 · p. 2

2 RELATED WORK

• Structured-agent and routing methods choose agents, message structures, models, or graphs before or during workflow construction; LW2S instead evaluates future components after a prefix exists.The cited approaches include debate and graph-based workflows (Du et al., 2023; Wang et al., 2024) and routing methods (Ong et al., 2024).Source: §2 · p. 2
• Selective prediction, confidence routing, consensus, verification cascades, and workflow compaction use observable signals, but those signals can confuse redundancy with correctness.Executors can share an incorrect MMLU option or structurally similar buggy MBPP code; LW2S pairs calibrated counterfactual evidence with domain-native guards.Source: §2 · p. 3
• Unlike counterfactual credit methods that evaluate alternative agent decisions for policy optimization, LW2S records single-component interventions as direct skip-safety supervision.Full trajectories alone do not reveal the reward consequence of omitting a verifier, summarizer, or later turn.Source: §2 · p. 3

3 METHODOLOGY

• The controller operates over observable intermediate outputs, using recorded rewards and costs as supervision while keeping the base LLM frozen.Source: §3 · p. 3

3.1 WORKFLOW SETUP

• A workflow state records the task and outputs so far; skip safety compares counterfactual reward with full-workflow reward, and savings equal full cost minus skip cost.The optimized cost sums prompt and completion tokens across executed components; latency is reported separately. Safety relative to the full workflow can retain an incorrect answer when that workflow fails.Source: §3.1 · pp. 3–4

3.2 COUNTERFACTUAL TRACE COLLECTION

• For each task, trace collection pairs the full workflow with single-component interventions, removing only the target component to attribute reward and cost differences to that skip.Source: §3.2 · p. 4

3.3 ACTION-SPECIFIC SAFETY MODELS

• Separate safety models predict reward preservation for each stage and action from task text, observed outputs, structured features, metadata, and available agreement or test signals.The implementation uses TF–IDF and structured features in class-balanced logistic regression, with thresholds selected on calibration traces.Source: §3.3 · p. 4

3.4 CALIBRATION AND ABSTENTION

• Held-out calibration selects action thresholds under a degradation-risk bound; with multiple sources, acceptance conditions must hold separately on every source.If no threshold is eligible, the action is excluded.Source: §3.4 · p. 5

3.5 DOMAIN-NATIVE GUARDS

• Domain-native guards require explicit state conditions: math verifier skips check numeric agreement, while MBPP verifier skips require both executor outputs to pass visible tests.MMLU option consensus can still reflect a shared misconception, and a failed guard blocks a skip despite a high model score.Source: §3.5 · p. 5

3.6 SELECTIVE SKIP POLICY

• At each stage, LW2S accepts only skip actions that meet calibrated safety thresholds and pass guards, choosing the largest estimated saving or continuing execution.In the sequential MBPP policy, a blocked verifier skip can be followed by a later summarizer-skip decision.Source: §3.6 · p. 5

4.1 WHY FULL-WORKFLOW LOGS ARE INSUFFICIENT

• Full-workflow observations do not identify skip safety because they reveal no reward for the counterfactual trajectory that omits a future component.Paired skip traces provide the missing outcomes for sampled tasks and prefixes; generalization to new prefixes remains the controller’s task.Source: §4.1 · pp. 5–6

4.2 INTERPRETATION OF EMPIRICAL CALIBRATION

• Equation 2’s pointwise Wilson bounds do not provide simultaneous coverage over threshold search or a guarantee under distribution shift.Coverage measures how often the policy acts, whereas task-level degradation measures observed reward consequences; a test slice can have full coverage.Source: §4.2 · p. 6

4.3 SINGLE-INTERVENTION ACCOUNTING

• Sequential fall-through selects only among valid single-stage interventions, so each task’s reward and cost correspond to an actually collected trace.Accounting uses the selected continuation, including verifier cost if execution reaches the later decision; savings from separate interventions are not added.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETUP

• The main matrix uses a five-role workflow and single-component skip interventions, evaluated on MATH, GSM8K, MMLU, and MBPP with two 7B instruction models.The base models remain unchanged; policies use the same traces, data split, reward parser, and token accounting.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

• Across six Qwen2.5-7B settings, LW2S reduced token cost with zero observed degradations in five and one on MMLU; Mistral-7B had zero across all six.Under the shared skip protocol, each of RouteLLM, GraphPlanner, AutoMix, and Prompt-LLM incurred degradation in at least one setting.Source: Table 1 · p. 7, §5.2 · p. 8, Figure 2 · p. 8

5.3 ATTRIBUTING THE GAIN TO COMPONENT-LEVEL CREDIT

• On MBPP slice1, LW2S selected verifier skips for 57 tasks and later summarizer skips for others, preserving full pass rate with the highest recorded saving.Fixed verifier skipping saved tokens but lost two pass-rate points.Source: Figure 3 · p. 8, §5.3 · p. 8

5.2 MAIN RESULTS

• On Qwen2.5-7B, LW2S improved MATH verifier-setting accuracy from 65.2% to 66.4% with 25.1% saving, and preserved MBPP slice1’s 63.0% pass rate with 31.2% saving.It also kept MMLU aggregate accuracy at 73.0% with 8.6% saving, although one task degraded.Source: §5.2 · p. 8

5.4 CODE GENERATION AS A DETERMINISTIC-CHECKER DOMAIN

• On unseen MBPP slice1, unguarded verifier skipping dropped pass rate from 63% to 61% when both executors produced the same incorrect code and the verifier repaired it.The code-execution guard blocked these failures; the two-stage policy preserved pass rate with 31.2% saving by also considering later summarizer omissions.Source: §5.4 · p. 9

5.5 WHAT THE BOUNDARY CASES SHOW

• Shared misconceptions in MMLU and shared buggy code in MBPP show that agreement alone can be unsafe; LW2S abstains or falls through when calibration, guards, and observed state disagree.AST equality alone selected 94 of 100 slice1 tasks and incurred two degradations.Source: §5.5 · p. 9

5.6 COMPONENT UTILITY UNDER TOPOLOGY SHIFT

• Under debate-refine topology shift, LW2S reached 68.0% MBPP pass rate with 16.65% saving, while fixed summarizer skipping reached the same operating point.Fixed verifier skipping preserved the 67.0% full-workflow pass rate with 27.42% saving; all three had zero observed degradations.Source: §5.6 · p. 9

• Across evaluated reasoning and coding tasks, LW2S reduces token cost while matching or improving aggregate full-workflow accuracy.Source: §6 · p. 9
• Controlled interventions expose the reward consequences of skipped steps, providing a learning signal absent from full-workflow logs alone.Source: §6 · p. 9
• Component and topology analyses distinguish broadly redundant calls from omissions that depend on workflow state, while shared-error cases show agreement alone is insufficient.Source: §6 · p. 9
• Sequential fall-through can recover savings when an earlier skip is rejected, motivating learning each component’s conditional utility for adaptive execution.Source: §6 · p. 9

• The appendix covers trace collection, intervention protocols, baseline implementations, and calibration before presenting scale-up, topology, and failure analyses.Source: A PROTOCOLS, DIAGNOSTICS, AND… · p. 11
• Accuracy reports aggregate task reward, while Deg. counts individual tasks on which a policy underperforms the full workflow.Source: A PROTOCOLS, DIAGNOSTICS, AND… · p. 11
• Because gains on some tasks can offset degradations on others, the paper reports both quantities alongside token savings.Source: A PROTOCOLS, DIAGNOSTICS, AND… · p. 11

• Each task-level JSONL record bundles full-workflow outputs with single-component skip interventions, parsed rewards, resource counts, and structured state metadata.Source: §A.1 · p. 11
• Thresholds are selected on calibration traces before held-out evaluation, with the degradation criterion required to hold separately across multiple calibration sources.Source: §A.1 · p. 11
• If no threshold qualifies, LW2S retains the full workflow or, in the MBPP two-stage setting, falls through to a later summarizer decision.Source: §A.1 · p. 11
• Token cost sums recorded prompt and completion tokens; reported latency sums component latency and is not used for threshold selection.Source: §A.1 · p. 11
• End-to-end wall-clock time is not reconstructed because runs may be interrupted, resumed, or scheduled differently; reported traces use open-weight models on one NVIDIA RTX PRO 6000 Blackwell Server Edition.Source: §A.1 · p. 11

• The main matrix fixes a planner–executor–executor–verifier–summarizer template to make intervention labels comparable across benchmarks.Source: §A.2 · p. 12
• The debate-refine workflow changes agent roles and information flow to test whether component-level skip opportunities persist under a different topology.Source: §A.2 · p. 12
• Table 3 lists the workflow templates, backbones, and tasks covered by the evaluations.Source: §A.2 · p. 12

A.3 BOUNDARY TO ADJACENT PARADIGMS

• LW2S asks whether a specific future component is still needed after observing a partially executed multi-agent workflow.The decision depends on the current workflow prefix rather than which model or agent graph to choose.Source: §A.3 · p. 12
• Unlike adjacent routing and early-exit methods, LW2S learns the reward consequences of individual omissions.Its sequential policy can retain an early component and reconsider a later omission as the workflow state evolves.Source: §A.3 · p. 12

A.4 INTERVENTION PROTOCOL; A.5 BASELINE EVALUATION PROTOCOL

• The intervention protocol fixes workflow settings within each task family and omits one future component; baseline rules share saved traces, data partitions, and reward and cost accounting.At each decision, only the observed prefix and then-available outputs are exposed, avoiding leakage from later components.Source: §A.4 · p. 12, Table 5 · p. 13, §A.5 · p. 13

A.6 QWEN2.5-14B SCALE-UP DIAGNOSTICS

• On MMLU, fixed verifier skipping raises accuracy from 82.0% to 83.0%, while executor-consensus skipping raises it to 84.0%; both still cause task-level degradations.Gains on other tasks offset losses of answers that the verifier repairs, so the shared-error boundary persists at larger scale.Source: §A.6 · pp. 13–14, Table 7 · p. 14

A.7 SECOND-WORKFLOW DEBATE-REFINE DIAGNOSTIC

• On the 100-task MBPP debate-refine slice, LW2S reaches 68.0% pass rate while saving 16.65% of token cost, with zero observed task-level degradations.This second workflow tests whether skip-credit signals extend beyond the original fixed-template workflow.Source: §A.7 · p. 14, Table 8 · p. 14

A.8 UNIFIED SAFETY-EFFICIENCY LEDGER; A.9 CALIBRATION CERTIFICATES AND UNCERTAINTY

• Calibration selects thresholds from calibration measurements before held-out testing; five settings have zero observed degradations, while MMLU has one among 100 tasks and a 5.4% upper bound.The zero-degradation 100-task MBPP slices have a 3.7% upper bound; Wilson bounds therefore retain sample-size-dependent uncertainty.Source: §A.8 · p. 14, §A.9 · p. 15, Table 9 · p. 15

A.10 LW2S COVERAGE AND DEGRADATION-RISK DIAGNOSTICS; A.11 COMPONENT-LEVEL CREDIT DIAGNOSTIC

• On MBPP slice0, LW2S selects 69 verifier and 31 summarizer skips at 70.0% accuracy and 28.5% saving; on slice1, it selects 57 and 43 at 63.0% and 31.2%.Both slices have zero observed task-level degradations; selected-task coverage and token savings measure different aspects of the policy.Source: §A.10 · p. 15, §A.11 · p. 16, Table 12 · p. 16

A.12 FAILURE AND ABSTENTION TAXONOMY

• Shared wrong consensus and superficial code agreement can make omissions unsafe, while redundant late summarization can leave a useful later skip after an early skip is blocked.The taxonomy links skip choices to the observed prefix and explains why a blocked early omission need not rule out a later one.Source: §A.12 · p. 16, Table 13 · p. 16

A.13 GUARD ABLATIONS

• On MBPP slice1, AST equality selects 94 tasks but incurs two degradations; requiring executable evidence blocks those failures.Verifier/summarizer AST equality is a hindsight diagnostic, not an online summarizer-skip condition.Source: §A.13 · pp. 16–17, Table 14 · p. 17

A.14 MMLU FAILURE CASE

• On the MMLU failure case, both executors select option A, while the verifier corrects the answer to C; skipping verification retains the shared error.Intervention labels distinguish this shared-error case from redundant verification during training, and source-specific calibration determines which predicted skip regions are accepted.Source: §A.14 · p. 17, Table 15 · p. 17