PaperlayerView exact revision on arXiv

Source-linked AI summary

CARE: Certifying Acceleration for Vision-Language-Action Inference

Rui Liu, Tong Zheng, Jindong Gu, Zhipeng Wang

arXiv:2610.08917v1cs.CLcs.LG

TL;DR

VLA inference remains costly, and average success does not isolate tasks that acceleration breaks. CARE certifies the fastest candidate whose paired-episode failure risk meets a user-defined budget, retaining the reference when evidence is insufficient. Across four LIBERO suites, it reports 9.0–10.8× speedups while preserving at least 85.8% of reference-solved episodes at 95% confidence.

  • Problem

    Latency and average task success do not identify which tasks acceleration breaks, while action changes can alter subsequent closed-loop states.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    CARE uses paired calibration rollouts to certify candidate failure risk against a user-defined budget, selects the fastest certified candidate, and retains the reference if none qualify.

    Source: §1 · pp. 2–3, §3.1 · p. 4

  • Results

    Across four LIBERO suites, CARE achieves 9.0–10.8× speedups over one-step replanning while preserving at least 85.8% of reference-solved episodes at simultaneous 95% confidence.

    Source: §1 · p. 3

  • Takeaways & Limitations

    CARE applies across policy classes and acceleration mechanisms, including flow-step reduction on LIBERO and compute-reduced language-model agents in Crafter.

    Source: §1 · p. 3, §5 · p. 10

    The selection objective is framed for a finite candidate family under a specified risk budget α and error level δ.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

While vision-language-action (VLA) models have advanced rapidly, running them at every control step remains expensive. Prior work accelerates VLA inference using techniques like action chunking and visual-token pruning, typically evaluating based on latency and average task success. However, acceleration may discard information and break tasks the original policy would solve, a risk hidden by average metrics. Measuring these failures is challenging because action deviations compound over closed-loop trajectories, meaning task failure is only observable across full episodes. We therefore define an acceleration-induced failure via paired rollouts from identical initial conditions, tracking when the reference succeeds but the accelerated policy fails. To manage this, we introduce CARE, an approach for certified accelerator selection. CARE uses paired rollouts on a calibration set to provide finite-sample guarantees that acceleration-induced failure risk stays below a user-specified budget. It deploys the fastest certified candidate, falling back to the reference if none qualify. By relying only on terminal outcomes and measured compute, CARE applies unchanged across diverse acceleration mechanisms, while sequential testing and failure-triggered reference rollouts keep certification affordable. On four LIBERO suites with OpenVLA-OFT, CARE certifies $9.0$--$10.8\times$ speedups while guaranteeing (at $95\%$ confidence) that at least $85.8\%$ of reference-solved episodes are preserved. Under tight budgets, selectors without guarantees exceed the budget in up to $75\%$ of trials, whereas CARE stays within budget and its sequential form uses $78.9\%$ fewer rollouts than exhaustive evaluation. CARE further generalizes to flow-step reduction for $\pi_{0.5}$, and to Qwen3.5-9B and Llama-3.1-8B agents in Crafter.

1 INTRODUCTION

VLA acceleration is commonly judged by inference cost and average success, which can miss tasks the accelerated policy breaks. CARE measures that risk on paired full-episode rollouts, then certifies the fastest candidate within a user-defined budget while retaining the reference when evidence is insufficient.

1 INTRODUCTION

  • Average success can hide acceleration-induced failures because shared failures count against the candidate while newly solved tasks can offset tasks it breaks.The risk is defined over paired, complete closed-loop episodes, where the reference succeeds and the accelerated policy fails.Source: §1 · pp. 1–2
  • CARE tests whether each candidate’s episode-level risk is below a user-defined budget, returns the fastest certified candidate, and otherwise retains the reference.It uses terminal outcomes and measured compute, treating candidates as black-box policies.Source: §1 · pp. 2–3
  • 9.0–10.8× speedups over one-step replanning preserve at least 85.8% of reference-solved episodes at simultaneous 95% confidence across four LIBERO suites.These OpenVLA-OFT candidates combine action chunking, visual-token pruning, and feature caching.Source: §1 · p. 3
  • 13–75% of resampled trials exceeded tight risk budgets under common selectors, while CARE stayed within budget in every trial.When evidence was insufficient, CARE retained the reference policy.Source: §1 · p. 3
  • CARE also certifies reduced flow steps for π0.5 on LIBERO and compute-reduced Qwen3.5-9B and Llama-3.1-8B agents in Crafter.The evaluation spans different policy classes and acceleration mechanisms.Source: §1 · p. 3
  • Task-balanced sequential testing and failure-triggered reference evaluation reduce rollout costs while preserving the finite-sample guarantee.On 24-configuration suites, CARE used 78.9% fewer rollouts than exhaustive evaluation.Source: §1 · pp. 2–3

2 RELATED WORK

Prior acceleration methods reduce computation but are generally assessed using latency and average task success. CARE instead certifies closed-loop acceleration risk under a specified budget, using paired terminal outcomes and measured compute.

2 RELATED WORK

  • Action chunking, token pruning, feature caching, dynamic inference, and flow-step reduction are among the approaches used to reduce multimodal-policy inference cost.These methods are primarily evaluated by latency and average task success, which do not reveal which tasks the faster policy newly fails.Source: §2 · p. 3
  • CARE selects the fastest candidate that can be certified under a specified acceleration-induced failure budget using paired terminal outcomes and measured inference cost.The procedure applies across action chunking, token pruning, feature caching, flow-step reduction, and combinations of these mechanisms.Source: §2 · p. 3
  • Finite-sample risk control and anytime-valid inference are related tools, while CARE applies certification to accelerator selection for closed-loop policies.Related work also includes safe policy improvement, risk-sensitive methods, and negative-flip analyses.Source: §2 · pp. 3–4
  • Unlike analyses over fixed inputs, CARE targets closed-loop episode risk, which cannot generally be inferred from marginal success rates or per-step guarantees.It uses task-balanced sequential certification and failure-triggered reference execution to reduce certification cost while retaining the deployment guarantee.Source: §2 · p. 4

3 APPROACH

CARE selects the fastest accelerator it can certify against an episode-level risk budget, using paired calibration rollouts and retaining the reference if none qualifies. Its procedure combines finite-sample testing with adaptive evaluation to reduce certification cost.

3 APPROACH

  • CARE evaluates acceleration configurations against a reference and deploys the fastest candidate it certifies, falling back to the reference if none qualifies.The candidate ordering is based on measured inference cost, and calibration choices preserve the finite-sample deployment guarantee.Source: §3 · p. 4, §3.4 · p. 6

3.1 PROBLEM FORMULATION

  • Acceleration-induced failure risk measures when acceleration turns a reference success into a failure, rather than overall success-rate differences.CARE uses this risk as its deployment constraint and also reports retention: the fraction of reference-solved episodes that acceleration preserves.Source: §3.1 · p. 4

3.2 CERTIFY THEN SELECT

  • For each candidate, CARE tests whether its risk meets the budget and controls family-wise error across candidates to limit false certifications.It certifies a candidate only after rejecting the hypothesis that its risk is at least the budget.Source: §3.2 · p. 5

3.3 TASK-BALANCED SEQUENTIAL CERTIFICATION

  • Task-balanced rounds let CARE inspect evidence across all tasks, while anytime-valid testing and early exit stop evaluation when a candidate is certified or cannot qualify.Rounds contain the same number of units per task; a candidate is certified when its evidence reaches the threshold.Source: §3.3 · p. 5

3.4 FAILURE-TRIGGERED REFERENCE EVALUATION

  • CARE runs the reference only when an evaluated candidate fails, reuses cached reference outcomes, and thereby preserves the decisions of exhaustive paired evaluation.The reference is executed at most once per unit, and early exit and failure-triggered evaluation affect calibration cost rather than candidate certification.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments test CARE on OpenVLA-OFT across four LIBERO suites, compare it with deployment selectors and broader candidate families, then assess transfer to π0.5 and Crafter agents. They also examine validity and evaluation-efficiency components.

4.1 CERTIFIED OPENVLA DEPLOYMENT

  • 9.02–10.81× speedups over H1 come with simultaneous 95% lower bounds preserving 85.8%–95.7% of reference-solved episodes across four LIBERO suites.CARE certified an accelerated policy on every suite.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 COMPARISON WITH DEPLOYMENT SELECTORS

  • CARE has no observed over-budget deployments, while other selectors exceed tighter AIF budgets in 13.0–75.0% of trials.CARE safely retains H1 when no accelerated policy has sufficient evidence for certification.Source: Table 2 · p. 7, §4.2 · p. 7
  • Sequential CARE keeps a 7.32× speedup with 26.9% fallback while reducing evaluation from 2,250 to 592 rollouts.Fixed-n CARE has the highest mean speedup among risk-controlling selectors at 8.88× with 11.7% fallback.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 EVALUATION WITH A BROADER CANDIDATE FAMILY

  • Across the four LIBERO suites, CARE alone certifies an accelerated policy on every suite, using 5,263 rather than 25,000 rollouts, 78.9% fewer than full-sample HB and fewer than each sequential baseline.It also reduces H1 evaluation from 1,000 to 113 rollouts.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 CROSS-ARCHITECTURE AND CROSS-DOMAIN EVALUATION

  • CARE selects two-step π0.5 policies on Spatial and Object and a four-step policy on Goal, while retaining the ten-step reference on Long.Across the four suites, CARE uses 1,276 rather than 4,000 rollouts, 68.1% fewer.Source: §4.4 · pp. 8–9, Table 5 · p. 9

4.5 DESIGN ANALYSIS AND ABLATIONS

  • The validity analysis finds that paired episode outcomes, family correction, and task-balanced acquisition address distinct risks in certification.Local and episode-level rates differ (4.3% and 1.0% on Object; 9.5% and 2.7% on Long); in a ten-candidate simulation, uncorrected selection deploys an over-budget policy in 46.0% of trials, versus at most 0.15% with family-corrected testing. Task-by-task acquisition can certify an over-budget policy before high-risk tasks are observed, unlike balanced round-robin acquisition in the tested profiles.Source: §4.5 · p. 9

4.4 CROSS-ARCHITECTURE AND CROSS-DOMAIN EVALUATION

  • In Crafter, collect wood schedules certify on every seed with 25.1%–30.6% less compute, while collect drink schedules certify on two of three seeds.The results cover Qwen3.5-9B and Llama-3.1-8B agents.Source: Table 6 · p. 9, §4.4 · p. 9

4.5 DESIGN ANALYSIS AND ABLATIONS

  • Sequential fastest-first evaluation and failure-triggered reference evaluation reduce total calibration from 9,000 to 1,997 rollouts and wall-clock time from 38.6 to 7.0 hours.That is a 77.8% rollout reduction and an 81.9% time reduction.Source: §4.5 · p. 9

5 CONCLUSIONS

CARE selects the fastest candidate whose acceleration-induced failure risk can be certified below a user-set budget, retaining the reference when evidence is insufficient. Its guarantees and efficiency rely on paired rollouts, task-balanced sequential testing, and failure-triggered reference evaluation.

5 CONCLUSIONS

  • Across OpenVLA-OFT and π0.5 on LIBERO and language-model agents in Crafter, CARE applies across policy classes and acceleration mechanisms, recovering acceleration when supported by data and avoiding over-budget heuristic selections.The results support treating acceleration as selectable with explicit finite-sample control over introduced failures, rather than only as a speed–accuracy trade-off.Source: §5 · p. 10

A.1 THE SEQUENTIAL PROCEDURE

  • CARE evaluates candidates fastest-first, stops at the first certified candidate, and runs the reference only after a candidate failure, reusing cached outcomes.Sequential tests can certify or rule out candidates early; if no candidate is certified, CARE retains the reference.Source: §A.1 · p. 13

A.2 ASSUMPTIONS AND THE HOEFFDING–BENTKUS p-VALUE

  • The guarantees require pre-specifying candidates and testing settings, independent calibration units sampled from the deployment task distributions, and execution randomness independent across runs.Sequential rounds contain the same number of units per task.Source: §A.2 · pp. 13–14

A.3 VALIDITY OF CARE

  • Under the stated assumptions, CARE bounds the chance of certifying any unsafe candidate by the sum of its error levels, and the fallback reference has zero acceleration-induced failure risk.The fixed-sample and sequential tests control their respective candidate-level error probabilities.Source: §A.3 · pp. 14–15

A.4 SUPERMARTINGALE PROPERTY UNDER TASK-BALANCED ROUNDS

Task-balanced rounds make the sequential test valid across tasks with differing risks: each round’s average conditional loss equals the overall candidate risk.

A.4 SUPERMARTINGALE PROPERTY UNDER TASK-BALANCED ROUNDS

  • Task-balanced rounds keep each round’s average conditional loss equal to the candidate’s overall risk, making the e-process a non-negative supermartingale under the unsafe-candidate null.A round drawn only from low-risk tasks could instead have an increment with expectation above one.Source: §A.4 · p. 15

A.5 IDENTIFICATION: WHY PAIRED, COMPLETE EPISODES ARE NEEDED

Neither marginal success rates nor isolated decision effects identify acceleration-induced failure risk; paired, complete episodes are needed to capture closed-loop trajectory changes.

A.5 IDENTIFICATION: WHY PAIRED, COMPLETE EPISODES ARE NEEDED

  • With reference and accelerated success rates both at 0.9, acceleration-induced failure risk can be anywhere from 0 to 0.1 because the success gap also reflects episodes the candidate repairs.The success gap equals acceleration-induced failure risk minus the probability of candidate-repaired episodes.Source: §A.5 · p. 15
  • Two accelerators can have zero single-decision effects yet episode-level risks of 1 and 0, so isolated decision measurements do not determine full-policy risk.The fully accelerated controller can reach persistent-error states that no single-decision intervention reaches.Source: §A.5 · p. 15

A.6 RETENTION BOUNDS AND RETENTION BUDGETS

CARE bounds retention—the fraction of reference-solved episodes also solved by an accelerated policy—using tests rather than a ratio of estimated risks. It can also certify a user-specified retention budget, with a different reference-rollout requirement.

A.6 RETENTION BOUNDS AND RETENTION BUDGETS

  • Retention is 1 minus acceleration-induced failure risk divided by reference success probability, so CARE bounds it by inverting tests rather than dividing estimates.This definition assumes the reference success probability is positive.Source: §A.6 · p. 16
  • Inverting the tests yields a simultaneous lower bound on each candidate’s retention and the selected candidate’s retention.The passage states the bound as Ret(λ) ≥ 1 − b̄ with probability at least 1 − δ′ for each candidate; simultaneous bounds hold with probability at least 1 − δ.Source: §A.6 · p. 16
  • A retention-budget test certifies Ret(bλ) > 1 − β with probability at least 1 − δ, but runs the reference on every calibration unit.Unlike failure-triggered evaluation, this mode needs reference outcomes even when the candidate succeeds.Source: §A.6 · p. 16
  • With failure-triggered evaluation, CARE instead estimates reference success on predesignated units and combines its lower bound with the risk certificate to bound retention.The reported combined confidence is at least 0.95 under the stated allocation of δ0 and the remaining error budget.Source: §A.6 · p. 16

B.1 DESIGN-CHOICE VALIDATION AND COST ABLATIONS

This section validates why CARE measures paired, complete episodes and corrects for candidate selection, then shows how its acquisition choices reduce certification cost.

B.1 DESIGN-CHOICE VALIDATION AND COST ABLATIONS

  • Local and episode-level AIF differ by factors of 4.31 on Object and 3.52 on Long, so local effects do not reliably predict full-episode risk.An accelerated action can alter subsequent states, observations, and actions.Source: §B.1 · p. 17, Table 7 · p. 17
  • 46.0% of trials deployed over-budget policies without family correction, versus at most 0.15% with family-corrected testing.The experiment searched over ten candidates.Source: Table 8 · p. 17, §B.1 · p. 17
  • Balanced round-robin acquisition stayed below the candidate-level error budget for every tested task-risk profile, while task-by-task ordering could certify prematurely.Task-by-task ordering may expose only low-risk tasks before certification, omitting tasks responsible for the true deployment risk.Source: Table 9 · p. 17, §B.1 · p. 17
  • 77.8% fewer total rollouts and 81.9% less wall-clock time than fixed-n test-all evaluation came from combining fastest-first and failure-triggered evaluation.Fastest-first evaluation reduced candidate rollouts by 75.6%; failure-triggered evaluation ran the reference only when a candidate failed.Source: Table 10 · p. 18, §B.1 · p. 18

B.2 SENSITIVITY ANALYSIS

CARE's operating point depends on the risk budget and calibration size: more evidence or a looser budget can reduce fallback and increase speedup.

B.2 SENSITIVITY ANALYSIS

  • 8.88× mean speedup at α = 5.0%, n = 250, with 11.7% fallback and no observed over-budget deployment.This is the reported default setting.Source: §B.2 · p. 18
  • At α = 0.025, increasing n from 500 to 2,000 reduced fallback from 56.3% to 1.4% and raised mean speedup from 4.59× to 9.99×.Source: §B.2 · p. 18
  • Tightening α increases fallback, while relaxing it allows faster policies to be certified; larger calibration evidence also reduces fallback and increases speedup.CARE maintained risk control across the reported budgets and calibration sizes.Source: §B.2 · p. 18

C BENCHMARKS

The benchmarks cover simulated tabletop manipulation and a procedurally generated survival game, with task success defined by each environment's goal conditions.

C BENCHMARKS

  • LIBERO uses four ten-task suites on a simulated tabletop arm: Spatial varies layouts, Object varies target objects, Goal varies goals, and Long chains subgoals.Each suite has 500 scenes drawn from 50 fixed initial states per task; success requires meeting the task goal before the step limit.Source: C BENCHMARKS · p. 18, Figure 2 · p. 19
  • Crafter is a procedurally generated 2D survival game with discrete actions and a status bar tracking health, food, drink, energy, and inventory.The two tasks are collect wood and collect drink; an episode succeeds when the agent unlocks the relevant achievement within the step budget.Source: C BENCHMARKS · pp. 18–19, Figure 3 · p. 20
  • Crafter tests whether CARE applies unchanged to language-model agents.Source: C BENCHMARKS · p. 19
Loading 2610.08917v1…