PaperlayerView exact revision on arXiv

Source-linked AI summary

Median Temporal Ensembling: Training-Free Robust Aggregation for Action-Chunked Visuomotor Policies

Yuhang Jiang

arXiv:2609.27167v1cs.ROcs.LG

TL;DR

Overlapping action chunks give temporal ensembling several predictions per timestep, but its weighted mean can be moved arbitrarily by one corrupted prediction. The paper substitutes a coordinate-wise median and evaluates it against sparse corruption and blank-frame faults. Across 25 configurations and corruption levels, the median is never worse than the mean and is significantly better in 15 after Holm correction.

  • Problem

    Temporal ensembling's weighted mean has breakdown point 0, so a single corrupted prediction can move the aggregate arbitrarily far.

    Source: §I · p. 1

  • Method

    The paper replaces the exponentially weighted mean with the coordinate-wise median of the same covering candidates, a one-line change requiring no retraining.

    Source: §V · p. 4

  • Results

    Across 25 configuration and corruption-level combinations, the median is never worse than the mean and is significantly better in 15 after Holm correction.

    Source: D. Results · p. 5

  • Takeaways & Limitations

    The median also recovers performance from blank camera frames and transfers to a second policy class.

    Source: D. Results · p. 5

    The results are in simulation, and the blank-frame fault is evaluated on only one task and one stride.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Action-chunked visuomotor policies predict overlapping trajectories, so every executed action is covered by several predictions. Temporal ensembling smooths execution by combining these predictions with an exponentially weighted mean. One corrupted prediction can move the aggregate without bound: its breakdown point is 0. We use adversarial corruption to stress this deployed aggregator and to compare two kinds of guarantee. A metric guarantee bounds the response to a perturbation of a given size. A combinatorial guarantee instead bounds the damage when at most q of the M candidates covering a timestep are corrupted, whatever their size. Encoder adversarial fine-tuning recovers 44% of the loss under the published patch attack, but only 7.3% after the attacker's step size is increased. By contrast, the coordinate-wise median of the same candidate set keeps its recovered fraction flat as attack optimisation increases. Median temporal ensembling costs one line and requires no retraining. Across 25 (configuration, corruption-level) combinations it is never worse than the mean and is significantly better in 15. It also transfers to a second policy class, and it recovers performance under a failure with no attacker in the loop at all: camera frames that arrive blank. Its effect on clean data is configuration-dependent, from -0.04 to +0.07. We also give the boundary: corruption that shifts every covering prediction by the same amount is invisible to this whole family of statistics, and no equivariant aggregator can remove it.

I. INTRODUCTION

Overlapping action chunks give temporal ensembling several predictions per timestep, but its weighted mean can be moved arbitrarily by one corrupted prediction. The paper replaces it with a coordinate-wise median and compares metric robustness with guarantees for sparse corruption.

I. INTRODUCTION

  • The weighted mean has breakdown point 0: one corrupted prediction can move its output arbitrarily far.This vulnerability belongs to the estimator, not to any particular adversary.Source: §I · p. 1

A. Contributions

  • Across 25 configurations and corruption levels, the median is never worse than the mean and is significantly better in 15 after Holm correction.The one-line replacement also recovers performance from blank camera frames and transfers to ACT.Source: A. Contributions · pp. 1–2
  • The median stays within the clean-candidate range exactly zero times out of 53,704 timesteps when fewer than half the covering candidates are corrupted; the mean leaves it in 81–93%.This directly measures the median's bracket property.Source: A. Contributions · p. 2
  • Encoder fine-tuning recovers 44% of the published patch-attack loss but only 7.3% after the attacker's step size increases.The encoder becomes less sensitive, but the defence fails against a scale-invariant optimiser.Source: A. Contributions · p. 2
  • Common-mode corruption shifts every covering candidate equally, making it invisible to consistency statistics and leaving no equivariant aggregator able to remove it.The predicted blind spot can leave monitors silent through a total task failure.Source: A. Contributions · p. 2

II. RELATED WORK

Prior work uses chunk overlap for action prediction, selection, and monitoring, while robust aggregation studies typically combine estimates from separate workers. This paper connects those settings by treating temporal ensembling's weighted mean as an estimator.

Action chunking, temporal ensembling and the monitors built on them

  • Action chunking and temporal ensembling combine overlapping action predictions, but some chunked policies do not combine overlapping predictions in the same way.OpenVLA emits one action per query, while π0 replans and discards unexecuted predictions; both are described as clean-performance choices.Source: §II · p. 2
  • TAS [3] trains a selector over cached predictions, whereas this paper's replacement needs no training.STAC/Sentinel, ACC [10], TIDE [11], and Falcon score agreement among overlapping chunks for monitoring or reuse.Source: §II · p. 2

Attacks and defences on visuomotor policies

  • Patch-position defences use masking and voting for a combinatorial guarantee over image regions, but are not evaluated here because their forward-pass cost compounds with diffusion sampling.The authors say cheaper variants have not, to their knowledge, been demonstrated on such a policy.Source: §II · p. 2

Robust aggregation

  • Robust aggregation studies such as Yin et al. treat estimates from workers, while temporal ensembling's weights were introduced for recency-based smoothing rather than estimator theory.On clean data, the aggregators are identical in five of eight configurations and differ by at most 0.07 in the others.Source: §II · p. 2

III. OVERLAPPING PREDICTIONS

Receding-horizon execution makes multiple action chunks cover the same control timestep. The number of covering predictions varies with timestep phase, and corruption is framed as sparse or common-mode.

III. OVERLAPPING PREDICTIONS

  • Under receding-horizon execution, each query emits a chunk but only h entries are executed, so an absolute timestep can be predicted by several queries.The executable part starts at index To − 1 because earlier entries align with observations already in the past.Source: §III · p. 2
  • For Tp=16, To=2, and h=2, the number of covering predictions alternates between 7 and 8 across timesteps.The two coverage counts occur at 48.8% and 48.2% of timesteps, respectively; M≤6 occurs only at episode boundaries (3%).Source: §III · pp. 2–3

IV. THREAT MODEL AND ATTACK CALIBRATION

  • The threat model distinguishes sparse corruption of some covering candidates from common-mode corruption that shifts every covering candidate by the same b(τ).The two regimes are defined by which predictions covering a timestep are corrupted.Source: §IV · p. 3

A. Sparse Query Corruption

Sparse query corruption selects some policy observations for perturbation while leaving others clean; the main schedule uses consecutive corrupted queries. Sensor faults provide a non-adversarial case, but the sparse-corruption guarantee has boundaries.

A. Sparse Query Corruption

  • The attacker perturbs selected query observations within an ℓ∞ ball of radius ϵ, leaving the remaining queries clean, and uses bursts of q consecutive queries per period.The schedule is chosen because a contiguous run maximises the corrupted fraction at the worst-hit timesteps.Source: A. Sparse Query Corruption · p. 3
  • The median guarantee requires 2q(τ) < M(τ); a tie can include a corrupted central order statistic, and the nominal schedule q can be exceeded at some timesteps.The covering set spans seven or eight consecutive queries while the schedule repeats every Mmin=7.Source: A. Sparse Query Corruption · p. 3

B. Sensor Faults and Intermittent Access

  • Blank frames take can success from 1.00 to 0.510 at q=2, whereas a stale frame, intermittent occlusion, motion blur, or a camera frozen for the burst costs at most 0.08.The blank-frame fault is described as comparable in damage to the calibrated attack at the same density.Source: B. Sensor Faults and Intermittent Access · p. 3
  • The condition 2q(τ) < M(τ) bounds the output to the clean range, not task success; the multi-camera or multi-agent case is not tested.Task success collapses at q=4 on can and q=5 on square.Source: B. Sensor Faults and Intermittent Access · p. 3

C. Calibrating the Attacks

The attack calibration finds that increasing patch step size materially strengthens the published attack, while the burst attack saturates by 200 steps. Median temporal ensembling replaces the weighted mean with a coordinate-wise median, whose bracket guarantee holds when fewer than half of the covering candidates are corrupted.

C. Calibrating the Attacks

  • Increasing only the patch step size lowers success from 0.185 to 0.025 on can and from 0.700 to 0.475 on square.The published update rule at its stated step size yields patches indistinguishable in strength from the released baseline.Source: C. Calibrating the Attacks · p. 3
  • At 200 PGD steps, the burst attack saturates; undefended success on can h=2, q=2 is 0.780, 0.560, and 0.640 at 50, 200, and 500 steps.The 10× sweep is retained as a robustness check, and effectiveness is reported as the fraction of attack-induced loss recovered.Source: C. Calibrating the Attacks · pp. 3–4

V. MEDIAN TEMPORAL ENSEMBLING

  • Replacing the exponentially weighted mean with the coordinate-wise median costs one line, requires no retraining or extra forward passes, and costs under 0.3% of one decision.The deployed discount parameter remains m=0.01.Source: §V · p. 4

A. The Bracket Guarantee

  • With fewer than half the covering candidates corrupted, the median stays within the range of clean candidates regardless of the corrupted values.The bracket argument applies to arbitrary corrupted values, including adaptive choices.Source: A. The Bracket Guarantee · p. 4
  • Even an oracle choosing corrupted values to move the median farthest does not make the task fail in the reported can and square settings.Success is 0.975 and 0.955 on can at q=2, 3, and 0.835 and 0.840 on square at q=3, 4; the table caption distinguishes independent movement from consistent drift but supplies no cell values.Source: A. The Bracket Guarantee · p. 4, Table II · p. 4

B. Weighting versus Statistic

The median’s advantage over the mean comes from the choice of statistic rather than the weighting scheme. Its clean-range guarantee is directly observed below the corruption threshold, while its task-level benefit can persist beyond that threshold.

B. Weighting versus Statistic

  • Uniform and exponential weights both yield 0.600 success, while every rank statistic gains at least 0.28 on the same candidate set.The paired comparison holds the candidate set and episode seeds fixed.Source: B. Weighting versus Statistic · p. 4
  • None of six pairwise comparisons separates the median from the two medoids after Holm correction; pooled evidence also misses the 0.017 threshold.At n=200, medoid point estimates are higher by 0.035 to 0.050.Source: B. Weighting versus Statistic · p. 4

C. Measuring the Breakdown Point

  • Across 53,704 timesteps in the guaranteed regime, the median leaves the clean-candidate range zero times, while the mean leaves it in 81–93%.The median’s maximum normalised excess is 0.0000; Table IV covers can and square at h=2 across the listed corruption levels.Source: C. Measuring the Breakdown Point · p. 4, Table IV · p. 5
  • Coordinate-wise medians can assemble an action no query proposed, but the output remains near the candidates and inside their coordinate-wise range.At can h=2, q=2, it coincides with a candidate in 0.03% of corrupted timesteps and is 0.13 of the mean pairwise distance from the nearest candidate on average.Source: C. Measuring the Breakdown Point · p. 5
  • The median still gains 0.240 at can h=1, q=8, despite every covering set exceeding the bracket condition.The breakdown threshold predicts the statistic’s guarantee, but not the magnitude of task benefit, which varies across configurations.Source: C. Measuring the Breakdown Point · p. 5

D. Results

Across the tested configurations and corruption levels, the median is never worse than the mean and is significantly better in 15 after Holm correction. The advantage also appears under blank frames, stronger attacks, and a second policy class, while clean-data effects vary by configuration.

D. Results

  • Across 25 cells, the median is never worse than the mean and is significantly better in 15 after Holm correction.The sweep is broad rather than deep because repeating one n=50 cell on the same seeds shifted it by 0.08.Source: D. Results · p. 5
  • Across 200 paired episodes in the largest cell, the median wins 78 while the mean wins 10; both succeed in 57 and both fail in 55.The attack and seed are identical, with only the aggregator changed.Source: D. Results · p. 5
  • On PushT, the median’s effect holds at every tested level on both backbones: +0.095 to +0.199 with the CNN and +0.039 to +0.078 with the Transformer.The Transformer has the fewest candidates in the sweep, M ∈{4, 5}.Source: D. Results · p. 5
  • With blank camera frames, the median raises can success from 0.510 to 0.690 at q=2 and from 0.120 to 0.185 at q=3; both gains survive Holm correction.At q=4 both aggregators score 0.000; the recovered fraction is 0.37 at q=2 and 0.07 at q=3 because blinded wrong actions can remain within the clean-candidate range.Source: D. Results · p. 5
  • On clean data, the effect ranges from −0.040 to +0.069 across configurations, and none of the three nonzero changes is significant.Lift is immune to the tested threat, with both aggregators succeeding on every episode.Source: D. Results · p. 5
  • Under 50, 200, and 500 attack steps, the median’s recovered fraction stays at 0.818, 0.750, and 0.778, with a significant paired advantage at each budget.The attack budget is increased 10× from 50 to 500 steps on can h=2, q=2.Source: D. Results · p. 5
  • On the ACT policy, the median raises success from 0.010 to 0.595 at q=2 and from 0.000 to 0.175 at q=3 under the recalibrated 20-step attack.At q=4 both aggregators score 0.000; the reported gain holds over attack budgets where the undefended policy still functions.Source: D. Results · p. 5

VI. A METRIC-TYPE DEFENCE: ENCODER FINE-TUNING

Encoder adversarial fine-tuning aims to reduce the response to a corrupted visual input, but its gains weaken as patch attacks are optimized more strongly. The section contrasts this metric-type defence with the median’s robustness to per-query corruption.

VI. A METRIC-TYPE DEFENCE: ENCODER FINE-TUNING

  • Fine-tuning the encoder for 3000 steps with the published patch and a clean-to-perturbed embedding consistency term targets small response to a corrupted input, not a bound on corrupted-input count.The diffusion head is frozen; this is a metric-type defence, unlike the patch defences’ combinatorial guarantee over image regions.Source: §VI · p. 6
  • Under the released patch, fine-tuning recovers 44.4% of attack-induced loss on can and 59.6% on square, with small nonzero clean costs.Clean success changes from 0.985 to 0.970 on can and from 0.940 to 0.935 on square.Source: §VI · p. 6
  • On can, recovered loss falls from 0.444 to 0.099 at α=5 × 10−4 and ends at 0.073 as patch-attack step size increases; on square it falls from 0.438 to 0.258.The can decline is 6.1×; the strongest-attack rows remain significant, so the effect collapses without being eliminated.Source: §VI · p. 6
  • The failure is not gradient masking: fine-tuning lowers gradient magnitude while preserving direction, and a scale-invariant sign optimiser can still take a full-size step.Regenerating the patch against the current encoder reduces recovered loss from 44.4% to 15.6% for the fixed attack and to 3.6% against the strong patch, no longer significant.Source: §VI · pp. 6–7
  • Against per-query burst attacks, the fine-tuned encoder recovers 0.091, 0.134, and −0.111 at 50, 200, and 500 PGD steps, while the median recovers 0.75–0.82 significantly in each cell.The encoder’s response to a fixed patch does not transfer to perturbations optimized per query.Source: §VI · p. 7

VII. THE COMMON-MODE BLIND SPOT

This section asks what consistency-based statistics can detect when corrupted predictions do not disagree with clean ones. It frames these statistics as a family invariant to common translation and relates that family to existing runtime monitors.

VII. THE COMMON-MODE BLIND SPOT

  • The common-mode blind-spot question is what remains when corrupted and clean predictions do not disagree.The section introduces the family of consistency statistics to formalize this case.Source: §VII · p. 7

A. A Family of Consistency Statistics

  • A statistic belongs to the Δ-family when translating every candidate by the same amount leaves its value unchanged.Pairwise differences are common, but distributional statistics require the broader invariance formulation.Source: A. A Family of Consistency Statistics · p. 7
  • TIDE, STAC, and Falcon’s reuse gate are in this family; ACC is in it only when bias is approximately constant over the overlap window.The passages describe TIDE’s fresh-chunk comparison, STAC’s translation-invariant distance, and Falcon’s difference norm.Source: A. A Family of Consistency Statistics · p. 7

B. The Common-Mode Null Space

A common translation of all covering candidates leaves every Δ-family statistic unchanged, regardless of the translation’s size. For equivariant aggregators, the same corruption shifts the output rather than being removed.

B. The Common-Mode Null Space

  • A common bias added to every covering candidate leaves every Δ-family statistic unchanged, regardless of its magnitude.The theorem assumes the same bias for all candidates covering the timestep.Source: B. The Common-Mode Null Space · p. 7
  • For an equivariant aggregator, comparing a fresh chunk with the ensembled plan is also Δ-family and inherits the blind spot.The mean, median, medoid, and any quantile satisfy the stated equivariance condition.Source: B. The Common-Mode Null Space · p. 7
  • Common-mode corruption shifts an equivariant aggregator’s output by exactly the shared bias, so only the non-common-mode component can be recovered.The premise is an exact common bias on every candidate; statistics comparing candidates against an external reference fall outside the definition.Source: B. The Common-Mode Null Space · p. 7

C. What the Blind Spot Costs

The blind spot has practical consequences: monitors can remain silent through task failure when corruption makes covering predictions agree. The paper limits its evidence to simulation and to settings with overlapping predictions that are combined.

C. What the Blind Spot Costs

  • With thresholds calibrated by their prescribed rules, TIDE never fires and ACC fires on 2% of episodes after the physical patch lowers can success to 0.04.Both monitors separate attacked from clean can episodes well by AUROC, but the prescribed operating point misses the failure.Source: C. What the Blind Spot Costs · p. 7
  • Dense bursts remain detectable, unlike the physical patch: the patch approximates equal shifts across queries, while untargeted PGD leaves unequal candidate displacements.With dense bursts, TIDE and ACC reach 98% / 98% at q=2 on can and 30% / 92% on square, while policy success falls to 0.00 and 0.18, respectively.Source: C. What the Blind Spot Costs · pp. 7–8

VIII. LIMITATIONS

  • The evidence is simulation-only, with fixed known strides h∈{1,…,4} and two policy families; the mechanism requires several predictions to overlap and be combined.The blank-frame fault is evaluated on one task and one stride; there is no hardware, real-optics, or lighting, viewpoint, or material study.Source: §VIII · p. 8

IX. CONCLUSION

  • The conclusion’s boundary is common-mode corruption: it is invisible to the Δ-family and cannot be removed by equivariant aggregation.An oracle attacker restricted to that bracket does not make the task fail, though it causes more damage when it pushes the aggregate in the same direction at consecutive timesteps.Source: §IX · p. 8

Related papers

Loading 2609.27167v1…