PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Test-Time Adaptation For Robot Policies

Motasem Alfarra, Pietro Mazzaglia, Markus Peschl, Daniel Dijkman, Christos Louizos

arXiv:2610.20318v1cs.ROcs.LG

TL;DR

Deployment shifts can degrade robot locomotion policies, while offline retraining is poorly suited to real-time on-device constraints. TEMPO stabilizes critic-guided test-time adaptation with entropy regularization and memory-guided sampling. It improves performance across simulated shifts and on a physical Go2 under a calf-payload perturbation.

  • Problem

    Robot locomotion policies can degrade under deployment shifts, while offline retraining interrupts operation and conflicts with real-time on-device compute and reliability constraints.

    Source: §1 · p. 1

  • Method

    TEMPO adapts the actor with a frozen critic, using entropy regularization and a score-based memory with importance sampling to focus updates on hard observations.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    TEMPO consistently improves over no adaptation and naive critic maximization across simulated shifts, with gains also observed on physical Go2; real-world mean score rose from 89.8 to 98.7.

    Source: §1 · p. 2, §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    Fully on-device adaptation improved performance over the non-adapted baseline on a physical Unitree Go2 with a calf payload.

    Source: §5.2 · p. 7, §6 · p. 9

    Adaptation uses no rewards or labels; task-level metrics are reserved for offline or benchmark reporting.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Reinforcement-learning policies for robot locomotion can fail at deployment when operating conditions shift (e.g., rough or slippery terrain, payload changes, sensing degradation), and retraining or manual recalibration is often infeasible under on-device latency and memory constraints. We study test-time adaptation (TTA) for robotic control: adapting a pretrained policy online and in a self-supervised manner during execution, without access to rewards. Our starting point is a lightweight baseline that updates only the actor using a frozen pretrained critic as a deployment-time performance surrogate. Building on this, we introduce TEMPO, a test-time entropy-regularized, memory-guided policy optimization method that stabilizes critic-driven updates under severe shift. TEMPO combines (i) an entropy-promoting regularizer on the critic's predictive distribution to mitigate overconfident extrapolation, and (ii) a score-based memory, inspired by prioritized replay, that prioritizes low-value ("hard") observations and applies importance sampling to focus limited computation on the most informative deployment conditions. Across Go1, Go2, and G1 in MuJoCo Playground under four representative shifts and multiple architectures, TEMPO yields consistent gains, reaching up to 80% improvement in a single-device setting. We further validate on a physical Unitree Go2 with an added calf payload, where fully on-device adaptation improves performance over the non-adapted baseline across real-world episodes.

1 Introduction

Deployment shifts can substantially degrade locomotion policies, while offline adaptation and architectural add-ons conflict with practical on-device constraints. The paper introduces TEMPO, which combines entropy-regularized critic-guided updates with a score-based memory, and reports consistent gains across simulated robots and shifts plus physical-robot validation.

1 Introduction

  • Real-world shifts can combine terrain, hardware, sensing, and environmental changes, substantially degrading policies trained under nominal conditions.Large-scale domain randomization is constrained by which variations can be specified and simulated, while offline retraining interrupts operation and strains on-device constraints.Source: §1 · pp. 1–2
  • The baseline adapts only actor parameters using a frozen pretrained critic as a performance surrogate, with a compact observation memory and no online rewards.The critic guides periodic test-time updates, though naive value maximization can overfit to overconfident estimates.Source: §1 · p. 2
  • TEMPO combines entropy regularization to temper critic overconfidence with a score-based memory and importance sampling to prioritize informative, low-value observations.The memory focuses limited computation on hard deployment conditions.Source: §1 · p. 2
  • TEMPO shows consistent gains across Go1, Go2, and G1 under four shifts and three architectures, with matching trends on a physical Go2 using a calf payload.The physical validation used fully on-device adaptation.Source: §1 · p. 2

2 Robot Adaptation as Test Time Adaptation

The paper formulates deployment as online adaptation from unlabeled observations under shifted dynamics, with evaluation metrics reserved for reporting rather than adaptation. It presents QMAX, a buffered critic-guided actor update that is lightweight and self-supervised, and can recover a significant portion of performance loss.

2.1 Problem Formulation and Evaluation Protocol

  • Deployment shifts change observations and transition dynamics, and the formulation assumes an actor–critic policy whose critic uses the actor's observations.The test-time rule may update policy parameters, critic parameters, observations, or a subset of them.Source: §2.1 · p. 3
  • Test-time adaptation uses online quantities from an unlabeled observation stream, while task metrics are used only for offline or benchmark evaluation.The adaptation mechanism does not use rewards or labels.Source: §2.1 · p. 3

2.2 Constructing a Baseline

  • Single-environment observations are strongly correlated and narrow, while abrupt shifts and latency limits constrain adaptation; QMAX therefore aggregates observations in a small memory.The buffer supports minibatches that are more stable and diverse than a single observation.Source: §2.2 · pp. 3–4
  • QMAX maximizes the frozen critic's predicted action value on buffered observations, then applies a gradient step to the actor.This uses critic predictions and test-time observations without rewards or labels.Source: §2.2 · p. 4
  • QMAX recovers a significant portion of the performance drop using a lightweight, self-supervised update guided by the pretrained critic.The critic remains frozen while actor parameters are updated.Source: §2.2 · p. 4

3 Test-time Entropy regularized, Memory guided Policy Optimization

TEMPO stabilizes critic-guided test-time policy updates by combining entropy regularization with a score-based memory that focuses adaptation on informative observations. It updates only policy parameters while keeping the pretrained critic frozen.

3 Test-time Entropy regularized, Memory guided Policy Optimization

  • TEMPO combines entropy-regularized critic maximization with score-based memory to stabilize and improve sample-efficiency of test-time policy updates.It updates only policy parameters while keeping the pretrained critic frozen.Source: §3 · p. 4

3.1 Entropy Maximization

  • Entropy regularization on the critic’s predictive distribution favors high-value updates while avoiding overconfident, overly concentrated predictions.The regularization strength is controlled by λ, and the method is intended to improve stability under large shifts.Source: §3.1 · p. 4

3.2 Score-Based Memory

  • TEMPO retains high-scoring observations, which correspond to low predicted value and harder deployment states, then samples minibatches proportional to their scores.The bounded buffer admits a new observation when its score exceeds the current minimum and evicts the lowest-scoring observation.Source: §3.2 · pp. 4–5
  • At each control step TEMPO records observations, and every K steps updates the policy once using a score-sampled minibatch and the entropy-regularized objective.The critic remains frozen during these on-device updates.Source: §3.2 · p. 5

4 Related Work

Prior robotics robustness work hardens policies or adds structure, while much test-time adaptation research has focused on perception. Robotics adaptation must also preserve stable closed-loop control within strict latency and memory budgets.

4 Related Work

  • Robotics robustness approaches use simulation hardening, domain randomization, exteroception, or auxiliary mechanisms, but deployment beyond training conditions can still require retraining or added structure.This motivates lightweight on-device adaptation of a single deployed actor through critic-guided updates.Source: §4 · p. 5
  • Most test-time adaptation progress has been on perception benchmarks, while robotics adds correlated on-policy streams and strict latency and memory constraints for stable control.The paper positions its contribution as lightweight, critic-guided TTA for on-device locomotion.Source: §4 · pp. 5–6
  • Across Go2 deployment shifts, TEMPO provides the largest performance gain, especially as shift severity increases.Figure 2 reports episodic return and average forward linear velocity across friction, calf-mass, rough-terrain, and torso-mass shifts.Source: Figure 2 · p. 6

5 Experiments

Across simulated shifts, TEMPO outperforms the non-adapted policy and QMAX; real Go2 trials, mixed corruptions, continual shifts, component ablations, and RMA tests also show gains over relevant baselines.

5 Experiments

  • The evaluation uses Go1, Go2, and G1 simulated locomotion tasks with three pretrained policy families and four shift types.Unless otherwise stated, the setup uses Fast-TD3 and Go2 on flat terrain without domain randomization.Source: §5 · p. 6

5.2 Real World Deployment

  • On physical Go2 trials, TEMPO improves mean score from 89.8 to 98.7 over the non-adapted baseline across tested commands.For backward walking at medium speed (vx = −0.5), the score rises 15%, from 98.7 to 113.5.Source: Table 1 · p. 7, §5.2 · p. 7

5.1 Effectiveness of TEMPO

  • Across four simulated shift families, TEMPO is consistently best across severity, with the clearest advantages in the hardest regimes and faster, more continuous locomotion than QMAX.The reported improvements appear in both episodic return and average forward linear velocity.Source: §5.1 · p. 7

5.3 Mixed Corruption Setting

  • Across eight mixed-corruption conditions, TEMPO averages 14.09 reward versus 11.18 for NO ADAPT and 12.48 for QMAX.The largest single-case gain is +4.1 (+39.0%) over QMAX under torso mass with rocky terrain at highest severity.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 Continual TTA

  • As shift severity increases during continual adaptation, TEMPO recovers faster and reaches a higher performance plateau than QMAX.Both methods recover source-domain performance instantly, without transient degradation or additional adaptation.Source: Figure 4 · p. 8, §5.4 · p. 8

5.5 Ablating Components of TEMPO

  • Across six payload and terrain settings, entropy regularization adds +0.5 to +1.6 return over QMAX, while score-based memory adds +1.0 to +2.3.Combining both components yields the best results in every column, improving over QMAX by +1.4 to +2.5.Source: Table 3 · p. 9, §5.5 · p. 9

5.6 RMA as an Adaptation-Aware Backbone

  • Under the evaluated shifted test streams, adding TEMPO to RMA’s native adaptation yields the best performance across shift severities.The comparison includes frozen RMA, native RMA adaptation, and RMA with TEMPO updates.Source: Figure 5 · p. 9, §5.6 · p. 9

6 Conclusions

The paper concludes that TEMPO improves reward-free, on-device adaptation under deployment shifts, with gains in simulation and on a physical Go2. The appendix outlines its online update loop, score-based memory, implementation, and sensor setup.

6 Conclusions

  • Across simulated shifts and on a physical Go2 with a calf-payload perturbation, TEMPO improves over no adaptation and naive critic maximization, with clearest gains under severe shifts.The real-world evaluation uses fully on-device adaptation.Source: §6 · p. 9

A Test-time Entropy regularized, Memory guided Policy Optimization

  • TEMPO uses the distributional critic output, which the appendix identifies as the critic network’s penultimate-layer output.The appendix supplies implementation algorithms because the main text omitted a full explicit algorithm for space reasons.Source: A Test-time Entropy regularized,… · p. 13
  • The online loop acts on each observation and adapts periodically when the memory is sufficiently populated, using importance sampling and one update step.Updates occur every K observations to limit added compute.Source: A Test-time Entropy regularized,… · p. 13, §B.1 · p. 13
  • TEMPO stores observations with scores, replacing the lowest-scoring memory item when a higher-scoring observation arrives after capacity is reached.The implementation can reuse the action already computed by the online loop to avoid an additional forward pass.Source: A Test-time Entropy regularized,… · p. 13

B.1 Setup.

  • The policy and critic use the same non-privileged proprioceptive and command-related observations, enabling adaptation directly from robot sensors.For the physical Go2, an Extended Kalman Filter estimates linear velocity from built-in foot force sensors.Source: §B.1 · p. 13

B.2 Generalization to Different Architectures and Environments

TEMPO transfers beyond its default Fast-TD3-on-Go2 setting without algorithm modifications or additional modules. The evaluation reports gains across policy families and on a humanoid embodiment.

B.2 Generalization to Different Architectures and Environments

  • TEMPO improves rocky-terrain performance on a Fast-TD3 policy trained with domain randomization, complementing robustness-oriented pretraining at deployment.The accompanying Table 4 reports Go1 and G1 results under mass perturbations, but the supplied passage gives no cell values.Source: §B.2 · p. 14
  • TEMPO remains effective with the SIMBA-V2 architecture without architecture-specific tuning, supporting applicability across actor–critic policies.Source: §B.2 · p. 14
  • TEMPO improves rocky-terrain performance on G1, suggesting transfer beyond quadrupeds to a humanoid with different morphology and contact dynamics.Source: §B.2 · p. 14

B.3 Sensitivity to Hyperparameters, Memory Size, Batch Size, and Adaptation Interval

TEMPO remains the best-performing method across memory-size, batch-size, and adaptation-interval sweeps, while offering configurations suited to tight on-device resource budgets. The results also report stable behavior across tested values of λ.

B.3 Sensitivity to Hyperparameters, Memory Size, Batch Size, and Adaptation Interval

  • Strong performance with small memories and minibatches and larger adaptation intervals supports low storage, limited per-update compute, and fewer updates.Very small memories can reduce informative-state coverage, while very large memories yield diminishing returns; larger intervals reduce overhead but slow reaction to abrupt changes.Source: §B.3 · pp. 14–15
  • TEMPO remains the best-performing method throughout memory-size, minibatch-size, and adaptation-interval sweeps under Go2 torso-mass corruption at 7.5 kg.The figure compares performance across the three deployment-critical hyperparameters.Source: §B.3 · p. 14, Figure 7 · p. 15
  • TEMPO shows stable behavior across λ values of 0.1, 0.25, 0.5, 0.75, and 0.9.Source: §B.3 · p. 15

B.4 Performance on Source Domain

After prolonged shifted deployment, QMAX and TEMPO recover source-domain performance immediately without additional recovery steps. Under shifts, TEMPO also provides the most stable and accurate angular-velocity tracking across severities.

B.4 Performance on Source Domain

  • QMAX and TEMPO immediately recover source-domain performance after shifts are removed, matching the non-adapted policy without transient degradation.The return-to-source evaluation covers calf- and torso-mass shifts with Fast-TD3; adapted parameters remain fixed during the final phase.Source: §B.4 · p. 15, Figure 9 · p. 16
  • On the source domain, no adaptation, QMAX, and TEMPO are numerically indistinguishable across reward signals: environment reward 25.6, Linear Velocity tracking reward 864, and Angular Velocity Tracking reward 457.The values are averaged across six seeds.Source: §B.4 · pp. 15–16

B.5 Improving Angular Velocity Tracking

  • Across four shift families and increasing severities, TEMPO provides the most stable and accurate angular-velocity tracking; gains are most pronounced on rocky terrain and torso-mass perturbations.QMAX also consistently improves tracking accuracy relative to no adaptation.Source: §B.5 · p. 16

B.6 Does TEMPO Improve Local Action Coverage?

On severe mixed corruption, TEMPO raises critic entropy relative to QMAX, alongside a higher return; the appendix also reports improved angular-velocity tracking and describes RMA target shifts and memory-score refresh variants.

B.6 Does TEMPO Improve Local Action Coverage?

  • 1.15 initial critic entropy fell to 1.05 with QMAX while TEMPO raised it to 1.41 (34% relative gap), alongside returns of 9.9 vs 14.0 (+41% relative).The passage links the entropy difference temporally with the return gap and says elevated critic entropy preserves broader gradient support and wider local action coverage.Source: §B.6 · pp. 16–17
  • TEMPO achieves the most stable and accurate Go2 angular-velocity tracking, particularly under severe shifts, compared with no adaptation.Figure 10 reports tracking error across four corruption families, averaged over five seeds.Source: Figure 10 · p. 16

B.7 RMA Setup

  • RMA’s Go2 flat-terrain training randomization excludes tested targets including friction 0.1–0.2, torso payload 5–10 kg, calf payload 3–5 kg, and roughness 0.05–0.1.The setup also randomizes friction, armature, center of mass, link mass, torso mass, and initial joint position within specified training ranges.Source: §B.7 · p. 17

B.8 Refreshing Scores of Memory

  • Periodic score refresh and age-decayed scoring add per-step compute without improving returns over minimum-score replacement in the tested simulations.The authors attribute this to limited simulation state diversity and expect stale-score handling to matter more in richer real-world deployments.Source: §B.8 · p. 17

B.9 Comparison to Test-Time Training

The comparison evaluates QMAX against Test-Time Training on Go2 under a mass-on-calf shift. Increasing TTT’s regularization coefficient monotonically lowers performance, with β = 0 recovering QMAX.

B.9 Comparison to Test-Time Training

  • TTT is compared with QMAX on Go2 under a mass-on-calf shift by sweeping β; its loss adds β-weighted KL regularization to LQMAX.The stated comparison is the proposed baseline QMAX versus TTT.Source: §B.9 · p. 17
  • For MC=4, return falls from 16.8 at β = 0 to 15.6 at β = 0.08, with monotonic decreases across the sweep.Intermediate returns are 16.3, 16.1, and 15.7 at β = 0.02, 0.04, and 0.06, respectively.Source: §B.9 · p. 17
  • For MC=5, return falls from 14.7 at β = 0 to 13.3 at β = 0.08, with monotonic decreases across the sweep.Intermediate returns are 14.4, 13.7, and 13.6 at β = 0.02, 0.04, and 0.06, respectively; β = 0 exactly recovers QMAX.Source: §B.9 · p. 17

C Compute Requirements, Limitations, and Broader Impact

Simulation experiments use a single V100 GPU, while real-world Go2 adaptation runs on four CPU cores. The paper notes dependence on an informative pretrained critic, added inference cost, and the need for testing and oversight in safety-critical deployment.

Compute Requirements

Limitations

  • TEMPO depends on a pretrained critic that remains reasonably informative under shift; severe miscalibration or a highly mismatched training domain may degrade adaptation.The method also adds a small memory and compute cost during inference.Source: C Compute Requirements, Limitations,… · p. 17

Broader Impact

  • Adaptive systems may pose risks without adequate testing and oversight, especially in safety-critical environments; the method is intended to complement careful system design.The authors encourage responsible use in real-world settings.Source: C Compute Requirements, Limitations,… · p. 17

Related papers

Loading 2610.20318v1…