PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO-MAX: Do Robot Policies Adapt When the World Changes?

Yunbei Zhang, Zijian Jin, Yuanzhe Liu, Janet Wang, Xilun Zhang, Yuyou Zhang, Zhenyu Zhang, Daoan Zhang, Shuaicheng Niu, Gen Li, Jianfei Yang, Jihun Hamm, Ismini Lourentzou, Weirui Ye, Bo Liu, Peter Stone, Marco Pavone

arXiv:2609.36518v1cs.ROcs.AI

TL;DR

Many simulation robustness benchmarks fix conditions at reset, leaving open whether policies remain effective when the world changes during execution. LIBERO-MAX compares matched rollouts with and without a mid-task event; across fourteen policies, events reduce success by 11.0–25.7 percentage points.

  • Problem

    Many simulation robustness evaluations configure external conditions before the first action and retain them throughout the rollout, leaving mid-execution adaptation unresolved.

    Source: §1 · p. 2

  • Method

    LIBERO-MAX pairs Base and Dynamic rollouts across eight event types, holding the task, initial state, policy seed, and pre-event action sequence fixed.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across fourteen policies, mid-task events reduce success by 11.0–25.7 percentage points, with the largest shared losses under geometry and observation changes.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The paired protocol and temporal diagnostics provide a testbed for diagnosing failures under mid-execution changes and measuring progress toward policies that remain effective as the world changes.

    Source: §5 · p. 11

    The main track evaluates simulated manipulation with one persistent event from a pre-grasp trigger family; transfer to physical robots, later task stages, and repeated or reversible changes needs further evaluation.

    Source: §I.2 · p. 41

Abstract

from arXiv · show

Robots must often continue a task after a target moves, the viewpoint shifts, or an obstacle appears, even though their earlier observations and committed actions reflect the previous scene. Many simulation robustness benchmarks fix external conditions at reset, leaving this temporal challenge underexamined. We introduce LIBERO-MAX, a benchmark of 8,000 paired cases spanning eight types of changes to geometry, observations, appearance, clutter, and paths. Each pair compares task execution with and without a mid-task event, holding the task, initial state, policy seed, and pre-event action sequence fixed. This controlled comparison distinguishes event-associated regressions from failures already present without the change. Across fourteen current VLA, hybrid, and world-action policies, events reduce success by 11.0-25.7 percentage points. Event profiles reveal shared vulnerabilities to geometry and observation changes, while policy-family rankings interleave. Camera controls show that robustness reflects both competence under the changed conditions and the trajectory from which they are encountered; varying query cadence does not eliminate the gap. Together, the paired protocol and temporal diagnostics establish LIBERO-MAX as a reproducible testbed for diagnosing failures under mid-execution changes and measuring progress toward robot policies that remain effective as the world changes.

1 Introduction

Many benchmarks fix conditions at reset, leaving unanswered whether policies adapt when a task changes after action has begun. LIBERO-MAX pairs identical pre-event histories to test that question.

1 Introduction

  • Mid-task changes differ from reset-time variations because policies inherit the state and action commitments from earlier execution, including queued actions before another observation.Success reflects both competence under changed conditions and the trajectory from which the change is encountered.Source: §1 · p. 2
  • Matching the pre-event history avoids conflating an event’s contribution with reset difficulty, policy randomness, or earlier action divergence.Paired outcome transitions also reveal Base successes that fail after the event.Source: §1 · pp. 2–3
  • LIBERO-MAX creates 8,000 rollout pairs across eight event types, holding task, initial state, instruction, policy seed, and pre-event actions fixed.Base continues without an added event; Dynamic receives one event and continues in the changed scene.Source: §1 · pp. 2–3
  • Across fourteen VLA, hybrid, and WAM policies, events reduce success by 11.0–25.7 percentage points; all paired gaps are negative and statistically resolved.The event breaks 20.8%–56.1% of episodes solved in Base.Source: §1 · p. 3
  • Event profiles show shared geometry and observation vulnerabilities, interleaved VLA/WAM rankings, and persistent success gaps at every tested query cadence.More frequent feedback does not automatically eliminate failures after events.Source: §1 · p. 3

2 Related Work

Prior simulation stress tests commonly set external conditions at reset, while LIBERO-MAX evaluates specific changes during execution using matched no-event and event rollouts.

2 Related Work

  • Earlier benchmarks test visual, embodiment, and other variations, but usually set external conditions at reset; LIBERO-MAX adds matched mid-task event comparisons.The shared executed prefix distinguishes an event’s contribution from differences in preceding execution.Source: §2 · p. 3
  • Chunked policies trade fewer policy calls for delayed feedback, and adaptive chunking adjusts action commitment across tasks; LIBERO-MAX also measures query-cadence effects.Its fixed event boundary supports future evaluation of methods that cancel pending actions or query again after a change.Source: §2 · pp. 3–4
  • Figure 2 compares Base and Dynamic success rates and paired percentage-point changes across policies; every policy’s success decreases after the controlled event.Each pair shares the task, initial state, policy seed, and executed prefix.Source: Figure 2 · p. 4

3 The LIBERO-MAX Benchmark

LIBERO-MAX pairs no-event and event rollouts after a shared executed prefix, then evaluates a balanced set of persistent changes during simulated manipulation.

3.1 Paired evaluation

  • Base continues without an added event, while Dynamic replays the same executed prefix before receiving the event; paired outcomes capture the average success-rate change and four outcome transitions.Assigned cases count toward success even if they do not reach the trigger or receive a post-event query.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Construction and evaluation tracks

  • The benchmark contains 8,000 pairs: 5,600 LIBERO-Plus cases and 2,400 LIBERO-PRO cases, with 1,000 pairs per event type.The 8,000 Dynamic episodes each have a no-event Base control.Source: §3.2 · p. 5, Table 1 · p. 5, Figure 3 · p. 6
  • LIBERO-MAX Lite uses 800 outcome-independent pairs and one tenth of Max’s rollouts per policy; across fourteen policies, success rates and paired gaps stay within 2.4 percentage points of Max.Max is retained for detailed event and policy comparisons.Source: §3.2 · p. 5

3.3 Events and temporal protocol

  • Eight persistent event types include target and receptacle relocation, camera shift, illumination and visual-theme changes, sensor noise, distractors, and obstacle insertion.Each event is introduced once at a deterministic pre-grasp proximity trigger and is not reversed before rollout end.Source: §3.3 · p. 5, Figure 3 · p. 6
  • Chunked policies may execute queued actions after an event but before a new observation; LIBERO-MAX records trigger timing, post-event queries, and this stale-action exposure.The evaluator restores the same initial state and verifies replay of the exact prefix before applying the event.Source: §3.3 · pp. 5–6

4 Results

Across fourteen policies, mid-task events reduce success, with losses concentrated in geometry and observation changes and no consistent ordering between policy families. Camera controls and query-cadence sweeps show that both the changed view and execution history matter, and cadence changes do not close the performance gap.

4.1 Evaluation protocol

  • Fourteen pretrained policies are evaluated on the same 8,000 pairs, with Base and Dynamic success, paired changes, conditional regression, and event-family breakdowns reported.Paired changes use 95% bootstrap intervals and exact McNemar tests; cross-policy Dynamic contrasts use common case IDs with Holm correction.Source: §4.1 · p. 6

4.2 Online events break previously successful episodes

  • Events reduce success by 11.0 to 25.7 percentage points and turn 20.8% to 56.1% of Base successes into failures across policies.Even MolmoAct2 and π0.5, which have strong Base performance, lose success after the event; paired outcomes distinguish these regressions from cases already unsuccessful in Base.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Policy families have no consistent ordering

  • The highest Dynamic success rates belong to VLAs, but remaining rankings and loss sizes show no consistent VLA–WAM ordering.The three highest Dynamic rates are from VLAs, followed by two WAMs; training, scale, and inference settings prevent isolating architecture effects.Source: §4.3 · p. 7

4.4 Geometry and observation changes cause the largest losses

  • Geometry losses span 21.2–52.4 points and observation losses 13.3–49.7, exceeding most appearance-and-clutter and path losses.Target relocation is most damaging for eleven policies; exceptions include camera shift for X-VLA and sensor noise for Fast-WAM and DiT4DiT.Source: §4.4 · pp. 7–9, Figure 5 · p. 9

4.5 Camera controls reveal the roles of viewpoint and execution history

  • Mid-task camera-shift success exceeds Reset by 11.1, 4.0, and 3.8 points for X-VLA, π0.5, and HiMem-WAM, respectively.The comparison indicates that starting in the original view can help policies make progress, but Reset and Mid-task also differ in visited states, time in the changed view, and remaining task budget.Source: §4.5 · pp. 9–10

4.6 Query cadence changes performance without closing the gap

  • Across all 18 query settings, Dynamic remains 11.1–23.0 points below Base, although the best query interval depends on the policy.For X-VLA, Dynamic success rises from 10.8% at Q = 2 to 39.2% at Q = 16; a smaller Base–Dynamic gap alone does not imply better task completion.Source: §4.6 · p. 11

5 Conclusion

LIBERO-MAX evaluates task completion after environmental changes during execution using paired Base and Dynamic rollouts. Its results show substantial losses under online events, while camera controls and a restoration intervention illustrate ways to diagnose and partly address them.

5 Conclusion

  • LIBERO-MAX pairs Base and Dynamic rollouts with the task, initial state, policy seed, and executed action prefix held fixed.Across eight event types and fourteen current policies, the largest shared losses occur under geometry and observation changes.Source: §5 · p. 11, Table 4 · p. 11
  • Camera controls show that both the changed viewpoint and the preceding trajectory affect task success.Changing query cadence leaves substantial losses at every tested setting, and no policy family consistently outperforms the others.Source: §5 · p. 11
  • A simple image-restoration module recovers part of the sensor-noise loss.The benchmark supports identifying disruptive changes and testing methods that help policies continue tasks.Source: §5 · p. 11, Table 4 · p. 11

A.1 Robot benchmarks and dynamic manipulation

The related work positions LIBERO-MAX as a complementary test of changes introduced after execution begins. The benchmark constructs matched Base/Dynamic cases, applies controlled interventions, and includes validity checks and a fixed Lite subset.

A.1 Robot benchmarks and dynamic manipulation; A.2 Object-centric generalization and changes during execution

  • Existing robustness benchmarks generally set external conditions before the first action, while DynamicVLA and DOMINO study continuous target motion and reactive control.LIBERO-MAX instead pairs rollouts with identical executed prefixes; the paper also describes an object-specific change during an ongoing task as a complementary extension to ORION.Source: §A.1 · p. 19, §A.2 · p. 19

A.3 Vision-language-action policies and action commitment; A.4 World models and world-action policies

  • VLA, world-action, and hybrid policies connect visual and language representations, prediction, and control, while LIBERO-MAX tests task completion after an external event.Its evaluation measures completion rather than the accuracy of predicted video or latent state.Source: §A.3 · p. 19, §A.4 · p. 20

B.1 Design requirements

  • The comparison requires an event after execution begins, a shared pre-event history, continued feasibility, and scoring every assigned case.Paired transitions and coverage diagnostics distinguish task completion, reaching the event, and completing after the change.Source: §B.1 · p. 20

B.2 Source selection and balancing

  • The 8,000 cases comprise 5,600 Plus-derived and 2,400 PRO-derived cases, with each event allocated 700 and 300 cases from those sources, respectively.Two fixed parameter variants are used per event, and the primary micro average weights all cases equally.Source: §B.2 · pp. 20–21

B.3 Events, triggers, and prefix replay

  • Eight events change targets, destinations, visual input, clutter, or paths, and the deterministic trigger applies each event during approach.Dynamic restores the initial state, replays the executed prefix, and applies the event at the trigger; chunked policies may still execute queued pre-event actions afterward.Source: §B.3 · p. 21

B.4 Validity and treatment of failures

  • All 5,600 Plus-derived and 2,400 PRO-derived cases pass automated preflight, but the human feasibility audit remains incomplete.Automated validity checks do not establish human-perceived naturalness or successful completion after every intervention.Source: §B.4 · p. 22

B.5 Qualitative intervention examples

  • Matched Base and Dynamic images show observation events changing visual evidence without moving task objects, while geometry, clutter, and path events modify physical state.The latter interventions are designed to preserve a feasible completion route.Source: §B.5 · p. 22, Figure 7 · p. 22, Figure 8 · p. 23

C.1 Outcome-independent subset

  • LIBERO-MAX Lite is a fixed 800-pair subset selected independently of checkpoint identity and policy outcomes, preserving equal allocation across eight events and a 70:30 source composition.The same fixed pool is used for the action-cadence analysis.Source: §C.1 · p. 23

C.2 Agreement with the full benchmark

LIBERO-MAX Lite is a rapid fixed-subset estimate of the full benchmark. It closely preserves reported rates and most policy orderings, while Max remains the primary track for more detailed analysis.

C.2 Agreement with the full benchmark

  • The largest absolute Lite–Max deviation is 2.4 points across reported Base rates, Dynamic rates, and paired gaps, and Lite preserves 88 of 91 pairwise Dynamic orderings.The three differing pairs are separated by at most 2.3 points on Max; DiT4DiT and Light-WAM tie on Lite.Source: §C.2 · p. 23, Table 7 · p. 24
  • Lite supports integration, debugging, ablation, and early comparison, while Max remains the primary reporting track.Max’s larger event cells support more detailed analysis with lower variance.Source: §C.2 · p. 23

D.1 Metrics and statistical comparisons

LIBERO-MAX compares paired outcomes while retaining all assigned cases, then analyzes released policies, event sensitivity, and response timing. Across the fourteen primary policies, events cause substantial losses, with distinct event-specific vulnerabilities.

D.1 Metrics and statistical comparisons

  • The benchmark reports success over all assigned cases, paired Dynamic-minus-Base percentage-point changes, conditional regression, retained performance, and event-family breakdowns.Cases that do not reach the trigger or a post-event query remain in the primary denominator.Source: §D.1 · p. 24

D.2 Released checkpoints and training

  • Fourteen primary policies span VLA, predictive VLA, and WAM designs; comparisons describe released checkpoints with differing scales, training data, and serving protocols.Uncertainty intervals measure evaluation precision, not differences in checkpoint training.Source: §D.2 · p. 24, §D.1 · p. 24

D.3 Action prediction and observation cadence

  • Policies execute Q actions between observations and queries, with each call returning at most H actions and Q ≤ H.The primary evaluations retain each policy’s serving protocol.Source: §D.3 · p. 25, Table 9 · p. 27

D.4 Partial-coverage Mimic-Video diagnostic

  • Mimic-Video is evaluated on 5,972 supported pairs and excluded from primary comparisons because its released checkpoint lacks a LIBERO-10 model.The 2,028 unsupported LIBERO-10 pairs are unavailable.Source: §D.4 · p. 26, Table 10 · p. 27

E.1 Paired outcomes and mechanism diagnostics

  • 20.8%–56.1% of Base successes become Dynamic failures across fourteen checkpoints; event-associated regressions and change-associated successes are counted separately.The smallest overall success-rate gap is 11.0 points for SmolVLA, which loses 56.1% of Base successes; MolmoAct2 retains the most successes.Source: §E.1 · p. 27, Table 11 · p. 28

E.2 Event-level robustness

  • Events cause losses from 21.2 to 52.4 points for geometry and 13.3 to 49.7 points for observation changes, generally exceeding appearance, clutter, and path losses.Target or receptacle relocation causes the largest loss for eleven policies; final outcomes do not reveal whether a policy detected the event or attempted the intended response.Source: §E.2 · p. 28

E.3 Robustness across source categories

Source-category and task-suite breakdowns show that losses persist across policies, but their size depends on category, task, and event parameters. Camera controls compare encountering a changed view after progress with seeing it from the start.

E.3 Robustness across source categories

  • Every checkpoint has lower Dynamic success and a smaller paired drop on PRO-derived than Plus-derived cases, but the subsets contain different task instances.Low Base success limits the possible loss; the aggregate difference does not establish that PRO increases sensitivity to events.Source: §E.3 · pp. 28–29, Table 13 · p. 30
  • Category-specific losses vary: X-VLA loses 37.2 points on Plus background texture and 47.9 on PRO noise and glare, while MolmoAct2 remains sensitive to sensor noise and object shape.Some PRO categories have low Base success, so small positive gaps do not establish successful responses.Source: §E.3 · pp. 29, 31, Figure 12 · p. 32, Figure 13 · p. 33
  • −10.7 to −23.4 points are the category-macro Dynamic-minus-Base deltas across all fourteen checkpoints, with each of 17 source categories equally weighted.Small or positive category gaps can reflect very low Base success rather than successful post-change responses.Source: §E.3 · pp. 29, 31, Figure 11 · p. 31

E.4 Task suites, severity metadata, and parameter sensitivity

  • Within-event Dynamic-success differences between two parameter variants range from 12.2 to 36.4 points across checkpoints, and the most affected task suite varies by policy.The macro deltas remain negative for all fourteen checkpoints when each source category receives equal weight.Source: §E.4 · p. 31, Table 14 · p. 34

F.1 Design and paired comparisons

  • The camera control shares 1,000 cases across X-VLA, π0.5, and HiMem-WAM to compare Base, Reset with the changed view from the first input, and Mid-task after the matched Base prefix.The same source-specific denominators apply across the three conditions.Source: §F.1 · pp. 34–35, Table 15 · p. 35

F.2 Timing effects and source consistency

Across three policies, Mid-task camera changes outperform Reset conditions, although changed views still challenge some policies. Trajectory diagnostics qualify this timing advantage, and milder shifts preserve a Mid-task advantage over Reset.

F.2 Timing effects and source consistency

  • Mid-task success exceeds Reset by 11.1 points for X-VLA, 4.0 for π0.5, and 3.8 for HiMem-WAM; changed views still reduce success for X-VLA and π0.5.Base, Reset, and Mid-task success are 68.2%, 1.3%, and 12.4% for X-VLA; 79.5%, 51.3%, and 55.3% for π0.5; and 72.5%, 68.8%, and 72.6% for HiMem-WAM.Source: §F.2 · p. 35, Figure 14 · p. 36
  • Mid-task exceeds Reset on both sources, but the PRO-only intervals for π0.5 and HiMem-WAM include zero.Resampling reference tasks jointly across sources gives positive Mid-task-minus-Reset intervals for all three policies over the observed cohort; additional seeds and training runs are outside the analysis.Source: §F.2 · p. 35

F.3 Paired outcomes and trajectory diagnostics

  • Reset failures can occur before or after the 18 cm approach threshold, which describes proximity rather than a verified grasp or required task stage.Among cases where Base and Mid-task succeed but Reset fails, Reset never enters the neighborhood in 74/106 X-VLA cases, 21/89 π0.5 cases, and 2/59 HiMem-WAM cases; some Mid-task failures occur despite a new query and no old queued actions.Source: §F.3 · pp. 36–37

F.4 Milder camera shifts

  • At 25%, 50%, and 100% camera strength, Mid-task exceeds Reset by +15.7, +22.1, and +11.1 points, respectively, while both changed-view conditions remain below Base.At 25% strength, Reset and Mid-task success are 36.0% and 51.7%, compared with 68.2% in Base; the timing gap peaks at 50%, not at the strongest shift.Source: §F.4 · pp. 37–38, Figure 15 · p. 37
  • At milder camera strengths, most Reset failures among cases solved in Base and Mid-task occur after reaching the approach threshold.At 25% strength, 18/189 such Reset trajectories never enter the neighborhood; at 50%, 55/221 do not.Source: §F.4 · pp. 37–38

G.1 Sweep design and complete results

The cadence sweep finds policy-specific serving settings but leaves a Dynamic deficit. A separate X-VLA restoration study improves Dynamic success while retaining a substantial gap.

G.1 Sweep design and complete results

  • Across cadence settings, Dynamic success remains 11.1–23.0 points below Base; policy-specific cadence optima differ.X-VLA rises from 21.9% to 60.4% Base success and from 10.8% to 39.2% Dynamic success as Q increases from 2 to 16.Source: §G.1 · p. 38, Table 16 · p. 39

G.2 Native stale-action exposure

  • Under native serving protocols, median stale-action exposure ranges from two to seven actions across six policies, despite eventual post-event queries in response-evaluable cases.Exposure measures delayed feedback, not internal change detection or physical safety; cadence changes do not eliminate the Dynamic deficit.Source: §G.2 · pp. 38–39

H.1 Study design and restoration module

  • The X-VLA restoration study compares native, always-on, and quality-gated processing on 300 fixed sensor-noise cases, with the checkpoint, seed, and serving protocol fixed.The module examines received images at policy queries; it is stateless and requires no training, clean image, simulator state, event flag, or ground-truth occlusion mask.Source: §H.1 · pp. 39–40, §H.2 · p. 40

H.2 Paired effects

  • Gated restoration raises Dynamic success by 7.0 points and reduces the Base/Dynamic gap by 6.7 points, with a 0.3-point Base change.It converts 33 native Dynamic failures to successes while losing 12 native successes; a 19.3-point gap remains, and the difference from always-on restoration is too uncertain to establish an advantage.Source: §H.2 · p. 40, Table 17 · p. 40

H.3 Coverage and processing cost

Gated restoration increases Dynamic success across both sources and corruption settings, with measured preprocessing costs differing between Base and Dynamic. The paired gap reflects both changed conditions and the trajectory from which the policy encounters them.

H.3 Coverage and processing cost

  • Gated restoration increases Dynamic success on both sources and at both corruption settings.Table 18 reports success rates and gated-versus-native gains in percentage points, with pointwise 95% bootstrap intervals.Source: §H.3 · p. 40, Table 18 · p. 41
  • Measured preprocessing averages 9.1 ms in Base and 48.8 ms in Dynamic; always-on means are 14.6 and 48.9 ms.These query-level measurements include quality estimation and restoration but exclude policy inference and simulation, so they are not end-to-end speedup estimates.Source: §H.3 · pp. 40–41

I.1 Interpreting progress on the benchmark

  • The paired gap reflects both changed conditions and the trajectory from which a policy must continue, while family rankings alone do not identify which design causes greater robustness.Reset controls and paired transitions help distinguish these demands; released policies also differ in training, scale, and action interfaces.Source: §I.1 · p. 41

I.2 Scope and limitations

LIBERO-MAX evaluates simulated manipulation with one persistent event from a pre-grasp trigger family, while transfer and intervention realism remain open questions. The discussion proposes broader event tracks and adaptive responses for future study.

I.2 Scope and limitations

  • The main track covers simulated manipulation with one persistent event from a pre-grasp trigger family; transfer to physical robots, later task stages, or repeated and reversible changes remains unevaluated.Geometry screening covers support, workspace, visibility, and collisions, but the human feasibility audit is incomplete; some interventions occur abruptly and need separate naturalness and physical-realization assessment.Source: §I.2 · p. 41
  • Camera controls cover three policies and 1,000 fixed camera cases per policy, while restoration covers one policy and one sensor-noise event.The studies do not establish internal change detection or deliberate replanning, and task completion alone cannot establish safe deployment because safety outcomes were not measured.Source: §I.2 · p. 41

I.3 Learning and evaluating responses to change

  • Future training could expose policies to dynamic-event trajectories and reward restored progress while accounting for interruptions, extra queries, and unsafe contacts.Adaptive VLA and WAM policies could learn when to preserve an action sequence, acquire fresh evidence, or revise the plan.Source: §I.3 · p. 42
  • A proposed task-conditioned gate compares a new observed representation with a latent state predicted from the preceding observation and action, then may continue, cancel pending actions, or stop before replanning.The proposal remains future work and should be compared with fixed-cadence execution and simpler triggers under matched observation and computation budgets, including effects on Base success.Source: §I.3 · p. 42
  • Separate supplemental tracks could test transient, recurrent, and compound events, with recovery latency, action cancellation, and contact measurements extending endpoint success.Base, reset-change, and mid-task controls could distinguish changed-object difficulty from execution history in feasible object-change cases.Source: §I.3 · p. 42
Loading 2609.36518v1…