PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Robot Metareasoning: Adaptive Computation Budgets for Real-Time Control

Lorenzo Magnino, Antonio Marino, Michael Amir, Daigo Shishika, Amanda Prorok

arXiv:2610.23662v1cs.RO

TL;DR

Fixed action commitments constrain how robots trade deliberation against feedback, although the useful computation budget varies with the state. The paper learns a semi-MDP policy that jointly selects execution horizons and solver budgets; across several robotic environments, it adapts budgets to task demands and benefits when budget variation can be exploited.

  • Problem

    Existing controllers fix action commitment and solver computation budgets before deployment, although the useful budget depends on state-specific dynamics, constraints, and optimization gains.

    Source: §I · p. 1

  • Method

    The paper formulates adaptive computation budgeting as a semi-MDP and learns an RL policy to choose execution horizons, using state, plan, and planner information.

    Source: §I · p. 2, §II · p. 3, §III · p. 3

  • Results

    Across three environments with distinct sources of difficulty, the policy adapts budgets to task demands, and online adaptation is highly beneficial when exploitable budget variation exists.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The results support adaptive computation budgets as a useful capability for embodied agents when different states call for different levels of deliberation and reaction.

    Source: §I · p. 1, §VII · p. 8

Abstract

from arXiv · show

Planning while moving faces an historical trade-off between deliberation and reactivity. A planner running concurrently with execution needs time to compute, so the robot must commit to an open-loop action chunk for that duration: a longer commitment buys a better plan at the cost of feedback frequency. Existing controllers fix this action commitment, and consequently the computation budget, before deployment. However, the optimal computation time is highly state-dependent (due to factors such as unmodeled dynamics, safety constraints, and marginal optimisation gains), i.e. dynamic, therefore offline static tuning risks leaving systems blind during critical maneuvers or unable to find global effective solutions. We cast the identification of the optimal computation budget as a semi-Markov decision process (SMDP) in which a single decision defines both the robot action commitments and the time allocated to the solver. To solve this, we train a reinforcement-learning policy conditioned on the current state, the executing plan, and inexpensive planner statistics. Across ground-robot navigation, agile quadrotor flight, and contact-rich manipulation, the learned policy changes its budget with the demands of the scenario, discovering efficient reasoning patterns that outperform fixed-budget baselines. https://lorenzomagnino.github.io/learning-robot-metareasoning-review/

I. INTRODUCTION

The introduction frames computation time as a state-dependent trade-off between deeper planning and timely feedback, which fixed execution commitments cannot address. It proposes learning adaptive commitments and situates this approach against chunking, MPC, and metareasoning methods.

I. INTRODUCTION

  • The optimal budget depends on unmodeled dynamics, proximity to state-space constraints, and the marginal optimality gained from further solver iterations.Its consequences also propagate to future states, so the current state alone is insufficient to assess a budget choice.Source: §I · p. 1
  • The paper casts adaptive action chunking as a semi-MDP, making one decision determine both how long the robot executes open-loop and the solver’s computation budget.An RL policy learns this decision from experience.Source: §I · p. 2
  • Across real-world and simulated environments, adaptive budgets match or outperform fixed-budget strategies, and planner statistics are strong policy signals.The policy trades reactivity against deeper solver optimization online.Source: §I · p. 2

Related Work

  • Unlike RTC [12], DEFLECT [13], and adaptive chunk-length methods [14], [15], this approach couples execution length to solver computation budget.The related-work passage notes that fixed chunk lengths do not let longer commitments be leveraged to compute a better plan.Source: Related Work · p. 2
  • Conventional metareasoning treats the world as frozen during deliberation and uses an explicit computation-cost penalty, whereas embodied systems evolve while the planner runs.Prior learned planning budgets [21] are described as restricted to discrete games and requiring a separate reflex policy during open-loop execution.Source: Related Work · p. 2

II. PRELIMINARIES

The control objective is to reach a desired state, but choosing an action-chunk length couples solver computation with how often the robot receives feedback. Longer open-loop commitments risk divergence under disturbances or model mismatch, while fixed horizons impose the same compromise across states.

II. PRELIMINARIES

  • During an h-step chunk, disturbances, stochasticity, or model mismatch can separate the true state from the predicted trajectory and risk constraint violations.The control objective is to drive the system into a desired state.Source: §II · pp. 2–3
  • A static chunk length rigidly trades frequent feedback and disturbance response against the deeper computation enabled by longer commitments.Small h limits solver budget and can yield myopic plans; large h leaves the system open-loop longer.Source: §II · p. 3
  • The paper addresses this trade-off by adapting commitment length h online, making deliberation versus reaction a decision rather than a fixed design constant.Real-time MPC uses h = 1, while action chunking executes a block of h actions.Source: §II · p. 3, Figure 3 · p. 3

III. ADAPTIVE COMPUTATION BUDGETS AS A SEMI-MDP

The paper formulates adaptive chunk duration as a semi-MDP whose decisions determine both open-loop commitment and the next decision time. Its chunk-level objective preserves the underlying step-level control objective, and its action values express the physical value of computation.

III. ADAPTIVE COMPUTATION BUDGETS AS A SEMI-MDP

  • At each decision epoch, the semi-MDP selects a duration h that determines the next decision time and the action chunk executed by the planner.The epoch reward accumulates discounted step-level rewards over the committed chunk.Source: §III · p. 3
  • Under the stated assumptions, maximizing the chunk-level semi-MDP objective directly maximizes the original MDP objective.The equivalence follows because epoch intervals partition environment time and each epoch reward is the exact discounted sum of its step-level rewards.Source: §III · pp. 3–4
  • The physical value of computation is an action value: it represents how much a deliberation budget is worth in the semi-MDP.The optimal budget is defined through relative return gain from the trivial solution h = 1.Source: §III · p. 4

IV. BOUNDED DIVERGENCE UNDER ADAPTIVE COMMITMENT

The section bounds open-loop return deviation under stated smoothness and model-mismatch assumptions. The resulting maximum commitment horizon informs the allowable action-space range for adaptive chunking.

IV. BOUNDED DIVERGENCE UNDER ADAPTIVE COMMITMENT

  • Under Assumption 1, a maximum open-loop commitment horizon exists for any acceptable degradation ϵ ≥ Lρe0, keeping expected cumulative return deviation bounded.The bound depends on initial prediction error and the specified acceptable degradation.Source: §IV · p. 4
  • Expected prediction error propagates recursively as e_j ≤ L_P e_{j−1} + δ, yielding a nondecreasing bound on cumulative return deviation as chunk length grows.The derivation accounts for nonzero initial divergence because the next plan is computed while the previous chunk executes.Source: §IV · p. 4
  • The theorem informs the action-space range: h = 1 is admissible when initial projection error is acceptable, while h_max should not exceed h(ϵ, e0).Source: §IV · p. 4

V. OPTIMIZATION AND TRAINING

The policy uses a richer-than-state context to choose its computation budget, combining the current state, previous budget, executing plan, and solver statistics. These statistics summarize search difficulty and plan safety and reliability; the navigation comparison reports that the learned budget dominates fixed commitments.

V. OPTIMIZATION AND TRAINING

  • The context combines the invocation state, previous budget, open-loop plan, and solver statistics to estimate how much deliberation the situation affords.The plan is the chunk executed while the solver thinks; the statistics summarize the search that produced it.Source: §V · p. 4
  • Solver statistics include rollout concentration, time-to-infeasibility, and mismatch between predicted and realised positions.These summarize search concentration, remaining safety margin, and model reliability, respectively.Source: §V · p. 4
  • The learned budget dominates the fixed commitments in pooled navigation results, where the best fixed commitment differs by metric.The best fixed commitment is h = 5 for success rate and h = 1 for total return.Source: Figure 4 · p. 5
  • The planner-statistics input distinguishes this policy from a state-conditioned schedule by exposing current solver difficulty to the policy.The statistics are observable to the planner but not to the state.Source: §V · pp. 4–5

A. Optimisation

The policy is optimised with PPO across decision epochs using duration-aware advantage estimates. Under exact advantages and exact surrogate maximisation, the stated proposition guarantees monotonic improvement in the original problem objective.

A. Optimisation

  • Each PPO buffer transition represents one decision epoch, with returns accumulated as defined for the semi-MDP.The supplied optimisation description introduces duration-aware GAE for advantage estimation.Source: A. Optimisation · p. 5
  • The GAE bootstrap and eligibility decay use elapsed environment time rather than decision-epoch count.The actor and critic share an encoder over the context, and the actor outputs a categorical distribution over commitment horizons.Source: A. Optimisation · p. 5
  • PPO updates are claimed to monotonically improve the control objective under exact advantages and exact maximisation of the surrogate.The proposition transfers improvement from the commitment-level objective to the underlying problem objective.Source: A. Optimisation · p. 5

VI. EXPERIMENTS

The experiments assess performance, state-dependent adaptation, and boundary behavior across ground navigation, cable-payload quadrotor flight, and contact-rich manipulation. They report success, return, and navigation harshness, using a shared control and compute setup across the evaluations.

VI. EXPERIMENTS

  • Success rate and mean total return are the primary metrics, with higher values indicating better performance.Navigation additionally reports harshness, where lower values indicate a more direct and smoother trajectory.Source: §VI · p. 5
  • The evaluation spans real-world holonomic ground navigation, cable-payload quadrotor flight, and contact-rich planar manipulation.The experiments ask whether adaptive budgets outperform fixed baselines, whether planner statistics support deliberation-versus-reactivity choices, and how the policy behaves when commitment needs vary little.Source: §VI · pp. 5–6
  • The RoboMaster merging-lane policy reduces commitment to h = 1 while passing a crossing obstacle and threading a close static-obstacle passage.The figure compares the learned policy with fixed commitments h ∈ {1, 4, 9} and overlays poses from one episode.Source: Figure 5 · p. 6
  • Experiments standardise the control and communication stack with ROS 2 and run planning and policy inference on one NVIDIA A100 GPU.Control periods vary by task: 0.1 s for navigation, 0.02 s for cable flight, and 0.04 s for PushT.Source: §VI · p. 6

A. Holonomic 2D navigation

In holonomic navigation, the adaptive policy achieves the strongest overall performance across success rate, total return, and harshness, while changing commitment with the demands of the route. Removing planner statistics reduces success, and the reported ablation indicates that state and solver signals are complementary.

A. Holonomic 2D navigation

  • The merging-lane regret plot tracks fixed-commitment regret across decision boundaries, with means and standard deviations over 10 episodes.Its shaded phase ends when the moving obstacle has been passed, and dotted rules indicate mean collision steps.Source: Figure 6 · p. 6
  • In cable-payload flight, the learned policy increases commitment during transit and reduces it near each target.The figure illustrates four stages from one episode.Source: Figure 8 · p. 7
  • The adaptive navigation policy achieves the strongest overall performance across success rate, total return, and harshness.In merging-lane navigation, it begins with a larger budget to identify a safe route, then reduces commitment for feedback while passing obstacles.Source: A. Holonomic 2D navigation · p. 7
  • The close agreement between simulated and RoboMaster navigation behavior indicates a small simulation-to-real gap for this task.The physical evaluation uses the same learned policy and budget-to-planner mapping as simulation, changing the robot, sensing, and execution stack.Source: A. Holonomic 2D navigation · p. 7
  • Removing planner statistics lowers navigation success from 98% to 86%, while planner statistics alone achieve 74% success.The full input achieves the highest success rate and total return, supporting the stated complementarity of state and solver signals.Source: Figure 7 · p. 6, A. Holonomic 2D navigation · p. 7

B. Quadrotor with a suspended cable payload

In cable-suspended-load flight, the learned policy succeeds where every fixed commitment fails, using longer budgets in stable transit and shorter commitments near targets. This two-stage pattern addresses the underactuated system’s open-loop divergence.

B. Quadrotor with a suspended cable payload

  • The policy succeeds in 80% of trials while every fixed commitment fails, and it achieves the highest return among fixed strategies.Success requires reaching and stabilizing at both goals under the stated settling criteria.Source: B. Quadrotor with a suspended cable… · p. 7
  • The policy uses larger budgets during stable transit and shorter commitments near targets, where the system must remain reactive.The cable-suspended load’s stiff, underactuated dynamics make open-loop staleness cause system divergence.Source: B. Quadrotor with a suspended cable… · p. 7
  • Planner statistics help the policy achieve its best performance in this task.Source: B. Quadrotor with a suspended cable… · p. 7

C. Contact-rich planar manipulation

In contact-rich planar Push-T manipulation, the policy settles on a constant budget rather than adapting. Under the nominal setting, it matches the best fixed baseline.

C. Contact-rich planar manipulation

VII. CONCLUSION

The paper introduces a semi-MDP framework for learning adaptive computation budgets in real-time robot control. Across three environments, adaptation responds to task demands and is beneficial when exploitable budget variation exists.

VII. CONCLUSION

  • The paper introduces a semi-MDP framework that learns adaptive computation budgets for real-time control using planner statistics and balancing plan execution length with computation time.Source: §VII · p. 8
  • Across three environments, the policy adapts budgets to task demands, using longer commitments for global routes and smooth transit and shorter ones near obstacles, critical passages, targets, and disturbances.Source: §VII · p. 8
  • In contact-rich planar manipulation, the policy converges to a constant budget matching the best fixed baseline when the nominal task offers little exploitable variation.Future work will explore richer planner statistics, learned world models, and more expressive planning algorithms.Source: §VII · p. 8

Related papers

Loading 2610.23662v1…