PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors

Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros, Manikatan Nambi, Lillian J. Ratliff, Dylan Randle

arXiv:2610.22852v1cs.ROcs.AI

TL;DR

Adapting frozen flow-matching policies through unstructured latent spaces makes exploration difficult, while deployment-specific reliability and throughput remain challenging. AnchorQ maps reference behaviors into state-conditioned latent anchors and refines them with critic-guided particles. It outperforms post-training baselines across simulated and real-world manipulation tasks, including 97% higher simulated task throughput than the closest baseline.

  • Problem

    Frozen generalist policies can remain imperfect under deployment changes, and unstructured continuous action or latent spaces make useful behaviors difficult to discover efficiently.

    Source: §1 · pp. 1–2

  • Method

    AnchorQ flow-reverses coarse reference behaviors into state-conditioned latent anchors, then uses critic-guided local particle refinement and selection to adapt a frozen policy.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    97% higher simulated task throughput than the closest baseline; AnchorQ also outperforms the frozen base policy and RL baseline on three real-robot tasks under the same interaction budget.

    Source: §1 · p. 3, §4.4 · p. 10

  • Takeaways & Limitations

    The findings highlight action representation as a key design choice for efficient adaptation across simulation and real-world manipulation benchmarks.

    Source: §5 · p. 10

    AnchorQ requires additional memory to score particles at every decision; distilling the adapted behavior could reduce its sampling-based inference needs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website https://anchorq-rl.com/.

1 Introduction

Pretrained VLA policies offer broad capabilities, but deployment-specific reliability and throughput remain challenging, while collecting demonstrations and fine-tuning can be costly. AnchorQ addresses inefficient exploration in unstructured action spaces with structured, refinable latent actions, and reports gains over steering baselines in simulation and on real robots.

1 Introduction

  • Pretrained VLAs can transfer across objects, scenes, and tasks, but changes in geometry, clutter, object placement, and execution can leave them imperfect for deployment.The passage notes that demonstrations covering these variations are costly and that fine-tuning can degrade useful pretrained behaviors.Source: §1 · p. 1
  • Unstructured continuous action or latent spaces make it difficult to discover useful behaviors, and DSRL and action-space residual RL fail on challenging tasks within the interaction budget.The section motivates an online representation that is compact when coarse behaviors suffice but refinable when richer behavior is needed.Source: §1 · p. 2
  • AnchorQ maps semantically meaningful reference actions to state-conditioned noise anchors, samples local particles, and uses critic-guided value tilting and selection to refine actions through a frozen policy.The critic is trained online with temporal-difference learning.Source: §1 · p. 2, Figure 1 · p. 2
  • The analysis says anchors reduce identification complexity, particle refinement trades support bias against local action-class complexity, and value tilting amplifies high-value refinements under critic error.Epistemic uncertainty adaptation calibrates exploitation to critic confidence.Source: §1 · p. 2
  • AnchorQ delivers 97% higher simulated task throughput than the closest baseline and outperforms the frozen base policy and RL baseline on three real-robot tasks under the same interaction budget.The real-robot tasks are described as challenging manipulation tasks.Source: §1 · p. 3

2 Related Work

Prior work adapts frozen generative policies through latent-space steering, action reranking, or reduced action spaces. AnchorQ differs by using flow inversion to map coarse reference behaviors into state-conditioned latent anchors, then refining them locally.

2 Related Work

  • Latent-space methods include reranking or steering sampled action chunks, learning over the full continuous noise space, and flow-inversion approaches paired with continuous latent actors or human corrections.The cited methods include DSRL, FRS, and UniSteer.Source: §2 · p. 3
  • AnchorQ combines flow inversion with local continuous particles around a fixed library of coarse reference behaviors, which a critic ranks.This contrasts with methods that learn an actor over the continuous latent space.Source: §2 · p. 3
  • Action-space reduction methods use discretization, learned primitives, or hierarchical refinement, typically constructing abstractions directly in action coordinates.AnchorQ instead maps reference actions through flow inversion and refines the resulting latents with sampled continuous perturbations.Source: §2 · p. 3

3 AnchorQ: Value Learning over Online Flow-Reversed Latents

AnchorQ treats latent noise as the action for a frozen flow-matching policy and builds state-dependent anchors by reversing the policy from coarse reference behaviors. It refines anchors with critic-weighted particles, selects among them online, and analyzes the tradeoffs in sample complexity, support bias, and critic reliability.

3.1 Online flow-reversed anchors

  • Flow reversal maps a fixed library of coarse reference action chunks into state-conditioned noise anchors, which AnchorQ reconstructs at each decision for critic evaluation.With a fixed ODE solver, the frozen policy deterministically maps latent noise to an action chunk, allowing the latent to serve as the RL action.Source: §3.1 · p. 3

3.2 Local particles and value-tilted refinement

  • Local particles perturb each noise anchor, and exponential value tilting weights them toward higher-critic-value candidates to form a refined latent within the particle cloud’s convex hull.The perturbation scale controls exploration reach, while the tilting strength controls preference for high-value particles.Source: §3.2 · p. 4

3.3 Action selection and critic training

  • AnchorQ selects among refined latents with epsilon-greedy exploration, executes the selected latent through the frozen policy, and trains its critic online with temporal-difference updates.It reuses cached anchors during critic updates and does not require a separately trained latent actor.Source: §3.3 · p. 4

3.4 Online noise anchors improve sample efficiency

  • When the best noise anchor is near-optimal, restricting search to C anchors can identify an epsilon-optimal action with probability at least 1 − alpha using O(C) episodes.The argument contrasts the restricted search with the cost of ruling out outside latent modes.Source: §3.4 · p. 5
  • Increasing the local search radius reduces support bias but enlarges the action class, trading representational coverage against the complexity of learning a selector.The value bound includes support gap, representation error, and selection error, divided by 1 − gamma.Source: §3.4 · pp. 4, 6
  • The value-tilting weights are an entropic, upper-tail-seeking reweighting of particles around each anchor.The analysis characterizes this as a risk-sensitive local refinement.Source: §3.4 · pp. 5–6
  • Value tilting increases weight on a truly high-value particle subset when its value margin exceeds twice the critic error, with amplification growing exponentially in the tilting strength and margin.Under local smoothness, the refined latent’s loss relative to the best local particle is controlled by critic error, finite-temperature weighting, and weighted particle variance.Source: §3.4 · p. 6
  • Because aggressive tilting can amplify critic errors, AnchorQ uses ensemble disagreement to reduce tilting when epistemic uncertainty is high and sharpen it as critic confidence increases.This uncertainty-based adaptation calibrates exploitation to critic confidence.Source: §3.4 · p. 6

4 Experiment Results

The experiments evaluate AnchorQ on held-out simulated manipulation suites and real-robot tasks, examining benchmark performance, ablations, particle tilting, and real-world steering.

4.1 Tasks & Setup

  • Evaluation spans held-out 15-task and hard-10 LIBERO-90 splits, six bimanual RoboTwin tasks, and three household robot tasks, with shared frozen-policy baselines.Simulation results average four seeds at 200K and 500K environment interactions.Source: §4.1 · pp. 6–7

4.2 Simulation Performance and Ablation Studies

  • AnchorQ averages 294 completions/hr across simulation tasks versus DSRL’s 149, a 97% throughput increase, and achieves at least 2× DSRL’s success rate on LIBERO-90 at matched budgets.AnchorQ also achieves the highest success rate across the three simulated suites.Source: Table 1 · p. 7, Figure 2 · p. 7, §4.2 · p. 8
  • Adaptive β adds 8.6 success-rate points, fixed β = 0.1 adds 4.0 over no tilting, and adaptive σ contributes a further 1.3 points.Across the tested ablations, performance still exceeds DSRL.Source: §4.2 · p. 8, Table 2 · p. 8, Figure 3 · p. 8

4.3 Analysis on anchors and particle tilting.

  • On LIBERO task 81, AnchorQ reaches 84% success, versus 22% for anchor-only and 0% for DSRL, while exploring beyond discrete anchors within their noise band.The analysis associates this finer steering with particle tilting.Source: §4.3 · p. 9, Figure 4 · p. 9

4.4 Real World Steering Experiments

  • With 100 online episodes per task, AnchorQ achieves higher success rates than frozen π0.5 and DSRL across three real-robot tasks.It is the only method to improve over the base policy on towel hanging.Source: Figure 5 · p. 10, §4.4 · p. 10

5 Conclusion

AnchorQ adapts frozen flow-matching policies by organizing latent-space exploration around state-conditioned reference anchors and local refinements. The paper positions this design against policy-updating and other steering approaches, and describes an online procedure that caches anchor banks and uses critic-guided selection and refinement.

5 Conclusion

  • Across simulated and real-world manipulation tasks, AnchorQ improves sample efficiency over post-training baselines, and action representation is a key design choice for efficient adaptation of frozen policies.The conclusion describes anchors, local particle tilting, and critic-guided selection as the method’s coarse-to-fine refinement approach.Source: §5 · p. 10

A Extended Related Work

  • Unlike policy fine-tuning methods, AnchorQ keeps the pretrained generator frozen and adapts behavior by selecting and refining its input latent.Related approaches update or distill the generator, while PLD uses a frozen VLA during online RL but later transfers improvements into its parameters.Source: A Extended Related Work · p. 17
  • AnchorQ uses inversion to define persistent, state-conditioned semantic coordinates for online learning, rather than using inverted motions only as data or guidance for a continuous noise actor.UniSteer uses online human corrections and trains a continuous noise actor; AnchorQ uses a fixed reference library and learns a critic over anchors and local neighborhoods.Source: A Extended Related Work · pp. 17–18
  • Unlike RLMM-Flow’s offline hierarchy over latent dimensionality, AnchorQ starts from inverted reference motions and performs online, Q-weighted local particle refinement.The related-work discussion distinguishes this hierarchy from action-coordinate discretization and interval-refinement approaches.Source: A Extended Related Work · p. 18
  • Uncertainty sets the strength of KL-regularized value tilting: high critic uncertainty softens particle weights, while low uncertainty permits sharper preference for high-scoring particles.The stated reliability guarantee requires local critic error to be controlled by the uncertainty scale; ensemble disagreement alone does not establish that calibration condition.Source: A Extended Related Work · p. 18, §B.1 · pp. 20–21

B.1 AnchorQ Pseudocode

  • The online procedure caches each observation’s anchor bank, refines candidates with fresh local clouds, and selects a refined latent using the critic ensemble mean.The selected latent is decoded by the frozen VLA; replay stores the executed latent, and cached banks can be reused during critic updates.Source: §B.1 · pp. 18, 19, 21

B.2 Implementation Details and Hyperparameters

The implementation uses a fixed set of reference motions, benchmark-specific critic designs, and different critic update choices for simulation and real-world training.

B.2 Implementation Details and Hyperparameters

  • Six cardinal Cartesian motions plus a defer candidate define the reference set; bimanual RoboTwin tasks use 36 motions plus defer, for 37 candidates.Simulation uses REDQ’s ensemble update scheme, while real-world training uses a three-step TD target and averages sampled target heads.Source: §B.2 · p. 21
  • For LIBERO-90, the critic combines frozen visual embeddings and proprioception with a state encoder and an encoder for the flattened 10 × 32 noise trajectory.The visual input comprises two 384-dimensional DINOv2-S embeddings; the proprioceptive vector is 8-dimensional.Source: §B.2 · p. 21
  • The simulation critic uses ten heads with independent trainable encoders and frozen random prior networks, whose disagreement supplies the local-adaptation uncertainty signal.The DROID critic instead uses shared trainable encoders with independent scalar output heads and no random priors.Source: §B.2 · p. 21

C Additional Experiment Results

Additional experiments report per-task results across simulation suites and describe where particle tilting separates from anchor-only learning and baseline RL.

C Additional Experiment Results

  • The additional results cover per-task analyses across simulation benchmarks and include a visual overview of LIBERO, RoboTwin, and real-world robot tasks.The benchmark hyperparameter table covers LIBERO-90, RoboTwin, and real-world experiments.Source: C Additional Experiment Results · p. 21, Table 3 · p. 22

C.1 Per-Task Results

  • On LIBERO-90, adaptive variants learn reliably across tasks, while fixed and no-tilting variants are less consistent; on hard-10, adaptive tilting succeeds on most tasks, though the hardest cases challenge every method.On RoboTwin, at least one particle-tilting variant improves over the base policy on every task, with particularly clear gains on handover and object placement.Source: §C.1 · p. 22

C.2 Additional LIBERO-90 Results

The additional LIBERO-90 analysis focuses on four tasks where particle tilting, anchor-only learning, and steering baselines differ most. These tasks require precise placement motions, and adaptive tilting variants perform consistently across them.

C.2 Additional LIBERO-90 Results

  • On four highlighted tasks, adaptive tilting variants perform consistently, while anchor-only learning is task-dependent and DSRL and Residual RL make little progress.The base policy performs poorly on all four tasks, which require precise end-effector motions to align objects with target placements.Source: §C.2 · p. 22
  • The four tasks include placing a frying pan on a cabinet and placing a book into different compartments of a caddy.The text identifies these as tasks 41, 73, 81, and 75.Source: §C.2 · p. 22
  • On task 81, the anchor-only rollout approaches the caddy but leaves the book outside, whereas adaptive tilting adjusts the approach and inserts it into the front compartment.Source: §C.2 · p. 22

D Overview of Supplemental Theoretical Results

The supplemental theory analyzes AnchorQ’s coarse-to-fine action hierarchy and its tradeoffs, alongside technical bounds on binary relative entropy. It establishes a boundary-point minimum and a separate estimate for small ϵ.

D Overview of Supplemental Theoretical Results

  • Restricting online learning to sufficiently accurate flow-reversed anchors can reduce identification samples, while full-class learners must rule out excluded modes even with exact anchor-mode reward laws.The comparison is isolated in a stylized one-decision latent-mode problem with C anchor modes among N modes.Source: D Overview of Supplemental Theoretical… · pp. 23–24
  • The anchor restriction has a support price: absent useful behaviors cannot be recovered with more critic data, while the resulting value loss is governed by local support deficiency.The theory separates this persistent structural support price from statistical selection error within the flow-reversed class.Source: D Overview of Supplemental Theoretical… · pp. 24–25
  • Expanding local particle reach can reduce support gaps but increases the behavioral action class and its sample burden, trading structural bias against learning complexity.The analyzed deterministic reach is a local envelope, not the implementation’s Gaussian particle scale; a separate result connects finite Gaussian clouds to envelope coverage.Source: D Overview of Supplemental Theoretical… · p. 25
  • Exponential tilting uniquely solves a KL-regularized value-improvement problem, increasing expected particle score and KL departure with tilt intensity while preserving candidate support.Finite-cloud guarantees also account for critic error and barycentric interpolation, which bounds the loss from executing the weighted-average latent instead of sampling a weighted particle.Source: D Overview of Supplemental Theoretical… · p. 26
  • AnchorQ’s sequential-control loss decomposes into support-gap, anchor-selection, particle-representation, finite-temperature, critic-error, and barycentric-interpolation terms.The decomposition separates whether good actions are represented from whether learning identifies and reliably exploits them.Source: D Overview of Supplemental Theoretical… · pp. 26–27
  • Uncertainty-adaptive tilting penalizes departure from the flow-reversed reference according to epistemic uncertainty, with calibrated critic error yielding weighting error O(uepi(s)).When the critic is poorly known, weighting remains soft; as uncertainty contracts, finite-temperature and critic-estimation terms contract together.Source: D Overview of Supplemental Theoretical… · pp. 27–28

E Online Flow-Reversal Abstraction and Latent Search Cost

  • The flow-reversal cost analysis uses a one-decision episodic latent MDP equivalent to fixed-confidence Bernoulli best-arm identification, with C anchor modes among N latent modes.The setup separates identifying a useful latent mode from state-distribution shift, temporal credit assignment, and function approximation.Source: E Online Flow-Reversal Abstraction and… · pp. 28–29

E.1 Technical lemmas

  • The technical results provide an adaptive change-of-measure lemma for learners with shared side information and randomized outputs, alongside binary relative-entropy bounds.The lemma relates transcript divergence to action-wise observation divergences and lower-bounds divergence for transcript events.Source: §E.1 · pp. 30–31
  • For p ≥ 1 − α and q ≤ α, the binary relative-entropy minimum occurs at (p, q) = (1 − α, α), with endpoint cases handled by limits.The proof uses the derivative signs on p > q; when q = 0 < p, the left-hand side is +∞.Source: §E.1 · p. 31
  • For ϵ ∈ (0, 1/8], the proof of estimate (7) sets x := 4ϵ and uses x² ≤ 1/4.It also uses the identities 1 − (1/2 − 2ϵ) = 1/2 + 2ϵ and 1 − (1/2 + 2ϵ) = 1/2 − 2ϵ.Source: §E.1 · p. 31

E.2 Main result on online flow-reversal abstraction sample complexity

The section establishes a sample-complexity separation between learning within the flow-reversal action abstraction and retaining the full latent class, then sets up the discounted-MDP framework for analyzing the restriction’s control cost.

E.2 Main result on online flow-reversal abstraction sample complexity

  • On a zero-bias base instance, each non-flow-reversed mode must be sampled at least kl(1 −α, α) / kl(1/2 −2ϵ, 1/2 + 2ϵ) times in expectation by a full-class learner.The lower bound applies even when the learner receives exact flow-reversed-mode laws before online interaction.Source: §E.2 · pp. 32, 34
  • A learner restricted to the flow-reversed modes can be (ϵ, α)-correct on the bounded-bias class using Cm online episodes.The base instance has zero control bias, while the full-class lower bound arises from ruling out alternatives with a superior non-flow-reversed mode.Source: §E.2 · pp. 32–33
  • The lower bound also applies to full-class learning initialized with prefill generated only from flow-reversed modes.The result is worst-case for latent modes without assumed smoothness or generalization; it does not imply every practical DSRL run must test N −C additional modes.Source: §E.2 · p. 34

F The general-MDP price of the persistent FRS abstraction

  • The general-MDP analysis separates an irreducible support error from a reducible selection error within the flow-reversal abstraction.The section formulates a discounted latent-action MDP and defines restricted Bellman operators with unique bounded fixed points and a deterministic stationary flow-reversal selector.Source: F The general-MDP price of the… · p. 34, §F.1 · p. 36

F.2 Main Result: Support gap and selection error

This section characterizes the value cost of persistent flow-reversal restriction by separating support gaps from selection errors. It bounds the loss of a learned selector in terms of both quantities.

F.2 Main Result: Support gap and selection error

  • The analysis separates support error from finite-data selection error within the flow-reversal abstraction.The support gap constrains the current latent to the flow-reversed bank, while selection error concerns choosing within that abstraction.Source: §F.2 · p. 37
  • For an arbitrary stationary flow-reversal selector, the value loss admits an exact decomposition into support and selection components.The theorem also gives bounds on the resulting loss for every state.Source: §F.2 · p. 37
  • With probability at least 1 −α, the learned selector’s value loss is bounded above by (¯∆F + ϵsel(n, α))/(1 −γ) and below by ∆F(s).The bound combines the support gap with the assumed finite-sample selection-error bound.Source: §F.2 · p. 38

F.3 From critic accuracy to selection error

This section analyzes how critic accuracy and action support shape selection error, distinguishing support bias from reducible learning error and framing local particle refinement as a bias–complexity tradeoff. It also characterizes reach-bounded control through a unique bounded fixed point and a support gap that decreases as local reach increases.

F.3 From critic accuracy to selection error

  • Selection error is bounded by twice critic error plus failure to maximize the critic, separating structural support error, statistical critic error, and candidate-selection error.More data cannot repair missing behavior in the flow-reversed bank, whereas improved critic learning can reduce statistical error; exact greedy selection sets ϵopt=0.Source: §F.3 · pp. 39–41

F.4 Sharpness and regime consequences

  • For every γ∈[0,1), support-error and selection-error losses can attain bounds scaling as 1/(1−γ), while one-step support gap can equal full control loss in a one-decision MDP.The bounds also yield bias regimes: if the initial support gap exceeds ϵctrl, no bank-restricted policy can meet that target; sufficiently small support bias plus selection error permits it with probability at least 1−α.Source: §F.4 · pp. 41–42

F.5 Interpretation and novelty

  • The support price is irreducible when useful behavior is absent from the bank, while critic estimation and candidate optimization contribute to reducible selection error.Support gaps matter in proportion to where the full-Q⋆-greedy flow-reversal selector visits, and the theorem gives a sharp bridge from local support gap to persistent restriction loss.Source: §F.5 · pp. 43–44

G AnchorQ traces a bias–complexity frontier

  • Increasing local reach ρ monotonically reduces one-step support gap while increasing behaviorally distinguishable candidates at any fixed resolution, placing particle AnchorQ on a bias–complexity frontier.The analysis uses deterministic reach ρ, which is distinct from the implementation’s Gaussian particle scale σp; a finite-sample bound includes both quantities.Source: G AnchorQ traces a bias–complexity… · p. 44

G.1 Continuous local action abstractions

  • For each state, the reach-ρ flow-reversed action set is a finite union of compact balls, so continuity ensures the reach-ρ Bellman maximum is attained.The section defines stationary reach-ρ policies as supported on that state-dependent set.Source: §G.1 · p. 45
  • The reach-ρ Bellman operator is a monotone γ-contraction with a unique bounded fixed point, which upper-bounds every stationary reach-ρ policy’s value.Source: §G.1 · p. 45
  • At zero reach, the admissible latent actions are exactly the command-conditioned anchors, and the support gap is nonnegative.Source: §G.1 · p. 46
  • Increasing local reach expands the admissible action set and weakly reduces the uniform support gap, which vanishes as reach grows without bound.Source: §G.1 · p. 46
  • The best restricted Bellman value equals the unrestricted optimum minus the statewise reach-ρ support gap, Δρ(s).Source: §G.1 · p. 47

G.2 Selection error within the reach-ρ abstraction

This section analyzes policy-value loss within the reach-ρ abstraction by separating support and selection error, bounding support bias under smooth local geometry, and characterizing candidate complexity. It also quantifies finite-discretization and finite-net selection terms, including a simultaneous high-probability guarantee across states for a learned selector with bounded selection error.

G.2 Selection error within the reach-ρ abstraction

  • The exact reach-ρ decomposition separates policy loss into support error and nonnegative selection error.At zero reach, the decomposition recovers the support–selection decomposition of Theorem 5.Source: §G.2 · pp. 48–49

G.3 A geometric rate for the support gap

  • Under smooth local optimal-value geometry, support bias is at most L/2 times the squared excess of the nearest-anchor distance over ρ, and vanishes when ρ reaches that distance.The distance is from a full-MDP optimal latent to its nearest flow-reversed anchor; the smoothness assumption applies along the segment between them.Source: §G.3 · pp. 49–50

G.4 Behaviorally distinguishable local candidates

  • The Bellman behavioral pseudometric measures whether latents differ in reward and transition consequences, and reach-ρ behavioral complexity is nondecreasing but may stay constant when added latents are redundant.Under behavioral Lipschitzness, complexity has a polynomial upper bound; local behavioral richness gives a lower bound when ℓBρ > εB.Source: §G.4 · pp. 50–52
  • Under behavioral Lipschitzness, finite behavioral nets exist, while local behavioral richness imposes a minimum net size for the reach-ρ action class.Packing and net sizes are related by a packing–net sandwich.Source: §G.4 · pp. 52–53

G.5 Finite behavioral discretization

  • Finite behavioral discretization adds at most εB one-step error.The finite net is chosen within the reach-ρ action class.Source: §G.5 · p. 53
  • The reach-ρ finite action set incurs value loss at most (Δ̄_ρ + ε_B)/(1 − γ) relative to the optimal value.The finite behavioral net preserves the best Q⋆ value within ε_B of the best value over Z_ρ(s).Source: §G.5 · pp. 53–54
  • The finite-net decomposition bounds every state's policy-value loss by (Δ̄_ρ + ε_B + ē_{ρ,ε_B}(ω))/(1 − γ), separating support, discretization, and selection errors.For a learned selector b_{ω_n}, the bound becomes (Δ̄_ρ + ε_B + ε_sel(n, α))/(1 − γ) simultaneously for every state with probability at least 1 − α.Source: §G.5 · p. 54

G.6 A finite-sample frontier under a generative model

The finite-sample analysis bounds learning over fixed behavioral nets under a generative model and characterizes how local reach trades support bias against candidate complexity. It also relates Gaussian particle coverage to behavioral coverage and representation error, while limiting those coverage guarantees to a fixed state unless confidence is adjusted across states.

G.6 A finite-sample frontier under a generative model

  • The finite-sample selection analysis specializes to synchronous empirical Q-iteration on fixed behavioral nets, with fresh generative-model samples at each iteration.The analysis makes dependence on the number of behaviorally distinguishable local candidates explicit.Source: §G.6 · p. 55
  • Increasing reach weakens the support-bias term but expands behavioral packing complexity, making the finite-sample upper bound nondecreasing in reach.For a prescribed support-bias budget, the smallest feasible reach minimizes behavioral packing complexity among larger feasible reaches.Source: §G.6 · pp. 55, 58
  • A target-accuracy reach choice must jointly bound support bias, behavioral resolution, and value-estimation error relative to the desired control tolerance.Among reaches satisfying the support-bias requirement, a smaller reach has no larger behavioral packing complexity.Source: §G.6 · p. 58

G.7 Relation to a finite Gaussian particle cloud

  • With sufficient particles, Gaussian clouds cover the bounded local latent envelope at a Euclidean resolution and, under behavioral Lipschitzness, at a corresponding behavioral resolution.The coverage probability applies simultaneously across commands for the fixed state considered.Source: §G.7 · pp. 59–60
  • Under the stated regularity conditions, particle coverage bounds representation error by the behavioral resolution, but the guarantee is for one fixed state.A simultaneous guarantee over states requires an additional union bound, such as replacing α with α/S; the reach radius is not the Gaussian scale.Source: §G.7 · pp. 60–61

G.8 Interpretation, novelty, and connection to the preceding results

The reach-based analysis connects exact-anchor search to broader local latent control by balancing support bias against behavioral complexity. The following section examines exponential value tilting as a local particle reweighting operation, with positive tilt favoring upper-tail scores around a selected anchor.

G.8 Interpretation, novelty, and connection to the preceding results

  • The reach parameter traces a bias–complexity frontier: larger local search can reduce support bias but increases the behavioral action class that must be learned.The resulting finite-sample bound separates support bias, behavior discretization, and value-selection error.Source: §G.8 · p. 61
  • The analysis organizes standard contraction, packing, concentration, and approximate-value-iteration arguments around a state-dependent local flow-reversed action family.It measures complexity by behaviorally distinguishable candidates and gives a monotone path from exact anchors toward broader latent control.Source: §G.8 · p. 61
  • The Gaussian coverage results connect finite particle clouds to representation error, but do not alone establish sample complexity for the implemented hierarchical policy.A separate policy bound accounts for anchor selection, particle weighting, critic error, and barycentric refinement.Source: §G.8 · p. 61

H Entropic weighting for local particle refinement

  • After a coarse anchor is selected, exponential tilting reweights only its local particle cloud, and the weights construct a barycentric latent refinement.Reach determines which local behaviors are available; value tilting reallocates probability among those available behaviors.Source: H Entropic weighting for local… · p. 62, §G.8 · p. 62

H.1 The local particle law and its entropic tilt

  • The entropic certainty equivalent evaluates local particle scores, while positive tilt increasingly emphasizes their upper tail.The weighting law is auxiliary: its particle weights shape the executed refinement rather than sampling one particle for execution.Source: §H.1 · p. 63
  • The exponential weights are the variationally optimal change of measure, favoring high-value local deviations while a KL penalty limits departure from the original particle law.The score is a local candidate score, not the environment return.Source: §H.1 · p. 63

H.2 The exact exponential-tilting theorem

This section analyzes when critic-based exponential weighting reliably selects high-value local particles, including the distinction between weighted-score guarantees and guarantees for executed barycentric refinements. It also characterizes hierarchical AnchorQ’s performance loss through support, representation, refinement, and selection errors, with guarantees tied to critic accuracy and local-candidate quality.

H.2 The exact exponential-tilting theorem

  • For each β > 0, exponential tilting uniquely optimizes the KL-regularized score objective while preserving the local proposal’s support.The weighting law is mutually absolutely continuous with the local proposal law.Source: §H.2 · pp. 64, 66
  • As β increases, the tilted score and KL path rise, and any region with a positive score margin gains weight exponentially relative to its complement.The score, KL, and certainty-equivalent paths are strictly increasing when the score is nonconstant almost everywhere under the local proposal.Source: §H.2 · pp. 64–65
  • Learned-score weighting remains reliable when its error is bounded, and strict high-value amplification is guaranteed when population-score separation exceeds 2ϵcrit.The learned weighting law is 2ϵcrit-optimal for the population KL-regularized objective.Source: §H.2 · p. 65

H.3 Risk-sensitive local weighting and the correct random variable

  • For small positive β, local risk-sensitive weighting rewards variance in candidate scores in addition to their mean, favoring upper-tail value opportunities.The variance is across local candidate behaviors, not uncertainty across critic heads.Source: §H.3 · p. 68
  • Using the conditional entropic return score makes the local entropic functional equal the realized-return certainty equivalent, but this identity applies to the weighting law, not AnchorQ’s executed barycentric latent.Relating the executed barycenter’s value to weighted candidate values requires an additional local regularity condition.Source: §H.3 · pp. 69–70
  • Local candidate-score risk is not generally realized-return risk: applying the outer entropic transform to conditional mean returns differs from transforming returns themselves.The two coincide only when conditional returns are almost surely constant for almost every local candidate.Source: §H.3 · p. 70
  • Conditional Jensen’s equality holds exactly when the relevant random variable Gs,c is conditionally almost surely constant given Zs,c.Source: §H.3 · p. 70

H.4 Sharpness of the critic-separation condition

  • If the population-score separation is below 2ϵcrit, a worst-case critic can reverse particle ordering; at equality it can eliminate strict preference, making the threshold necessary for a uniform strict-amplification guarantee.This worst-case result concerns the weighting law, not every instance; guarantees for the executed barycenter additionally require local regularity.Source: §H.4 · p. 71

H.5 True-score improvement under a learned local weighting

  • Under the stated condition, learned local weighting has strictly larger average population score than the untilted local law, but this does not alone guarantee improvement in the executed barycentric latent.Transferring the weighted-score guarantee to the barycenter requires local smoothness and finite-cloud conditions.Source: §H.5 · p. 72

H.6 Finite realized local particle clouds

  • For a finite realized cloud, exponential tilting yields positive normalized entropic weights, while the executed barycenter’s gap to the best particle is bounded using temperature, critic error, and local smoothness.Relative to the best action in the ideal reach-ρ neighborhood, the executed-refinement gap decomposes into finite-particle representation, critic, finite-temperature weighting, and barycentric interpolation errors.Source: §H.6 · pp. 73, 75, 76

H.7 Sub-optimality bound for hierarchical AnchorQ

  • The hierarchical policy forms one value-tilted barycentric candidate per anchor’s local particle cloud, then uses the critic to select a candidate for execution through the frozen VLA.The analysis distinguishes refined-candidate selection error and critic error at candidates from particle-level errors, since particle accuracy alone does not ensure barycenter accuracy.Source: §H.7 · pp. 76–77
  • Theorem 8 bounds policy loss through support gap, representation error, log(M + 1)/β, particle-level critic error, within-cloud variance, and refined-candidate selection error.The selection error is at most 2 times the candidate-level critic error.Source: §H.7 · pp. 78, 80
  • Refining every anchor before selection avoids an extra loss term for committing to one anchor, while particle clouds incur representation error over the union of anchor neighborhoods.The support gap measures the best reachable latent’s one-step gap from the unrestricted optimum; the representation term measures how well the finite clouds represent that neighborhood union.Source: §H.7 · p. 80
  • The guarantee is conditional on critic accuracy relative to Q⋆: particle- and candidate-level errors are distinct, may include estimation or bootstrap bias, and need not vanish with Bellman-target fit.The selection error vanishes when the selector chooses a truly highest-valued refined candidate.Source: §H.7 · p. 80
  • Increasing reach ρ can reduce support gap but enlarges the set particles must represent; adding particles cannot increase representation error, while larger β lowers the explicit soft-weighting term.These are representation-term and bound tradeoffs, not guarantees of monotonic improvement in the full bound or executed policy; refinement also requires accurate local value rankings.Source: §H.7 · p. 80
  • The decomposition also permits a shared adaptive tilt β(s, b) and ϵ-greedy execution, with the latter’s exploration loss represented separately from critic error.Adaptive tilting retains both critic-error terms, while ϵ-greedy execution averages the selection error over selection randomness and particle clouds.Source: §H.7 · p. 80

H.8 Connection to Theorems 1–3

AnchorQ’s coarse-to-fine analysis separates the limits of reachable behavior from the errors of local refinement and value-based selection. It further explains how uncertainty-adaptive tilting regulates that selection as critic reliability changes.

H.8 Connection to Theorems 1–3

  • Anchor selection remains a coarse choice among FRS anchors; particles restore expressivity only around the selected anchor, so local weighting cannot reach better behavior outside those regions.The local-refinement guarantee depends on the critic distinguishing useful refinements from inferior ones.Source: §H.8 · pp. 80–81
  • For a fixed local particle cloud, increasing tilt intensity raises expected particle score and KL departure from the reference law without expanding its support.Reach controls which local behavior is accessible; tilt controls how strongly particle values affect refinement.Source: §H.8 · p. 81
  • Theorem 8 separates sequential-control error into support bias, anchor selection, finite-particle representation, critic, finite-temperature, and barycentric interpolation terms.Greater reach can reduce support bias but makes the neighborhoods harder to represent; stronger tilting reduces the explicit temperature term only when critic errors do not undermine weighting.Source: §H.8 · pp. 81–82

H.9 Interpretation and novelty

  • AnchorQ’s novelty lies in applying KL-regularized entropic weighting to local refinements and connecting those weights to the barycentric latent executed by the robot.Particle parameters determine represented refinements, whereas β controls how aggressively their values influence the result.Source: §H.9 · p. 82
  • Positive β seeks the upper tail of local particle values, not coarse commands, critic-head uncertainty, or realized environment returns.The return interpretation applies only under the conditional transformation stated in Proposition 10.Source: §H.9 · p. 82
  • A local value margin greater than 2ϵcrit is required for the stated worst-case-sharp guarantee that tilting amplifies a high-value particle subset.Aggressive tilting is theoretically justified only when the critic reliably resolves the relevant local differences.Source: §H.9 · p. 82
  • Local smoothness bounds the gap between high weighted particle value and the value of the executed barycentric latent by an interpolation penalty tied to weighted particle variance.The theory distinguishes the quality of the weighting rule from the geometry of latent interpolation.Source: §H.9 · p. 82

I Uncertainty Adaptation: Epistemically adaptive local tilting

  • Uncertainty adaptation weakens tilting when epistemic uncertainty is large and strengthens it as uncertainty contracts, while critic error sensitivity under fixed tilting scales with βϵcrit.The monotonicity claim during learning is conditional on a fixed local score landscape; observed statistics need not be monotone as scores and particle clouds change.Source: I Uncertainty Adaptation:… · p. 83

I.1 Calibration of the epistemic scale

The analysis makes uncertainty adaptation’s guarantees conditional on ensemble-derived uncertainty tracking critic error. Under that calibration, inverse-uncertainty tilting yields scale-free sensitivity, finite-cloud bounds, and a conditional conservative-to-aggressive refinement schedule.

I.1 Calibration of the epistemic scale

  • Ensemble disagreement is a practical uncertainty surrogate, but adaptive guarantees require it to track critic error up to a state-independent factor.The paper explicitly notes that ensemble disagreement is not automatically a calibrated confidence radius for deep Q-learning.Source: §I.1 · p. 83
  • With calibrated uncertainty and β(s) = 1/uepi(s), adaptive tilting makes weighting-law sensitivity to critic error scale-free.The guarantee assumes the ensemble-derived scale tracks critic error up to a state-independent multiplicative constant.Source: §I.1 · pp. 83–85
  • The learned adaptive weighting law uniquely optimizes an uncertainty-weighted variational objective, with epistemic uncertainty penalizing departure from the untilted FRS particle law.This makes uncertainty the coefficient governing the KL departure penalty.Source: §I.1 · p. 84

I.2 Finite-cloud implication

  • For fixed M and calibration constant, adaptive finite-cloud weighting error is O(uepi(s)); the executed barycentric refinement has an additional interpolation term.The interpolation contribution depends on local smoothness and weighted particle variance.Source: §I.1 · p. 85, §I.2 · p. 86

I.3 Automatic annealing with learning

  • For a fixed local score landscape, contracting epistemic uncertainty necessarily strengthens the tilt, but actual training statistics need not be monotone as scores and particle clouds change.The uncertainty floor caps the maximum tilting intensity.Source: §I.3 · p. 86

Related papers

Loading 2610.22852v1…