PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortized Feedback Planning: Turning Model-Based Rollouts into Executable Policies

Jeonggyu Huh

arXiv:2609.35012v1math.OCcs.SYeess.SY

TL;DR

Closed-loop planning must account for future observation-dependent actions, but repeated optimization can be expensive at deployment. BG refines current actions through conditional rollouts and stores them in executable policies; the paper reports competitive comparisons and theoretical accuracy results within stated assumptions and tested settings.

  • Problem

    Closed-loop planning must account for how future observations change later actions, a response omitted by optimizing a fixed future action sequence.

    Source: §1 · p. 1

  • Method

    BG differentiates conditional rollouts through an existing actor, corrects current actions, and stores the corrections while reusing deployed future feedback in a backward sweep.

    Source: §1 · p. 1, §7 · p. 11

  • Results

    Richer maps narrow several student–BG cost gaps, BG needs less additional construction in tested arm and docking implementations, and thrust approaches the tested feedback planner while the arm retains a planner gap.

    Source: §1 · p. 1, §7 · p. 11

  • Takeaways & Limitations

    The theory and task studies expose the roles of learned feedback, local correction, and storage in executable control.

    Source: §7 · p. 11

    The stated guarantees do not certify implementation-wide rates or all-state accuracy, and scaling to large models and long horizons is untested.

    Source: §7 · p. 12

Abstract

from arXiv · show

Closed-loop planning accounts for future observation-dependent actions but can be expensive to repeat at deployment. Bellman-gradient (BG) refinement differentiates conditional rollouts through an existing actor, corrects the current action, and stores the result in an executable policy. A backward sweep reuses deployed future feedback; full-horizon rollouts in an identified Gaussian belief model need no learned value critic. A nonlinear error recursion links quadratic continuation error, local correction, and policy storage. A controlled non-LQG example exhibits second-order policy accuracy with consistent storage, while ideal affine LQG admits exact backward recovery. In nonquadratic thrust, BG reduces actor cost by 2.47-3.20% and remains within 0.13-0.53% of the tested feedback MPC; original policies execute in 5-6 microseconds in a seed-0 native audit. With matched storage, BG attains competitive plant costs at about one ninth (arm) and one sixtieth (docking) of feedback-teacher-plus-student construction time on one GPU, excluding shared learning and node preparation. Richer common maps substantially narrow some student-BG gaps. These results expose the roles of learned feedback, local correction, and storage in executable control.

1 Introduction

Closed-loop planning must account for how future observations change later actions, but repeatedly optimizing future feedback is costly. The paper uses a trained actor for future feedback, locally corrects actions with model derivatives, and stores the corrections as executable policies.

1 Introduction

  • A controller must anticipate how an observation changes later actions; optimizing a fixed future action sequence omits that response even if replanning follows each measurement.The passage locates this distinction within dual control.Source: §1 · p. 1
  • BG differentiates conditional rollouts through an available continuation policy, corrects only the current action, and reuses already-corrected later policies in a backward sweep.The model and actor remain frozen during refinement; the full-horizon construction needs no fitted cost-to-go function.Source: §1 · p. 1
  • Learning supplies broad feedback, model derivatives add local action precision, and regression makes the refined decisions reusable at deployment.Deployment requires a belief update and one policy evaluation.Source: §1 · p. 1
  • Richer maps narrow several student–BG cost gaps, and BG needs less additional construction in the tested arm and docking implementations.Figure 1 compares matched-storage BG and feedback-MPC distillation using paired plant-cost gaps and additional-construction comparisons.Source: §1 · p. 1, Figure 1 · p. 2
  • The central result is a nonlinear error recursion connecting quadratic continuation error with local correction and policy storage.It includes derivative, safeguard, and fixed-ridge terms; the paper also reports exact backward recovery for ideal affine LQG and second-order policy accuracy in a controlled non-LQG example.Source: §1 · pp. 1–2

2 Related work and positioning

The paper positions BG among explicit and learned MPC, policy-improvement and storage methods, and conditional-rollout approaches. Its distinction is local action refinement through deployed future feedback, followed by regression into executable policies.

2 Related work and positioning

  • Explicit MPC moves optimization offline, while learned approximate MPC fits executable policies and differentiable predictive control learns through model rollouts [Hertneck et al., 2018; Drgoňa et al., 2022a,b].BG adds current-action refinement to a base actor rather than globally retraining its parameters.Source: §2 · p. 2
  • Prior work studies Newton interpretations of policy improvement, LQ policy iteration, and local quadratic convergence for discounted stationary problems [Puterman and Brumelle, 1979; Kleinman, 1968; Hewer, 1971; Santos and Rust, 2004].Finite-horizon performance iteration also learns current policies through fitted future feedback; BG instead tracks one backward sweep and actual storage in a weighted C2 policy norm, including fixed-ridge bias.Source: §2 · p. 2
  • Guided policy search learns from optimized trajectories or actions [Levine and Koltun, 2013, 2014; Zhang et al., 2016], while POPLIN guides online planning with policies.BG’s matched-storage comparison uses restricted feedback-MPC teachers.Source: §2 · p. 3
  • Unlike stochastic value gradients, belief-space DDP, and learned actor-value or short-planning methods, BG retains future-feedback state derivatives and regresses current-action targets across belief nodes.The paper also distinguishes its observation-based backward policy construction from fixed-latent open-loop BPTT adjoints and notes that Gaussian inference requires explicit information closure.Source: §2 · p. 3

3 Learning a Gaussian belief model

The identified Gaussian model supplies conditional rollouts, while the plant remains the data-generating system. Information closure supports Gaussian filtering; the resulting belief and task state support conditional simulation for planning.

3 Learning a Gaussian belief model

  • The identified model supplies conditional rollouts, while the plant is the data-generating system, so fitted-model and plant costs remain distinct.Source: §3 · p. 3

3.1 Dynamics and information closure

  • Gaussian closure requires a specified predictor update and the stated noise and initial-state assumptions; arbitrary encoders do not ensure that closure.Under those assumptions, the latent state conditioned on the information history is Gaussian, with mean and covariance following Kalman recursions.Source: §3.1 · p. 3
  • Innovation likelihood identifies model parameters through the filter and recurrent state, after which parameters are frozen; fitted covariance does not capture parameter uncertainty or misspecification.Source: §3.1 · p. 3

3.2 Conditional simulation and decision state

  • Conditional rollouts sample from the fitted Gaussian, propagate fitted dynamics and observations, and apply the same observations to filtering, task updates, and costs.Unknown future predictors require a causal model; the differentiable EKF extension adds restart and filtering errors, and Gaussian closure does not certify it.Source: §3.2 · pp. 3–4

4 Model-grounded policy correction

BG evaluates current actions through conditional rollouts with fixed future feedback, then stores accepted local corrections in policies. In the backward construction, each date uses the already-deployed later policies as its continuation.

4 Model-grounded policy correction

  • The method combines baseline feedback, local derivative-based action refinement, and regression storage, while retaining future observation response in conditional rollouts.Complete conditional rollouts evaluate cost-to-go functions without a learned critic.Source: §4 · p. 4, §4.1 · p. 4

4.1 Conditional derivatives with a fixed future policy

  • For conditional derivatives, BG fixes the continuation tail but retains its dependence on state inputs; fresh rollout averages are unbiased conditional on the pre-sample information.The pathwise result freezes model parameters and base noise, and does not differentiate policy construction.Source: §4.1 · p. 4

4.2 Local correction and acceptance

  • BG locally corrects the baseline action using the continuation’s gradient and Hessian, with a safeguarded proposal for unrestricted actions.The implementation may reject the proposal after paired Monte Carlo comparison against the baseline using the same tail and comparison noise.Source: §4.2 · pp. 4–5

4.3 Backward policy construction on sampled belief nodes

  • In the nonquadratic experiment, the executed policy limits extrapolation relative to the baseline.The cap bounds displacement, not cost, and acceptance and storage errors remain part of the method’s regularity and deployment analysis.Source: §4.3 · p. 5
  • The backward sweep corrects sampled belief nodes against the completed future-policy tail, then fits date-specific maps and evaluates the frozen policy on held-out episodes.Acceptance precedes regression, so stored targets and executed actions may differ; fitting before a nonlinear action transform can also differ from fitting executed actions.Source: §4.3 · p. 5

5 Guarantees and evaluation criteria

The section gives conditions for backward policy improvement and deployment error, then analyzes how local correction and policy storage affect nonlinear accuracy. It also establishes an exact affine LQG reference and clarifies the scope of model-transfer evaluation.

5.1 Backward improvement and deployment error

  • Nonpositive backward-comparison defects imply nominal cost nonincrease when the hybrid-policy costs and defects are integrable.The comparison evaluates adjacent policies with a shared baseline prefix and corrected tail; acceptance errors and storage interpolation contribute to the defect budget.Source: §5.1 · pp. 5–6

5.2 Nonlinear backward accuracy

  • Under the stated local regularity and storage conditions, a consistent regime yields O(ϵ^2) policy error and O(ϵ^4) nominal value error after one backward sweep.These are sufficient theoretical rates, not rates established for learned controllers; fixed storage budgets do not certify the required representation or ridge scaling.Source: §5.2 · p. 7

5.3 Local action correction and alignment

  • A positive metric can support strict local descent without estimating the population Hessian, although further projection may not preserve that guarantee.The alignment bound concerns the population residual after storage and safeguards; exact selected-tail minimization can still retain continuation error.Source: §5.3 · p. 8

5.4 Affine LQG as an exact reference

  • In fitted affine LQG with unrestricted actions, one exact Newton step per date and exact affine storage recover the policy after one backward sweep.The result excludes damping, curvature modification, and clipping.Source: §5.4 · p. 8

5.5 Model transfer and evaluation

  • Plant improvement is guaranteed only when nominal improvement exceeds the separate model-transfer error bounds for both controllers.Evaluation uses each deployed policy’s own observation history, and policy and model/filter discrepancies have a cross term rather than additive squared errors.Source: §5.5 · p. 8

6 Experiments

The experiments evaluate executable feedback in thrust, active-sensing docking, and a nonlinear arm using fitted models and paired physical trajectories. They compare open-loop replanning, restricted feedback MPC, and original task policies, with timing reported under distinct audit conditions.

6 Experiments

  • The experiments test executable feedback in thrust, active-sensing docking, and a nonlinear arm.Each task uses eight models fitted to action–observation data without latent-state labels; filtering is exact for thrust and docking and uses an approximate EKF for the arm.Source: §6 · pp. 8–9
  • Open-loop feedback MPC optimizes action sequences and replans after observations, whereas restricted feedback MPC optimizes future belief-dependent actions.Neither baseline uses DPO/BG initialization or continuation, and costs use paired physical trajectories with each controller’s own belief history.Source: §6 · p. 9
  • Figure 2 compares costs and available latencies of the original task policies, while Figure 1 separately compares common-map controllers.Reported timing markers use different hardware and audit scopes; cross-marker quotients are not algorithmic speedups.Source: §6 · p. 9

LQG reference

The thrust experiment provides a quadratic reference against fitted Riccati feedback.

LQG reference

  • BG reduces fitted-optimal action MSE by over 99.99% and matches Riccati-feedback plant costs at reported precision in quadratic thrust.The analytic controller remains cheaper and more accurate when available.Source: LQG reference · p. 9

Nonquadratic cost and the value of local calculation

The task evaluations test BG under nonquadratic costs and compare it with feedback planners and distilled students under matched storage. Results vary by task, while richer maps narrow several student–BG cost gaps.

Nonquadratic cost and the value of local calculation

  • In nonquadratic thrust, backward BG lowers DPO cost by 2.47%/3.20% and is within 0.13%/0.53% of tested restricted feedback MPC.The two values in each pair correspond to mild and anisotropic thrust conditions.Source: Nonquadratic cost and the value of… · p. 9, Table 2 · p. 9
  • With future thrust fixed, OLF assigns no value to sensing and chooses zero power, whereas DPO and BG retain observation-dependent actions.A validated OLF solver succeeds in all 98,304 calls but retains the zero-sensing continuation’s poor performance.Source: §6.2 · p. 9
  • In regular/late-noise docking, backward BG reduces DPO cost by 0.03796/0.06746 and raises success by 1.22/2.63 points; feedback MPC costs 0.01335 more and 0.07158 less than BG, respectively.The regular cost gain repeats on fresh episodes; the late-noise change is unresolved.Source: §6.2 · pp. 9–10, Table 3 · p. 10
  • Backward BG closes 53.9% of the eight-seed DPO-to-DP mean cost gap, but terminal-node regression loses precision and exact terminal-thrust replacement lowers full-policy cost.The reported diagnosis is a storage bottleneck at those nodes, not an attribution of all three-state docking error to regression; a second sweep gives no reliable improvement.Source: §6.2 · p. 10
  • For the arm, backward BG lowers DPO cost by 0.09382 and raises attainment by 10.74 points; feedback MPC lowers BG cost by 5.81%.Fresh episodes using the same model seeds give cost 6.68845.Source: §6.3 · p. 10, Table 4 · p. 11

6.4 Matched policy-storage comparison

  • In matched rich maps, student–BG mean-cost gaps shrink 59–61% in the two arm streams and 61% in regular docking; thrust resolves no method difference without establishing equivalence.Late-noise docking favors BG in both maps, but the map effect is unresolved; intervals are exploratory and unadjusted for multiplicity.Source: §6.4 · p. 11
  • On one local GPU, rich-map BG construction takes 9.34/82.78 seconds in arm, 5.09/313.39 in regular docking, and 4.99/298.16 under late noise versus teacher-plus-student fitting.These budget-specific times exclude shared model/actor training, node preparation, evaluation, and export; thrust lacks a matched total-time ratio.Source: §6.4 · p. 11

7 Discussion and conclusion

The conclusion presents BG as a stored local correction through deployed future feedback, with task results that differ across thrust, docking, and arm. Its guarantees and tested scope remain bounded by model and implementation conditions.

7 Discussion and conclusion

  • BG refines current actions through deployed future feedback and stores the corrections; its nonlinear recursion tracks continuation, derivative, safeguard, and representation errors.The controlled non-LQG example demonstrates consistent second-order policy accuracy, while affine LQG gives exact backward recovery.Source: §7 · p. 11
  • Thrust BG approaches the tested feedback planner, docking shows future-observation response and sensitivity to physical-action fitting, and the arm retains a planner gap.Richer matched maps narrow student–BG differences in arm and regular docking; late-noise docking favors BG, while thrust resolves no difference.Source: §7 · p. 11
  • The evaluated tasks use supplied low-dimensional model families, two actions, and 12–16 decisions, and the arm uses an approximate EKF.Finite-node checks certify neither all-state accuracy nor plant transfer; large-model and long-horizon scaling is untested, and full rollouts cost O(NMT^2) model steps before derivative/storage overhead.Source: §7 · pp. 11–12

Declaration of generative AI and AI-assisted technologies

The appendix records mathematical references for the paper’s local and backward comparisons, LQG error propagation, nonlinear stored-policy recursion, regularity conditions, and controlled non-LQG accuracy example.

Declaration of generative AI and AI-assisted technologies

  • Under the author’s direction, ChatGPT/Codex and Claude assisted with idea and argument examination and manuscript editing; ChatGPT/Codex also assisted implementation and debugging.The author retains responsibility for all content.Source: Declaration of generative AI and… · p. 12

Technical Appendix

  • The technical appendix organizes proofs across local and backward comparisons, finite-horizon LQG error propagation, nonlinear stored-policy recursion, regularity conditions, and a controlled non-LQG accuracy test.Main-paper equation and result numbers are retained.Source: Technical Appendix · p. 13

A.1 Proof of Proposition 3.2

  • For the controlled model, substituting coefficients evaluated at the known context and action lets the same Gaussian conditioning formulas apply.The predictive likelihood differentiates through the filter and recurrent predictor state; the observation score alone need not be the joint likelihood.Source: §A.1 · p. 13

A.2 Conditional derivatives and action alignment

  • Differentiating the rollout composition includes the state derivative of every future continuation action, with model and policy parameters fixed.The sampled, safeguarded Newton-step bound combines Taylor remainder, estimation, regularization, and feasible-action residual terms under its stated smoothness conditions.Source: §A.2 · pp. 14–15

A.3 Baseline continuation and descent

  • Corrected-tail acceptance does not imply the baseline-continuation sign condition; Theorem 5.1 addresses that continuation directly.With approximate action-value comparisons, the stated zero-margin bound limits deterioration but does not guarantee improvement.Source: §A.3 · pp. 15–16

A.4 Backward comparison and regression error

  • The backward comparison proof sums datewise cost differences, which telescope to the policy-cost difference and yield the stated inequalities under their sign and bound conditions.The proof uses a common baseline prefix and the same corrected tail after each date.Source: §A.4 · p. 16

A.5 Inexact metrics and implemented safeguards

The appendix bounds implemented local steps using realized gradient and metric errors, then connects accepted corrections to exact affine recovery and model-to-plant transfer conditions. Its guarantees depend on explicit error bounds and assumptions, not merely on independent acceptance samples or positive-definite metrics.

A.5 Inexact metrics and implemented safeguards

  • Strict local decrease follows when the quadratic margin is positive and the step length satisfies the stated Lipschitz-gradient condition.If L∇ = 0, every feasible step length 0 < α ≤ 1 gives strict decrease.Source: §A.5 · p. 16
  • A floored positive-definite metric supplies a metric, not a population-curvature error bound; the result instead requires a Lipschitz population gradient and realized-error control.Unbiasedness of the gradient estimate alone does not establish the needed error bound, and a capped continuation need not meet the regularity assumptions automatically.Source: §A.5 · p. 16
  • A negative integrated bound certifies fitted-model improvement, but a nonnegative bound only limits possible deterioration and plant transfer requires a separate bound.The comparison uses the same actual corrected continuation and accepted action; independent acceptance samples alone do not establish the required error bounds.Source: §A.5 · p. 17

A.6 Proof of Proposition 5.7

  • Under the fixed context schedule and exact fitting, backward construction recovers the affine LQG minimizer at every date and preserves it under the exact acceptance test.The target is affine, so full-column-rank fitting gives zero residual; pooling schedules requires a representation containing their optimal coefficients and a full-rank pooled design.Source: §A.6 · p. 17

A.7 Model-transfer and action-error accounting

  • Plant improvement transfers when nominal improvement exceeds the sum of separate plant-model error bounds for the two policies.Those bounds must cover the complete corrected controller and fitted filter; a one-action probe followed by DPO is insufficient.Source: §A.7 · p. 17

B Finite-horizon error propagation and LQG structure

  • The finite-horizon analysis tracks continuation, local correction, and representation errors through the actual corrected affine tail, while fixing the fitted model and horizon.Small finite-panel action MSE alone does not establish the theorem’s coefficient-error premise; the finite-horizon constant need not be uniform in horizon.Source: B Finite-horizon error propagation and… · p. 18

B.1 Finite-horizon affine refinement

For a fixed fitted affine LQG model, the appendix analyzes finite-horizon correction around a baseline and then tracks errors through refreshed affine tails and numerical deployment. The bounds require coefficient control and do not claim horizon-uniform constants.

B.3 Riccati reference coefficients

  • The optimal augmented affine recursion agrees with the Riccati recursion, while its augmented matrix is distinct from the physical-state block.Source: §B.1 · p. 18

B.1 Finite-horizon affine refinement

  • Under the theorem’s coefficient-control premise, finite-horizon frozen-continuation affine refinement has an O(ϵ²) coefficient-error bound, with exact correction at the final decision.The bound is finite for each fixed horizon, but its constant can grow rapidly with horizon; small finite-panel action MSE does not establish the premise.Source: §B.1 · pp. 18–19

B.2 Propagation through a refreshed affine tail

  • With an actual corrected affine tail, zero local errors yield exact backward recovery, while nonzero local errors and corrected-tail coefficients determine earlier-date errors.Node-fit MSE alone does not bound the local error term, and nonlinear or capped maps require additional approximation arguments.Source: §B.2 · p. 20

B.4 Numerical deployment of the finite-horizon correction

  • Deployment error must be controlled separately, including sampling, solving, regularization, clipping, and rejection; a feasible constant mixture improves when its stated bracket is below baseline error.Fitted excess cost is curvature-weighted action error under the policy visitation law, so total cost, plant evaluation, and common-history action MSE use different measures or baselines.Source: §B.4 · pp. 20–21

C Nonlinear backward accuracy and storage

  • The nonlinear backward analysis closes by measuring stored-policy error in the same norm used for earlier-date continuation calculations.Prescribed local radii must satisfy the stated checks; the finite-node diagnostics do not certify those sufficient conditions.Source: C Nonlinear backward accuracy and… · p. 21

C.1 Weighted continuation cancellation

  • Weighted continuation cancellation bounds action-gradient and action-Hessian differences by constants times continuation error, under derivative, integrability, and regularity conditions.Gaussianity alone does not imply the required envelope conditions; hard-cap crossings need separate treatment.Source: §C.1 · pp. 21–22

C.2 Node correction, acceptance and policy storage

The appendix links node corrections and acceptance to stored-policy accuracy, then closes a backward error recursion and identifies a fixed-ridge consistency obstruction. These results are conditional on local regularity and storage assumptions, and the exact-belief conditions remain distinct requirements.

C.2 Node correction, acceptance and policy storage

  • Exact acceptance preserves second-order target accuracy even upon rejection, and under the stated alternatives accepted targets remain second-order accurate.This concerns target actions; acceptance precedes regression, so targets and executed actions may differ.Source: §C.2 · pp. 22–23
  • The sequential polynomial-kernel storage map fits direct actions and then a kernel to residuals after an intermediate cap, so replacing that prediction changes the analyzed operator.The bound applies to the realized frozen candidate or uniformly over the candidate list; finite-node smoothness alone does not guarantee second-order policy accuracy.Source: §C.2 · pp. 23–24

C.3 Closing the recursion

  • The stored-map norm closes the backward recursion, and under the stated scaling assumptions O(ϵ²) policy errors imply O(ϵ⁴) nominal value error for a fixed finite horizon.The constants need not be uniform in horizon.Source: §C.3 · p. 24

C.4 Storage consistency and fixed ridge

  • Positive fixed ridge penalties can leave a nonzero storage consistency defect even for optimal baseline labels, obstructing an unconditional asymptotic rate claim.The lower bound may be very small and is not an estimate of experimental performance loss; a sufficient alternative requires representation and conditioning assumptions.Source: §C.4 · pp. 24–25

D Nonsmooth safeguards and belief regularity

  • The derivative and belief analyses impose separate conditions: nonsmooth safeguards can invalidate population-Hessian identities, while exact filtering requires modeled predictor information and retained state dynamics.The main learned-controller experiments retain hard safeguards, and the stated sufficient growth and covariance conditions have not been certified for those tasks.Source: D Nonsmooth safeguards and belief… · p. 25

D.1 Derivative chain rule and nonsmooth safeguards

The appendix distinguishes valid population derivatives from branchwise curvature under hard safeguards and gives a smooth alternative with explicit regularity requirements. Exact belief construction also requires predictor information and dynamics to be represented, not merely filtering the latent state.

D.1 Derivative chain rule and nonsmooth safeguards

  • If noise paths cross a hard-cap boundary as the action varies, branchwise automatic-differentiation curvature lacks an established population-Hessian unbiasedness guarantee.In the clipped-normal example, the pathwise second derivative is zero almost surely at fixed action, but the population derivative equals −0.222548 at u = 1/2.Source: §D.1 · pp. 25–26
  • A smooth radial cap can support pathwise derivative identities when the complete deployed continuation is C2 and sampled derivatives have integrable local envelopes; smoothing only one safeguard is insufficient.The evaluated main learned-controller experiments retain hard safeguards, and the sufficient growth and covariance conditions have not been certified for those tasks.Source: §D.1 · pp. 26–27

D.2 Exact-belief conditions

  • New predictor information must enter the likelihood or satisfy the stated assumption, and recurrent predictor states or random future covariates need their own retained states and transition laws.Filtering the latent state alone does not supply omitted predictor dynamics.Source: §D.2 · p. 27
  • Exact filtering, derivatives, local optimization, and action-value decrease are separate requirements; regressed policies retain an interpolation term.The associated improvement results also require their acceptance and model conditions.Source: §D.2 · p. 27

E A controlled non-LQG quadratic-accuracy regime

This section examines deployed-policy accuracy and storage in a controlled non-LQG perturbation setting, alongside feedback-student comparisons under finite teacher budgets, deployment and numerical audits, and a scalar active-sensing DP comparison. It also reports reference-planner checks and quadratic-thrust plant-cost comparisons, including the distinction between terminal BG corrections and regression into an executable policy.

E A controlled non-LQG quadratic-accuracy regime

  • Decreasing residual ridge yields approximately second-order deployed policies at both dates and fourth-order continuation-gradient error; fixed direct ridge instead leaves a nonzero error floor.At ϵ = 0.00625, date-zero target/deployed errors are 4.76×10−5/1.26×10−4 and date-one errors are 7.72 × 10−5/1.09 × 10−4. The experiment uses a representable residual direction and smooth deployment; its sampled compact-domain errors do not establish global bounds or learned-controller conditions.Source: E A controlled non-LQG… · pp. 28–30, Table E.1 · p. 29

S1.1 Active-sensing docking

  • Docking OLF-MPC optimizes open-loop thrust sequences and analytically eliminates sensing, so replanning does not restore the omitted ex ante value of future information.The separate feedback MPC optimizes datewise sensing and affine belief-mean thrust feedback, but is not an exact POMDP solution.Source: S1.1 Active-sensing docking · p. 32

LQG reference

  • BG reduces fitted-optimal LQG action MSE by more than 99.99%, to 1.94 × 10−6/1.72 × 10−6 in mild/anisotropic conditions.Plant-optimal MSEs are 0.002029/0.001966 and approach the fitted-model–plant oracle discrepancy; squared errors do not provide an additive error decomposition.Source: LQG reference · p. 33

S1.4 Nonlinear robot arm

  • Enabling future belief response lowers mean arm plant cost from 6.36050 to 6.28979, a paired difference of −0.07071 [−0.08023, −0.06118].Extra open-loop optimization alone gives an unresolved difference of −0.02133 [−0.05262, +0.00996]; the fixed-budget ablation supports a feedback-response contribution, not nonlinear optimality.Source: S1.4 Nonlinear robot arm · p. 34

S2 Policy storage and target validation

  • On fresh mild/anisotropic thrust panels, backward BG costs 1.606786/2.052751 versus static BG’s 1.609062/2.057187; its original-panel cost exceeds online feedback MPC by 0.002213/0.011277.The fresh-panel paired backward-minus-static differences are −0.00228 [−0.00364, −0.00091] and −0.00444 [−0.00577, −0.00310]. These are exploratory intervals over eight paired fitted-model seed means.Source: S2 Policy storage and target validation · p. 35
  • In regular/late-noise docking, backward BG costs 0.822330/4.025040 on original panels and 0.821157/3.940916 on fresh panels, while restricted online feedback MPC is cheaper by 0.01335 in regular docking.Original-panel success is 82.57%/50.68% and fresh-panel success is 82.25%/49.80%; backward BG is costlier than restricted online feedback MPC by 0.07158 under late noise. Comparisons are descriptive eight-seed intervals, not comparisons to an optimal belief policy.Source: S2 Policy storage and target validation · p. 35
  • For the two-link arm, backward BG costs 6.678051 on original and 6.688447 on fresh panels, versus 6.289789 for restricted online feedback MPC; stored maps average 31,065 bytes.Original/fresh attainment is 59.23%/57.08%, and target-node action MSE is 0.043954, a storage diagnostic rather than plant performance.Source: S2 Policy storage and target validation · p. 35

S3.1 Independent OLF solver verification

  • All 256 nonquadratic-thrust OLF solver batches pass, with maximum gradient 9.70 × 10−6 and suboptimality bounded by 1.69 × 10−8; this verifies the OLF solve, not feedback optimality.The bound uses effort-coefficient Hessian eigenvalue lower bounds of µ = 0.12 or 0.02.Source: S3.1 Independent OLF solver verification · p. 35

S3.2 Restricted feedback-MPC distillation

  • Arm student cost barely changed from 6.73788 to 6.73866 as feedback iterations rose from 30 to 320, while construction time increased from 57.83 to 436.23 seconds.Thrust student costs improved substantially between 16- and 80-iteration teacher checkpoints, with little further change at 320; student costs were near task-section BG means, but storage maps differed. Regular-docking student cost minus online feedback MPC was −0.00701 [−0.01610, +0.00208], versus +0.09887 [+0.05022, +0.14752] under late noise.Source: S3.2 Restricted feedback-MPC… · pp. 36–37, Table S2 · p. 36, Table S3 · p. 37

S3.3 Matched policy-storage comparison

  • Matched storage narrowed student–BG gaps in the arm (∆ change −0.014407) and regular docking (∆ +0.006401 to +0.002487), while thrust intervals resolved neither superiority nor equivalence.In late-noise docking, BG had lower cost with both maps, but the change in ∆ was unresolved. Teacher-plus-fit/BG time ratios were 8.86 for the arm, 61.63 for regular docking, and 59.81 for late-noise docking; these reflect tested implementations and finite budgets, not minimum computation for common accuracy.Source: S3.3 Matched policy-storage comparison · pp. 38–39, Table S4 · p. 38, Table S5 · p. 39

S4.1 Historical deployment and offline stage accounting

  • Historical BG/student preparation subtotals ranged from 79.8/214.9 seconds in mild thrust to 118.8/168.0 seconds in the arm, but mixed hardware and exclusions preclude a speed ratio.The accounting includes recorded learning and construction stages but excludes node preparation, data collection, evaluation, native export, and exploratory tuning; it is historical stage accounting rather than a synchronized same-host benchmark.Source: S4.1 Historical deployment and offline… · p. 39, Figure S1 · p. 40

S4.2 Native policy kernels and acados

  • Native C and float64 PyTorch evaluations of matched frozen maps agreed within 2.30×10−14 across 147,456 policy–state inputs.For thrust SQP, OLF succeeded on all 48 audited beliefs with maximum objective discrepancy 7.11 × 10−15, while feedback SQP reached its iteration limit in 32 of 48 cases; RTI performed one QP step, not a nonlinear convergence certification.Source: S4.2 Native policy kernels and acados · pp. 41–42, Table S6 · p. 41, Table S7 · p. 41

S4.3 Derivative and numerical checks

  • Arm test NLL improved from −1.32966 to −1.54294, only 0.00025 above the plant-coefficient EKF score; approximate filters and persistent planner gaps do not identify a unique cause.Common-noise derivative checks found refreshed-tail gradient errors of 2.07 × 10−5/3.66 × 10−5 in thrust and 3.13×10−4 in the arm; the arm refreshed-tail Hessian discrepancy reached 0.0874, so it did not validate a smooth Hessian.Source: S4.3 Derivative and numerical checks · p. 42

S5 Numerical DP reference for scalar active-sensing docking

  • Across eight seeds, BG reduced conditional cost from 0.070864 to 0.068430, closing 53.9% of the DPO-to-numerical-DP gap.The numerical DP value was 0.066349; BG lowered cost in all eight seeds, with mean paired reduction 0.002434 ± 0.000325. The DP reference remains numerical, and this scalar companion is not an exact reduction of the three-state docking experiment.Source: S5 Numerical DP reference for scalar… · p. 43
  • At 96.5% of construction nodes, target thrust was within 0.01 of the exact action, but its mean one-step cost excess was less than 1% of the fitted policy’s.This isolates a representation bottleneck at terminal construction nodes, not a full allocation of policy cost gap among theoretical error terms.Source: S5 Numerical DP reference for scalar… · pp. 43–44, Table S9 · p. 43
  • Replacing only the terminal BG thrust with the exact action reduced mean conditional full-policy cost from 0.068430 to 0.067486 across eight seeds; all eight improved.The paired mean improvement was 0.000944 ± 0.000182, about 45% descriptively of the mean BG-to-Bellman gap; this is not an additive regret decomposition.Source: S5 Numerical DP reference for scalar… · p. 44
  • A second backward sweep changed mean conditional cost from 0.068430 to 0.068711, with only three of eight seeds improving.The paired difference was +0.000281±0.000180; the experiment does not support reliable improvement from repeated sweeps.Source: S5 Numerical DP reference for scalar… · p. 44

S6 Quadratic thrust reference panel

  • DPO-to-BG plant-cost differences were −0.03694 [−0.03948, −0.03439] (mild) and −0.06831 [−0.07126, −0.06536] (anisotropic).These are exploratory 95% paired t intervals across eight seeds.Source: S6 Quadratic thrust reference panel · p. 44, Table S10 · p. 44
  • Riccati kernel readings were near the measurement floor, while fitted-controller residuals of 10−11–10−10 arose from differing floating-point reference and execution paths.The fitted controller’s mathematical self-error is zero; the plant oracle uses its own plant belief, so its fitted-reference error need not vanish.Source: S6 Quadratic thrust reference panel · p. 44

Related papers

Loading 2609.35012v1…