PaperlayerView exact revision on arXiv

Source-linked AI summary

VIGOR-MPC: Vision-Guided Objective Regulation MPC

Jelena Trisovic, Rahel Rickenbach, Marek Landert, Panagiotis Minos, Melanie N. Zeilinger

arXiv:2610.20333v1cs.ROcs.SYeess.SY

TL;DR

Vision-based control must cope with noisy, variable-length, partially occluded observations while sharing limited onboard compute between perception and control. VIGOR-MPC distills a privileged, field-of-view-aware long-horizon MPC into a short-horizon student that infers cost parameters from vision. It approaches teacher performance, reduces computation time, and generalizes to unseen tracks.

  • Problem

    Vision-based control must track references using partial, noisy observations while field-of-view changes, unknown environments, and limited compute complicate safe, timely control.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    VIGOR-MPC trains a short-horizon MPC student to infer cost parameters from perceived context and imitate a privileged, field-of-view-aware long-horizon teacher.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    VIGOR-MPC approximates privileged-teacher behavior despite limited view and imperfect perception, with a 94.2% reduction in computation time.

    Source: §4.1 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    VIGOR-MPC successfully generalizes to unseen track layouts and obstacle configurations in closed-loop application.

    Source: §4.1 · p. 7

    Strong theoretical guarantees require exact imitation across the full feasible state space.

    Source: §6 · p. 10

Abstract

from arXiv · show

Controlling an agent in an environment solely perceived through onboard vision is challenging: perception yields noisy, variable-length, and partially occluded context, while onboard compute is shared between perception and control, limiting the planning horizon executable in real time. We propose VIGOR-MPC, a modular student-teacher framework, agnostic to the perception pipeline, which addresses reference tracking across navigation tasks by distilling a privileged, field-of-view-aware long-horizon MPC into a computationally lightweight short-horizon MPC whose cost parameters are determined online from vision. The teacher operates in simulation with full state knowledge and a multi-trajectory formulation that guarantees the agent can stay within its field of view. The student observes only the noisy, occluded context produced by a perception module and learns, via a multi-level recurrent mapping, to infer cost parameters that aim to recover the teacher's behavior despite missing information. We evaluate VIGOR-MPC on vision-based autonomous racing in simulation and on hardware. Using only onboard perception of the track, VIGOR-MPC generalizes to tracks unseen during training and approaches the lap times of the privileged long-horizon teacher while reducing per-step computation by over 90%.

1 Introduction

Vision-based control must handle uncertain, incomplete observations and limited compute while generalizing to unknown environments. VIGOR-MPC combines a privileged, field-of-view-aware teacher with a memory-equipped student that adapts MPC costs from perceived context.

1 Introduction

  • Vision-based control faces noisy perception, limited and occluded views, compute shared with perception, and unknown deployment environments.The introduction notes that end-to-end learning approaches tend to generalize poorly to unseen layouts or new information sources.Source: §1 · p. 1
  • VIGOR-MPC distills a privileged, field-of-view-aware MPC teacher into a lightweight student that adapts a predefined controller from noisy, partially occluded context.The student uses a memory-capable neural network to generate MPC cost parameters from observations produced by a perception module.Source: §1 · p. 2, Figure 1 · p. 2
  • The framework is designed to adapt online to perceptual uncertainty and demonstrated in simulation and on hardware for racing without prior track knowledge.Its modular mapping is intended to remain agnostic to the perception pipeline.Source: §1 · p. 2
  • Prior partially learned controllers largely rely on clean state-based context, leaving perception-derived noisy, occluded, variable-length context unaddressed.VIGOR-MPC builds on structured imitation of optimization-based controllers to address this gap.Source: §1 · p. 2

2 Problem Formulation

The problem is to track references safely when an agent has only noisy, partial camera observations and limited compute. The controller must manage changing observation horizons, retain temporarily hidden context, and generalize across unknown environments and perception pipelines.

2 Problem Formulation

  • The agent’s state is known, but reference and environmental context are extracted from partial, noisy onboard RGB-D observations.The formulation distinguishes dynamics-relevant context from other environmental information that may affect constraints, such as obstacles.Source: §2 · p. 3
  • A changing field of view makes reference and context sequences variable in length, sometimes empty, and liable to disappear from view.The controller must plan safely without assuming a fixed or minimum lookahead and retain past observations when they become occluded.Source: §2 · p. 3
  • Perception errors include noise and outliers with generally unknown, non-stationary distributions, motivating compensation from data without prescribed error bounds.The errors may vary with viewing angle, lighting, or scene content.Source: §2 · p. 3
  • The controller should generalize to unseen environments and remain agnostic to the application-dependent context and perception pipeline.The formulation also calls for low-latency control because perception already consumes substantial computational resources.Source: §2 · p. 3

3 Method

VIGOR-MPC uses a privileged long-horizon teacher to train a short-horizon student whose learned costs adapt a predefined MPC to partial, noisy observations. Its teacher explicitly plans for field-of-view safety, while recurrent processing preserves context across missing observations and time.

3.1 Preliminaries: Model Predictive Control

  • MPC takes the current state, reference, and context, solves over a prediction horizon subject to dynamics and constraints, then applies only the first optimal input.The optimization uses stage costs with parameters and may include context-dependent state constraints.Source: §3.1 · p. 4

3.2 Imitation Learning from Perfect Knowledge

  • The student retains a predefined short-horizon MPC structure and learns objective parameters to imitate the teacher using partial, noisy observations.The framework is intended to transfer from representative training environments to unseen ones.Source: §3.2 · pp. 4–5
  • The privileged teacher uses long-horizon MT-MPC to jointly predict a safe trajectory constrained to remain in view and a performance trajectory coupled to it at the first step.The teacher has ground-truth references and context; the safe trajectory guarantees a feasible sequence that stays within the observable field of view.Source: §3.2 · p. 5
  • Context entries are padded with non-informative values and paired with binary visibility masks before recurrent processing.The recurrent network maintains memory so previously observed context remains usable after it becomes occluded.Source: §3.2 · p. 5, Figure 3 · p. 6
  • Multi-level recurrence processes context within each frame and carries hidden state across frames to produce the student’s MPC cost parameters.Attention pooling forms a fixed-dimensional frame representation, which is combined with the current state before the higher-level recurrence.Source: §3.2 · p. 5
  • Training minimizes mean-squared error between predicted teacher and student trajectories using sequences of consecutive frames to account for recurrent memory.The loss is evaluated over the first N_D steps, and only sequence starts with a teacher-feasible solution over the full sequence are retained.Source: §3.2 · pp. 5–6

4 Experiments

Experiments evaluate VIGOR-MPC in simulation and on hardware, focusing on teacher matching, obstacle safety, generalization, and the contributions of its FOV-aware components. The framework approaches privileged-teacher performance with substantially lower computation and transfers to unseen tracks.

4 Experiments

  • The evaluation compares VIGOR-MPC with a perfect long-horizon teacher, an image-informed long-horizon MPC, and a short-horizon MPC without context-dependent costs.Simulation and hardware results are complemented by an ablation study of the method components.Source: §4 · p. 6, §4.1 · p. 7

4.1 Simulation Results

  • VIGOR-MPC reduces imitation error by 57.53% over short-horizon MPC, nearly matches the privileged teacher’s lap time, and cuts computation time by 94.2%.The comparison uses the obstacle-free Figure 4 track; imitation error is RMSE over the first five prediction steps.Source: Table 1 · p. 7, §4.1 · p. 7
  • Across 100 random obstacle configurations, VIGOR-MPC avoided every obstacle in 99 and stopped safely rather than crashing in the remaining case.The environments use randomly generated obstacle configurations.Source: §4.1 · p. 7, Figure 4 · p. 7
  • VIGOR-MPC successfully completed closed-loop runs on test track layouts beyond the sole training layout, with lap times reported for two unseen tracks.Figure 5 shows three layouts, with only the leftmost used for training.Source: §4.1 · p. 7, Figure 5 · p. 8, Table 2 · p. 8
  • In harder obstacle tests, cross-frame time recurrence raises lap completion to 94%, while masking and context recurrence together reach 46%.The 100 configurations contain more and wider obstacles than the Figure 4 tests; the two components are reported as complementary.Source: §4.1 · p. 8, Table 3 · p. 9

4.2 Hardware Experiments

  • Hardware experiments ran perception and control locally at 30 Hz, using a model trained entirely in simulation without real-world training data.The setup used an onboard camera and a modified F1Tenth platform.Source: §4.2 · p. 9

5 Conclusion

VIGOR-MPC combines a privileged long-horizon teacher with a memory-capable student that adapts a short-horizon MPC’s cost parameters from partial, noisy observations. In autonomous racing, it approximates teacher behavior in simulation and the real world despite limited vision and imperfect perception.

5 Conclusion

  • VIGOR-MPC distills a privileged, perfect-information long-horizon MPC into a short-horizon MPC whose cost parameters are generated from partial, noisy observations.A memory-capable neural network produces the student’s parameters.Source: §5 · p. 9
  • In autonomous racing, VIGOR-MPC approximates privileged-teacher behavior in simulation and the real world despite limited field of view and imperfect perception.The hardware deployment is visualized in Figure 7.Source: §5 · p. 9, Figure 7 · p. 9
  • The framework combines optimization-based control structure with learned adaptation to address challenges in vision-based control.The evaluation in this paper uses autonomous racing as its example application.Source: §5 · p. 9

6 Limitations

The limitations concern state estimation, obstacle representation, and the strength of guarantees. The extended related work positions VIGOR-MPC among modular controllers that preserve optimization structure while adapting to context.

6 Limitations

  • Velocity estimates currently rely on motion capture, although track-relative position and orientation come directly from onboard perception.Point-cloud registration with an Extended Kalman Filter is proposed as a possible alternative for velocity estimation.Source: §6 · p. 9
  • The current context restricts obstacles to either side of the track, and exact imitation across the full feasible state space would be needed for strong theoretical guarantees.Richer context representations or MPC parameterizations could handle more complex obstacle configurations.Source: §6 · p. 10

A Extended Related Work

  • End-to-end policies map camera observations directly to actions and can acquire complex sensorimotor behaviors, but require substantial data and compute, lack explicit constraint satisfaction, and typically need retraining after non-trivial deployment-domain or task changes.Robotics applications needing fast training, modest compute, and reuse of existing perception modules therefore favor modular architectures; modularity is empirically dominant across robotics.Source: A Extended Related Work · p. 13
  • Modular classical controllers preserve safety and interpretability but are rigid to noisy, occluded, or variable-length context; uncertainty-aware alternatives often tighten constraints instead of learning controller adaptation.Examples include sampled-curvature MPC [4], tube-based MPC [15], and distributionally robust MPC [16].Source: A Extended Related Work · p. 13
  • Partially learned controllers retain optimization structure while learning selected ingredients, and VIGOR-MPC addresses the gap of applying this approach to noisy perception output.Earlier methods were developed almost exclusively with clean, low-dimensional context; VIGOR-MPC builds on [6].Source: A Extended Related Work · p. 13

B Additional Methodological Preliminaries

The appendix details the field-of-view-aware teacher, the student’s gradient-based recurrent imitation training, and the simulation perception and control pipeline. Its implementation combines a computationally efficient vehicle model with visual estimates of track geometry and obstacle-constrained corridors.

B.1 Multi-Trajectory Model Predictive Control

  • The MT-MPC jointly plans safe and performance trajectories, coupling their first steps so the applied input balances safety and performance while the safe trajectory remains within the field of view.Given initial feasibility, the formulation provides a feasible control sequence at each subsequent step; its terminal set ensures the agent stays within view.Source: §B.1 · p. 14

B.2 Differentiable Model Predictive Control

  • The student’s recurrent mapping is trained by propagating imitation-loss gradients through its MPC, using differentiable MPC based on KKT differentiation and iterative iLQR convex approximations.The method is used because differentiating through the optimization is nontrivial and the approach supports general MPC cost formulations.Source: §B.2 · p. 14

B.3 Recurrent Neural Networks

  • The RNN recursively updates a hidden state from sequential inputs and maps it to the objective-parameter adaptation vector, retaining information from earlier observations.The hidden state serves as memory, making recurrent networks suitable for partially observable settings where relevant context can disappear from view.Source: §B.3 · p. 15

C Detailed Imitation Learning Algorithm

  • Training minimizes imitation loss between teacher performance and student trajectories, using recurrently predicted corrections to warm-start student cost parameters and Adam to update network weights.The loss uses MSE over the first ND ≤ NS prediction steps; training samples environments and state sequences from teacher rollouts.Source: C Detailed Imitation Learning Algorithm · pp. 15–16

D.1 Example Simulation Setup

  • The racing setup uses a track-relative Frenet frame and a kinematic bicycle model, trading reduced accuracy during high-slip maneuvers for lower MPC evaluation cost.The authors motivate the model for the F1TENTH platform’s low-to-moderate velocities and acceleration profiles.Source: §D.1 · p. 16

D.2 Perception Pipeline Details

  • The perception pipeline filters pixels against an expected floor-depth map, then applies Edge Drawing to extract ordered lane-marking edges within the masked region.Distance-dependent thresholds account for increasing stereo-depth uncertainty; the mask also reduces edge-tracing cost by excluding obstacles and reflections.Source: §D.2.1 · p. 18, §D.2.2 · p. 18

D.2.3 Centerline Estimation from Lane Markings

  • Centerline estimation groups projected lane-marking edges by proximity and orientation, pairs opposite sides to estimate midpoints, and offsets visible unpaired markings by the known half-lane width.Recursive refinement handles local mismatches in point sampling when pairing markings.Source: §D.2.3 · pp. 18–19

D.2.4 Obstacle Detection

  • Obstacle detection projects obstacle points onto the centerline’s curvilinear coordinates and tightens the corresponding lateral corridor over each obstacle’s progress span.Where clusters overlap in progress, the tightest bound prevails; unobserved regions retain the nominal track half-width.Source: §D.2.4 · pp. 19–20

D.3 Learning Framework Setup and Implementation Details

The learning setup supplies the recurrent network with visible environmental context and selected vehicle state, then trains and validates it against teacher trajectories. The real-world platform uses onboard RGB-D sensing and runs perception and control locally.

D.3 Learning Framework Setup and Implementation Details

  • The network receives field-of-view-masked context and vehicle speed, lateral deviation, and orientation, but not track progress, to support generalization across tracks.Variable-length context is padded, and missing or occluded entries receive a noninformative value and negative mask.Source: §D.3 · p. 20
  • A GRU processes variable-length context, attention pooling combines its sequence features with global state features, and a feedforward network predicts MPC cost parameters.The GRU has hidden size 32; the feedforward network has six fully connected layers with 512 hidden units each.Source: §D.3 · pp. 20–21
  • The imitation loss sums weighted state and input trajectory MSEs over the first ND prediction steps, while model selection uses lap performance on a fixed training-track rollout.The selected model is the one with the best validation-rollout lap performance.Source: §D.3 · p. 21

E.1 Hardware Setup: Car and Camera

  • Real-world tests use a modified F1TENTH car with an onboard RGB-D camera and local computation; the foam-mat track has a 1.10 m lane and physical obstacles.The camera supplies aligned color and depth at 848 × 480 resolution and 90 FPS.Source: §E.1 · p. 21, §E.2 · p. 21

E.3 Experiment Setup

The experiment setup describes the real-world perception and control arrangement, while the following appendix section introduces additional simulation and hardware results and training visualizations.

E.3 Experiment Setup

  • The real-world system solves MPC at 30 Hz, estimates the centerline and lane markings from onboard RGB-D streams, and handles obstacles with a separate detector.An AprilTag calibration estimates camera pose relative to the floor; ground-truth state estimates come from external motion capture.Source: §E.3 · p. 21

F Additional Results

  • The appendix’s additional-results section covers further simulation results, real-world results, and visualizations of training behavior.Figure 10 depicts closed-loop VIGOR-MPC employment across randomly placed obstacle configurations.Source: F Additional Results · p. 22, Figure 10 · p. 22

F.1 Additional Simulation Results

Additional simulations report lower imitation error, obstacle avoidance on unseen layouts, and a constraint-satisfaction gap against an end-to-end policy. Teacher horizon length trades lap time against solve time, with reduced control cost freeing compute for perception.

Extended Heavy-Tail Statistics for Imitation Error

  • VIGOR-MPC reduced median imitation loss by 51.4% relative to MPCNS, while its p90 tail performance was comparable to the teacher.The difference from MPCNS was significant under a Wilcoxon signed-rank test (p = 7.6·−12).Source: §F.1 · p. 22, Table 6 · p. 23

Further Results on Obstacle Avoidance

  • Across 100 obstacle configurations on a track unseen during training, VIGOR-MPC completed laps without crashing; it also handled intentionally placed, unseen obstacle patterns.One tested configuration did not complete the lap, but the car stopped safely rather than crashing.Source: §F.1 · pp. 23–24, Figure 11 · p. 23, Figure 12 · p. 24

Comparison with an End-to-End Learned Policy

  • The end-to-end policy crashed into an obstacle or track boundary in 89 of 100 runs, while VIGOR-MPC's MPC explicitly enforces dynamics and boundaries.The end-to-end baseline had a larger backbone and identical supervision; representative laps are shown in Figure 13.Source: §F.1 · p. 24, Figure 13 · p. 25

Effect of Teacher Horizon Length NL on Lap Time and Per-step Solve Time

  • Longer teacher horizons improve lap time until about NL ≈30 while increasing solve time; shortening the horizon reduces solve time but degrades lap time.The teacher is used offline, and cutting control cost by more than 90% frees resources for perception.Source: §F.1 · p. 25, Table 7 · p. 25

F.2 Additional Real-World Results

Real-world F1-Tenth experiments show obstacle handling, a comparison with a short-horizon MPC baseline, and generalization from a single training track.

Further Obstacle Configurations

  • In a continuous hardware run, VIGOR-MPC handled multiple obstacle configurations while some obstacles were removed and replaced.Figure 14 shows additional successfully handled configurations.Source: §F.2 · p. 26, Figure 14 · p. 26

Comparison with Short Horizon MPC

  • With the same centerline-estimation and obstacle-detection pipeline, VIGOR-MPC completed a lap while MPCNS crashed into the track boundary at multiple instances.The closed-loop comparison is shown in Figure 15.Source: §F.2 · p. 26, Figure 15 · p. 26

Generalization Capabilities

  • A model trained solely on the leftmost track was run on the hardware track, with obstacles dynamically replaced during execution.Figure 16 visualizes three laps with different obstacle configurations.Source: §F.2 · p. 26, Figure 16 · p. 26

F.3 Illustration of Training Behavior

The training visualizations show VIGOR-MPC's behavior progressing from short-horizon MPC behavior toward the long-horizon MT-MPC teacher.

Illustration of Training Behavior

  • As training progresses, VIGOR-MPC's behavior shifts from short-horizon MPC toward long-horizon MT-MPC behavior.Figure 17 compares open-loop predictions against both baselines, while Figure 18 shows trajectory evolution across epochs.Source: §F.3 · p. 27, Figure 17 · p. 27, Figure 18 · p. 28
  • Figure 17 tracks VIGOR-MPC's open-loop predictions against the long-horizon MT-MPC teacher and short-horizon MPC as training progresses.The teacher prediction is solid blue and the short-horizon prediction is dashed green.Source: Figure 17 · p. 27
  • Figure 18 visualizes how VIGOR-MPC trajectories evolve across training epochs.The reported progression is from short-horizon MPC behavior toward long-horizon MT-MPC behavior.Source: §F.3 · p. 27, Figure 18 · p. 28

F.4 Extension to Quadrotor Dynamics

The paper extends VIGOR-MPC to quadrotor dynamics and demonstrates both 3D racing and reference tracking in simulation. In 3D racing, its lap time approaches the long-horizon MPC result and improves on the short-horizon baseline at the same horizon.

Extension to Quadrotor Dynamics

  • The quadrotor model neglects motor and attitude dynamics, assumes a fast inner attitude loop, sets path torsion to zero, and decouples heading from translation.The resulting state and inputs use tangential, normal, and binormal motion with commanded yaw rate; the setup adds track and input bounds.Source: §F.4 · p. 28
  • The quadrotor model constrains lateral offset to the track and bounds tangential speed, accelerations, and yaw rate.The reported bounds include amax = 8 m/s2 and ωψ,max = 3 rad/s.Source: §F.4 · p. 28

Racing in 3D

  • VIGOR-MPC completed the 3D racing lap in 7.82 s at horizon 6, versus 7.62 s for MPCMT*,NL at horizon 25 and 9.97 s for MPCNS at horizon 6.Figure 19 visualizes the racing application and reports the lap times.Source: §F.4 · pp. 28–29, Figure 19 · p. 29

Reference Tracking in 3D

  • For 3D reference tracking, the method connects reference points with trajectory segments and guides a quadrotor between pillars to stop at the final point.The task and its simulated execution are shown in the middle and right images of Figure 19.Source: §F.4 · p. 29, Figure 19 · p. 29

Related papers

Loading 2610.20333v1…