PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Perception Representations for Off-road Dynamics Learning

Micah Nye, Samuel Triest, Matthew Sivaprakasam, Isaiah Adu, Sebastian Scherer, Wenshan Wang

arXiv:2610.22900v1cs.RO

TL;DR

High-speed off-road dynamics models must account for terrain interactions, but learned models often underuse environmental observations and their spatial structure. The paper queries BEV features around predicted states at each step and evaluates performance offline and on hardware. It reports reduced prediction error and improved worst-case tracking performance in the evaluated settings.

  • Problem

    High-speed off-road control needs accurate dynamics models, yet terrain interactions and environment-dependent trajectory variation are difficult for simplified or state-only models to capture.

    Source: §1 · pp. 1–2

  • Method

    SpatialBEV conditions recurrent dynamics predictions on geometric and semantic BEV features queried around predicted states at each step.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Dense-map SpatialBEV reduces prediction error versus the strongest baseline by 17.1% on Full, and has the lowest worst-case CTE on the geometry course.

    Source: §5.1 · p. 6, §5.3 · p. 8

  • Takeaways & Limitations

    The paper concludes that perception representations should be part of off-road dynamics models and that added spatial structure further improves modeling on high-speed and geometrically challenging courses.

    Source: §7 · p. 9

    Map querying and interpolation add computation: the model takes 28.3 ms for 825 particles versus 12.9 ms for vanilla BEV, leaving little room for more samples or a longer horizon.

    Source: §6 · p. 9

Abstract

from arXiv · show

Autonomously navigating off-road terrain at high speed demands accurate control, which often relies on dynamics models that capture complex interactions between the robot and terrain. In off-road regimes, terrain interaction is a major component of the vehicle's dynamics. Prior work attempts to capture this effect by training learned models of the vehicle's dynamics, conditioned on its perception of the local environment. However, said conditioning is often overly simplistic, ignoring the spatial structure of the dynamics problem. Our approach, SpatialBEV, exploits this spatial structure. At each step, we query the bird's-eye-view (BEV) features around the predicted state, conditioning the next prediction on current geometric and semantic cues. We additionally present a benchmarking approach that mines a dataset's most difficult scenarios. On datasets collected across seasons, SpatialBEV reduces prediction error by 17.1%, compared to physics-based and learned baselines. We further evaluate on hardware on high-speed ($\geq$10 m/s) and geometrically challenging courses with a 25.3% reduction in maximum cross-track error. More details can be found at https://theairlab.org/spatial-bev-dynamics .

1 Introduction

High-speed off-road MPC depends on accurate dynamics models, but simplified physics and state-only learned models do not capture terrain-driven variation well. The paper proposes spatially structured perception conditioning and a difficult-scenario testing methodology.

1 Introduction

  • High-speed off-road navigation challenges MPC because simplified physics models struggle to capture complex terrain interactions that may lead to task failure.Learned state-only models can represent differing trajectories as uncertainty, although the paper characterizes that uncertainty as epistemic because environmental physics are unobserved.Source: §1 · pp. 1–2
  • State-only learned dynamics models often neglect environmental observations, leaving terrain-driven trajectory variation as epistemic uncertainty.Identical actions and initial conditions can produce different trajectories on surfaces with different geometry or friction.Source: §1 · pp. 1–2
  • The paper conditions predictions on geometric and visual-semantic BEV cues at every step, rather than using BEV as a single fixed prior.It also proposes clustering similar actions to identify high-variance scenarios where environment context may be needed for accurate prediction.Source: §1 · p. 2

2. A benchmarking approach to mine the most challenging sequences of dynamics datasets

The paper evaluates its approach through offline experiments and online hardware experiments at high speeds.

2. A benchmarking approach to mine the most challenging sequences of dynamics datasets

  • The paper reports extensive offline and online hardware experiments at speeds of at least 10 m/s to demonstrate the approach’s efficacy.The contribution passage identifies both offline experiments and online hardware experiments at high speeds.Source: §2 · p. 2

2 Related Works

Prior dynamics work spans physics-informed and learned models, with perception inputs added in several forms. The paper distinguishes its approach by using queried BEV context for both state prediction and correction.

2 Related Works

  • Physics-informed tire models offer structured learning but depend on physical assumptions, while residual learning supplements a physics model and retains dependence on its accuracy.The cited work addresses nonlinearities from undulated terrain by learning force residuals.Source: §2 · p. 2
  • Learned dynamics use direct forward-state prediction, and perception inputs have included FPV images, point clouds, and several BEV map types.Most prior approaches treat perception as fixed context encoded once at rollout start, while Lee et al. use a BEV crop.Source: §2 · p. 2
  • Figure 2 depicts geometric and semantic BEV features queried at prior or predicted positions during each rollout step.The caption describes the queries as providing context for dynamics rollouts.Source: Figure 2 · p. 3
  • Unlike Gibson et al., who use queried tire-point features only as a physics residual, this approach uses BEV context for state prediction and correction.The paper also formalizes the spatial representation and evaluates it through ablations and quantitative and qualitative experiments.Source: §2 · p. 3

3 Method

The method combines a proprioceptive recurrent dynamics backbone with BEV features queried along predicted motion. Geometric sampling supplies localized context to recurrent cells, with the sampling footprint as the platform-specific component.

3 Method

  • The method is designed to leverage perception for predicting forward motion over challenging terrain before the robot traverses it.The overview presents the proprioceptive backbone and structured BEV representation as the two method components.Source: §3 · p. 3

3.1 Proprioceptive Dynamics Model

  • The dynamics backbone uses recurrent predictor and context networks, allowing the spatial representation to be inserted between recurrent cells.The RNN is chosen over a Transformer for sequential recurrence and a small memory footprint.Source: §3.1 · p. 3
  • The predictor estimates state differences and adds them to the current state, localizing its learning target near zero.A context network initializes the predictor’s hidden state from state and action history and trains shared state and action encoders.Source: §3.1 · p. 3
  • The method augments proprioceptive dynamics with local observations because undulating terrain and variable ground cover require information about terrain ahead.The model’s state transition also takes prior state-action context and a local observation, such as a BEV map, as inputs.Source: §3 · p. 3, §3.1 · p. 4

3.2 Structured Perception Representations for Environment Context

  • The method embeds terrain height, surface normals, and visual foundation-model features in structured BEV maps, built from LiDAR geometry and FPV imagery.Prior fixed-context approaches do not fully exploit BEV spatial structure, motivating localized context at each prediction step.Source: §3.2 · p. 4
  • The recurrent BEV cell samples map features over the predicted state’s tire-track footprint, then uses that context to correct the hidden state and augment the next cell’s input.The context network instead queries using known state history and omits the correction step.Source: §3.2 · p. 4, Figure 3 · p. 4
  • The footprint-based sampling is a spatial-attention variant whose grid is fixed by geometry; replacing the footprint can adapt it to other platforms.The paper identifies quadrupedal, bipedal, and tracked contact patterns as alternatives to the wheeled footprint.Source: §3.2 · p. 4
  • Figure 4 contrasts trajectories for similarly clustered actions: the downhill case travels farther, flat dirt moves least, and low-friction ice travels farther on flat ground.The caption attributes the differences to terradynamics highlighted by perception.Source: Figure 4 · p. 5

4 Benchmarking

The benchmark targets difficult off-road interactions that aggregate dataset metrics can obscure. It ranks groups of similar samples by trajectory variance and evaluates on the highest-ranked groups.

4 Benchmarking

  • Aggregate metrics are dominated by benign interactions and can mask long-tail failures that matter for real-world safety.The benchmark mines the dataset for state sequences models are most likely to mispredict.Source: §4 · p. 5
  • Candidates are grouped by L2 distance between normalized action sequences and initial conditions, with the threshold chosen by mean silhouette coefficient.The resulting graph is partitioned into cliques; mean pairwise Hausdorff distance across each group’s trajectories measures variance and difficulty.Source: §4 · p. 5
  • Dhard is the union of the top p percent of groups ranked by trajectory variance, while errhard evaluates the unchanged error metric over Dhard.Comparing errhard with errD can reveal long-tail failures that aggregate evaluation conflates with benign data.Source: §4 · p. 5

5 Experiments

Experiments assess structured BEV perception, long-tail benchmark mining, and hardware control across seasonal datasets and challenging courses. Together, the evaluations test prediction accuracy and its relationship to closed-loop performance.

5.1 Effect of Structured Perception Representations

  • Vanilla BEV activations fall on irrelevant map regions, while SpatialBEV queries follow its prediction near the ground-truth path.The activation maps visualize the difference between the two representations.Source: Figure 5 · p. 6, §5.1 · p. 7
  • 17.1% lower full-set prediction error with dense maps than the strongest baseline; dense-map SpatialBEV outperforms all baselines across the datasets.The reported reductions versus the strongest baseline are 17.1% on Snow, 8.5% on Slopes, and 19.9% on Speed.Source: §5.1 · p. 6, Figure 6 · p. 7, Table 1 · p. 7
  • The recurrent BEV correction reduces full-dataset error by 22.4% with sparse maps and 4.7% with dense maps, with error reduced in all but one dataset-map setting.The reported reductions reach 30.8% on Snow with sparse maps and 12.1% with dense maps.Source: §5.1 · p. 7, Table 2 · p. 7
  • Tracks is the best footprint across sub-datasets and reduces full-dataset error by 14.2% versus the next-best footprint.Its dense tire-path coverage matters most at speed, when the robot moves farther than the map resolution per step; broader Full Body and Lookahead footprints also sample away from the tire path.Source: §5.1 · pp. 7–8

5.2 Mining Long-tail Failures

  • Worst-case prediction error declines from the highest-variance decile to the lowest, and FPV and State baselines have much higher error on the hardest partition.The deciles are formed by sorting trajectory variance, with the highest-variance runs in the first decile.Source: Figure 7 · p. 8, §5.2 · p. 8

5.3 Closed-loop Control on Hardware

  • On hardware, prediction error was 16.6% lower on average than the best baseline, but closed-loop performance matched learned baselines and did not always correlate with open-loop error.Dense-map BEV methods could not be run; SpatialBEV had the lowest worst-case CTE on the geometry course.Source: §5.3 · p. 8, Table 3 · p. 8, Table 4 · p. 9

6 Limitations

The paper identifies four limitations: adaptation, data collection, mapping latency, and possible controller-related false interventions.

6 Limitations

  • Online adaptation could further improve robustness and generalization, despite strong performance on held-out datasets and environments.The authors identify adaptation as a potential next step.Source: §6 · p. 9
  • Real-world training avoids a sim-to-real gap but is expensive, sample-inefficient, and vulnerable to hardware failures.Combining simulation and real data is proposed as a natural next step.Source: §6 · p. 9
  • With all components running, SpatialBEV takes 82 ms per rollout, leaving little room for more particles or a longer horizon despite sufficing for 10 Hz control.Vanilla BEV takes 40 ms under the same all-components condition.Source: §6 · p. 9
  • A better line-tracking algorithm may be needed because the current approach produced a false intervention during testing.The authors identify this as a future testing improvement.Source: §6 · p. 9

7 Conclusion

The paper concludes that terrain geometry and visual semantics should be part of learned off-road dynamics models, with spatial structure further improving prediction. Additional experiments report qualitative performance across varied scenarios and expose difficult terrain cases that remain unresolved.

7 Conclusion

  • Structured perception representations make terrain geometry and visual semantics core components of learned off-road dynamics, with spatial structure further improving modeling on high-speed, geometrically challenging courses.The conclusion draws on offline evaluation and online hardware results.Source: §7 · p. 9

A Additional Experiments

  • The additional experiments examine qualitative predictions, Transformer-based alternatives, and seed ablations.The section provides qualitative and quantitative evaluation beyond the main experiments.Source: A Additional Experiments · p. 16

A.1 Open-loop Qualitative Analysis

  • Across snow, ice, marshes, slopes, and varied maneuvers, SpatialBEV consistently predicts trajectories accurately at low, medium, and high speeds.On a slight snowy turn, all perception-based methods perform comparably well, indicating appearance cues alone can provide enough context.Source: §A.1 · p. 16
  • On uphill scenarios, SpatialBEV stays closest to ground truth, but no method predicts the ice-rut event that constrains the vehicle’s motion.In the ice case, all methods overshoot toward the left; on the uphill, SpatialBEV slightly undershoots.Source: §A.1 · p. 16

A.2 Extended Open-loop Results

Extended offline results compare Transformer-based dynamics models across datasets, while seed ablations measure variability. Transformers perform unevenly across datasets, and the structured representation has its clearest advantage on Speed trajectories with large displacements.

A.2 Extended Open-loop Results

  • The extended evaluation adds Transformer architectures to the LSTM-based comparisons, including a state-only Transformer, a BEV Transformer, and AnyCar.AnyCar is trained on the same data as the LSTMs, but predicts 5-second sequences at 10 Hz rather than 1 second at 50 Hz.Source: §A.2 · pp. 16–17
  • Transformer baselines achieve low prediction error on Snow and Slopes but degrade on Speed; the BEV Transformer performs worst across nearly all methods and datasets.The authors suggest explicit sampling grids are more effective spatial attention, while noting the Transformer architecture is relatively simple and could improve.Source: §A.2 · p. 17, Table 5 · p. 17
  • On Speed trajectories with large displacements, SpatialBEV achieves the best average and worst-case performance, supporting the proposed value of explicit spatial structure.Table 5 extends the offline position-prediction results across three datasets and the full test set.Source: §A.2 · p. 17, Table 5 · p. 17

A.3 Seed Ablations

  • Across four random seeds, prediction error has a standard deviation of 0.07 m2.The authors report this as a seed ablation.Source: §A.3 · p. 17

B Dataset Details

The appendix documents the real-world dataset, BEV perception pipeline, MPPI control setup, and hardware used for evaluation. Together, these details specify the terrain coverage, map construction, controller costs, and deployment constraints.

B Dataset Details

  • The real-world dataset covers 16.3 hours across two sites and all four seasons, with terrain ranging from loose gravel and snow to ruts and steep slopes.Top speed is 13 m/s, and maximum pitch is 19.1° (32.2% grade).Source: B Dataset Details · p. 17, Table 6 · p. 18
  • Test subsets isolate Snow, Slopes, and Speed using complete snow cover, maximum pitch of at least 0.1 radians, and average speed of at least 6.0 m/s, respectively.Post-processing removes reverse motion, high-odometry-covariance samples, and braking samples because braking is outside the robot’s action space.Source: B Dataset Details · p. 18

C.1.1 Perception

  • The mapping pipeline extracts camera features, projects them onto LiDAR points, and aggregates the resulting feature point cloud into a scrolling voxel grid.The mapping flow is shown in Figure 10.Source: §C.1.1 · pp. 18–19, Figure 10 · p. 18
  • A Markov random field estimates the terrain surface; geometry and visual foundation-model features are then computed relative to that surface.Maps run at 5 Hz on the onboard GPU, with ego-alignment and elevation limiting applied before inference.Source: §C.1.1 · p. 19

C.1.2 Control

  • MPPI uses the learned dynamics models and samples candidate actions from Gaussian walks with varying noise plus 441 uniformly sampled throttle-steer sequences.The action sequences are evaluated at every MPPI iteration.Source: §C.1.2 · p. 19
  • The MPPI cost combines speed and lateral-acceleration limits with path tracking and progress, while the authors note that more tuning could reduce controller-related failures.The path computations are in 2D, and the tracking weight is set higher than the progress weight to discourage corner-cutting.Source: §C.1.2 · pp. 19–20

C.2 Hardware

  • The vehicle has steering and throttle but no brake; terrain resistance and steering-motor limitations cause an approximately 5-second delay from maximum-steer command to maximum steer.The model implicitly learns this steering delay.Source: §C.2 · p. 20
  • The vehicle uses LiDAR and an RGB stereo camera at 10 Hz, RTK localization at 25 Hz, and a laptop with an RTX 5080 GPU for perception and control.The laptop also has a 24-core Intel Core Ultra 9 275HX CPU and 32 GB RAM.Source: §C.2 · p. 20

C.3 MPPI Failure Analysis

The failure analysis distinguishes a control failure from a dynamics-modeling failure. The appendices describe the perception encoding and tire-track footprint used by the spatial method.

C.3 MPPI Failure Analysis

  • MPPI’s intervention was attributed to cost-function tuning, most likely a low tracking-cost weight and sparse action sampling.The selected path undercut the target and approached dense vegetation after 2,000 ms of following those solutions.Source: §C.3 · pp. 20–21, Figure 11 · p. 21

D.1 Network Architecture

  • Vanilla perception methods encode perception once into a hidden state that initializes the context LSTM alongside the state latent.The FPV and BEV LSTM encoders use CNNs with three and four downsampling blocks, respectively.Source: §D.1 · p. 21
  • The spatial method queries BEV features at the current predicted state and uses the encoded context to condition the next prediction and correction step.It samples 40 tire-track points and encodes their features with an MLP.Source: §D.1 · pp. 21–22

D.2 BEV Footprint

  • The BEV footprint comprises two tracks, separated laterally by 1 m and spanning 4 × 0.42 m with 10 × 2 points.The method does not explicitly condition on velocity, steering, suspension, or roll; slip and weight transfer are represented implicitly in the recurrent latent.Source: §D.2 · p. 22

D.2.1 Ablation Footprints

The appendices describe alternative footprints for ablation, how activation maps are produced, and the state-estimation sources used for training and deployment.

D.2.1 Ablation Footprints

  • The footprint ablation compares the tracks footprint with 4-point, Full Body, and Lookahead alternatives, each representing a different spatial sampling choice.4-point follows Gibson et al.; Full Body ignores tire locations, while Lookahead covers feasible robot travel to remove prediction-based querying.Source: §D.2.1 · p. 22

D.3 Activation Maps

  • GradCAM activation maps trace the first predicted state’s x position back to the Vanilla BEV CNN to visualize spatial feature importance.The method weights channel activations using pooled gradients, applies ReLU or absolute value, then upsamples the map.Source: §D.3 · p. 23

D.4 Training

  • Training uses SuperOdometry state estimates, while real-world deployment uses RTK because RTK was unavailable throughout dataset collection.The authors note this mismatch highlights robustness to the state-estimation solution.Source: §D.4 · p. 23

Related papers

Loading 2610.22900v1…