TerrainForge: Physics-Grounded road geometry Editing for Counterfactual Autonomous Driving

1 Introduction

• Road geometry and surface friction alter wheel loading and tire forces, changing vehicle motion, camera viewpoint, and the clearance between vehicles.Different speeds or encounter times can produce different responses to the same road feature.Source: §1 · p. 1
• TerrainForge replays recovered vehicle controls on edited roads, recomputing trajectories and camera motion while holding initial states and controls fixed for paired comparisons.The framework uses reconstructed driving episodes and supports crests, sags, speed humps, and surface-condition changes.Source: §1 · p. 2, Figure 2 · p. 3
• 1.52 m for crests and 1.41 m for sags were the median peak ego-lead gap discrepancies when surrounding vehicles kept recorded motion; held-out terminal-gap prediction had 22–40% lower mean absolute error than predicting no change.The multi-vehicle comparison covers 18 episodes; the surrogate result concerns held-out scenes.Source: §1 · p. 2
• A shared road field drives Gaussian-scene deformation and coupled four-wheel vehicle dynamics, keeping edited terrain and multiple vehicles physically synchronized.Updated vehicle poses and camera motion also drive rendering.Source: §1 · p. 2, Figure 2 · p. 3
• Intervention–response pairs train a first-stage surrogate to screen candidate road edits cheaply before full multi-vehicle rollout.It uses original driving context and edit parameters to predict ego terminal-gap changes.Source: §1 · p. 2

Controllable driving-world generation

• Controllable driving-world generation uses structured scene representations to specify traffic and road structure [Hu et al., 2023, Wen et al., 2024, Gao et al., 2024, Li et al., 2024, 2025, Rowe et al., 2025, Li et al., 2026], while TerrainForge treats road modification as a geometric intervention.Recent methods also support substantial changes to road layouts and rare scene configurations.Source: §2 · p. 3

Physics-aware driving-world generation

• Physics-aware driving-world models condition image or video generation on trajectories, motion priors, occupancy, or simulator states [Yang et al., 2024, 2026, Zhou et al., 2026b, Rahimi et al., 2026], but rarely model vehicle response to road changes explicitly.Their emphasis is mainly geometric and kinematic consistency.Source: §2 · p. 3

Reconstruction-based approaches

• Reconstruction-based methods couple photorealistic 3D scenes with simulators or physics models [Huang et al., 2026b,a, Zhou et al., 2026a]; TerrainForge applies a continuous road model across recorded multi-vehicle episodes.muSync-GS couples road elevation and weather-dependent friction to ego-vehicle dynamics and camera motion [Chen et al., 2026].Source: §2 · p. 3

Vehicle dynamics and multi-vehicle simulation

• Vehicle simulators model tire-road interaction and coupled motion [Dosovitskiy et al., 2017, Gulino et al., 2023, Kazemkhani et al., 2024, Zhu et al., 2026, Mechanical Simulation Corporation, 2026], but generally lack recorded-scene appearance and context.TerrainForge embeds continuous road interventions and vehicle dynamics in reconstructed real-world episodes.Source: §2 · p. 3

3.1 Overview and counterfactual formulation

• Each paired counterfactual run fixes initial states, controls, timestamps, and horizon while changing only declared road variables.The primary setting replays controls inferred from recorded motion; prescribed-control and speed-tracking variants are secondary experiments.Source: §3.1 · p. 4

3.2 Unified road model

• The unified road model applies the same intervention field to vehicle dynamics and Gaussian-scene deformation, with ego motion determining the counterfactual camera trajectory.Each wheel queries the shared road surface, gradient, and friction; vehicles traveling in opposite directions experience the corresponding reversed longitudinal grade.Source: §3.2 · p. 4

3.3 Factual Grounding and Coupled Vehicle Rollout

• Vehicle controls are inferred from recorded motion on the source road and frozen for replay; the coupled model represents planar motion, heave, pitch, roll, wheel dynamics, and tire forces.Multi-vehicle episodes replay each vehicle’s own plan-view path, exercising longitudinal and vertical coupling rather than lateral response.Source: §3.3 · pp. 4–5

3.4 Physics-synchronized scene rendering

• The edited road deforms road and sidewalk Gaussians, while counterfactual vehicle poses and calibrated ego-camera motion drive a joint re-rendering rather than a warp of recorded frames.Clearance and terminal gap are computed from metric counterfactual poses, not rendering-only contact and depth corrections.Source: §3.4 · p. 5

3.5 Surrogate prediction

• The first-stage surrogate screens edits from ego-only rollouts; selected candidates proceed to full multi-vehicle rollout for the final result.It predicts terminal-gap and minimum-clearance changes plus consequential-edit probability from pre-simulation features; training and evaluation use scene-disjoint splits.Source: §3.5 · p. 6

4.1 Vehicle-model validation against CarSim

• On four vertical-alignment holdouts, speed RMSE is 0.004–0.007 m/s and pitch RMSE is 0.051–0.061°; steered-holdout lateral-position RMSE is 0.035–0.11 m.Normal-load extrema differ by less than 0.002 of static load.Source: §4.1 · p. 6

4.2 Real-road validation of the recovered road profile

• Against an independent LiDAR ground estimate, recovered road profiles have 0.14° error versus 0.45° for a flat-road assumption.On the held-out split, world-frame pitch RMSE is 1.03°/0.24°/0.26° for flat, kinematic, and half-car predictions, respectively.Source: §4.2 · p. 6

4.3 Scene realization and multi-vehicle counterfactual propagation

• Across 18 usable ego–lead episodes, ego-only replay differs from all-vehicle propagation by median peak gap changes of 1.52 m for crests and 1.41 m for sags.Across ten Waymo scenes, 88 curves and 30 humps have median apex-depth and curvature errors of 0.17% and 0.05%, with 1.2 mm surface RMS; 257/266 traversing vehicles respond, while all 87 non-traversers remain unchanged.Source: §4.3 · pp. 6–7

4.5 Screening candidate road edits

• Held-out terminal-gap MAE is 0.27 m for sags and 0.57 m for humps, 40% and 22% lower than predicting no change; consequential-edit AP is 0.85.Training used 15,758 edits over 983 braking episodes, with only the ego re-simulated.Source: Table 2 · p. 8, §4.5 · p. 8

4.4 Comparison with Video Editing Models

• No text-only, projected-line, or road-only-depth output matches the target; whole-frame depth passes once on Cosmos (RMSE 0.10 m, correlation 0.98) but alters off-road content by 44.6/255.Camera-trajectory consistency is used as a necessary condition for realizing the prescribed edit.Source: §4.4 · pp. 7–8

• TerrainForge applies a shared road intervention to vehicle dynamics, scene geometry, and camera motion in reconstructed driving episodes.Source: §5 · p. 8
• Surrounding-vehicle responses can change the estimated ego–lead interaction under a road edit.Source: §5 · p. 9
• A scene-disjoint surrogate predicts terminal-gap changes to support inexpensive screening of candidate edits.Source: §5 · p. 9

A Additional method details

• Continuous alignments use parabolic vertical curves and compact features use raised-cosine profiles; the authored profile is shared across dynamics, scene deformation, camera motion, and geometry verification.The compact feature returns to zero height and grade at its footprint endpoints.Source: A Additional method details · p. 11
• At each integration substep, a 32-state four-wheel model queries contact height, gradient, and friction, coupling tire and suspension forces to vehicle motion.The state includes planar motion, heave, pitch, roll, unsprung masses, tire slip, and wheel rotation.Source: A Additional method details · p. 11
• Road and sidewalk surfels follow displacement and rotate to the new surface normal, supported objects move according to footing, and unrelated or out-of-corridor geometry remains fixed.Surface-resting rigid objects are also rotated about their footing to avoid penetration or floating.Source: A Additional method details · p. 12
• Tracked vehicles are reconstructed once as rigid objects rather than re-estimated frame by frame, which the authors say avoids floating, popping, and flicker.Vehicles below the 120-pixel minimum mask-size threshold are not reconstructed or rendered.Source: A Additional method details · p. 12
• Metric counterfactual poses advance recorded paths by edited-minus-baseline rollout changes, while separate display corrections seat vehicles on the reconstructed road and adjust depth.Safety quantities use metric poses, not display corrections; rendering depth factors are measured on the unedited render and reused across cases.Source: A Additional method details · pp. 12–14
• Weather appearance and friction are authored separately: friction changes vehicle response, while weather is an optional appearance layer applied to finished geometry and motion.Rain and snow rendering use edited-scene frames and rollout camera poses.Source: A Additional method details · pp. 11, 14
• Safety evaluation uses metric-world planar boxes, checks contact every 5 ms, and measures terminal gap at the stop time or the common horizon endpoint.Post-contact response is not modeled; colliding rollouts are labeled by contact time and relative speed.Source: A Additional method details · pp. 14–15
• Intervention candidates are retained even when ineffective, harmful, or unsupported, but final labels come from the full coupled rollout after validity checks over the supported horizon.The checks include collision, low clearance, wheel contact, friction utilization, road support, and numerical convergence.Source: A Additional method details · pp. 15–16

Vertical alignment

• All four unseen vertical-alignment profiles pass every gate, including sustained grades, a crest, a SAG, and a speed-hump holdout.The crest and SAG coast from 50 km/h without braking or steering, isolating the road input's effect.Source: B Additional CarSim results · p. 16
• On crest and SAG cases, full-trace total-pitch RMSE is 0.058° and 0.051°, respectively; removing the pre-event static offset reduces it to 0.012° and 0.016°.The model reproduces CarSim's total normal-load extremes within 0.002 of static.Source: B Additional CarSim results · p. 16

Lateral response

• The three steered holdouts have lateral-position RMSE of 0.03–0.11 m and heading RMSE of at most 6.3 mrad, with all gates and 36 convergence checks passing.The holdouts were evaluated after model parameters were frozen; the front lateral tire-force scale was fitted on two development brake-turn cases.Source: B Additional CarSim results · p. 16, Table 7 · p. 17

Residual biases

• On the sinusoidal-steer holdout, yaw-rate NRMSE is 0.032, within its 0.20 limit but higher than on the brake-turns.Steady-state yaw rate and lateral acceleration are 3–4% above CarSim, and end drift is 0.02–0.15 m after 9.2 s.Source: B Additional CarSim results · p. 16
• Roll angle is 11% below CarSim, and roll per unit lateral acceleration is 14% lower, locating the reported bias in the roll submodel rather than the tire.The four scene-supported SAG cases pass all integrity and dynamics gates, including lateral-state checks, 192 initialization gates, and 72 timestep-convergence gates.Source: B Additional CarSim results · p. 17

SAG stress and the reverse-motion boundary

• Five of six additional SAG stress cases pass every gate; the sixth fails after both simulators fall below the 0.5 m/s guard and CarSim reverses while TerrainForge holds at rest.The resulting speed RMSE is 0.157 m/s against a 0.12 limit, and sideslip RMSE is 1.003 rad against a 0.015 limit; the authors identify this as a model-boundary mismatch, not a numerical failure.Source: B Additional CarSim results · p. 17

Protocol

• Measured speed and accelerations condition the inherited half-car rollout, which predicts body pitch and heave against OXTS signals without time alignment or ZOD parameter fitting.The evaluation uses 96 recording-balanced events from 13 recordings and compares flat-road, kinematic-camera, and recovered-road half-car predictions.Source: C Additional ZOD results · p. 17

ZOD prediction results

• On the pre-declared ZOD test split, the full model's world-frame pitch RMSE is 0.13–0.47°, compared with 0.31–2.30° for the flat baseline.The test split contains three recordings; the full model uses the recovered road, and no parameter is fitted on ZOD.Source: C Additional ZOD results · p. 18, Table 8 · p. 18

Results by event type

• Road input removes most world-frame pitch error where the road has vertical curvature, while road-relative pitch and heave predictions are close across methods and event types.The flat baseline is never worse than the full model for road-relative pitch and heave; all methods have higher errors on harder event types.Source: C Additional ZOD results · p. 18

Independent LiDAR check

• Against LiDAR road pitch, the OXTS-recovered road has 0.14° error versus 0.45° for the flat-road assumption, a 68% reduction.LiDAR is available for 79 of 96 events; the comparison uses a fixed ground-patch and thresholds, and the authors note the LiDAR ground-fit noise is about 0.14°.Source: C Additional ZOD results · p. 18

Motivation

• The geometry path is checked in the scene itself because the dynamics and camera read the authored profile analytically, while the deformed Gaussian scene results from attachment and deformation rules.Rendered-pixel alignment with the profile is examined separately through the camera in the following section.Source: D Additional geometry results · pp. 18–19

Setup

• The test applies symmetric crests and SAGs plus 50, 75, and 100 mm humps across ten Waymo scenes, without per-scene tuning; curves that do not fit with 10 m clearance at both ends are left unaltered.The evaluated set contains 88 vertical curves and 30 humps; the measured profile is recovered from road-Gaussian heights binned along the route.Source: D Additional geometry results · p. 18

Results

• Across 88 vertical curves, apex-depth error has a 0.17% median and 3.5% maximum, while vertical-curvature error has a 0.05% median and 2.8% maximum.Across all curves, grade error is at most 0.06 percentage points; all 118 edits were deformed and measured without failure or warning.Source: D Additional geometry results · p. 19
• Every road Gaussian in the lateral core of a vertical-curve edit is displaced to within 0.06 mm of the authored amount; humps have zero residual within their footprint, and no Gaussian outside it moves.The median RMS residual surface error for vertical curves is 1.2 mm, with a 2.5 mm maximum; the authors identify it as the measurement floor.Source: D Additional geometry results · p. 19

Population and selection

• The bank selects 20 episodes from 394 eligible episodes in a population of 3,791 audited Waymo training-split episodes, with eligibility requiring a sufficiently long ego route and at least four moving vehicles.Episode eligibility also requires a serialized package, geometric audit, factual ego–lead association, and exclusion from other experiments.Source: §E.1 · p. 19

Vehicles and validity

• The 353 valid vehicles reproduce their factual baseline exactly on a zero edit, but pedestrians, cyclists, and parked vehicles are not simulated.Of 387 admitted vehicles, 34 are refused before edits because their recorded tracks stop and restart, which the model cannot relaunch from rest.Source: §E.1 · p. 19

Response definition

• Vehicles count as traversing when an axle query enters the edit support, and as responding when peak speed or total-normal-load change reaches its stated threshold against that vehicle's flat-road rollout.Response thresholds are 0.05 m/s for peak speed change or 0.005 of static total normal load; the response onset uses the 0.1 s output clock.Source: §E.1 · p. 19

Pairing and edit

• Each episode pairs the ego and factual lead under edited and flat-road rollouts; the ego-only estimate keeps the lead on its flat-road motion, and disagreement is the maximum framewise difference between the two gap changes.The edit arms use symmetric crests or sags at 2% and 8%, with friction fixed at μ = 1, zero steering, and replayed plan-view paths.Source: §E.1 · p. 19

Results

• The ego–lead gap disagreement exceeds 0.5 m in every episode at both edit severities; minimum-gap changes are smaller and absent in 8 of 18 episodes whose closest approach precedes the edit.The bank reports 18 valid ego–lead pairs in every arm after excluding one episode where an edited vehicle was latched at rest.Source: §E.1 · p. 20, Table 9 · p. 20

E.2 The reconstructed episode 109409

• Each of 17 moving tracked vehicles besides the ego is grounded against its own recorded trajectory, while replayed plan-view paths isolate longitudinal and vertical coupling rather than lateral response.The identified wheel torque is then frozen, and vehicles use their own recorded path distance when encountering the edit.Source: §E.2 · p. 20
• Among vehicles reaching the edit, speed changes reach 6.2 km/h for the crest and 5.6 km/h for the SAG, while path-distance changes reach 7.3 m and 5.8 m.Vehicles not reaching the edited stretch have exactly zero speed and path change.Source: §E.2 · p. 20
• The differences are 0.009 and 0.011 of static weight; remaining differences occur at curvature steps, where the model spreads loading over the wheelbase instead of applying it at the center of mass.Source: §E.2 · p. 21
• With all moving vehicles responding, the crest shortens the along-path gap by up to 3.4 m and the SAG lengthens it by up to 3.0 m; ego-only replay reverses the sign (+5.3 and −4.7 m).The minimum gap changes by less than 0.5 m from a 30.1 m flat-road value.Source: §E.2 · p. 21

E.3 A prescribed-input case

• Figure 16 tests the same braking command on four roads toward a static lead vehicle, with brake onset chosen to expose the braking mechanism.Source: §E.3 · p. 21

Protocol and measurement

• The text-only protocol tests three generators on three scenes, while camera consistency is measured from recovered motion separately from SSIM source preservation.Trajectory thresholds are fixed using ten controls; projected lines and target depth are tested as additional geometry-conditioning inputs.Source: F Generic-generator comparison:… · pp. 21, 22, 24

Results: text only

• For the Scene 22 text-only sag, VACE nearly copies the input while Cosmos and LTX regenerate the street, and none moves the horizon; crest and grade requests show the same pattern.TerrainForge’s authored sag and crest produce far-field shifts linear in road pitch, with 8.8 px per degree and correlation 0.97.Source: F Generic-generator comparison:… · p. 22, Table 10 · p. 23

Further scenes

• Across three further scenes, generator far-field motion is at most 1.9 px, versus 29–47 px for TerrainForge at the steepest stations.For this comparison, VACE changes the lane region beyond adjacent-frame differences, so the copy label does not transfer.Source: F Generic-generator comparison:… · p. 22, Table 12 · p. 24

Results: conditioned generators

• No text-only, projected-line, or road-only-depth video matches the target camera trajectory; whole-frame depth does so only at weight 0.6, while the preselected fidelity setting of 1.0 and weight 0.3 fail.At weight 0.6, recovered camera height has RMSE 0.10 m, correlation 0.98, and peak-to-peak ratio 1.04; the trajectory rule is met by neither the fidelity-selected setting nor weight 0.3.Source: F Generic-generator comparison:… · pp. 22, 24, Table 11 · p. 24

Conditioned generators

• At weight 0.6, the generator meets the camera-trajectory criterion but changes surrounding appearance, with outside-road difference 44.6/255 versus 4.5–15.5/255 for text-only outputs.This result indicates that a generator can texture a scaffold that already carries authored geometry.Source: F Generic-generator comparison:… · p. 22

Our arm

• TerrainForge’s authored arm deforms reconstructed scenes with prescribed profiles and renders from a camera following road height and pitch on the original station schedule.The sag, crest, and flat frames are earlier renders; uphill and downhill frames use current code, with the sag re-render differing by 1.4/255 on average.Source: F Generic-generator comparison:… · p. 23

Vehicle response of the sag and crest

• For the crest and sag, body pitch reaches 5.2° in magnitude against 4.6° road pitch, while total normal load remains 0.94–1.06 of static weight.Speed differs from the flat road by about ±1 km/h because the driver tracks the recorded speed.Source: F Generic-generator comparison:… · p. 24

Sustained grades

• Under the same driver intent, sustained uphill and downhill grades change speed by up to −6.7 km/h and +3.7 km/h relative to flat, although their early far-field shifts are only +3 and −2 px.At 36 m, far-field shifts grow to +26 and −22 px as the camera rises or falls by up to 1.9 m.Source: F Generic-generator comparison:… · p. 24

Protocol

• The primary analysis was restricted to scenes whose baseline keeps vx ≥1 m/s through the edit window, because the model cannot relaunch a vehicle brought to rest; all 200 scenes were run either way.An identity phase on 20 scenes reproduced the bank records exactly across all 11 scalars (|diff| = 0).Source: G Lateral response to a road-geometry… · p. 25
• The edit is a bounded symmetric 60 m crest or sag, with K values of 7.5, 3.75, and 1.875 m per percent at grades 2%, 4%, and 8%; only 2% meets the internal plausibility threshold.Source: G Lateral response to a road-geometry… · p. 25

Full results

• Across both friction levels, response quantities change monotonically with edit strength; halving µ doubles tire-utilisation change while leaving kinematic and load changes unchanged.Curve-versus-control separation appears in lateral acceleration, not load quantities; crest unloading follows v^2H′′/g without lateral excitation.Source: G Lateral response to a road-geometry… · p. 25
• Of 1,200 edited runs at each friction level, 48 runs raise a hard-failure flag; 141 of 142 total flag instances are route-support exits, and one is tire saturation at µ = 0.4.Flag counts grow with edit strength and are the same at both friction levels.Source: G Lateral response to a road-geometry… · p. 25

Obstacle results

• The change in terminal gap is predicted with 22–40% lower error than predicting no change, with rank correlation 0.81–0.89.The test table reports MAE in meters over held-out edits and Spearman rank correlation pooled over test edits.Source: Table 14 · p. 26, H Additional screening results · p. 26

Screening setup

• The obstacle bank contains 15,758 edits from 983 braking episodes, re-simulating only the ego while replaying the recorded target.The obstacle evaluation protocol, including models, baselines, and metrics, was set before fitting on the combined banks; test intervals use scene-level bootstrap resampling.Source: H Additional screening results · p. 26

Physics prior

• The closed-form prior estimates terminal-gap effects from elevation-induced speed changes under an equal-retarding-force assumption, but omits wheel unloading over a hump.It ranks SAG edits at 0.68 but does not estimate their size correctly; it carries no information about hump effects.Source: H Additional screening results · pp. 26–27

Obstacle results

• AP 0.85 for consequential-edit detection exceeds 0.20 for random ranking and 0.35 for geometry alone; reviewing the top-ranked fifth recovers 80% of effective edits.Detection AP rises from 0.78 to 0.85 as training scenes increase from 25% to 100%, with no plateau yet.Source: H Additional screening results · p. 26, Table 15 · p. 27

Lateral results

• R2 is 0.76–0.98 for ten of eleven lateral peak quantities and 0.58 for path deviation with the better trajectory-based learner; static descriptors reach at most 0.58.The sequence model significantly outperforms the summary learner on five quantities, underperforms on two, and is indistinguishable on four.Source: H Additional screening results · pp. 25, 27, Table 16 · p. 27

Obstacle results

• Minimum-clearance change is not predicted better than no change; most edits leave the closest approach unchanged, and no model beats that baseline.The fixed squared-error regressor also fails to beat no change for terminal-gap prediction, with MAE ratios of 1.27 for SAG and 1.49 for humps.Source: H Additional screening results · p. 27