HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

Motivation

• Driving scenes couple multiple actors, lane geometries, and infrastructure, so global appearance edits must balance instruction alignment with preservation of local structure.The paper identifies unpaired real-world supervision, unreliable synthetic edits, and conditioning-induced temporal shock as three bottlenecks.Source: §1 · p. 2
• The paper frames its central question as how to distill faithful traffic semantics into driving-scene editing to achieve spatiotemporal consistency.Source: §1 · p. 2

Approach

• HorizonComposer combines temporally aligned real-world videos with synthetic edits, dense pixel-level rewards for offline reinforcement learning, and thinking frames to smooth appearance transitions.The stated aim of the thinking frames is to reconcile static image conditioning with video dynamics without corrupting multi-agent dynamics.Source: §1 · p. 3

Contributions

• Up to 20% improvement in content preservation and downstream gains of 5% in road line segmentation and 7.14% in multi-object detection are reported for HorizonComposer.The authors also report a 21% relative preference advantage over the second-best method overall.Source: §1 · p. 3

Instruction-guided video editing

• General-purpose editors lack native guarantees for traffic semantics, while HorizonComposer adapts intermediate reasoning frames to anchor multi-agent dynamics.The related work distinguishes this use from ChronoEdit’s focus on single-image physics.Source: §2 · p. 3

Driving scene synthesis and editing

• Driving-scene methods often use reconstruction or external simulators; HorizonComposer edits video directly and uses dense pixel-level rewards to preserve ODD structure.The passage notes that existing approaches still struggle to preserve vehicle identities under appearance edits.Source: §2 · p. 3

Reinforcement learning for diffusion

• HorizonComposer extends inference-time guidance with pixel-aligned reward indicators, addressing scalar signals that miss within-frame structural variation.The passage contrasts this with costly on-policy methods and offline weighted regression that collapses to filtered imitation.Source: §2 · p. 3

3.1 Overview

• The framework uses a hybrid dataset for structure and appearance supervision, extracts spatiotemporal rewards from it, and buffers misaligned reference frames with temporal conditioning.The source clip, edited reference image, depth sequence, and target instruction form the editing conditions; the depth branch is frozen while the visual branch is finetuned.Source: §3.1 · pp. 3–4

3.2.1 Real-World Data Harvesting through Structural Alignment

• Pose-based pairing across repeated routes is followed by filtering freeze frames and large timestamp gaps, then converting scene-attribute differences into editing instructions.These real-world pairs retain scene structure but cannot guarantee identical traffic dynamics.Source: §3.2.1 · pp. 4–5

3.2.2 Synthetic Appearance Generation

• Synthetic pairs edit an initial frame with Qwen Image Editing, then propagate the change with Cosmos-Transfer2.5 using depth and visual branch weights of 0.9 and 0.1.The visual weight is kept small because a larger weight can over-anchor to the source appearance and ignore the target edit.Source: §3.2.2 · p. 5

3.3 Semantics Alignment with Fine-grained Reward and Offline-RL Fine-tuning

• Because real pairs have traffic misalignment and synthetic pairs can distort safety-critical semantics, the training approach scores reliability at the pixel level rather than discarding whole videos.The reward maps combine editing metrics and driving-specific semantic checks to use reliable signals while suppressing artifacts.Source: §3.3 · p. 5, Figure 3 · p. 5

3.3.1 Constructing Fine-grained Traffic-Semantic Rewards

• The dense reward map scores content preservation, temporal consistency, and instruction alignment, using scene-change detection, semantic checks, and frame-level metrics broadcast across pixels.Safety-critical elements such as traffic lines, crosswalks, and vehicles are explicitly checked for geometric corruption.Source: §3.3.1 · pp. 5–6

3.3.2 Offline Reinforcement Learning

• Offline RL extends advantage conditioning from a single action-wide indicator to a pixel-aligned indicator derived from reward-map scores, enabling localized improvement in otherwise noisy examples.At inference, the model sets the indicator to 1 globally; the dense formulation requires no architectural change to the underlying objective.Source: §3.3.2 · p. 6

3.4 Thinking Frames for Smoothing Temporal Shock

• Two thinking frames and a temporally reversed prefix of driving controls give generation a warm-up period to reconcile reference-image artifacts with video dynamics.The prepended frames are cropped after denoising, leaving the output at its original length.Source: §3.4 · pp. 6–7

4.1 Settings

• The evaluation measures temporal consistency, ODD preservation, instruction alignment, and photorealism on withheld Boreas, nuScenes, and Waymo videos and geographically non-overlapping nuPlan test images.Metrics include optical-flow-based FF-α and FF-β, actor and background stability, road-line IoU, car-appearance preservation, prompt alignment, and image quality.Source: §4.1 · p. 7

4.2 Main Results on Driving-Scene Video Editing

• Figure 4 contrasts input videos with CCEdit, UniVideo, SyntheticPip, and HorizonComposer; SyntheticPip adds a disappearing vehicle, while HorizonComposer and UniVideo retain “STOP AHEAD.”The figure also reports a hallucinated yellow traffic light from SyntheticPip, hazy artifacts from CCEdit, and distorted realism from UniVideo.Source: Figure 4 · p. 8, §4.2 · p. 8
• HorizonComposer improves in-domain temporal consistency and safety-critical road-marking and car-appearance preservation, and is the only evaluated model maintaining these gains out of distribution.It retains competitive instruction alignment and image quality; SyntheticPip scores higher on some metrics in those dimensions.Source: §4.2 · pp. 7–8, Table 1 · p. 8
• Users preferred HorizonComposer across all four criteria, with relative preference increases over SyntheticPip of 39% in content preservation and 12% in temporal consistency.The reported increases were approximately 9% in instruction following and 30% in photorealism; 429 criterion-level responses were pooled.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Extensions to Local Editing

• Applying HorizonComposer after HorizonForge combines a local ego-motion or lane-change edit with a global appearance shift while retaining the edited traffic layout.Figure 5 illustrates an autumn transformation that preserves the newly edited BEV trajectory and local semantics.Source: §4.3 · p. 9, Figure 5 · p. 9

4.4 Downstream Task: Lane Marking Segmentation

• On SeRM lane-marking segmentation, HorizonComposer-generated data raises mIoU from 37.48% to 40.27% and mAcc from 50.04% to 55.62% versus SyntheticPip-generated data.The method outperforms SyntheticPip across all reported metrics.Source: §4.4 · p. 10, Table 3 · p. 10

4.5 Downstream Task: Multi-Object Detection

• On BDD100K object detection, adding 200 HorizonComposer-generated videos to 200 real videos raises mAP from 0.167 to 0.175, mAP@50 from 0.316 to 0.324, and mAP@75 from 0.154 to 0.165.The two fine-tuning settings start from the same pretrained checkpoint.Source: §4.5 · p. 10

• HorizonComposer combines hybrid real/synthetic supervision, dense reward-conditioned offline reinforcement learning, and temporally robust thinking frames for instruction-guided driving video editing.Source: §5 · p. 10
• The framework achieves strong spatiotemporal consistency while maintaining faithful instruction alignment and improving preservation of safety-critical structures such as lane markings.Source: §5 · p. 10
• Edited videos improve downstream lane-marking segmentation, highlighting their potential for scalable autonomous-driving simulation and data augmentation.Source: §5 · p. 10

A Limitations

• Dense reward guidance is bounded by reward-model accuracy, and noisy evaluations of small, distant, or occluded agents can misdirect improvement signals.The authors suggest joint or iterative training of reward and diffusion models as a possible mitigation.Source: A Limitations · p. 15
• HorizonComposer is designed mainly for global appearance edits; localized editing is supported through composition with existing pipelines, while direct localized editing remains future work.Source: A Limitations · p. 15

B Broader Impacts

• The authors identify high-fidelity, spatiotemporally consistent driving edge cases as a potential way to enrich simulations and train safer autonomous systems without physical testing risks.Source: B Broader Impacts · p. 15

C Additional Videos

• Additional video results are available in the supplementary materials.The demo website is accessible by opening index.html in the unzipped supplementary archive.Source: C Additional Videos · p. 15

D Additional Results

• In qualitative comparisons, HorizonComposer preserves road markings, turn signals, crosswalks, vehicles, and overpass clearance, while CCEdit shows haze and UniVideo distorts realism.Only UniVideo and HorizonComposer preserve the solid white line and turn signal in one example.Source: Figure 6 · p. 16

D.1 Ablation: Robust Supervision & Conditioning

• Dense reward conditioning improves lane and car preservation on ID and OOD splits, while thinking frames yield the highest subject and background consistency and improve FF-α.The full method retains competitive imaging quality and instruction alignment in qualitative and human evaluations, despite a slight automated-metric trade-off.Source: §D.1 · p. 17

E Detailed Related Works

• General-purpose editors lack native guarantees for driving semantics; unlike ChronoEdit’s single-image setting, thinking frames anchor continuous multi-agent dynamics against temporal shock.Source: E Detailed Related Works · p. 17
• The approach uses video-domain editing with hybrid data pairing and dense pixel-level rewards, in contrast to methods relying on explicit 3D/4D reconstruction or sample-level scalar rewards.The cited discussion notes that policy-gradient video sampling is costly and scalar indicators miss within-frame variation in structural reliability.Source: E Detailed Related Works · p. 18

F Real-World Dataset Construction

• Real-world pairs are formed by matching frames from different traversals of the same route using camera pose, then filtering static segments and timestamp gaps before fixed-length segmentation.Matches are accepted only below a pose-disparity threshold; the pipeline also generates prompts from global scene attributes using a VLM.Source: F Real-World Dataset Construction · p. 18
• The VLM prompt requests descriptions of weather, lighting, road layout, and environment, with constrained vocabularies and “None” for absent or unseen attributes.Source: F Real-World Dataset Construction · p. 19

G Synthetic Appearance Generation Details

• Synthetic pairs edit the first frame of sampled 45-frame clips and retain the remaining frames, using randomly selected feasible weather, season, and time-of-day conditions.The pipeline uses Waymo, Boreas, and nuScenes clips and depth control to help preserve geometry and layout.Source: G Synthetic Appearance Generation… · pp. 19–20

H Reward Map Construction

• The reward map combines video-editing metrics with driving-specific checks to identify reliable supervision and safety-critical regions.Source: H Reward Map Construction · p. 20
• Content-preservation rewards penalize structural deviations and assess traffic lines, crosswalks, arrows, and vehicle body-paint preservation using segmentation, instance matching, and VLM checks.Source: H Reward Map Construction · p. 20
• Temporal rewards use FF-α for optical-flow-based intensity consistency and FF-β for motion-orientation consistency; instruction success is broadcast spatially across each frame.Source: H Reward Map Construction · p. 20

I Evaluation Protocol

• The protocol combines general video-editing metrics, multimodal alignment scores, and driving-aware structural checks across four evaluation axes.Source: I Evaluation Protocol · p. 20
• FF-α measures intensity consistency by optical-flow backward warping, while FF-β assesses motion orientation when large displacements make warping unreliable.Source: I Evaluation Protocol · p. 20
• Preservation metrics track foreground and background stability, road-line mask IoU, and vehicle body-paint retention after vehicle-mask matching.Source: I Evaluation Protocol · p. 20
• Instruction alignment uses CLIP-based success and similarity scores plus MLLM judgments, while photorealism uses aesthetic and image-quality predictors.Source: I Evaluation Protocol · p. 21

J User Study Details

• The user study compares HorizonComposer with CC-Edit, UniVideo, and SyntheticPip using randomized output order and one randomly selected criterion per trial.It collected 429 responses from 44 participants, each answering 10 questions.Source: J User Study Details · p. 21

K.1 Training & Architecture Details

• A frozen depth branch preserves geometric layout, while the visual branch receives sharp RGB inputs in the Cosmos-Transfer2.5 diffusion transformer.At inference, an externally edited image context guides generation.Source: §K.1 · p. 21
• Training uses the hybrid dataset with a standard v-prediction diffusion objective; inference generates 85 edited frames and 8 transition frames.The described setup uses 28,000 steps and 480p resolution.Source: §K.1 · p. 21

K.2 Image Context Robustness via Augmentation

• With probability 0.75, Poisson blending composites a randomly selected donor frame into the reference context using an elliptical mask.The augmentation creates a locally edited, photometrically coherent reference while preserving global scene structure.Source: §K.2 · p. 22
• The augmentation exposes the model to localized context perturbations so it can tolerate partial mismatches between reference appearance and target trajectory.Source: §K.2 · p. 22

L Dataset Impacts

• Hybrid-data training improves temporal consistency and traffic-semantic preservation, outweighing minor drops in instruction alignment.Source: L Dataset Impacts · p. 22
• Table 8 compares synthetic-only and real-world-only training against the hybrid approach on out-of-distribution data.Source: Table 8 · p. 22
• Table 9 reports relative improvements over hybrid data across temporal stability, instruction alignment, ODD preservation, and overall video quality on out-of-distribution data.Source: Table 9 · p. 22

M.1 Synthetic Paired Videos

• Synthetic samples pair 45-frame source and target clips with an instruction and first-frame reference, providing explicit editing supervision absent from raw driving logs.The synthetic dataset draws on Boreas, nuScenes, Waymo, and nuPlan.Source: §M.1 · p. 22, Table 10 · p. 23
• The synthetic dataset has 28,348 training clips, with held-out clips for in-domain testing and nuPlan reserved for out-of-distribution testing.Instructions edit weather, time of day, and season.Source: §M.1 · p. 23, Table 11 · p. 23, Table 12 · p. 23

M.2 Real-World Paired Videos

• Pose-matched Boreas traversals produce 33,750 unique real-world pairs, used bidirectionally as 67,500 directional training examples.Condition labels are normalized to the synthetic dataset taxonomy for transition reporting.Source: §M.2 · p. 23, Table 13 · p. 24

M.3 Overall Editing Coverage

• Synthetic clips commonly edit two or three attributes, while 9,534 of 67,500 Boreas videos have no valid condition change or missing annotations.The combined data covers all target weather, time-of-day, and season categories, with natural imbalance inherited from driving logs.Source: §M.3 · pp. 23–24, Table 14 · p. 24

N Reward Impacts

• Combining temporal-consistency, instruction-alignment, and content-preservation rewards yields the strongest overall OOD generalization while maintaining stable imaging quality and competitive instruction alignment.Source: N Reward Impacts · p. 24

O.1 SeRM Full-Resolution Pair Reconstruction

• Verified SeRM crop-to-frame matches are reconstructed as full-resolution image–annotation pairs, with labels placed in the bottom 320 rows of a 672 × 1280 canvas.The procedure first retrieves candidates using image features, then verifies exact pixel matches in route order.Source: O Downstream Experiment Details · p. 24, §O.1 · pp. 24–25

O.2 Video Clip Construction and Target Assignment

• The study partitions verified frames into 93-frame clips, assigns appearance targets, and excludes 8 thinking frames so generated frames can reuse original annotations.Figure 8 compares original images, annotation overlays, SyntheticPip augmentation, and the authors’ augmentation; the text reports better lane-marking preservation than SyntheticPip.Source: §O.2 · p. 25, Figure 8 · p. 27

O.3 Training

• PIDNet is fine-tuned on original SeRM frames alone or together with SyntheticPip-generated or HorizonComposer-generated frames using the original annotations.The text notes that generated clips that remove or distort lane markings may misguide training.Source: §O.3 · p. 26

P Offline Reinforcement Learning

• The offline-RL method treats hybrid-data pairs as off-policy rollouts and uses reward-conditioned fine-tuning to learn an editing policy from source-video and prompt inputs to edited clips.Training represents unconditional and improvement-indicator-conditional distributions; inference samples with the indicator set to 1 to favor higher-reward outputs.Source: P Offline Reinforcement Learning · pp. 26–27

Q Reward Granularity

• 18.81% relative gain in lane preservation and 3.51% in car preservation come from dense versus scalar reward conditioning.Dense rewards also improve FF-β, Aes., and Img.; remaining metrics show small differences.Source: Table 19 · p. 27, Q Reward Granularity · p. 27

NeurIPS Paper Checklist

• The checklist says results support the abstract and introduction’s dataset, reward-guidance, and temporal-augmentation claims.Source: NeurIPS Paper Checklist · p. 28

2. Limitations

• Company restrictions prevent open access to data and code.Source: §2 · p. 30
• Error bars are reported in Table 5, and evaluation covers two datasets with 150 examples each across multiple ablation experiments.Source: §2 · p. 30

10. Broader impacts

• The crowdsourced visual-preference study lacked formal IRB approval; authors report minimal-to-no participant risk and no PII or demographic data.Source: §10 · p. 33