PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenLongTail: Generative Scaling of Long-Tail Driving Data

Lulin Liu, Nuo Chen, Yan Wang, Bangya Liu, Wenyan Cong, Hezhen Hu, Boris Ivanovic, Hao Wang, Ziyao Zeng, Xinyu Gong, Yang Zhou, Zixiang Xiong, Dilin Wang, Zhangyang Wang, Weisong Shi, Ruohan Zhang, Marco Pavone, Zhiwen Fan

arXiv:2607.09655v1cs.CV

TL;DR

Long-tail driving events are important for reliable autonomy, but heterogeneous in-the-wild videos often lack the multi-view coverage needed for policy training. OpenLongTail converts such videos into pose-grounded, synchronized multi-view assets, improving policy robustness under long-tail events while supporting view synthesis and ego-trajectory recovery.

  • Problem

    Heterogeneous long-tail driving videos often lack the multi-view coverage and pose information needed to train reliable policies for safety-critical situations.

  • Method

    OpenLongTail recovers metric-scale ego trajectories and uses pose-informed, geometry-grounded view synthesis to convert monocular videos into synchronized multi-view assets.

  • Results

    OpenLongTail-synthesized assets consistently improve policy robustness under long-tail events across closed-loop evaluation, view synthesis, and ego-trajectory recovery.

  • Takeaways & Limitations

    The results support heterogeneous monocular videos as a scalable training source for long-tail driving policy learning.

  • Takeaways & Limitations

    The benefit of external-source augmentation is alignment-conditioned, with performance regressing on the complex-intersection slice.

Abstract

from arXiv · show

Scaling robust driving policies is fundamentally bottlenecked by the scarcity of edge cases in curated datasets. While the real world continuously captures these critical events, such long-tail events remain underutilized when collected from heterogeneous sources. Specifically, diverse but valuable in-the-wild long-tail videos lack the full view coverage required for training policy models, often missing multi-view poses or originating solely from monocular dash cameras. This modality gap prevents these ubiquitous observations from being converted into scalable training data for long-tail generalization. We introduce OpenLongTail, an open-source generative data engine for scaling autonomous driving policies under long-tail events. To transform heterogeneous data sources into view-aligned and temporally coherent multi-view assets that are useful for policy learning, we develop a pose-informed extrapolative view synthesis pipeline that generates the missing views. We further enhance cross-view consistency and the temporal alignment for the newly generated views by injecting Plücker ray geometry into the scalable generation engine. By synthesizing heterogeneous long-tail data, we observe a significant improvement in closed-loop driving robustness in handling long-tail events. By measuring the extrapolative view synthesis and pose metrics, we validate the effectiveness of OpenLongTail in visual fidelity, cross-view consistency, and ego-trajectory recovery.

1 Introduction

OpenLongTail addresses the scarcity and underuse of long-tail driving data by converting heterogeneous, often monocular videos into pose-grounded, synchronized multi-view assets. Its geometry-aware generation and validation target scalable policy learning and improved robustness to long-tail events.

  • Motivation: Large-scale datasets have advanced VLA driving policies on common scenarios, but heterogeneous long-tail videos remain difficult to use for policy training.These videos often lack full view coverage, including multi-view poses, or originate solely from monocular dash cameras.
  • OpenLongTail: OpenLongTail is an open-source generative data engine that converts heterogeneous long-tail driving videos into pose-grounded, spatially aligned, temporally coherent multi-view assets.It recovers a metric-scale ego-trajectory from weakly annotated videos and uses the recovered motion to guide pose-informed extrapolative view synthesis.
  • Method: Plücker-ray camera geometry and a cross-view memory bank provide spatially grounded latent context for stronger consistency under the target camera rig.The system synthesizes target side and rear views that may observe regions absent from the input video.
  • Validation: OpenLongTail is evaluated through closed-loop VLA driving, view synthesis, and ego-trajectory recovery, with generated assets improving long-tail driving policy robustness.The evaluation is conducted in AlpaSim and positions ubiquitous monocular long-tail videos as a scalable training source.

2 Related Work

Related work spans vision-language-action models for autonomous driving and controllable video generation for coherent, multi-view, long-horizon, and closed-loop simulation. However, sparse overlap in surround-view rig cameras leaves target regions unobservable, motivating OpenLongTail’s pose-aware geometry recovery and controllable world generation.

  • Vision-Language-Action Models For Autonomous Driving: Vision-language-action models unify perception, language-conditioned reasoning, and end-to-end planning for autonomous driving.Recent approaches increasingly leverage reinforcement learning, diffusion, and explainable policies.
  • Controllable Video Generation: World foundation models provide generative priors for coherent visual synthesis, supporting driving methods for multi-view, long-horizon, and closed-loop simulation.The cited driving-oriented methods apply controllable video generation to autonomous-driving simulation.
  • Controllable Video Generation: Surround-view rig cameras exhibit minimal overlap, leaving massive target regions unobservable and unrecoverable through standard geometric projection or warping.OpenLongTail addresses this limitation by coupling pose-aware geometry recovery with controllable world generation to hallucinate unobservable rig views while enforcing cross-view geometric alignment and temporal coherence.

3 Methodology

OpenLongTail converts monocular long-tail driving videos into synchronized multi-view assets through pose-informed generation under a target camera rig. Its method combines metric-scale trajectory stabilization with Plücker-ray geometry, temporal depth warping, cross-view memory, and flow-matching training.

  • Pipeline formulation: OpenLongTail formulates multi-view synthesis as a two-stage process conditioned on a continuous ego-trajectory and target camera rig.The pipeline transforms heterogeneous monocular front-view videos into motion-consistent multi-view assets for scalable long-tail data construction.
  • Camera trajectory recovery: MapAnything recovers metric camera poses, while forward-backward filtering and smoothing preserve metric ego-motion and reduce frame-level jitter.The forward Kalman filter suppresses local noise, and the backward Rauch-Tung-Striebel smoother stabilizes the global path using the complete clip.
  • Multi-view generation: The generator synthesizes five non-front views with a frozen Wan 2.1 VAE and Wan 2.1-VACE DiT 1.3B backbone, training only LoRA adapters and conditioning modules.The trainable components are LoRA adapters on Wan-DiT/VACE self-attention layers plus the Geometry Encoder, Temporal Depth Warp, and Cross-View Memory Bank.
  • Geometry conditioning: Plücker-ray geometry provides unified token-wise camera conditioning across the DiT, VACE control branch, and memory bank in a consistent 3D ray frame.The Geometry Encoder also incorporates camera identity, stream role, and ego-motion features.
  • Temporal depth warp: Temporal depth warping supplies pose-aligned RGB priors, using frontal reprojection for lateral cameras and view-specific temporal offsets for rear cameras without direct overlap.The warp relies on frozen DepthCrafter depth and analytic operations; view-specific offsets maximize valid spatial coverage across target cameras.
  • Cross-view memory: A directed autoregressive memory graph propagates neighboring-view information, while dense and semantic memories preserve pixel correspondence and robustness to spatial misalignment.The five target cameras are synthesized in topological order, with up to three conditioning latents and a 64-query Perceiver-style resampler.

4 Evaluation

OpenLongTail is evaluated first by downstream closed-loop driving performance, then by the view-synthesis and ego-trajectory components enabling that performance. The evaluation covers unseen scenes, external monocular sources, and metric-scale or Sim(3)-aligned trajectory recovery.

  • Evaluation scope: The evaluation prioritizes whether synthesized long-tail multi-view data improves closed-loop driving, followed by analyses of view synthesis and ego-trajectory recovery.The study evaluates closed-loop performance in AlpaSim before assessing generation quality and pose recovery.
  • Closed-loop driving: OpenLongTail-generated assets are evaluated in closed-loop simulations on 53 long-tail events in AlpaSim, where rendered multi-camera observations feed policy action prediction.The simulation deploys the fine-tuned VLA policy as an ego-driver whose observations are regenerated according to its current ego state.
  • External-data scaling: Augmenting NV Syn. with Waymo E2E assets improves uncommon vehicles (0.717→0.765), cyclists (0.662→0.689), and work zones (0.935→0.950), but complex intersections regress.The external-data benefit is alignment-conditioned, with Waymo-derived assets recovering several NV-only failure cases while regressing the complex-intersection slice.
  • Multi-view generation: View synthesis is tested on seen scenarios, unseen scenarios, and external monocular sources using image, video, and epipolar-filtered keypoint-consistency metrics.The unseen split contains 218 clips from entirely unseen scenes, and Table 2 reports stronger GeoKPM µ cross-view geometric consistency than baselines.
  • External monocular sources: Nexar evaluation uses only monocular front-view streams from heterogeneous consumer dashcams with unknown intrinsics, rolling-shutter and compression artifacts, and no reliable metric camera pose.OpenLongTail recovers ego-motion entirely through its pose estimation module before synthesizing target views.
  • Trajectory recovery: Ego-trajectory recovery is evaluated on 218 clips from 109 scenes against five recent methods under metric-scale and Sim(3)-aligned protocols.The compared methods are DROID-W, MegaSAM, VGGT, MapAnything, and ViPE.

5 Conclusion and Limitations · A Dataset Curation and Splits

OpenLongTail is presented as an open-scaling generative framework that converts heterogeneous, unposed videos into synchronized, pose-grounded multi-view assets for driving-policy scaling. Its synthesized assets improve robustness to long-tail events across closed-loop evaluation, extrapolative view synthesis, and ego-trajectory recovery.

  • 5 Conclusion and Limitations: OpenLongTail transforms heterogeneous, unposed videos into synchronized, pose-grounded multi-view assets.The framework is designed for scaling driving policies.
  • 5 Conclusion and Limitations: The framework supports scaling driving policies from heterogeneous video sources.This establishes a generative approach for converting diverse video data into policy-training assets.
  • 5 Conclusion and Limitations: OpenLongTail-synthesized assets consistently improve policy robustness under long-tail events.This result is reported across closed-loop evaluation in AlpaSim NVIDIA et al. (2025).
  • 5 Conclusion and Limitations: The assets are evaluated through closed-loop driving evaluation in AlpaSim NVIDIA et al. (2025).The passage groups this evaluation with extrapolative view synthesis and ego-trajectory recovery.
  • 5 Conclusion and Limitations: OpenLongTail evaluates extrapolative view synthesis as part of its reported validation.The passage identifies extrapolative view synthesis alongside closed-loop evaluation and ego-trajectory recovery.
  • 5 Conclusion and Limitations: The framework evaluates ego-trajectory recovery and positions ubiquitous monocular videos as a scalable training source.This follows the reported robustness gains under long-tail events.

A.1 Dataset Sources · A.2 Train/Test Split Protocol

OpenLongTail curates approximately 200K 41-frame clips from about 50K scenes, primarily using PAV and supplementing it with PandaSet and nuScenes for broader diversity. Evaluation uses temporally disjoint windows within seen scenes, held-out scene IDs for unseen scenarios, and Waymo E2E exclusively as an external monocular-source test set.

  • A.1 Dataset Sources: The dataset contains approximately 200K 41-frame clips from about 50K scenes, curated from PAV, PandaSet, and nuScenes.PAV supplies most training clips, while PandaSet and nuScenes add sensor, camera, weather, scene-layout, and location diversity.
  • A.1 Dataset Sources: PAV provides the primary training data because it offers diverse, large-scale driving logs and rich long-tail scenarios.
  • A.1 Dataset Sources: PandaSet and nuScenes supplement PAV to improve sensor, camera, weather, scene-layout, and location diversity.
  • A.1 Dataset Sources: Waymo E2E is reserved for external monocular-source evaluation and is never used for training.Its front-view videos are inputs for synthesizing missing target-rig views.
  • A.2 Train/Test Split Protocol: Seen-scenario evaluation assigns disjoint 41-frame temporal windows within the same scene to training and testing.The test clips share the training scene ID and global environment but contain strictly unseen frames.
  • A.2 Train/Test Split Protocol: Unseen-scenario evaluation splits by scene ID, excluding all clips from held-out scenes during training.This prevents exposure to the held-out scenes’ layouts, camera trajectories, and visual appearances, and supports the main extrapolative view evaluation.
  • A.2 Train/Test Split Protocol: Waymo E2E serves as an external evaluation set, with no clips used for training, validation, hyperparameter tuning, or checkpoint selection.Front-view videos are used to synthesize missing target-rig views with OpenLongTail.

B Generative Scaling across Diverse Sources

OpenLongTail scales long-tail driving data by converting heterogeneous videos into policy-compatible, target-rig multi-view assets for downstream training. Its benefits depend on source alignment: external data can broaden diversity, but category mismatch limits gains for specific evaluation slices.

  • Generative scaling across diverse sources: OpenLongTail expands the usable source pool by transforming videos from different datasets, camera setups, and collection protocols into a shared target-rig format.Generative scaling therefore increases source diversity rather than merely multiplying samples from a fixed distribution.
  • Generative scaling across diverse sources: Waymo E2E front-camera videos are converted into policy-compatible multi-view assets under the target camera rig, enabling their use in downstream VLA training.This conversion supports practical evaluation of external data in closed-loop long-tail scenarios.
  • Closed-loop benefits: Generated assets improve closed-loop rollout behavior across uncommon vehicles, cyclists, and complex intersections, with NV and external synthesized data producing more scene-consistent trajectories than the base model.The result indicates that heterogeneous generated data can expand useful long-tail coverage rather than simply increasing sample count.
  • Source alignment: 671 NV examples versus 52 Waymo E2E examples cover temporary path reassignment and weakly marked complex-intersection cases, while human-guided traffic-control cases contain 62 versus 0.These counts show that NV and Waymo emphasize different sub-distributions within the broad complex-intersection category.
  • Source alignment: Transfer efficiency depends heavily on source-category alignment: aligned external data yields direct benefits, whereas mismatched sub-distributions broaden diversity but offer limited gains for a specific slice.Effective scaling requires both robust conversion and distribution-aware source selection.

C Implementation Details … E Evaluation Protocol

OpenLongTail fine-tunes Wan2.1-VACE-1.3B with parameter-efficient adapters and conditioning modules, while sigma-dependent memory gates regulate cross-view information during denoising. Its inference pipeline recovers scene geometry and ego motion before autoregressively synthesizing five missing camera views.

  • C.1 Generation Model Fine-Tuning: OpenLongTail fine-tunes Wan2.1-VACE-1.3B as its video-generation backbone.
  • C.1 Generation Model Fine-Tuning: The Wan VAE remains frozen, while trainable parameters are limited to LoRA adapters in Wan-DiT/VACE self-attention layers and proposed conditioning modules.The conditioning modules comprise a geometry encoder, temporal-depth-warp interface, and memory bank module.
  • C.1 Generation Model Fine-Tuning: Sigma-dependent gates modulate dense and semantic memory branches according to the flow-matching noise level, balancing semantic context with spatial correspondence across denoising stages.At each Memory layer, a layer-specific MLP produces the gates from the sampled noise level.
  • C.1 Generation Model Fine-Tuning: The per-sample gate outputs are scalars in the range (0, 1) and scale their corresponding dense and semantic memory updates.
  • C.1 Generation Model Fine-Tuning: Zero-initialized final projections with negative biases produce an initial gate value of approximately 0.2, making memory a weak residual pathway at training onset.The six Memory layers use unshared gates, allowing each layer to learn its own noise-dependent injection schedule.
  • D Inference Pipeline: Given a 41-frame front-view video, inference recovers ego trajectory, estimates depth, builds temporal depth-warp conditions, and encodes target-camera Plücker rays.
  • D Inference Pipeline: The system autoregressively synthesizes the five non-front views in topological order, adding each generated view to cross-view memory for subsequent target cameras.

E.1 Image and Video Fidelity Metrics … F Additional View Synthesis Results

The evaluation defines image, video, and cross-view geometric metrics for generated target views, using standardized inputs and inference conditions across methods. GeoKPM complements appearance metrics by testing whether generated correspondences satisfy camera-induced epipolar geometry.

  • E.1 Image and Video Fidelity Metrics: PSNR, SSIM, and LPIPS compare generated target-view frames with ground-truth target-view frames at a common evaluation resolution.FID compares generated and ground-truth frames across the evaluation split, while FVD uses 41-frame clips.
  • E.1 Image and Video Fidelity Metrics: FID is computed over generated and ground-truth frames aggregated across the evaluation split, whereas FVD is computed over 41-frame clips.Unless otherwise stated, the metrics are computed over the full target-camera image.
  • E.2 Cross-View Geometric Consistency: GeoKPM evaluates cross-view consistency by measuring the fraction of confident generated correspondences that satisfy the epipolar geometry induced by camera parameters and recovered ego pose.Confident matches are first obtained with LoFTR, then filtered according to geometric validity.
  • E.2 Cross-View Geometric Consistency: OpenLongTail inference recovers and smooths the ego trajectory, estimates front-view depth, and generates target views sequentially with Plücker-ray conditioning and memory-bank dependencies.The procedure synthesizes a multi-view rollout from one front-view video using the pose-conditioned Wan2.1-VACE diffusion model.
  • E.2 Cross-View Geometric Consistency: Unlike raw KPM, GeoKPM penalizes front-view copying because copied-view matches can be abundant while violating the target-camera epipolar constraint.This makes GeoKPM a more direct measure of geometric consistency for generated views.
  • E.3 Baseline Evaluation Fairness: Cross-view geometric evaluation additionally standardizes camera parameters, recovered ego pose, resolution, LoFTR confidence threshold, and epipolar threshold across methods.The specified LoFTR confidence threshold is 0.6 and τepi = 10 pixels.
  • E.3 Baseline Evaluation Fairness: All methods use the same front-view inputs, target camera rig, recovered ego trajectory, output resolution, and clip length, with 50 diffusion sampling steps where applicable.Baselines with fixed public schedules retain their official settings, and no method receives additional target-view images at test time.

F.1 In-Distribution Seen-Scenario Novel View Evaluation

In seen-scenario novel-view evaluation, cross-left and cross-right cameras achieve the strongest image-level fidelity because they overlap more with the observed front view. Rear-facing views require greater viewpoint extrapolation, while 41-frame clips retain relatively stable temporal coherence despite lower single-frame fidelity.

  • View-dependent fidelity: Cross-left and cross-right cameras obtain the strongest image-level fidelity, with higher PSNR and SSIM and lower LPIPS than rear-facing cameras.Their stronger spatial overlap with the observed front camera provides more directly pose-grounded visual evidence.
  • View-dependent fidelity: Rear-left, rear-right, and rear-tele views have lower image-level similarity because they require larger viewpoint extrapolation and include weakly observed or unobserved regions.Rear-tele is particularly challenging because its narrower field of view and larger directional change increase sensitivity to pose and temporal alignment errors.
  • Temporal coherence: FVD remains relatively stable across target views, indicating reasonable temporal coherence in generated 41-frame clips even when extrapolative cameras have lower single-frame fidelity.This stability persists despite the greater pose and temporal alignment sensitivity of rear-tele.

F.2 Additional Qualitative Results on Unseen Scenes … H.1 Policy Backbone and Input Construction

OpenLongTail generalizes to unseen scenes by generating temporally coherent multi-view assets across diverse domains and extrapolation levels. The downstream policy setup standardizes Alpamayo-R1 inputs and compares ground-truth versus synthesized non-front views under controlled fine-tuning protocols.

  • F.2 Additional Qualitative Results on Unseen Scenes: Unseen-scene examples span rural, residential, and snowy urban environments, testing generalization beyond memorized scene geometry.The examples use scene UUIDs not observed during training and include diverse road layouts, appearance domains, and weather conditions.
  • F.2 Additional Qualitative Results on Unseen Scenes: Views with greater geometric overlap preserve sharper appearance and road boundaries, while rear-facing extrapolation makes fine texture recovery harder.More extrapolative views require hallucinating larger unobserved regions.
  • F.2 Additional Qualitative Results on Unseen Scenes: Generated clips remain temporally coherent across the 41-frame horizon despite challenging unseen-scene appearance shifts.The qualitative results show visually usable multi-view assets under snowy and residential domain changes.
  • G Pose Recovery Protocol: Pose recovery applies MapAnything followed by forward Kalman filtering and backward Rauch–Tung–Striebel smoothing.The smoothed trajectory drives Plücker-ray geometry, temporal depth warps, target-rig poses, and policy inputs.
  • G Pose Recovery Protocol: Metric-scale evaluation measures absolute trajectory usefulness, whereas Sim(3)-aligned evaluation measures relative geometry after removing global rotation, translation, and scale.The two protocols assess different properties and should not be directly compared as the same metric.
  • H.1 Policy Backbone and Input Construction: Alpamayo-R1 is fine-tuned using four camera views and ego-motion under a fixed policy interface.The front view is original, front-tele uses a deterministic 30◦ FoV-Z crop, and cross-left/right views are ground-truth or synthesized depending on the recipe.
  • H.1 Policy Backbone and Input Construction: The two-stage SFT configuration trains the VLM first, then freezes it while training only the diffusion expert.All variants use the same architecture, camera interface, action format, optimization protocol, and evaluation environment; recipes vary in additional data and non-front-view source.

H.2 SFT Data Composition · H.3 AlpaSim Closed-Loop Evaluation

H.2 evaluates fine-tuning recipes that vary only in added long-tail data, including ground-truth and OpenLongTail-synthesized multi-view assets. H.3 uses AlpaSim to assess policies in photorealistic closed loop by rendering observations from the evolving ego state.

  • H.2 SFT Data Composition: All SFT recipes fine-tune Alpamayo-R1 for three epochs with the same optimization setup, differing only in their additional training-data mixture.The nominal dataset contains 10K randomly sampled driving trajectories.
  • H.2 SFT Data Composition: Long-tail mixtures combine PAV and Waymo E2E clips with either available ground-truth multi-view data or OpenLongTail-synthesized multi-view assets.This design compares heterogeneous long-tail sources and their ground-truth or synthesized multi-view forms.
  • H.2 SFT Data Composition: GT recipes use synchronized calibrated multi-camera logs, whereas Syn recipes synthesize target views from front-view videos under the same target-rig camera protocol.Keeping the policy interface and SFT procedure fixed isolates the effect of synthesized multi-view long-tail assets.
  • H.3 AlpaSim Closed-Loop Evaluation: AlpaSim is a photorealistic closed-loop simulator that re-renders sensor inputs from the policy’s current simulated ego state at every rollout step.This exposes policies to the consequences of their own actions, including compounding errors, recovery behavior, and off-road motion.
  • H.3 AlpaSim Closed-Loop Evaluation: AlpaSim constructs reusable assets from synchronized multi-camera data, calibration, ego poses, object tracks, and maps, reconstructing scenes with NuRec as neural 3D assets typically represented by 3D Gaussian Splatting.The reconstruction preserves high-fidelity visual appearance such as road texture.
  • H.3 AlpaSim Closed-Loop Evaluation: Each asset combines neural visual reconstruction with explicit driving structure, including ground meshes, drivable surfaces, lane geometry, road boundaries, routes, and dynamic-agent tracks.These layers support collision checking, off-road detection, lane compliance, and progress measurement while neural rendering supplies realistic camera observations.
  • H.3 AlpaSim Closed-Loop Evaluation: During closed-loop rendering, AlpaSim places the virtual camera rig at the current ego pose, synthesizes multi-camera observations, receives a policy action or future trajectory, updates ego state, and repeats.Vehicle dynamics or trajectory-following models drive the state update between rendering steps.
  • H.3 AlpaSim Closed-Loop Evaluation: Novel-view synthesis enables feedback-loop evaluation by rendering photorealistic camera views from arbitrary nearby viewpoints when the ego vehicle moves beyond poses observed in the original log.This supports evaluation of VLA models using image-level cues such as lane markings, cones, signs, workers, police officers, occlusions, and irregular intersections.
Loading 2607.09655v1…