PaperlayerView exact revision on arXiv

Source-linked AI summary

Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation

Akos T. Kopeczi-Bocz, Tian Mi, Gabor Orosz, Denes Takacs

arXiv:2609.05523v1cs.CVcs.LGeess.IV

TL;DR

Monocular roadside perception must infer 3D vehicle geometry from a single image, while existing infrastructure pipelines often separate visual detection from geometric state estimation. This paper modifies YOLOv5 to predict vehicle states directly in a ground-fixed frame and reports trajectory and orientation recovery, with limited extrapolation to unseen trajectory configurations.

  • Problem

    Monocular 3D detection must infer depth and geometry from one image, and existing infrastructure approaches often separate image-plane detection from ground-plane state estimation.

    Source: §I · pp. 1–2

  • Method

    A modified YOLOv5 detector uses perspective-image features to jointly predict vehicle ground-plane position, dimensions, and yaw in a ground-fixed frame.

    Source: §I · p. 2

  • Results

    The network recovers vehicle trajectories and orientations in training and in-context tests, generalizing to unseen recordings with familiar maneuvers.

    Source: C. Training results · p. 7, §V · p. 9

  • Takeaways & Limitations

    The method estimates vehicle states from monocular roadside imagery without explicit 3D reconstruction or geometric post-processing during inference.

    Source: §V · p. 9

    Previously unseen trajectory configurations are only partially recovered under distribution shift, and closely associated vehicle movements can be conflated.

    Source: §V · p. 9

Abstract

from arXiv · show

This paper presents a one-stage learning framework that maps monocular roadside-camera images directly to vehicle states in a ground-fixed coordinate frame. Unlike conventional approaches that first detect vehicles in the image plane and subsequently apply geometric post-processing, the proposed method leverages features from a pretrained object detector to jointly estimate each vehicle's ground-plane position, dimensions, and yaw angle. The framework therefore uses visual features not only for vehicle detection but also for direct spatial and orientation estimation. To support model training and evaluation, we develop a data-collection and label-generation pipeline based on synchronized video from a roadside camera and an unmanned aerial vehicle (UAV). Acting as a temporary top-view sensing platform, the UAV provides vehicle trajectories, dimensions, and orientations, which are transformed into the ground-fixed coordinate frame and temporally aligned with the roadside-camera images to generate ground-truth labels. The framework is evaluated using data collected during multiple experiments at the Mcity Test Facility. Results show that the proposed method can recover vehicle trajectories and orientations from monocular roadside imagery without a separate geometric post-processing stage, demonstrating its potential as a scalable approach to infrastructure-based perception at urban intersections.

I. INTRODUCTION

Infrastructure-based perception can complement onboard sensing at challenging intersections, while monocular roadside cameras offer a practical alternative to costly LiDAR. This paper proposes a one-stage framework that directly estimates vehicle states in a ground-fixed frame, using synchronized UAV and roadside-camera data for ground truth.

I. INTRODUCTION

  • Two-stage infrastructure methods separate image-plane detection from geometric state estimation, while monocular inference must resolve depth and 3D geometry from one image.Monocular predictions are sensitive to occlusion, perspective ambiguity, and calibration errors.Source: §I · p. 1
  • A key design challenge is preserving perspective-image features while directly estimating physically meaningful vehicle states in a ground-fixed frame.Ground-plane feature priors and image-to-ground warping are described as alternatives, but the latter may distort appearance and hinder reuse of perspective-pretrained features.Source: §I · pp. 1–2
  • The framework directly predicts each vehicle’s ground-plane position, dimensions, and yaw from roadside-camera images in a single detection stage.It redesigns the YOLOv5 detection head and loss while retaining transferable features learned from perspective images.Source: §I · p. 2
  • Synchronized UAV and roadside-camera measurements provide ground-truth vehicle states aligned with roadside observations for training and evaluation.The proposed method requires only a monocular roadside camera at inference.Source: §I · p. 2
  • The evaluation covers in-context and out-of-context scenarios using quantitative errors and qualitative trajectory visualizations.The paper is organized around dataset construction, network design, evaluation, and discussion of limitations and future work.Source: §I · p. 2

II. GROUND TRUTH DATA GENERATION

The study builds a synchronized UAV and roadside-camera dataset at Mcity to train and validate roadside vehicle-detection algorithms. Five vehicles performed varied maneuvers across four scenarios, with five repeated runs per scenario.

A. Experiment description

  • The UAV processing approach extracts vehicle positions, yaw angles, and other kinematics using image stabilization and correlation-based template matching.The earlier method was validated against onboard RTK GPS and produced smooth trajectories and accurate kinematic states.Source: A. Experiment description · p. 2
  • A UAV flew above the intersection to record 1920 × 1080 video for ground-truth kinematics, while a lamppost-mounted roadside camera covered most of the intersection.The data collection was designed to support training roadside-camera vehicle-detection algorithms.Source: A. Experiment description · p. 2
  • The dataset comprises 20 synchronized top-view and roadside-view recordings from five vehicles across four scenarios, each repeated five times.Scenarios included straight crossings, left and right turns, following, and other vehicle interactions.Source: A. Experiment description · p. 2

B. Data collection and processing

Ground-truth states are derived from stabilized UAV observations in a ground-fixed frame, with velocity and yaw rate calculated from measured positions and yaw. The processing filters noisy measurements before deriving kinematics and retains varied vehicle motion for training.

B. Data collection and processing

  • The ground-fixed frame is right-handed, centered at the intersection, assumes a flat ground plane, and uses UAV footage stabilized to that frame at approximately 60 Hz.The frame interval is approximately 17 ms.Source: B. Data collection and processing · pp. 2–3
  • The measured vehicle state contains its geometric-center coordinates and yaw; velocity components and yaw rate are derived from changes in position and yaw over time.Yaw is the angle between the vehicle’s longitudinal axis and the frame’s east-pointing x axis; speed is the velocity-vector magnitude.Source: B. Data collection and processing · p. 3

C. Ground truth data processing

  • Raw position and yaw measurements are filtered with a second-order Savitzky-Golay filter using a 41-point window, about 0.68 s at 60 Hz.The filtering comparison for a turning truck shows greater noise in speed because it is obtained by numerically differentiating position.Source: C. Ground truth data processing · p. 3, Figure 2 · p. 3
  • Filtered yaw and speed traces across five vehicles include stationary, accelerating, and turning behavior, providing varied kinematics for training.The illustrated run was repeated five times for its scenario and similarly repeated for the other three scenarios.Source: Figure 3 · p. 3, C. Ground truth data processing · p. 4

D. Homography transformation

The homography maps top-view points into roadside-image locations, supporting label visualization and definition of the camera’s observable area. It is used for dataset construction and visualization; the network instead learns the mapping from raw roadside-image features.

D. Homography transformation

  • The homography H maps points from the top-view image plane to corresponding pixel locations in the roadside image.The mapping supports view-consistent training-label construction and visualization.Source: D. Homography transformation · p. 4
  • The transformation visualizes ground-truth boxes on roadside video and defines the geometric boundaries of the camera’s observable area.A virtual grid shown in both views illustrates correspondences between grid vertices.Source: D. Homography transformation · p. 4
  • The neural network receives raw roadside-image features, not data mapped to the roadside view, and learns to predict vehicle positions in ground-fixed frame F.The homography is used for dataset construction and visualization, rather than as a network input transformation.Source: D. Homography transformation · p. 4

E. Spatial filtering

Spatial filtering keeps training labels aligned with what the roadside camera can observe: it removes vehicles outside the camera’s field of view and excludes occluded data points. The section also introduces the network architecture and the search for a feature-freezing point.

E. Spatial filtering

  • The camera’s field of view is projected onto the top-view plane, and vehicle data outside it are omitted during training.In Fig. 6, trajectories inside the field of view are colored, while those outside it are gray.Source: E. Spatial filtering · pp. 4–5, Figure 6 · p. 4
  • Occluded data points are excluded so the model is trained solely on visible features.The example describes Vehicle 5 becoming partially occluded by Vehicle 1 as the scene evolves.Source: E. Spatial filtering · p. 5

III. NETWORK TOPOLOGY AND TRAINING

  • The network section covers YOLOv5 architecture, modified detection heads, loss formulation, and training losses under different layer-freezing configurations.It seeks a freezing point that maximizes reuse of pretrained YOLO features while adapting to the new task.Source: §III · p. 5

A. Base model and modifications

The method preserves YOLOv5’s backbone and neck while redesigning its detection heads to predict vehicle states on a ground-fixed plane from roadside images. Its loss combines objectness with weighted errors for position, size, and yaw instead of image-plane IoU regression.

A. Base model and modifications

  • The network retains YOLOv5’s convolutional backbone and spatial-pyramid-pooling feature extraction, Path Aggregation Network neck, and task-specific detection heads.Each head predicts bounding-box quantities at spatial locations on variable grid sizes.Source: A. Base model and modifications · p. 5, Figure 7 · p. 5
  • The backbone and neck remain unchanged; only the task-specific detection head is redesigned for ground-plane states and vehicle orientation.The output reduces class predictions to one class and adds orientation in the ground-fixed frame.Source: A. Base model and modifications · p. 5
  • Predictions are parameterized in the ground-fixed frame, where the network estimates vehicle position, dimensions, and yaw from roadside-image input.The output grid is fixed to the ground plane, and the network learns the geometry and homography implicitly.Source: A. Base model and modifications · p. 5

B. Loss function formulation

  • Rotated-box IoU is replaced by weighted mean-squared error on predicted quantities in frame F because exact IoU requires computationally expensive polygon intersection.The box loss weights position, size, and yaw errors; the overall loss combines it with unchanged objectness loss.Source: B. Loss function formulation · pp. 5–6
  • The loss weights are α = 5, β = 1, and γ = 10, and may be adjusted because the goal is viability rather than hyperparameter fine-tuning.Source: B. Loss function formulation · p. 6
  • Classification loss is omitted because the task focuses on vehicle parameters without multi-class categorization.Source: B. Loss function formulation · p. 6

C. Training results

Training losses decrease across freezing configurations, but validation results favor limited freezing. Freezing layers 1–2 reduces validation box loss versus the unfrozen baseline, while deeper freezing can impair adaptation.

C. Training results

  • The backbone and neck start from pretrained YOLOv5 weights, while modified detection heads start randomly; freezing earlier layers reduces trainable parameters.The training data use runs 1–4 for training and validation, with run 5 reserved for in-context tests and Scenario 3 for out-of-context testing.Source: C. Training results · pp. 6–7
  • Box and objectness training losses decrease monotonically, while validation losses rise slightly after 10–15 epochs, indicating overfitting.Objectness loss saturates quickly, while position and yaw components continue improving longer.Source: C. Training results · p. 7
  • Freezing through layer 10 leaves final error largely unchanged, whereas freezing through layer 10 and above can increase training errors by excluding critical parameters.The validation box loss saturates at different levels depending on the freezing point.Source: C. Training results · p. 7
  • Freezing layers 1–2 reduces validation box loss by 5% versus the unfrozen baseline (1.8855 vs. 1.9788).Freezing through layer 15 gives a lower validation loss of 1.6929, but with training error an order of magnitude higher.Source: C. Training results · p. 7
  • Qualitative evaluation reports recovery of vehicle positions and orientations in the ground-fixed frame for images not used during training.Source: C. Training results · p. 7

IV. RESULTS

The results section evaluates performance on training and validation data, unseen runs from familiar scenarios, and a fully unseen scenario. It combines quantitative errors with trajectory visualizations.

IV. RESULTS

  • Table II lists end-of-training-epoch-30 box and objectness losses for training and validation datasets.Source: Table II · p. 7
  • Evaluation spans training and validation conditions, unseen runs of familiar scenarios, and extrapolation to a completely unseen scenario.Source: §IV · p. 7
  • The section reports quantitative errors relative to ground truth alongside trajectory visualizations in top-view and roadside perspectives.Source: §IV · p. 7

A. Training dataset

On training and validation examples, predicted trajectories align closely with ground truth, and yaw and speed estimates track the reference signals. The reported errors are generally small across the illustrated time series.

A. Training dataset

  • Predicted and ground-truth trajectories align closely in the top view, with no visible degradation between training and validation instances.The figure contrasts gray ground-truth trajectories with dark network predictions.Source: A. Training dataset · p. 7, Figure 10 · p. 8
  • Yaw-angle error remains predominantly under 4 degrees, while speed error stays below 0.5 m/s for most of the time.Source: A. Training dataset · p. 8, Figure 10 · p. 8
  • The yaw and speed predictions closely follow ground truth for both training and validation examples.The figure uses light colors for ground truth inside the camera field of view, gray outside it, and dark colors for predictions.Source: A. Training dataset · pp. 7–8, Figure 10 · p. 8

B. In-context test set

On an unseen run from a familiar scenario, predicted trajectories generally track ground truth, though one vehicle near the camera’s field-of-view edge is missed and errors rise during entry or exit. In the unseen scenario, familiar trajectory segments are reconstructed more reliably than unfamiliar ones.

B. In-context test set

  • On the in-context test run, predicted trajectories closely track ground truth for four vehicles, but a vehicle near the field-of-view edge is not detected.Source: B. In-context test set · p. 8
  • In-context yaw and speed outputs generally align with ground truth, but errors increase most during vehicle entry into or exit from the camera’s field of view.Source: B. In-context test set · p. 8

C. Out-of-context test set

  • In the out-of-context test, familiar trajectory segments are reconstructed correctly, while unfamiliar segments are only partially recovered and can be assigned to a training-set trajectory.Vehicle 5 is incorrectly associated with Vehicle 1; Vehicle 3’s first segment is projected onto a familiar path with perpendicular orientation.Source: C. Out-of-context test set · p. 8
  • For correctly recovered out-of-context segments, yaw and speed follow ground truth, with errors of the same order of magnitude as in the in-context test.Source: C. Out-of-context test set · p. 8

V. CONCLUSION

The conclusion presents a roadside monocular framework that directly estimates ground-fixed vehicle states and reports recovery of trajectories and orientations in training and in-context scenarios. It also identifies weaker yaw estimation near field-of-view boundaries and limited extrapolation to underrepresented motion patterns, motivating broader data and evaluation.

V. CONCLUSION

  • The framework directly maps roadside images to ground-fixed vehicle states, avoiding explicit 3D reconstruction and inference-time geometric post-processing.Its output includes ground-plane position, dimensions, and yaw angle.Source: §V · pp. 8–9
  • The network recovers trajectories and orientations in training and in-context tests, but yaw estimation falls near field-of-view boundaries as partial visibility and occlusion reduce visual information.Fig. 11 shows trajectories and yaw and speed comparisons for a held-out run from the same experimental scenarios.Source: Figure 11 · p. 9, §V · p. 9
  • Under distribution shift, familiar trajectory segments remain reliable, but unseen configurations are only partly recovered and associated vehicles’ trajectories can be conflated.The paper attributes limited extrapolation to motion patterns inadequately represented in training.Source: §V · p. 9
  • The approach is presented as a practical, scalable framework for extracting vehicle states and trajectory-level traffic information from monocular roadside cameras.Source: §V · p. 9
  • Future work targets generalization through more diverse training data, temporal and kinematics-based models, visibility-aware augmentation, and evaluation across broader intersections, viewpoints, and traffic conditions.Source: §V · p. 9
Loading 2609.05523v1…