PaperlayerView exact revision on arXiv

Source-linked AI summary

Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems

Yu Yao, Mingze Xu, Chiho Choi, David J. Crandall, Ella M. Atkins, Behzad Dariush

arXiv:1809.07408v2cs.CVcs.RO

TL;DR

The paper addresses predicting nearby vehicles’ future locations and scales from an ego-vehicle’s first-person view. It uses a multi-stream RNN encoder-decoder with bounding-box and optical-flow inputs, and reports better results than baselines on HEV-I and KITTI.

  • Problem

    Predicting nearby vehicles’ future locations matters for collision avoidance, but vehicle prediction from first-person views must handle perspective and ego-motion.

    Source: §I · p. 1

  • Method

    A multi-stream RNN encoder-decoder uses past bounding boxes and dense optical flow to predict future boxes, with optional future ego-motion input.

    Source: §I · p. 1, §III · p. 2, C. Future Location-Scale Decoding · p. 3

  • Results

    On HEV-I and KITTI, the proposed method outperforms the baselines; on HEV-I, RNN-ED-XO has FDE 25.56 vs 29.06 for Conv1D.

    Source: D. Results on HEV-I Dataset · p. 6, E. Results on KITTI Dataset · p. 6

  • Takeaways & Limitations

    The paper introduces the HEV-I intersection-video dataset, containing over 2,400 vehicles in 230 videos, for evaluating future vehicle localization.

    Source: §I · p. 1

    Future work could incorporate scene context, traffic signs and signals, depth, vehicle-environment interactions, and vehicle-to-vehicle or vehicle-to-pedestrian relationships.

    Source: §V · p. 6

Abstract

from arXiv · show

Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixel-level observations for future vehicle localization. We show that incorporating dense optical flow improves prediction results significantly since it captures information about motion as well as appearance change. We also find that explicitly modeling future motion of the ego-vehicle improves the prediction accuracy, which could be especially beneficial in intelligent and automated vehicles that have motion planning capability. To evaluate the performance of our approach, we present a new dataset of first-person videos collected from a variety of scenarios at road intersections, which are particularly challenging moments for prediction because vehicle trajectories are diverse and dynamic.

I. INTRODUCTION

The paper targets future vehicle localization from an ego-vehicle’s first-person camera, where safety systems need nearby vehicles’ future positions and overhead-view approaches have practical access constraints. It introduces a multi-stream RNN encoder-decoder and evaluates it on intersection videos and KITTI.

I. INTRODUCTION

  • Overhead-view trajectory prediction can require external cameras or LiDAR uncommon in production vehicles, or aerial imagery transferred over a network.The paper instead uses forward-facing cameras, which also capture object appearance and interactions with the ego-vehicle.Source: §I · p. 1
  • The method predicts nearby vehicles’ relative future locations and scales as bounding boxes from egocentric video, including urban scenarios with diverse multi-vehicle interactions.The paper evaluates the approach on the new intersection dataset and KITTI, reporting state-of-the-art results.Source: §I · p. 1, Figure 1 · p. 1
  • The contributions include a multi-stream RNN-ED using dense optical flow and future ego-motion, plus the HEV-I intersection dataset with over 2,400 vehicles in 230 videos.The paper reports state-of-the-art results on HEV-I and KITTI.Source: §I · p. 1

II. RELATED WORK

Prior trajectory prediction often uses bird’s-eye views, while egocentric vision research spans several tasks and includes methods for predicting people’s future locations and actions. This paper distinguishes its vehicle-localization approach by using vehicle locations and image features from first-person views.

II. RELATED WORK

  • Egocentric vision research has addressed activity recognition, object detection, person identification, summarization, gaze anticipation, and ego-action estimation or prediction.Examples include predicting a camera wearer’s future location, basketball players’ behavior, and egocentric basketball motion sequences.Source: §II · p. 2
  • For egocentric pedestrian prediction, one method models location uncertainty without image features, while another uses pose, scale, and ego-motion in a Conv1D framework.The paper notes that pose is specific to people and that Conv1D does not explicitly update temporal states along each trajectory.Source: §II · p. 2
  • Prior vehicle prediction approaches include motion-feature models, social pooling, scene-context attention, and multimodal RNN-based methods.These methods model trajectories and context from bird’s-eye views in static-camera settings.Source: §II · p. 2
  • Vehicle trajectory methods commonly use bird’s-eye views in static-camera settings, whereas monocular first-person views require indirect distance cues and include ego-motion effects.The proposed approach uses past vehicle location, scale, optical flow, and future ego-motion to predict future locations.Source: §II · p. 2

III. FUTURE VEHICLE LOCALIZATION FROM FIRST-PERSON VIEWS

The method predicts future vehicle bounding boxes from past egocentric observations and box trajectories. Its multi-stream encoder-decoder separates location-scale information from pixel-level observations and can incorporate future ego-motion.

III. FUTURE VEHICLE LOCALIZATION FROM FIRST-PERSON VIEWS

  • Given past image evidence and a vehicle’s bounding-box trajectory, the goal is to predict its future bounding-box trajectory in the first-person view.The method treats perspective-image motion differently because pixel distance does not directly represent physical distance and ego-motion creates apparent object motion.Source: §III · p. 2
  • The multi-stream RNN encoder-decoder encodes past box trajectories for location and scale, encodes dense optical flow for pixel-level information, and decodes future boxes with optional future ego-motion.Future ego-motion may be available from an intelligent vehicle’s planner; the decoder updates future boxes from the encoded hidden state.Source: §III · p. 2

1) Location-Scale Encoding:

  • Bounding boxes capture both location and scale because the same pixel location can represent a nearby lead vehicle or a distant vehicle across an intersection.Scale can also represent depth and orientation: distant vehicles tend to have smaller boxes, while crossing vehicles tend to have larger aspect ratios.Source: 1) Location-Scale Encoding · pp. 2–3, Figure 2 · p. 3

2) Motion-Appearance Encoding:

The motion-appearance stream represents pixel-level vehicle motion and appearance changes using dense optical flow. Two recurrent encoders model the box and image streams before late fusion.

2) Motion-Appearance Encoding:

  • Dense optical flow estimates motion vectors at every pixel, helping distinguish moving objects from background and capture appearance changes across object pixels.Unlike sparse Lucas-Kanade flow, dense flow provides pixel-level estimates.Source: 2) Motion-Appearance Encoding · p. 3
  • The method bilinearly pools flow features from an expanded vehicle-box region, encoding relative motion with surrounding environmental context.The resulting vector concatenates pooled horizontal and vertical flow components.Source: 2) Motion-Appearance Encoding · p. 3
  • Separate GRU encoders temporally model each input stream, and their representations are combined using late fusion.The GRUs use linear projections with ReLU activations and maintain hidden-state vectors over time.Source: 2) Motion-Appearance Encoding · p. 3

B. Future Ego-Motion Cue

The model uses planned future ego-motion as a cue for predicting nearby vehicles’ relative positions. It represents that motion through camera-frame transformations and a future-motion feature.

B. Future Ego-Motion Cue

  • Planned ego-vehicle motion can aid prediction of nearby vehicles’ relative positions and may help anticipate interaction-driven responses, such as yielding or passing.The paper assumes motion planning, such as trajectory generation, is available for autonomous vehicles.Source: B. Future Ego-Motion Cue · p. 3
  • Future ego-motion is represented by 2D rotation matrices and translation vectors that describe camera-frame transformations between successive times.Pairwise transformations can be composed to estimate motion across the prediction horizon from the current frame.Source: B. Future Ego-Motion Cue · p. 3
  • The future-motion feature includes yaw and translations expressed in a right-handed coordinate system whose positive x-axis aligns with vehicle heading.Estimated future motion is then input to the trajectory decoder.Source: B. Future Ego-Motion Cue · p. 3

C. Future Location-Scale Decoding

The decoder starts from fused representations of past bounding-box and optical-flow observations. It predicts future box location and scale relative to the current frame rather than generating absolute boxes directly.

C. Future Location-Scale Decoding

  • The future-box decoder is initialized from the final fused hidden state of the bounding-box and optical-flow encoders.The paper uses another GRU for decoding future bounding boxes.Source: C. Future Location-Scale Decoding · p. 3
  • The decoder’s hidden state is used to generate future bounding-box outputs.The passage identifies the decoder hidden state but does not describe its additional role here.Source: C. Future Location-Scale Decoding · p. 3
  • The decoder predicts each future bounding box’s relative location and scale from the current frame instead of generating absolute boxes directly.The output is shifted to have zero initial, which the paper says improves performance.Source: C. Future Location-Scale Decoding · pp. 3–4

IV. EXPERIMENTS

The experiments use HEV-I, an intersection-video dataset with varied vehicle and ego-vehicle motion, and report evaluation on HEV-I and KITTI. The supplied passages describe the data split and evaluation-table scopes, but not the tables’ results.

IV. EXPERIMENTS

  • HEV-I contains 230 intersection videos from the San Francisco Bay Area, recorded from a windshield-mounted RGB camera at 10 fps.The videos range from 10 to 60 seconds and are reduced to 1280×640 for the paper’s experiments.Source: A. Dataset · p. 4
  • Table I compares HEV-I with KITTI, while Tables II and III report proposed-method and baseline results on HEV-I and KITTI, respectively.Table II names FDE, ADE, and FIOU as its metrics; the supplied passages do not provide table values.Source: Table I · p. 4, Table II · p. 4, Table III · p. 4
  • The evaluation uses the past 1 second to predict bounding boxes for the next 1 second, with a random 70/30 video split yielding about 40,000 training and 17,000 testing samples.The dataset procedure uses 2-second samples to balance vehicle inclusion and travel length.Source: A. Dataset · p. 4
  • HEV-I includes varied ego turns and vehicle maneuvers, with many short tracks but also numerous trajectories longer than 100 pixels.The paper notes that longer trajectories are generally more difficult to predict.Source: A. Dataset · p. 4

B. Implementation Details

The implementation computes dense optical flow and estimates ego-vehicle motion, then trains a GRU-based model with Adam. The supplied passages give the principal model and training settings.

B. Implementation Details

  • Dense optical flow is computed with FlowNet2.0, and ORB-SLAM2 estimates ego-vehicle motion from first-person videos.The flow features are produced using a 5 × 5 ROIPooling operator.Source: B. Implementation Details · p. 4
  • The qualitative-results and failure-case figures are identified as Fig. 4 and Fig. 5, respectively.Their captions specify HEV-I examples but do not provide implementation settings.Source: Figure 4 · p. 5, Figure 5 · p. 5
  • The model uses GRUs with hidden size 512, Adam at learning rate 0.0005, batch size 64, and training for up to 40 epochs.The best models are selected; the paper notes that GRUs have fewer parameters than LSTMs and train faster without affecting performance.Source: B. Implementation Details · pp. 4–5

C. Baselines and Metrics

The section defines the evaluation metrics and compares simple extrapolation and Conv1D baselines with progressively richer RNN encoder-decoder variants. The ablations isolate the contributions of additional input streams and future ego-motion.

C. Baselines and Metrics

  • Linear extrapolates future boxes assuming linear location and scale changes, while ConstAccel assumes constant horizontal and vertical acceleration in the camera frame..Source: C. Baselines and Metrics · p. 5
  • Conv1D is adapted from using past bounding boxes and dense optical flow instead of location-scale and pose input streams.Source: C. Baselines and Metrics · pp. 5–6
  • The authors implement simpler RNN-ED baselines to assess the contribution of each model component.Source: C. Baselines and Metrics · p. 5
  • RNN-ED-X uses past bounding boxes alone; RNN-ED-XE adds future ego-motion, and RNN-ED-XO uses a two-stream encoder-decoder.RNN-ED-XOE is identified as the best model and includes awareness of future ego-motion.Source: C. Baselines and Metrics · pp. 5–6
  • FDE and ADE evaluate location prediction, while FIOU measures predicted-to-ground-truth bounding-box overlap at the final frame.ADE emphasizes overall prediction accuracy along the horizon.Source: C. Baselines and Metrics · p. 6

D. Results on HEV-I Dataset

On HEV-I, RNN-ED-XOE outperforms the naive baselines and Conv1D, with gains also reported for optical flow and future ego-motion. Qualitative examples illustrate strengths and remaining failure cases.

D. Results on HEV-I Dataset

  • FDE of 24.92 vs. 72.37 vs. 58.00: RNN-ED-XOE beats Linear and ConstAccel on all cases and improves about 15% over Conv1D.The improvement over baselines is more significant on challenging cases with complex trajectories.Source: D. Results on HEV-I Dataset · p. 6
  • Adding dense optical flow reduces FDE from 34.04 to 25.56 for RNN-ED-XO, with the benefit especially evident on challenging cases.Source: D. Results on HEV-I Dataset · p. 6
  • RNN-ED-XO has FDE 25.56 vs 29.06 for Conv1D; the authors attribute the difference to recurrent predictions using previous hidden states.Future ego-motion further improves performance.Source: D. Results on HEV-I Dataset · p. 6
  • Qualitative examples show better predictions for noisy boundary boxes, ego-motion-driven direction changes, and curved or long trajectories than Conv1D.Source: D. Results on HEV-I Dataset · p. 6
  • The method misses curvature caused by uneven road surfaces and can be misled by optical flow from pedestrians occluding the target vehicle.The authors suggest modeling the full traffic scene and participant relations as future work.Source: D. Results on HEV-I Dataset · p. 6

E. Results on KITTI Dataset

The paper reports that its method outperforms all baselines on KITTI, while noting weaker learning-model performance than on HEV-I. The conclusion presents the approach and dataset contributions and identifies additional scene and interaction cues for future work.

E. Results on KITTI Dataset

V. CONCLUSION

  • The conclusion frames the work as a new task of predicting target-vehicle location and scale in first-person video, evaluated on an intersection dataset and KITTI.The multi-stream RNN encoder-decoder incorporates future ego-motion and is assessed through component ablations.Source: §V · p. 6
  • Future work could incorporate scene context, traffic signs and signals, depth, vehicle-environment interactions, and vehicle-to-vehicle or vehicle-to-pedestrian relations.Source: §V · p. 6
Loading 1809.07408v2…