Source-linked AI summary
Situation Awareness for Intelligent Data Distribution in Connected Vehicles
Falk Dettinger, Akshay Narla, Michael Weyrich
TL;DR
Redundant V2X transmissions and occlusion-related perception limits motivate direct traffic-situation identification for connected vehicles. The paper combines semantic segmentation, Cam2BEV projective transformation, and SIN classification on BEV images, evaluating the concept in CARLA and real-world urban datasets. The reported results show situation inference in simulation and portability to real-world data, while supporting situation-aware sensor-data selection.
Problem
Redundant V2X transmissions increase bandwidth pressure, while existing perception methods do not directly identify traffic situations from surrounding objects.
Method
The concept semantically segments camera images, transforms them into BEV with Cam2BEV, and uses SIN to classify the ego-vehicle’s traffic situation.
Results
The system achieved 67.3%–80.2% correct situation inference across CARLA towns, 63.4% overall success with the RGB camera, and 61% overall success on urban real-world scenarios.
Takeaways & Limitations
The approach provides a basis for selecting relevant sensor data for transmission or retrieval according to the ego-vehicle’s traffic situation, supporting V2X redundancy mitigation.
Abstract
from arXiv · showhide
The limitations of on-board sensors and blind spots caused by occlusion cause the reduction of perception quality in autonomous vehicles. In such cases, cooperative perception provides additional data via Vehicle-to-Everything communication to enhance local perception, causing a large volume of data transmission. The vehicle can focus on acquiring and utilizing relevant data according to the prevailing road context by identifying the current traffic situation. To achieve this, we propose a concept for the situation identification of the vehicle using Bird's-Eye-View images. Firstly, the situation around the vehicle is identified using object detection with semantic segmentation, followed by understanding the context of the traffic using a situation identification module consisting of an open-source projective transformation network Cam2BEV and a situation identification neural network. The concept was evaluated and validated by running the software on the CARLA simulator using the in-built RGB camera and the semantic segmentation camera. Additionally, the portability of the situation identification module for real-world applications was verified on Cityscapes and nuScenes urban driving datasets. Overall, the proposed situation identification approach enables efficient sensor data management by prioritizing relevant data to the current traffic situation. The source code is available in the following link: https://github.com/akshaynarla/DySi_Select
I. INTRODUCTION
Connected-vehicle data exchange can create redundant transmissions and bandwidth pressure, while existing perception methods do not directly identify traffic situations. The paper motivates a BEV-based approach to explicit situation identification for more targeted data distribution.
- Motivation: Increasing connected-vehicle functions and traffic density can amplify redundant messages, reducing per-subscriber bandwidth and increasing transmission time.The introduction identifies targeted data-transmission optimization as necessary in vehicular environments.
- Motivation: Current perception methods detect 3D objects but do not directly identify the traffic situation, requiring computationally costly interpretation of objects, signs, and signals.The proposed direction uses the ego-vehicle’s top-down camera projection as input for direct situation identification.
- Related Work: Prior work addresses scene understanding, risk assessment, adverse-lighting perception, trajectory prediction, or BEV object identification without explicitly identifying the surrounding traffic situation.The reviewed approaches include scene graphs, RGB-thermal fusion, motion prediction, and Cam2BEV-based BEV transformation.
- Contribution: The paper presents a machine-learning concept intended to identify road traffic situations directly from the ego-vehicle’s BEV, which the authors state is not provided by other reviewed methods.The work frames explicit situation identification as a missing capability in the reviewed literature.
III. CONCEPT
The proposed pipeline first detects environmental objects, then transforms the camera view into BEV to support understanding of the ego-vehicle’s surrounding traffic situation.
- Concept: Semantic segmentation identifies objects in the camera image before projective transformation produces a BEV representation of the ego-vehicle’s vicinity.The sequence follows the stated progression from environmental object identification to overall context understanding.
A. Semantic Segmentation Network (SemSeg)
The concept combines semantic segmentation, Cam2BEV transformation, and SIN classification to map camera imagery to one of five defined traffic situations.
- Semantic Segmentation Network (SemSeg): DeepLabv3+ performs semantic segmentation on front-camera RGB frames and supplies the resulting image to Cam2BEV.The segmentation module uses an off-the-shelf algorithm to identify surrounding objects.
- Projective Transformation Network (Cam2BEV): Cam2BEV projectively transforms semantically segmented front-view features into a corrected BEV image and includes a semantic class for occlusions.The transformation uses a spatial transformer unit with a uNetXST network.
- Situation Identification Network (SIN): SIN classifies the Cam2BEV output into one of five configured situation classes representing the ego-vehicle’s road context.The BEV representation is used because it provides clearer information about the surroundings than the front view.
- Situation Classes: The five classes cover free and occluded intersections, free drive, free drive with parked-vehicle or building occlusions, and occluded drive.The definitions distinguish intersection context, moving-vehicle occlusion, and static environmental occlusion.
- Training Data: The SIN dataset contains 2500 training images, 250 validation images, and 100 testing images, equally divided among the situation classes.The dataset is derived from the open-source Cam2BEV dataset and uses 1936 × 968 BEV images.
2) Dataset:
The situation-identification network uses a lightweight CNN architecture suited to a small, visually similar BEV dataset, within a concept illustrated by the proposed traffic-situation-identification figure.
- Dataset and Model: VGGNet-16 is selected as the SIN backbone because the dataset is small, visually similar, and has minimal foreground data.The paper states that these dataset characteristics make a complex or state-of-the-art network unnecessary.
3) Neural Network Architecture:
The architecture uses three cascaded neural networks trained separately for application-specific performance. Its identified situation can support rule-based decisions in other vehicle applications.
- 3) Neural Network Architecture:: The identified situation can support rule-based decisions intended to improve system robustness and reliability.The paper states that suitability for real-world data is evaluated later.
- 3) Neural Network Architecture:: Three cascaded neural networks are trained separately on different datasets for application-specific performance.The open-source and out-of-the-box model performance is not discussed in detail.
1) SemSeg module:
The evaluation uses a pre-trained semantic-segmentation module and supporting training components across simulation and real-world urban settings. The concept is validated in CARLA and assessed for portability with nuScenes and Cityscapes.
- 1) SemSeg module:: DeepLabv3+ pretrained on Cityscapes performs semantic segmentation without application-specific training.Its pretrained performance is excluded from evaluation of the SemSeg module.
- 1) SemSeg module:: Cam2BEV training uses simulated ground-truth BEV images covering approximately 50 m x 25 m and ten semantic classes, including occluded.The network uses one-hot encoding and is trained with the open-source loss function.
- 1) SemSeg module:: The SIN module uses transfer learning with ImageNet-initialized VGGNet-16 layers and is trained for 25 epochs.Training uses Adam with a 0.0001 learning rate and batches of size 15.
- 1) SemSeg module:: Although situation classes apply across road types, validation is conducted in urban CARLA, nuScenes, and Cityscapes settings.The evaluation setup covers both simulation and real-world urban datasets.
1) Simulation:
The concept is validated in CARLA using an urban town map and a scripted ego-vehicle client. The simulation tests camera configurations while running the neural-network pipeline to identify the ego-vehicle’s traffic situation.
- 1) Simulation:: CARLA provides a predefined urban town environment for validating accurate traffic-situation identification.The simulation goal is to demonstrate correct identification of the ego-vehicle’s situation.
- 1) Simulation:: A Python-controlled CARLA world synchronizes clients while the DySi Select client spawns the ego vehicle, attaches a camera, and runs the neural-network pipeline.Two camera settings are tested in the simulation.
2) Real-world Portability:
The software pipeline is evaluated in simulation and on real-world urban datasets using accuracy and inference time. Results show successful situation prediction, with performance depending on camera input and processing configuration.
- 2) Real-world Portability:: Real-world frames are processed sequentially, with situation prediction performed every 10th frame to reduce redundant processing.This assumes that vehicle conditions remain consistent at normal 30 FPS.
- 2) Real-world Portability:: Evaluation measures situation-prediction accuracy and average inference time, with predictions manually cross-verified against ego-vehicle ground-truth images.The same evaluation criteria are applied to simulation and real-world data.
- 2) Real-world Portability:: 67.3% to 80.2% correct situation inference is achieved across CARLA towns with the semantic-segmentation camera.The average inference time for semantically segmented inputs is 0.26s.
- 2) Real-world Portability:: 63.4% success and 0.5s average inference time are reported for the CARLA RGB-camera setup.The RGB setup takes almost twice as long because it includes semantic-segmentation inference, while segmentation also struggles with distant object boundaries.
2) Real-world Portability:
The situation-identification software transfers to real-world urban-driving datasets, performing best on Cityscapes and less reliably on nuScenes, especially at night.
- About 67% success was achieved on Cityscapes across three urban scenarios in Stuttgart.
- 60% performance was obtained on nuScenes daytime-urban scenarios, with the drop attributed to semantic segmentation pre-trained on Cityscapes.
- 61% overall success was achieved in nuScenes urban areas when nighttime scenarios were included.
- 34% success was achieved for nuScenes nighttime traffic situations, mainly from correctly predicting well-lit road areas.
- Average situation inference took about 0.37s-0.38s, with semantic segmentation consuming about 50% of the time.
V. CONCLUSION
The paper presents a lightweight BEV-based approach for classifying the ego-vehicle’s traffic situation and evaluates it in simulation and on real-world datasets. It positions situation identification as a basis for selecting relevant V2X data while identifying directions for broader sensing and scenario coverage.
- The approach classifies semantically segmented BEV images into five defined traffic-situation classes using a Situation Identification Network.
- The concept uses simple and lightweight neural networks to identify the vehicle’s traffic situation directly.
- The concept was verified with a prototype implemented in the CARLA simulation framework.
- Real-world portability was validated using images from the nuScenes and Cityscapes datasets.
- The identified ego-vehicle situation provides a basis for selecting relevant sensor data for transmission or backend retrieval in V2X.
- Future development includes larger scenario datasets and integrating sensors such as LiDAR or Radar into the situation-identification network.