PaperlayerView exact revision on arXiv

Source-linked AI summary

RELLIS-3D Dataset: Data, Benchmarks and Analysis

Peng Jiang, Philip Osteen, Maggie Wigness, Srikanth Saripalli

arXiv:2011.12954v4cs.CVcs.RO

TL;DR

Existing autonomy datasets largely represent urban environments or lack multimodal off-road data needed for semantic scene understanding. RELLIS-3D provides synchronized multimodal off-road data with extensive image and LiDAR annotations and benchmarks current segmentation models. The dataset exposes substantial challenges from off-road structure and class imbalance, motivating further research.

  • Problem

    Autonomous navigation research lacks a large multimodal off-road dataset with extensive ground-truth annotations, despite the importance of semantic understanding in such environments.

  • Method

    The paper introduces RELLIS-3D, five synchronized off-road sensor sequences with raw RGB, LiDAR, stereo, GPS, and IMU data, annotations, benchmark splits, and segmentation evaluations.

  • Results

    RELLIS-3D presents challenges for current semantic segmentation algorithms, including extreme off-road class imbalance and environmental irregularity.

  • Takeaways & Limitations

    RELLIS-3D provides resources for developing advanced semantic segmentation and multimodal methods for autonomous navigation in off-road environments.

Abstract

from arXiv · show

Semantic scene understanding is crucial for robust and safe autonomous navigation, particularly so in off-road environments. Recent deep learning advances for 3D semantic segmentation rely heavily on large sets of training data, however existing autonomy datasets either represent urban environments or lack multimodal off-road data. We fill this gap with RELLIS-3D, a multimodal dataset collected in an off-road environment, which contains annotations for 13,556 LiDAR scans and 6,235 images. The data was collected on the Rellis Campus of Texas A\&M University and presents challenges to existing algorithms related to class imbalance and environmental topography. Additionally, we evaluate the current state-of-the-art deep learning semantic segmentation models on this dataset. Experimental results show that RELLIS-3D presents challenges for algorithms designed for segmentation in urban environments. This novel dataset provides the resources needed by researchers to continue to develop more advanced algorithms and investigate new research directions to enhance autonomous navigation in off-road environments. RELLIS-3D is available at https://github.com/unmannedlab/RELLIS-3D

I. INTRODUCTION

RELLIS-3D addresses the lack of large, multimodal off-road datasets by providing synchronized sensor data, extensive annotations, and benchmark evaluations for semantic segmentation.

  • Motivation: Off-road scenes have unstructured boundaries, uneven terrain, strong textures, irregular features, and environment-dependent class distributions that limit direct transfer from urban models.These properties make off-road semantic segmentation distinct from indoor and urban settings.
  • Dataset: RELLIS-3D was collected on the Rellis Campus of Texas A&M University using a Clearpath Robotics Warthog in diverse off-road areas.The facility includes runways, aprons, terrain, forests, bushes, pastures, and lakes.
  • Dataset: The dataset provides synchronized raw RGB, LiDAR, stereo, GPS, and IMU data, enabling multimodal and fusion algorithms to be developed without new collection.The raw autonomy data are synchronized using Precision Time Protocol and include the full sensor suite.
  • Annotations: 6,235 pixel-wise image annotations and 13,556 full LiDAR scans are provided across five collected sequences.The annotations support semantic segmentation in both image and point-cloud modalities.
  • Benchmark: The authors establish training, validation, and testing splits and evaluate state-of-the-art image and point-cloud semantic segmentation algorithms.The initial results are intended to expose off-road challenges and identify research directions.

II. RELATED WORK

Existing off-road datasets provide useful but incomplete coverage, leaving a gap in multimodal data and rich annotations for autonomous navigation research.

  • Existing datasets: Most autonomous-vehicle datasets represent urban environments and commonly provide only 2D RGB annotations.Examples include CamVid, Cityscapes, Mapillary Vistas, D2-City, and BDD100k.
  • Off-road datasets: RUGD offers rich off-road RGB annotations but lacks multiple modalities, while Freiburg Forest provides multimodal multispectral images but only 366 annotated images.These datasets cover important aspects of off-road perception but remain limited in modality or annotation scale.
  • Off-road datasets: YCOR includes image and point-cloud data but annotates only images, and the NREC dataset provides single-class bounding-box annotations for person detection.These annotation choices restrict their use for broader multimodal semantic segmentation.
  • RELLIS-3D: RELLIS-3D fills the off-road data gap with a full stack of multimodal sensors and multimodal annotations using a rich ontology.The dataset is presented as a resource for advancing off-road autonomous navigation research.

III. SENSOR SETUP AND CALIBRATION

The Warthog platform combines LiDAR, stereo and RGB cameras, and GPS/IMU sensing to collect synchronized multimodal autonomy data.

  • LiDAR: The sensor setup includes an Ouster OS1 64-channel LiDAR operating at 10 Hz with 2048 horizontal resolution and a 45° vertical field of view.This provides 3D laser measurements for the platform.
  • Cameras: A Nerian Karmin2 with SceneScan provides 3D stereo camera data at 10 Hz.The stereo system contributes depth-related visual sensing.
  • Cameras: A Basler RGB camera records 1920x1200 images at 10 Hz using a 16mm/F18 Edmund Optics lens.The RGB stream supplies dense color imagery for visual perception.
  • Navigation and computing: A VectorNav VN300 dual-antenna GNSS/INS provides GPS at 300 Hz and IMU measurements at 100 Hz.The platform uses separate navigation and vision computers for control, collection, and sensor processing.
  • Navigation and computing: The navigation and vision computers communicate over Ethernet and run Ubuntu Linux with ROS Kinetic.Their roles are divided between robotic control and data collection or sensor processing.

B. Synchronization

Sensor streams are synchronized primarily with PTP, while GPS/IMU data are aligned to LiDAR and camera frames by nearest timestamp.

  • Synchronization: PTP synchronizes the vision computer with the navigation computer, Ouster LiDAR, stereo camera, and RGB camera.This provides a common timing basis across the main sensor network.
  • Synchronization: Because GPS/IMU cannot use PTP, the system selects the GPS/IMU measurement with the closest timestamp to each LiDAR and camera frame.The GPS/IMU system updates at 100 Hz, supporting this nearest-timestamp alignment.
  • Annotations: RELLIS-3D ground-truth examples use dense pixel-wise labels for images and point-wise labels for LiDAR scans across 20 visual classes.The same ontology is applied to both modalities.

C. Camera Calibration

RELLIS-3D calibrates its cameras under a pin-hole projection model and estimates camera–LiDAR extrinsics using established methods. Without ground truth, calibration is checked against factory stereo parameters and mean line re-projection error.

  • Camera model: The cameras are modeled as pin-hole systems using perspective projection, with intrinsic calibration obtained through the ROS Camera Calibrator library.The calibration accounts for radial and tangential lens distortion through distortion coefficients.
  • Extrinsic calibration: Camera–LiDAR extrinsic calibration is determined using established calibration methods because cross-sensor calibration is required for multimodal fusion.
  • Validation: In the absence of ground truth, estimated calibration parameters are compared with factory stereo calibration and evaluated using mean line re-projection errors.MLRE is computed by projecting LiDAR-frame points on planar-target edges into the camera image.
  • Data collection: The dataset contains five traversal sequences collected on three non-paved trails at Texas A&M University’s Ground Research facility.Sequences cover bush- and tree-covered trails, pasture and forest, and a hill surrounded by a lake and highway.
  • Ontology: RELLIS-3D defines object and terrain classes derived largely from RUGD while adding classes such as mud, man-made barriers, rubble piles, puddle, and deep water.

B. Annotations

RELLIS-3D provides verified pixel-wise image annotations and LiDAR point-wise annotations generated through camera–LiDAR projection and refinement across overlapping scans. The image labels cover 6,235 annotated frames after downsampling and removing stationary duplicates.

  • Image annotations: Pixel-wise image labels were created by trained crowdsourced annotators, with one annotator assigned to each video sequence.Several rounds of in-house verification corrected missing, incorrect, or inconsistent labels.
  • Label distribution: Image label distributions are dominated by sky, grass, tree, and bush classes.
  • Point-cloud annotations: Point-wise LiDAR annotation was initialized by projecting more than 6,000 image annotations through camera–LiDAR calibration.Annotations were then refined for multiple overlapping LiDAR scans using the SemanticKITTI point-cloud annotation application.

C. Dataset Statistics

RELLIS-3D has highly imbalanced labels across both image and point-cloud modalities, with different dominant classes caused partly by sensor characteristics. The benchmark evaluates semantic segmentation using mean intersection-over-union across classes.

  • Class distribution: 94% of labeled image pixels belong to sky, grass, tree, and bushes, while grass, tree, and bushes comprise 80% of point labels.Differences in resolution, viewing angle, and sensing mechanism produce divergent label distributions between modalities.
  • Class imbalance: Off-road class imbalance depends strongly on the particular environment and is more severe within the dataset than across urban datasets.This makes semantic segmentation of rare classes more challenging than in urban environments.
  • Class distribution: Grass, tree, and bush dominate the point-cloud label population.
  • Evaluation metric: Mean intersection-over-union (mIoU) is the evaluation metric for semantic segmentation.It averages class-wise intersection-over-union over C classes.
  • Evaluation metric: For class c, mIoU uses TPc, FPc, and FNc to quantify overlap from true-positive, false-positive, and false-negative predictions.C denotes the number of classes.

B. Image Semantic Segmentation

The image segmentation benchmark evaluates HRNETV2+OCR and Gated-SCNN on defined RELLIS-3D splits. Both models perform substantially below their reported Cityscapes results, with class imbalance and unclear off-road boundaries identified as key challenges.

  • Models: The benchmark evaluates HRNETV2+OCR and Gated-SCNN for 2D image semantic segmentation.
  • Models: HRNet maintains high-resolution representations, while OCR aggregates features within object regions to improve pixel representations.
  • Models: Gated-SCNN uses separate classical segmentation and shape-processing streams to incorporate shape information.
  • Experimental setup: The image split contains 3,302 training images, 983 validation images, and 1,672 testing images, using 19 classes including void.The dirt class is omitted because it is extremely sparse in the annotations.
  • Results: The evaluation table reports results for single images with 20 classes and scans with 16 classes across all baselines.
  • Results: 52.92% mIoU for Gated-SCNN and 51.55% mIoU for HRNet+OCR fall below their Cityscapes results of 74.7% and 81.1%, respectively.The paper associates degradation mainly with serious class imbalance and also identifies unclear off-road boundaries as a challenge.

C. LiDAR Point Cloud Semantic Segmentation

The paper evaluates SalsaNext and KPConv for LiDAR point-cloud semantic segmentation on RELLIS-3D, adapting the experiments to classes observable in the scans. Both models perform substantially below their reported SemanticKITTI results, with class imbalance and unstructured features identified as key challenges.

  • Methods: SalsaNext and KPConv are evaluated as representative projective and point-based LiDAR segmentation methods.SalsaNext operates on projected LiDAR data, while KPConv applies local 3D convolution kernels to nearby points.
  • Experimental setup: 15 classes, including void, are used because LiDAR cannot establish points for sky and some distant objects.The point-cloud experiments retain the same train, validation, and test splits as the image experiments.
  • Results: 43.07% mIoU is achieved by SalsaNext, compared with 19.07% mIoU for KPConv on RELLIS-3D.The reported test split contains 3,343 scans, following training on 7,800 scans and validation on 2,413 scans.
  • Results: 59.5% mIoU for SalsaNext and 58.8% for KPConv on SemanticKITTI exceed their corresponding RELLIS-3D results.KPConv exhibits the more obvious degradation relative to its SemanticKITTI performance.
  • Discussion: Extreme class imbalance and unstructured off-road features are identified as the main causes of degraded point-cloud performance.The paper notes that KPConv does not learn on the whole LiDAR scan during training.

D. Discussion

The discussion argues that off-road navigation benefits from combining semantic, visual, inertial, depth, and registered LiDAR information. Examples involving bushes, hills, and partially occluded fences illustrate why geometry or single-scan perception can be insufficient.

  • Discussion: Extreme class imbalance challenges current semantic segmentation algorithms, while sparsely visible objects can remain important for navigation decisions.The discussion connects rare classes with autonomous decision making and navigation planning.
  • Discussion: RGB images provide denser and richer visual features, producing better image-based segmentation performance than LiDAR in the reported experiments.The paper nevertheless describes LiDAR as critical because it supplies complementary 3D information.
  • Discussion: Hill scenes show that combining point-cloud and image semantics can help distinguish bushes and support traversal decisions.Geometry alone may classify bushes as non-traversable obstacles, whereas semantic information can be used by higher-level autonomy components.
  • Discussion: Registered LiDAR scans can reveal a fence hidden behind bushes when image data or a single scan makes it difficult to detect.The paper links this integrated observation to avoiding incorrect or hazardous navigation decisions.
  • Discussion: The discussion proposes more abstract labels such as traversable, non-traversable, and up/down-hill to complement object-based semantic classes.It also states that inertial, visual, and depth information are vital and complementary for off-road tasks.

VI. SUMMARY AND FUTURE WORK

RELLIS-3D is introduced as a multimodal off-road dataset with pixel-wise and point-wise annotations, designed to expose the limitations of current segmentation models. Future work targets richer semantic labels, broader environmental diversity, and odometry benchmarks.

  • Summary: RELLIS-3D provides pixel-wise and point-wise annotations representative of off-road environments for semantic segmentation.The dataset is presented as a large-scale multimodal resource.
  • Summary: Current state-of-the-art models degrade significantly on RELLIS-3D because of imbalanced classes and unstructured off-road features.The authors use this result to characterize off-road navigation as distinct from conventional segmentation settings.
  • Future work: Future dataset extensions will include higher-order labels such as traversable, non-traversable, up-hill, and down-hill.These labels are intended to expand the dataset beyond object and terrain categories.
  • Future work: Because all data were collected at one outdoor facility, maintaining environmental diversity across train, validation, and test sets is a challenge.The authors plan to add different off-road environments to improve diversity.
  • Future work: Future work will investigate odometry benchmarks and whether semantic information can improve odometry estimation.The dataset’s accurate GPS and stereo imagery provide supporting resources for these investigations.
Loading 2011.12954v4…