DriveOSS: Dataset and Benchmark for 3D Anomaly Segmentation and Open-set Semantic Segmentation in Autonomous Driving

1. Introduction

• Closed-set 3D methods assign every point to a predefined class, while open-set perception must recognize ID classes and detect OOD objects.Unknown obstacles such as construction materials or fallen cargo may be misclassified as known classes, potentially making downstream decisions unreliable.Source: §1 · p. 1
• 3D open-set progress and standardized benchmarks remain limited, especially for adverse weather and illumination conditions.Existing studies often evaluate closed-set segmentation and anomaly detection separately, while many driving datasets underrepresent nighttime, rain, snow, and reduced visibility.Source: §1 · p. 1
• The dataset provides synchronized LiDAR and surround-view RGB data with point-wise annotations across diverse weather and illumination conditions.It separates ID scenes containing known classes from OOD scenes containing unknown objects not used during training.Source: §1 · p. 1, Figure 1 · p. 2
• DriveOSS establishes benchmarks for 3D anomaly segmentation and open-set semantic segmentation, alongside closed-set semantic evaluation on the ID subset.It adapts open-IoU from pixels to 3D points to jointly evaluate known-class recognition and unknown-object detection.Source: §1 · p. 2

2. Related Work

• Many automotive 3D segmentation datasets use closed-set taxonomies and are not designed to evaluate unknown-object recognition; adverse visibility is also underrepresented.The comparison table defines DriveOSS as having 32 raw labels, with 19 ID classes retained after merging labels and excluding ignored labels; outlier is additionally used for anomaly and open-set semantic segmentation.Source: §2 · p. 2, Table 1 · p. 3
• 2D benchmarks have advanced anomaly detection and open-set scene understanding, but extending open-set perception to 3D is harder because LiDAR is sparse and sensor-dependent.LiDAR measurements are affected by range, beam patterns, occlusion, and sensor-specific characteristics.Source: §2 · p. 2
• Synthetic 3D benchmarks enable scalable evaluation but retain a domain gap from real sensor measurements.Mixed real–synthetic OOD benchmarks insert objects into real scans, but cannot fully reproduce material-dependent reflections, occlusions, noise, and sampling patterns.Source: §2 · p. 2

3.1. Data Acquisition

• The vehicle carries six Basler cameras, an Ouster OS1-128 LiDAR, and GPS/IMU, synchronized to a common GPS-derived time reference.The cameras operate at 10 Hz, the LiDAR at 20 Hz, and the GPS/IMU at 600 Hz.Source: §3.1 · p. 3, Figure 2 · p. 3
• Data cover diverse urban, rural, pedestrian, parking, and industrial settings in clear, cloudy, rainy, snowy, daytime, and nighttime conditions.DriveOSS-OOD includes both safely placed controlled anomalies and naturally occurring anomalies encountered during regular driving.Source: §3.1 · p. 3

3.2. Data Annotation

• Anomalies are unknown physical obstacles posing potential collision or navigation risks; non-ID objects on or immediately beside drivable surfaces are labeled outlier.Permanent off-road fixtures outside the ID taxonomy are labeled other-object, while noisy returns and points outside the region of interest are unlabeled.Source: §3.2 · p. 4
• DriveOSS retains fine-grained known-class annotations alongside an outlier class, rather than reducing scenes to inlier, outlier, and unlabeled categories.After merging dynamic counterparts and excluding ignored labels, 19 ID classes aligned with SemanticKITTI remain alongside the 20th outlier class.Source: §3.2 · p. 4
• Ground truth starts from coarse automatic pre-labels, then receives manual refinement and independent cross-checking by at least one additional annotator.Annotators use ground/plane removal, class-isolation filters, and surround-view images during refinement.Source: §3.2 · p. 4

3.3. Data Analysis

• DriveOSS contains 9 ID training, 10 ID validation, 17 OOD validation, and 16 OOD test sequences, comprising 12,661 LiDAR frames and 1,522,748 annotated OOD points.Outlier occurs exclusively in DriveOSS-OOD.Source: §3.3 · p. 5

4.1. Closed-Set 3D Semantic Segmentation

• Closed-set segmentation assigns each LiDAR point to a predefined ID class and is evaluated on 19 classes using class-wise IoU and mean IoU.The benchmark includes LiDAR-only models and LiDAR–camera fusion models.Source: §4.1 · p. 6

4.2. 3D Anomaly Segmentation

• Anomaly segmentation predicts a continuous per-point anomaly score and is evaluated with AUROC, AP, and FPR@95 for ID-versus-OOD discrimination.Methods include post-hoc scores on closed-set models and LIDO, which jointly produces semantic predictions and feature-based anomaly scores; all are trained on ID data only.Source: §4.2 · p. 6

4.3. 3D Open-Set Semantic Segmentation

• Open-set segmentation assigns points to one of 19 ID classes or an outlier class, combining semantic predictions with anomaly scores and thresholding on validation data.The fixed validation-calibrated threshold is applied to test predictions; evaluation reports class-wise open-IoU, mOpen-IoU over ID classes, and outlier-class IoU.Source: §4.3 · p. 6

5.1. Implementation Details

• Models train on ID classes with SemanticKITTI and Panoptic-CUDAL augmentation; each method's anomaly threshold is calibrated on OOD validation at 95% OOD true positive rate and fixed for test.Neither DriveOSS-OOD samples nor anomaly annotations are used during training.Source: §5.1 · pp. 6–7

5.2. Benchmark Results

• 58.35% and 40.99% are LIDO's highest mIoU values on DriveOSS-ID validation and DriveOSS-OOD test, respectively; all methods degrade on the OOD split.Fusion gains vary by model: LCPS improves over LCPS-lidar by 2.11 points on ID but 0.25 on OOD, while M4F3D* underperforms M4F3D by 2.65 points on OOD.Source: §5.2 · p. 7, Table 4 · p. 7
• 91.70% AUROC and 32.66% FPR@95 make LIDO the most balanced anomaly segmenter, while M4F3D+RbA's 12.57% AP coexists with 100% FPR@95.Post-hoc anomaly scoring depends strongly on the underlying representation, and no score dominates across architectures.Source: §5.2 · p. 7, Table 5 · p. 7
• 22.98% mOpen-IoU and 1.07% outlier IoU are LIDO's best reported values, yet its mOpen-IoU remains 18.01 points below its closed-set mIoU on the same test scenes.Neither closed-set accuracy nor threshold-independent anomaly ranking reliably predicts open-set performance.Source: §5.2 · pp. 7–8
• Multimodal fusion improves in-domain closed-set recognition but provides no consistent advantage in anomaly or open-set segmentation.The results expose a mismatch between known-class recognition, anomaly ranking, and thresholded open-set prediction.Source: §5.2 · p. 8
• 0.25 ms per frame is the maximum post-hoc scoring overhead; M4F3D* takes 106.5 ms per frame versus 180.0 ms for LCPS.M4F3D* adds 2.14 ms over M4F3D, while LCPS adds 94.36 ms over LCPS-lidar; low fusion overhead does not ensure robust ID–OOD separation.Source: §5.2 · p. 8

• DriveOSS provides synchronized multimodal data and point-wise annotations for known classes and OOD objects across diverse locations, weather, and illumination.Source: §6 · p. 8
• The paper adapts open-IoU to 3D point clouds and benchmarks LiDAR-only and LiDAR-camera fusion models with post-hoc anomaly scoring and a feature-based method.Source: §6 · p. 8
• Current approaches still struggle to detect unknown objects while preserving accurate semantic predictions for known classes.Source: §6 · p. 8

• The current benchmark focuses on single-frame perception with a fixed sensor configuration and does not explicitly exploit temporal information.Source: §7 · p. 8
• Future work will investigate temporal and unified open-set segmentation models that jointly optimize known-class recognition and unknown-object detection.Source: §7 · p. 8
• The authors plan to extend DriveOSS to open-set occupancy prediction for voxel-level reasoning over free space, known classes, and unknown objects.Source: §7 · p. 8

A.1. Sensor Setup; A.2. Privacy Protection

• DriveOSS uses one LiDAR, six surround-view RGB cameras, and a GPS/IMU system.Before release, faces and license plates are blurred automatically and checked manually at the frame level.Source: §A.1 · p. 12, §A.2 · p. 12

B.1. Closed-Set 3D Semantic Segmentation

• Closed-set segmentation evaluates IoU and mIoU on ID classes, excluding outlier and ignored points.The supplement defines class IoU as TP divided by TP, FP, and FN combined.Source: §B.1 · p. 12

B.2. 3D Anomaly Segmentation

• Anomaly segmentation treats ID-versus-OOD detection as binary discrimination and evaluates continuous scores with AUROC, AP, and FPR95.Ignored points are excluded from evaluation.Source: §B.2 · p. 12

B.3. 3D Open-Set Semantic Segmentation

• Open-set segmentation combines per-point ID-class predictions with an outlier label, using a validation-calibrated threshold fixed for test evaluation.Open-IoU accounts for errors involving OOD points, and the reported final score is mean open-IoU over ID classes.Source: §B.3 · p. 12

C. Query-Level Camera-LiDAR Fusion

• The camera–LiDAR fusion model adds image cross-attention to decoder queries after LiDAR cross-attention, while retaining the sparse 3D backbone and query initialization.The design injects appearance information without explicit point-to-pixel projection and adds little latency over the LiDAR-only model.Source: C. Query-Level Camera-LiDAR Fusion · pp. 12–13

D. Implementation Details

• Models train on ID data only, while OOD validation and test use geographically disjoint locations and no OOD samples or annotations enter training.Every weather condition is represented in both DriveOSS-ID training and closed-set validation splits.Source: D. Implementation Details · p. 13

E. Additional Results

• Additional results include class-wise closed-set segmentation on ID validation and OOD validation and test, plus anomaly and open-set segmentation on OOD validation.The supplied table captions identify these result scopes but do not provide numerical outcomes.Source: E. Additional Results · p. 13, Table S2 · p. 14, Table S3 · p. 14

F.1. Maximum Softmax Probability (MSP)

• MSP scores uncertainty as one minus the highest softmax class probability; lower maximum probabilities indicate uncertain or potentially OOD regions.The method uses the maximum softmax probability as its confidence measure.Source: §F.1 · p. 14

F.2. Maximum Logit (MaxLogit)

• MaxLogit uses the negative highest pre-softmax logit, so higher scores indicate greater uncertainty or OOD likelihood.It bypasses probability calibration and operates directly on pre-softmax activations.Source: §F.2 · p. 14

F.3. Energy Score

• Energy summarizes activation magnitude across classes; higher energy indicates uncertain or OOD samples.The method describes lower energy as characteristic of confident in-distribution predictions.Source: §F.3 · pp. 14–15

F.4. Entropy

• Entropy measures uncertainty in the class-probability distribution, reaching its maximum when classes are equally likely and zero for a fully confident prediction.The probabilities are softmax outputs.Source: §F.4 · p. 15

F.5. Rejected by All (RbA)

• Rejected by All assigns high anomaly scores to points rejected by every known-class prediction, and scores are oriented so larger values indicate higher OOD likelihood.Source: §F.5 · p. 15

G. Societal Impact

• DriveOSS has finite geographic, sensor, and anomaly coverage, so benchmark performance does not establish deployment readiness and should complement broader safety validation.Released RGB images undergo automatic anonymization and manual verification.Source: G. Societal Impact · p. 15

H. Qualitative Results

• Closed-set segmentation assigns each point to an ID class, anomaly segmentation highlights candidate OOD regions, and open-set segmentation combines both while labeling detected unknowns as outliers.Source: H. Qualitative Results · p. 15
• Figure S2 organizes projected LiDAR points by clear, night, cloudy, rainy, and snowy conditions and by ID, OOD validation, and OOD test splits.The projections also qualitatively verify LiDAR–camera calibration.Source: H. Qualitative Results · p. 15
• Rainy examples show visibly fewer road-surface LiDAR returns, illustrating a sensing challenge for LiDAR and multimodal perception.Source: H. Qualitative Results · p. 15

I. Annotation Workflow and Quality Control

• A SemanticKITTI-pretrained Mask4Former generates coarse known-class labels, and KISS-ICP registers consecutive scans to provide temporal context during labeling.Source: §I · p. 15
• Annotators use synchronized surround-view images, ground and plane removal, and class-isolation filters to refine boundaries and correct mislabeled points.Source: §I · p. 16
• Unknown objects are manually labeled as outliers because the pre-labeling model predicts only closed-set classes, and every sequence receives an independent annotator cross-check.The workflow uses a unified class definition and spatial-hazard policy.Source: §I · p. 17
• Figure S3 shows temporally registered point clouds alongside RGB images, while Figure S4 compares raw pre-labels with human-refined ground truth across ID and OOD splits.Displayed examples include corrections to pre-labels and manual annotation of unknown road hazards.Source: §I · p. 17, Figure S3 · p. 18, Figure S4 · p. 19