Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes

1 Introduction

• Anomalies can be misclassified as inlier classes in urban scenes, making their recognition important for road safety.For example, a rock or animal mistaken for road could lead to a potentially fatal collision.Source: §1 · p. 1
• Uncertainty-based methods can miss anomalies and flag hard inliers, while reconstruction methods require an extra network that is inefficient for real-time embedded systems.The introduction also notes that some reconstruction methods require retraining when the inlier segmentation model changes under distribution shift.Source: §1 · pp. 1–2
• PEBAL jointly optimizes pixel-wise abstention learning and an energy-based model to learn an adaptive pixel-level anomaly class alongside the inlier classes.The model fine-tunes only the segmentation model’s last block for efficient training.Source: §1 · p. 3
• Energy-based pixel-wise penalties adapt to scene variation, while smoothness and sparsity constraints model local and global pixel dependencies.The paper links these constraints to reducing false-positive and false-negative anomaly predictions.Source: §1 · p. 3
• PEBAL achieves state-of-the-art classification accuracy on all relevant Fishyscapes benchmarks and the best classification results on LostandFound and Road Anomaly test sets.The introduction says these results significantly surpass competing methods.Source: §1 · p. 4

Uncertainty-based Anomaly Segmentation

• Uncertainty-based methods can misclassify object boundaries and have low pixel-wise detection accuracy or unreliable uncertainty estimates without proxy-outlier fine-tuning.They may assign high uncertainty to challenging inliers or low uncertainty to outliers after overconfident misclassification.Source: §2 · p. 4

Reconstruction-based Anomaly Segmentation

• Reconstruction-based methods depend on accurate segmentation and face complex, slow-to-train and slow-to-run reconstruction models, as well as low-quality reconstructions.Source: §2 · p. 4

Anomaly Segmentation via Outlier Exposure

• Outlier-exposure methods use auxiliary outlier datasets, but maximizing outlier uncertainty can degrade inlier segmentation and disproportionate outlier data can bias predictions toward anomalies.The paper says its approach respects the assumption that anomalies are rare, occupy a small part of training data, and appear around inliers.Source: §2 · p. 4

Abstention Learning

• Abstention learning adds a reserve anomaly class, but its single fixed penalty does not adapt to varying pixel-level anomalies; PEBAL instead learns pixel-wise energy-based penalties.The cited passage distinguishes this from uncertainty methods, which can predict anomalies even when the model is uncertain but confident.Source: §2 · pp. 4–5

Energy-based Models

• The energy-based model assigns low energy to inlier training samples and high energy to outliers, with logsumexp-based scores avoiding partition-function estimation.Source: §2 · p. 5

3.1 Training Set

• The training outlier set uses an anomaly class reserved as Y + 1, and its anomaly types do not overlap with those in testing.Source: §3.1 · p. 5

3.2 Pixel-wise Energy-biased Abstention Learning (PEBAL)

• PEBAL jointly trains PAL and EBM, with EBM estimating a pixel-specific abstention penalty instead of using one tuned value across training samples.The penalty is designed to be low for outlier pixels and high for inlier pixels.Source: §3.2 · p. 6, Figure 2 · p. 6
• Energy-margin losses separate inlier and outlier pixel energies, while regularization encourages sparse, spatially smooth anomaly maps.The energy margins create a gap between normal and abnormal pixels; regularization weights smoothness and sparsity across neighboring pixels and the energy map.Source: §3.2 · p. 7

3.3 Training and Inference

• AnomalyMix pastes labeled anomalous objects into inlier images to retain inlier context, and training fine-tunes only the final classification block.This avoids retraining the whole segmentation model; the paper also states that this fine-tuning keeps the original dataset's inlier segmentation accuracy.Source: §3.3 · p. 8
• At inference, PEBAL scores each pixel using inlier free energy, then applies Gaussian smoothing to the energy map.Source: §3.3 · p. 8

4.1 Datasets

• LostAndFound contains real-world small obstacles, while Fishyscapes and Road Anomaly include more diverse objects and domain shifts from Cityscapes.LostAndFound has 1,203 test images from 13 street scenes and 37 anomalous-object types; Road Anomaly has 60 internet-sourced images.Source: §4.1 · pp. 8–9, Remarks – Superior Performance on… · p. 12

4.2 Implementation Details

• PEBAL fine-tunes the final classification block of Cityscapes-trained DeepLabv3+ models with WideResnet38 or ResNet101 backbones.Training runs for 20 epochs; COCO outliers are mixed into Cityscapes images using AnomalyMix.Source: §4.2 · p. 9

4.3 Evaluation Measures

• Evaluation reports AUROC, AP, and FPR95, using AP and FPR95 for Fishyscapes leaderboard comparisons.Source: §4.3 · p. 9

4.4 Comparison on Anomaly Segmentation Benchmarks

• On LostAndFound, PEBAL improves AP by 15% to 40% and reduces FPR95 by about 5% to 70% against prior state-of-the-art methods.It reports 1.03% FPR95; fine-tuning sacrifices 0.2%–0.7% Cityscapes mIoU.Source: Comparison on LostAndFound · p. 10, Table 1 · p. 10, Comparison on Fishyscapes validation… · p. 12
• PEBAL surpasses prior approaches across the reported Fishyscapes and Road Anomaly benchmarks, with results tested using WideResnet38 and Resnet101 backbones.On the Fishyscapes leaderboard, it surpasses same-category methods by about 13% to 42% on LostAndFound and 40% to 50% AP on Static; the reported gains on the three validation/testing benchmarks are 20% to 50%.Source: Comparison on Fishyscapes Leaderboard · p. 10, Comparison on Fishyscapes validation… · p. 11, Table 2 · p. 11

4.5 Ablation Study

• Joint EBM and pixel-wise abstention training reaches 77.19% AP and 1.19% FPR on LostAndFound, exceeding the individual-module results.The reported individual EBM and abstention results are AP=70.2, FPR=8.9 and AP=72.7, FPR=3.8, respectively; smoothness and sparsity losses further improve performance.Source: §4.5 · pp. 12–13, Table 5 · p. 13

4.6 Outlier Samples and Computational Efficiency

• PEBAL maintains AP and FPR performance across varied COCO outlier classes and training-sample amounts, including training with one outlier class.Source: Outlier Diversity and Efficiency · pp. 13–14, Table 7 · p. 14
• PEBAL uses 1.3M trainable parameters, trains in about 12 minutes per epoch, and has a 0.55-second mean inference time per image.The comparison reports 137.1M parameters and 0.85 seconds for Meta-OoD, and 157.3M parameters and 1.95 seconds for Synboost.Source: Computational Efficiency · p. 14

• PEBAL fine-tunes a segmentation model’s last block and jointly optimizes pixel-wise abstention learning with an energy-based model.Source: §5 · p. 14
• Across four benchmarks, the authors report accuracy and robustness regardless of the amount or diversity of exposed training outliers.Source: §5 · p. 14
• PEBAL is less effective on Road Anomaly, which contains more diverse and realistic anomalous objects; improving generalization to unknown anomalies is future work.Source: §5 · p. 14

• The examples show effective detection of small and distant objects.Source: A Qualitative results · p. 19
• The examples include objects at different scales across rows 1 to 5.Source: A Qualitative results · p. 19
• Small and distant objects appear in rows 6 and 7 of Figure S1.Source: A Qualitative results · p. 19

• PEBAL achieves consistently state-of-the-art AUC regardless of outlier-class selection or the number of outlier training samples.Source: B More AUC results · p. 19, Table S1 · p. 20, Table S2 · p. 20
• Table S1 reports Fishyscapes AUC results across different outlier-class diversity levels, averaged over six random seeds.Source: Table S1 · p. 20
• Table S2 reports Fishyscapes AUC results across different amounts of outlier training samples, averaged over six random seeds.Source: Table S2 · p. 20

C Hyper-parameters Selection

• PEBAL reaches AP∈[76.9, 78.3] and FPR∈[0.8, 1.3] on LF test set across the reported EBM margins.The ranges are for min ∈[−12, −22] and mout ∈[−2, −8].Source: C Hyper-parameters Selection · p. 20
• AP=78.29 for λ = 0.01 and AP=77.15 for λ = 0.1 on the LF test set.The paper describes this as a small performance gap.Source: C Hyper-parameters Selection · p. 20
• Testing reports a small performance gap between λ ∈{0.1, 0.01} on the LF test set.Source: C Hyper-parameters Selection · p. 20

D Training Details on Cityscapes

• The WideResnet38 checkpoint was pretrained on Mapillary Vista and fine-tuned on Cityscapes using label relaxation loss and sdc-aug label propagation, with an alternate validation split.Its validation split is {cv2: monchengladbach, strasbourg, stuttgart}, rather than the standard cv0 split.Source: D Training Details on Cityscapes · p. 20
• The Resnet101 checkpoint uses the standard cv0 Cityscapes train/validation split and default DeepLabv3+ formulations.The passage reports 80.3 mIoU on Cityscapes Val for this checkpoint.Source: D Training Details on Cityscapes · p. 20
• Table S3 covers Fishyscapes LostandFound and Static validation sets and the Road Anomaly test set with WideResnet38 under the standard cv0 split.Source: Table S3 · p. 21

E Results Based on Different DeepLabv3+ Checkpoint

• PEBAL surpasses previous SOTA SML by 40%, 50%, and 20% of AP on the three datasets, respectively.Source: E Results Based on Different… · p. 21
• PEBAL outperforms previous methods by a large margin on all three benchmarks regardless of backbone, segmentation accuracy, or Cityscapes train/validation split.Source: E Results Based on Different… · p. 21
• PEBAL achieves the best AUC and FPR results on all datasets.Source: E Results Based on Different… · p. 21