HSI-Road Relabeled: Surface-Aware Road-Scene Segmentation

1. INTRODUCTION

• Surface-aware road understanding matters for traction control and road-condition monitoring, as well as distinguishing paved terrain, standing water, and drivable grassy patches.Existing binary road labels support free-space detection but not these surface-oriented tasks.Source: §1 · p. 1
• NIR may complement RGB because materials have different reflectance across visible and near-infrared wavelengths, including surfaces that appear similar in RGB.The text notes that robustness requires empirical evaluation for each class and downstream architecture.Source: §1 · p. 1
• HSI-Road’s binary masks do not support material-level segmentation, despite containing varied drivable surfaces such as asphalt and dirt.The dataset had not previously been extended with multiclass surface-oriented annotations.Source: §1 · p. 1
• The work contributes six-class surface annotations, an RGB-to-NIR registration pipeline, four input configurations, and a multilabel-stratified train/validation/test split.The configurations are original-resolution RGB, registered low-resolution RGB, NIR, and channel-stacked registered RGB–NIR.Source: §1 · p. 1

2.1. RGB and NIR for Road Scene Understanding

• Explicit matched-resolution RGB-versus-NIR comparisons for surface-oriented multiclass segmentation remain underexplored in autonomous driving.This gap makes it difficult to separate multimodal gains from registration artifacts and spatial-resolution mismatch.Source: §2.1 · p. 1
• Remote sensing combines NIR and RGB to discriminate materials such as vegetation, moisture, and mineral surfaces, while their exploration in autonomous driving remains limited.The passage notes that spatial misregistration can severely degrade detection performance.Source: §2.1 · p. 1

2.2. HSI-Road Dataset and HSI Segmentation in AD

• HSI-Road contains 3,799 paired RGB and 25-channel NIR images with binary Background and Road masks, covering asphalt, concrete, and dirt track.Over 60% of images are on rural roads.Source: §2.2 · p. 1
• The binary labeling scheme limits HSI-Road to drivable-area segmentation, motivating road-surface-oriented annotations.The dataset’s binary labels do not distinguish its varied road surfaces.Source: §2.2 · pp. 1–2
• Prior HSI road-scene studies have used pixel-level ANN classification and baseline models including UNet, DeepLabV3+, and SegFormer [3, 6–8].The passage characterizes deep learning in HSI for autonomous driving as relatively limited.Source: §2.2 · p. 2

3.1. Six-Class Surface-Oriented Taxonomy

• The relabeled taxonomy contains six mutually exclusive pixel classes: Background, Asphalt, Concrete, Dirt, Water, and Grass.Annotations were manually created using original-resolution RGB images and a custom segmentation tool.Source: §3.1 · p. 2
• Relabeling is not a strict subdivision of the original Road class: it redraws surface boundaries and can assign formerly Background pixels to Water or Grass.Thus the new annotations change both semantic granularity and the spatial extent of labeled surfaces.Source: §3.1 · p. 2

3.2. Spatial Registration: RGB-to-NIR Modality

• Registration forms a pseudo-NIR grayscale from RGB channels; when initial NCC is below 0.70, remaining channel triplets are tested against that threshold.If none reaches the threshold, the highest-NCC triplet is retained.Source: §3.2 · p. 2
• ORB with SIFT fallback and RANSAC estimate an affine alignment, then ECC refinement estimates a residual transform that is composed with it.Across mapped pairs, mean ± SD NCC is 0.73 ± 0.12 and mean inlier reprojection residual is 1.22 ± 0.15 px on the NIR grid.Source: §3.2 · p. 2
• The RGB-to-NIR mapping is also applied to semantic masks using nearest-neighbor interpolation to preserve categorical labels.Source: §3.2 · p. 2
• A checkerboard of alternating RGB and NIR patches visualizes spatial alignment.Source: Figure 1 · p. 2

4.1. Segmentation Backbones

• The evaluation uses six segmentation models, including two UPerNet encoder variants; the UPerNet comparison holds the decoder fixed while changing the encoder.For NIR and RGBNstk, the model input layers accept 25 and 28 channels, respectively.Source: §4.1 · pp. 2–3

4.2. Data Split, Model Training and Evaluation

• The 3,799 images are split into 2,659/570/570 train/validation/test images using multilabel stratification, and each model is trained three times from scratch.RGBreg, NIR, and RGBNstk share 192 × 384 resolution; RGBori-to-RGBreg comparisons combine resolution and registration effects.Source: §4.2 · p. 3

4.4. Effect of Spatial Mapping: RGBori vs. RGBreg

• RGBori-to-RGBreg mapping reduces mIoU by 4.01–5.97 points across models; averaged across models, Water drops 12.68 IoU points and Grass drops 8.89.The RGBori–RGBreg comparison combines effects of resizing, registration, interpolation, residual misregistration, and mask transformation.Source: §4.4 · p. 3, §4.2 · p. 3

4.5. Matched resolution: NIR vs. RGBreg vs. RGBNstk

• RGBNstk beats RGBreg in mean mIoU for five of six models, with gains of 1.24–3.52 points; Water improves by 4.02–10.20 points for all six.NIR alone beats RGBreg in two models and decreases performance in four; Grass gains are less consistent than Water gains.Source: §4.5 · pp. 3–4, Table 5 · p. 4

4.3. Overall Performance

• Six-class mIoU spans 73.39–82.73% and mF1 81.70–89.43%; UPerNetMiT-B0 leads RGBori, RGBreg, and RGBNstk, while UPerNetEN-B0 leads NIR.At matched resolution, RGBNstk exceeds RGBreg by 2.63 mIoU points for UPerNetMiT-B0.Source: §4.3 · p. 3, Table 5 · p. 4

4.6. Backbone Comparison

• UPerNetMiT-B0 leads on RGBori, RGBreg, and RGBNstk, while UPerNetEN-B0 leads on NIR; SegFormer with RGBNstk is the strongest non-UPerNet configuration.On NIR, the UPerNet variants differ by 3.95 mIoU and 3.45 mF1 points despite sharing a decoder.Source: §4.6 · p. 4

4.7. Computational Efficiency

• RGBNstk reduces peak GPU memory by 72.0% for DeepLabV3+ to 84.5% for SegFormer compared with RGBori, while accuracy, latency, parameters, and memory trade off.SegFormer has the lowest parameter count and latency; UPerNet variants achieve higher six-class accuracy but use 2.87–3.12× as many parameters.Source: §4.7 · pp. 4–5, Table 6 · p. 4

Training from scratch

• All models are trained from scratch; the authors identify HSI-specific pretrained vision and hyperspectral foundation models as future work.The scratch-training choice is made to maintain a consistent comparison across input modalities.Source: §5 · p. 5

RGB–NIR stacking

• RGBNstk uses channel stacking, while intermediate, late, or attention-based fusion and band selection or dimensionality reduction remain to be investigated.The authors cite adjacent-band redundancies as a reason to investigate dimensionality reduction.Source: §5 · p. 5

Registration and resolution are confounded

• The study does not isolate spatial resolution, class distribution, boundary complexity, or data-splitting strategy as separate factors.The authors leave isolation of these confounds for future work.Source: §5 · p. 5

• The work adds manually annotated six-class surface labels, an RGB-to-NIR registration pipeline, and a reference multilabel-stratified train/validation/test split.The six-class taxonomy reveals clearer differences among modalities and SSMs than the original binary masks.Source: §6 · p. 5
• 82.73% mIoU and 89.43% mF1: RGBori with UPerNetMiT-B0 achieves the highest overall result, both across SSMs and per class.Source: §6 · p. 5
• RGBori has 12× RGBreg’s pixel count; mapping it to RGBreg reduces mIoU by 4.01–5.97 points and mF1 by 3.57–4.80 points across SSMs.Source: §6 · p. 5
• At 192×384, RGBNstk improves over NIR for all six SSMs and RGBreg for five of six, with Water showing the most consistent class-level improvement.Gains over RGBreg reach 3.52 mIoU and 2.92 mF1 points.Source: §6 · p. 5
• The authors recommend matched-resolution modality comparisons; RGB–NIR gains are measurable but SSM- and class-dependent, and encoder choice should reflect deployment constraints as well as accuracy.Source: §6 · p. 5