PaperlayerView exact revision on arXiv

Source-linked AI summary

CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids

Toomas Tahves, Mauro Bellone, Raivo Sell

arXiv:2609.09881v1cs.CVcs.RO

TL;DR

Autonomous-driving segmentation must detect tiny, safety-critical VRUs from camera and LiDAR streams despite modality gaps and strict latency constraints. CLFTv2 uses hierarchical Swin encoders with per-scale residual fusion and a lightweight coarse-to-fine decoder, achieving competitive multi-dataset accuracy and efficiency while retaining a limitation on dense LiDAR integration.

  • Problem

    Tiny VRUs, complementary camera-LiDAR sensing gaps, and strict latency constraints create a challenging multi-modal segmentation problem.

  • Method

    CLFTv2 combines parallel hierarchical Swin encoders with per-scale residual fusion and a coarse-to-fine FPN-style decoder in the 2D projection domain.

  • Results

    Across three driving datasets, CLFTv2 achieves competitive accuracy against query decoders while demanding substantially lower GFLOPs and delivering 2.2× higher throughput.

  • Takeaways & Limitations

    Lightweight residual fusion performs on par with computationally demanding query-matching decoders under sparse and noisy supervision, while prioritizing VRU recall.

  • Takeaways & Limitations

    Local receptive fields limit cross-modal integration on geometrically dense point clouds such as Waymo, motivating hybrid architectures with sparse long-range token mixing.

Abstract

from arXiv · show

Semantic segmentation for autonomous driving requires reliable detection of vulnerable road users (VRUs) despite heavy class imbalance. We introduce CLFTv2, a hierarchical camera-LiDAR fusion framework replacing global ViT attention with a Swin-based multi-scale encoder and a lightweight FPN-style residual decoder. Operating in the 2D perspective domain, CLFTv2 integrates multi-scale geometric cues through shifted-window attention and per-scale residual fusion, avoiding the computational overhead of query-matching decoders. Across three driving datasets, CLFTv2 consistently improves VRU recall. On ZOD, CLFTv2-Large achieves 53.5\% mIoU, improving pedestrian IoU from 35.5\% to 44.9\% over the prior CLFT model. On Waymo, CLFTv2 reaches 61.7\% mIoU. Additionally, a modality-isolation study suggests ViT's global receptive field yields stronger fusion gains only under dense LiDAR returns. Compared to a Swin-based Mask2Former adaptation, CLFTv2 requires 1.4$\times$ fewer GFLOPs and delivers 2.2$\times$ higher throughput, while achieving comparable overall accuracy. These results demonstrate that hierarchical local-attention fusion offers an efficient, scalable alternative to global-attention and query-based decoders for real-time on-vehicle perception in intelligent transportation systems. Source code is publicly available.

I. INTRODUCTION

CLFTv2 addresses the difficulty of detecting tiny, safety-critical VRUs from complementary but computationally demanding camera and LiDAR streams. It replaces CLFT’s fixed-resolution global ViT design with hierarchical Swin fusion and a lightweight residual decoder for efficient multi-scale perception.

  • VRUs occupy only 0.3-0.4% of perspective-image pixels, while cameras lack metric depth and LiDAR becomes sparse with distance.These complementary sensing gaps make distant VRU feature extraction especially challenging.
  • Dense multi-modal processing must remain lightweight because autonomous-driving systems face strict latency constraints.
  • CLFT’s ViT encoder provides cross-modal context but incurs quadratic complexity and produces only a single spatial resolution.These properties make generating multi-scale pyramids for small-object detection expensive.
  • CLFTv2 uses shifted-window Swin attention, parallel weight-shared encoders, per-scale residual fusion, and a coarse-to-fine FPN-style decoder in the 2D projection domain.Projected LiDAR is encoded as a three-channel coordinate map for direct processing alongside camera input.
  • The architecture avoids the computational overhead of universal query-based decoders for closed-set semantic segmentation.
  • Across ZOD, Waymo, and ISEAuto, CLFTv2 achieves competitive accuracy while requiring substantially fewer GFLOPs and delivering 2.2× higher throughput.
  • A modality-isolation study identifies a trade-off: Swin improves small-object resolution, but ViT better exploits dense, localized LiDAR returns.The authors suggest hybrid global-local architectures as future work.
  • Cross-dataset initialization accelerates convergence and improves early-epoch accuracy across differing sensor characteristics and geographical datasets.

B. Hierarchical Vision Transformers for Dense Prediction

Hierarchical dense-prediction architectures combine Swin’s efficient local attention and multi-scale feature pyramid with FPN-style decoding. CLFTv2 applies this hierarchy to camera-LiDAR fusion while avoiding query-matching machinery that is unnecessary for fixed-class segmentation.

  • B. Hierarchical Vision Transformers for Dense Prediction: Swin uses shifted-window attention for linear image-resolution complexity and patch merging to construct a four-stage feature pyramid.Its hierarchy addresses the single-scale limitation of global ViT encoders.
  • B. Hierarchical Vision Transformers for Dense Prediction: FPNs normalize channels with lateral 1×1 projections and merge coarse semantic information with fine spatial detail through top-down upsampling.
  • B. Hierarchical Vision Transformers for Dense Prediction: CLFTv2 replaces the ViT-DPT stack with a Swin-FPN hierarchy and adds per-scale residual camera-LiDAR fusion before coarse-to-fine accumulation.
  • B. Hierarchical Vision Transformers for Dense Prediction: MaskFormer-style query matching introduces training overhead that does not directly correspond to closed-set, fixed-class semantic segmentation.The camera-LiDAR adaptation uses the same per-scale residual fusion method.
  • B. Hierarchical Vision Transformers for Dense Prediction: CLFTv2 contains Siamese Swin encoders, a four-scale coarse-to-fine residual decoder, and a lightweight head that restores the original input resolution.
  • B. Hierarchical Vision Transformers for Dense Prediction: The encoder partitions inputs into 4×4 patches and processes them through four hierarchical stages with local M×M attention windows.The main benchmarks use M=16 for Tiny at 256×256 and M=12 to 24 for Base and Large at 384×384.
  • B. Hierarchical Vision Transformers for Dense Prediction: The decoder starts at Stage 4 and progressively upsamples and sums features through Stages 3, 2, and 1 before producing the full-resolution prediction.
  • B. Hierarchical Vision Transformers for Dense Prediction: The architecture alternates shifted windows to permit cross-window information flow while maintaining local attention.

B. Siamese Feature Encoding

CLFTv2 encodes camera appearance and projected LiDAR geometry through parallel, parameter-sharing Swin streams. Per-level projections align channel dimensions while preserving native spatial resolution for consistent multi-scale fusion.

  • B. Siamese Feature Encoding: Camera and projected LiDAR tensors are processed by two Swin Transformer backbones sharing identical encoder parameters.
  • B. Siamese Feature Encoding: The shared encoder maps each three-channel modality input to a four-stage feature pyramid with strides 4, 8, 16, and 32.
  • B. Siamese Feature Encoding: Projected LiDAR represents each pixel with 3D coordinates (X, Y, Z), forming a dense three-channel geometry map compatible with standard vision backbones.
  • B. Siamese Feature Encoding: A learned 1×1 convolution projects every backbone level to D=256 while preserving its native spatial resolution.This normalizes feature scales and enables consistent fusion without resampling at that stage.

2) Coarse-to-Fine Residual Fusion:

CLFTv2 fuses camera and LiDAR features from coarse to fine using modality-specific residual refinement and accumulated upsampled context. The decoder combines coarse semantic context with fine spatial detail before producing the finest-scale representation.

  • The decoder processes four levels from coarsest stride 32 to finest stride 4, accumulating a fused representation at each scale.
  • At each scale, separate RGB and LiDAR ResConv units refine projected modality features before fusion through a shared output ResConv.
  • Finer levels bilinearly upsample the preceding coarser representation by 2× and add it to the refined modality features.This supplies top-down semantic context at the current spatial resolution.
  • The accumulated representation combines global semantic context propagated from the coarsest level with bottom-up fine-grained detail.The design follows the top-down pathway pattern of FPN while aligning visual and geometric features before linear combination.
  • The per-scale fusion block independently refines 1×1 projected features, sums them with upsampled context, and applies a final shared ResConv.
  • Each ResConv unit uses two consecutive 3×3 convolutions with ReLU activations and an identity skip connection.

3) Ablation Experiments for Design Choices:

The ablations show that coarse-to-fine recurrence is the dominant decoder design choice, while more elaborate fusion weighting provides little additional benefit under the tested supervision conditions. The decoder produces full-resolution class logits after lightweight feature refinement.

  • Fusion design: 12.8% absolute ZOD mIoU loss results when removing the recurrent upsampled context, decreasing mIoU from 35.9% to 23.1%.The ablation identifies the coarse-to-fine recurrence as the dominant contributor to accuracy.
  • Fusion design: ±0.7% is the maximum mIoU change from replacing element-wise summation with five gated or attention-weighted alternatives.These results indicate that extra fusion parameters were not warranted under the tested supervision conditions.
  • Segmentation head: The segmentation head applies 3×3 convolution, Batch Normalization, ReLU, and 1×1 convolution before upsampling predictions to the original resolution.The output consists of per-pixel logits for the semantic classes.
  • Evaluation setting: The evaluation uses three autonomous-driving datasets spanning different regions, sensor configurations, annotation methods, and class distributions.Waymo uses front-facing camera and LiDAR captures with four annotated classes, while ISEAuto uses two foreground classes and manually verified labels.

3) Zenseact Open Dataset (ZOD):

ZOD combines challenging driving conditions with semi-automatically generated four-class masks, while the preprocessing pipeline aligns camera and LiDAR inputs and applies weighted cross-entropy training.

  • Dataset: ZOD uses 2,300 frames split evenly between training and validation, with four-class masks generated through a semi-automated SAM-based projection pipeline.The masks remain pseudo-labels and may contain minor local inaccuracies, especially for occluded or truncated objects.
  • Dataset: ZOD masks should be interpreted as upper-bound estimates constrained by geometric visibility rather than pure sensor-level detection capability.The generation pipeline has a higher rejection rate for occluded or heavily truncated objects.
  • Input preprocessing: Camera images are resized to H×W and normalized with ImageNet statistics before processing.The same pretrained-backbone normalization is used across the camera inputs.
  • Input preprocessing: Projected LiDAR returns replace RGB channels with per-pixel x, y, and z coordinates, with dataset-specific quantization for 8-bit storage.The closest projected return is retained for each pixel, while empty pixels receive a neutral value.
  • Optimization: Training uses weighted cross-entropy with class weights derived from per-class pixel frequencies.The weighting schedule uses an approximate inverse-square-root rule.

A. Experimental Setup

Experiments use a shared PyTorch implementation and common batch-size regime while adapting training configurations and baseline implementations to each architecture.

  • Training configuration: All models are implemented in PyTorch and trained on NVIDIA A100 GPUs, with more than 800 hours of total training time.Training configurations were tailored to each architecture’s optimal setup.
  • Training configuration: A common batch size of 8 and base learning rate of 8×10^-5 are preserved across model setups.Optimizers, schedules, epochs, and input resolutions vary by architecture and dataset.
  • Training configuration: Data augmentation includes random horizontal flipping, ±20° rotation, and random resized cropping.The reported probabilities are 0.5, 0.4, and 0.3, respectively.
  • Training configuration: Training schedules differ by architecture to match convergence, with fewer epochs used for models that plateau earlier.CLFT results come from the unified codebase and supersede prior publications.
  • Baseline implementations: MaskFormer and Mask2Former are re-implemented natively in PyTorch to reduce framework-induced variance.The adapted implementations retain their original decoder designs while using the common camera-LiDAR pipeline.

4) Evaluation Metrics:

The evaluation emphasizes foreground performance under severe class imbalance and compares CLFTv2 with prior and query-based models across ZOD, Waymo, and ISEAuto.

  • Evaluation metrics: Foreground mIoU excludes background and focuses on vehicle, human, and sign classes because background can dominate standard mIoU.Frequency-Weighted Foreground IoU additionally weights classes by pixel frequency.
  • Evaluation protocol: Top-10 validation checkpoint averaging reduces single-epoch noise and is applied consistently across models.Dispersion is reported where available.
  • Dataset results: 53.5% mIoU is achieved by CLFTv2-Large on ZOD, versus 46.8% for CLFT-Large.Human IoU increases from 35.5% to 44.9%, while Sign IoU increases from 33.2% to 43.2%.
  • Dataset results: Query-based models lead on ISEAuto, with MaskFormer-Large at 75.6%, Mask2Former-Large at 75.4%, and CLFTv2-Large at 73.3%.The dataset ranking is therefore dependent on the evaluation dataset.
  • Dataset results: 61.7% mIoU is achieved by CLFTv2-Large on Waymo, while CLFT-Large and CLFT-Base reach 68.3% and 66.3%, respectively.The ViT-based CLFT models therefore outperform the Swin-based architecture on Waymo.
  • Fusion analysis: Global attention provides stronger fusion on Waymo’s dense LiDAR point clouds, whereas single-modality inputs show nearly identical performance across architectures.This comparison isolates the fusion interaction mechanism rather than backbone-only performance.

C. Ablation Studies

The ablations examine window size, modality contribution, and fusion design, revealing trade-offs among context, compute, sensor complementarity, and residual propagation. Residual fusion improves boundary recovery and accuracy over simpler alternatives.

  • Window-size ablation: 6.73% mIoU is lost on ZOD when CLFTv2-Base changes from window 24 to 16, while FLOPs fall from 119.0G to 52.7G.The comparison also changes input resolution, so the gap reflects both window size and resolution.
  • Modality ablation: All 32 modality settings improve over RGB-only with fusion, while Waymo fusion gains range from +4.76% to +11.30%.Waymo is geometry-dominant, with LiDAR-only exceeding RGB-only in 5/6 transformer settings.
  • Modality ablation: +10.52% is CLFT’s mean Waymo fusion gain over RGB-only, compared with +4.85% for CLFTv2.On ISEAuto, CLFTv2 gains only +0.34% to +1.21%, indicating limited marginal benefit from LiDAR under dense, simpler-label conditions.
  • Fusion design: Residual fusion joins optical boundaries with sparse depth and suppresses structural artifacts that remain under RGB-only, LiDAR-only, or simple averaging.The qualitative examples show recovered boundaries and reduced false-class or boundary artifacts.
  • Fusion design: 35.9% mIoU results from adding residual propagation, compared with 22.8% for simple averaging.Latency rises from 10.8 ms to 13.2 ms, a +22.2% increase; gating did not improve mIoU in this setup.

D. Transfer Learning and Cross-Dataset Convergence

The experiments assess transfer initialization, safety metrics, environmental robustness, and inference efficiency across datasets and conditions. CLFTv2 favors recall and efficiency, but its relative accuracy depends on dataset and weather setting.

  • Transfer learning: Transfer-initialized CLFTv2-Tiny models consistently reach higher early-epoch mIoU than ImageNet-pretrained models across ZOD, Waymo, and ISEAuto.Related-dataset initialization is especially beneficial for ZOD, where learning from scratch is slow.
  • Safety and robustness: 96.1% and 95.7% are CLFTv2-Large’s Waymo pedestrian recall values for Day Fair and Night Rain, respectively.CLFT-Large instead has the highest precision in those conditions: 72.1% Day Fair and 71.7% Night Rain.
  • Safety and robustness: 2.3–2.5% is the Day Fair-to-Night Rain mIoU drop on Waymo for CLFTv2-Base/Large, versus 8.8–9.4% on ZOD and 15.9–17.1% on ISEAuto.On ZOD, CLFTv2-Large leads most weather splits, while Mask2Former-Base leads Night Rain.
  • Efficiency: 2.2× faster is CLFTv2-Tiny than Mask2Former-Tiny, with latency of 22.0 ms versus 48.4 ms and about 40% lower memory.At larger scale, CLFTv2-Large also uses 204.4G versus 265.3G FLOPs and 81.8 ms versus 118.2 ms.

G. Limitations and Future Work

The paper identifies limitations involving dense point-cloud fusion, pseudo-label supervision, and hardware scope. It frames CLFTv2 as an efficient alternative while reserving broader deployment claims for future validation.

  • Local-window fusion: Dense Waymo point clouds expose a limitation of local receptive fields: CLFTv2 trails ViT-based architectures despite shifted-window efficiency.The paper proposes hybrid backbones with sparse long-range token mixing as future work.
  • Supervision quality: ZOD evaluation uses SAM-generated pseudo-labels, so results incorporate varied supervision quality and may not isolate architectural capability from noisy labels.The paper calls for evaluation on high-quality human-annotated datasets.
  • Deployment scope: Hardware efficiency results are currently measured on an NVIDIA A100, leaving validation on lower-power edge devices and micro-controllers as future work.The limitation bounds immediate conclusions about real-time vehicular deployment.
  • Supported conclusion: CLFTv2 maintains performance on safety-critical classes while reducing latency and compute overhead relative to leading universal segmentation architectures.Across three datasets, lightweight residual fusion is reported as on par with computationally demanding query-matching decoders under sparse and noisy supervision.
  • Future direction: Local-window efficiency and global cross-modal alignment remain practical design trade-offs for scalable real-time perception architectures.The conclusion connects the efficiency benefits of local attention with its dense-point-cloud fusion limitation.
Loading 2609.09881v1…