PaperlayerView exact revision on arXiv

Source-linked AI summary

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion

Suhani Grover, Astik Srivastava, Viswas Dinesh, Avinash Sharma, K. Madhava Krishna

arXiv:2609.36844v1cs.CVcs.RO

TL;DR

Transparent surfaces are difficult for robotic perception because optical cues and depth measurements can be unreliable on glass, especially in challenging lighting. GlassFormer fuses a radar-guided spatial prior with RGB-D sensing in a transformer-based segmentation framework. It improves low-light mIoU over a fine-tuned SegFormer backbone, while the current setup’s effective range is limited by its RGB-D camera.

  • Problem

    Optical methods for transparent-surface segmentation have limited reliability because visual cues degrade under low light or glare and depth sensors can return background or invalid measurements at glass.

    Source: §I · p. 1

  • Method

    GlassFormer projects dominant mmWave radar range into the image plane, combines it with RGB-D depth inconsistencies as a spatial prior, and injects that prior into SegFormer-B2 through RadarAttention.

    Source: §I · p. 2

  • Results

    0.590 vs. 0.491 mIoU in low light for RadarAttention and the fine-tuned SegFormer backbone, respectively.

    Source: D. Effect of RadarAttention · p. 7

  • Takeaways & Limitations

    The radar prior improves low-light performance while the approach maintains real-time performance and uses lightweight modifications to an existing segmentation backbone.

    Source: D. Effect of RadarAttention · p. 7, §VI · p. 7

    The current sensor setup’s effective operating range is approximately 6 m, limited by the Intel RealSense D455 depth range, despite radar ranging up to 13 m.

    Source: §VI · p. 7

Abstract

from arXiv · show

Transparent surfaces are ubiquitous in built environments, yet they remain a persistent failure case for robotic perception. RGB cameras perceive the background behind glass rather than the surface itself, while depth sensors such as LiDAR, time-of-flight, and RGB-D often return invalid or background measurements in transparent regions. As a result, systems that rely solely on optical sensing may misinterpret glass walls, doors, or mirrors as free space, compromising safe and reliable navigation. Existing glass segmentation approaches address this by learning visual cues such as reflections, boundaries, and semantic context from RGB images. While effective under favourable lighting and viewing conditions, these cues degrade in low-light environments, under glare, or when glass surfaces are featureless or partially occluded. In this work, we propose a multimodal framework that fuses millimetre-wave radar with RGB-D sensing for real-time transparent surface segmentation. Radar reflects strongly off glass surfaces, providing a geometric cue that remains reliable precisely where vision and depth fail. We exploit this cross-modal inconsistency to generate a radar-guided spatial prior, which is integrated into a lightweight transformer-based segmentation network, GlassFormer, via cross-modal attention. We report results on a mixed-condition test split covering all scene types and a dedicated low-light split designed to stress vision-only methods. GlassFormer achieves 0.88 mIoU on the mixed split, and 0.59 mIoU on the low light split, demonstrating substantial robustness gains over vision-only baselines while maintaining real-time performance on resource-constrained platforms.

I. INTRODUCTION

Glass is difficult for robotic perception because it is visually ambiguous and often produces invalid or background depth measurements. GlassFormer uses mmWave radar as a complementary spatial cue, fusing a radar-guided prior with RGB features for transparent-surface segmentation.

I. INTRODUCTION

  • Glass is difficult to detect because it lacks intrinsic color or texture and conventional depth sensors may return invalid or background measurements at transparent interfaces.RGB appearance is often determined by the scene around and behind the glass, while depth-sensing failures persist irrespective of ambient illumination.Source: §I · p. 1
  • GlassFormer injects a radar-guided spatial prior into SegFormer-B2 through cross-modal RadarAttention, targeting real-time transparent-surface segmentation on resource-constrained platforms.The prior combines mmWave range measurements with RGB-D depth inconsistencies to localize candidate regions in the image plane.Source: §I · p. 2
  • The approach is reported to gain substantially over state-of-the-art methods under low-light and high-glare conditions where vision-only approaches degrade.The introduction also identifies featureless conditions as a setting where visual cues can be unreliable.Source: §I · pp. 1–2
  • The synchronized RGB-D-radar dataset contains annotated frames across diverse scenes, including low-light and high-glare conditions underrepresented in existing benchmarks..Source: §I · p. 2
  • An open-source ROS2 driver supports real-time integration of pulsed coherent radar in robotic perception pipelines.Source: §I · p. 2

II. RELATED WORK

Prior glass-segmentation work has largely used RGB cues, with methods targeting contextual features, reflections, boundaries, and coarse-to-fine refinement. These approaches still assume favorable lighting and limited specular reflection.

A. Vision-Based Transparent Surface Segmentation

B. Multimodal Perception for Transparent Surfaces

Multimodal approaches combine optical sensing with other modalities, but depth-based methods inherit depth sensors’ failures on transparent surfaces. The paper describes radar returns as a complementary cue and uses radar range profiles to localize reflecting surfaces.

B. Multimodal Perception for Transparent Surfaces

  • RGB-D multimodal methods fuse depth with RGB features through cross-modal attention or weighted fusion, but still rely on depth that fails on transparent surfaces.Source: B. Multimodal Perception for… · p. 2
  • Other sensing approaches trade off range, spatial resolution, noise sensitivity, cost, or specialized hardware requirements.LiDAR approaches use intensity variance or multi-echo processing; ultrasonic, polarization, and thermal sensing offer other complementary cues.Source: B. Multimodal Perception for… · p. 2
  • Compared with FuseGrasp [25], which targets transparent-object grasping with higher-dimensional radar processing, this approach uses a 1D radar signal for real-time scene segmentation on low-compute platforms.Source: B. Multimodal Perception for… · p. 2

A. Pulsed Coherent Radar and Signal Model

  • At millimeter-wave frequencies, glass produces measurable radar returns, and planar glass surfaces generate amplitude peaks while structured-light and stereo depth sensors can return invalid or inconsistent boundary measurements.The radar’s magnitude profile across range bins is used to localize dominant reflecting surfaces.Source: A. Pulsed Coherent Radar and Signal… · p. 3

B. Sensor Geometry and Configuration

The radar configuration samples a range profile and uses a beamwidth-defined conical sensing volume for mask generation. The table is identified as the radar configuration for data acquisition.

B. Sensor Geometry and Configuration

  • The setup samples 200 range bins over approximately 0.30 m to 13.0 m, applies HWAAS = 64, and aggregates three sweeps per frame.The stated averaging is intended to improve measurement stability while preserving responsiveness for real-time perception.Source: B. Sensor Geometry and Configuration · p. 3
  • The radar’s approximate 40° half-power beamwidth defines the conical sensing volume projected into the image plane for radar-guided mask generation.Source: B. Sensor Geometry and Configuration · p. 3
  • Table I is titled “Radar Configuration for Data Acquisition.”Source: Table I · p. 3

IV. METHODOLOGY

The method selects the strongest radar range return as the dominant reflector, then projects the radar field of view into the RGB image to define where depth reasoning will occur. The projection uses a rectangular image region and is not restricted to frontal views.

1) Range Peak Extraction:

  • The dominant reflector is estimated by selecting the range bin with maximum radar-profile intensity.Its radial distance represents the most prominent surface detected along the radar boresight.Source: 1) Range Peak Extraction · p. 3
  • Peak selection assumes a dominant single-interface reflection, so thick glass or multipath may shift the selected return from the near air-glass interface.The selected peak may not correspond exactly to the near interface.Source: 1) Range Peak Extraction · p. 3

2) Field-of-View Projection:

  • The projected radar field of view defines an RGB-image region of interest centered at the camera principal point.The projection uses the calibrated camera focal length, principal point, and radar field-of-view angle.Source: 2) Field-of-View Projection · p. 3
  • The projection approximates the conical radar beam as a rectangular image region for computational simplicity and pixel-grid alignment.Subsequent depth reasoning is restricted to this region.Source: 2) Field-of-View Projection · p. 3

3) Depth Consistency Filtering:

Within the projected radar field of view, the method marks pixels as transparent candidates when RGB-D depth does not corroborate the radar-confirmed surface. It then cleans the binary prior before passing it to the segmentation network.

3) Depth Consistency Filtering:

  • Invalid depth or depth sufficiently beyond the radar-confirmed range marks a pixel as a transparent candidate.The tolerance margin accommodates calibration error, surface thickness, and multipath effects.Source: 3) Depth Consistency Filtering · pp. 3–4
  • The mismatch identifies likely transparent interfaces because depth sensing can observe background geometry while radar detects the physical surface.The resulting binary mask is a radar-derived transparency prior.Source: Figure 2 · p. 4, 3) Depth Consistency Filtering · p. 4
  • Morphological opening and closing with a 5×5 kernel suppress isolated artifacts and fill small discontinuities in the mask.The refined per-frame prior is supplied to the segmentation network.Source: 3) Depth Consistency Filtering · p. 4

B. GlassFormer Architecture

GlassFormer builds on SegFormer-B2, whose hierarchical transformer encoder produces multiscale RGB features for a lightweight MLP decoder. The backbone uses depth-wise convolutions for implicit positional information and reduced key/value sequence lengths for efficient attention.

B. GlassFormer Architecture

1) SegFormer Backbone:

  • The Mix-FFN embeds a 3×3 depth-wise convolution to provide positional information without fixed positional encodings.This avoids fixed encodings that degrade when test resolution differs from training resolution.Source: 1) SegFormer Backbone · p. 4
  • Reduced key and value sequence lengths lower attention complexity from O(N^2) to O(N^2/R), enabling operation at high spatial resolutions.The reduction ratio is applied across encoder stages.Source: 1) SegFormer Backbone · p. 4
  • The decoder unifies channels across all four stages, upsamples and concatenates features, then uses MLP layers to predict a segmentation mask.Its design uses the transformer's effective receptive field instead of heavy context modules such as ASPP.Source: 1) SegFormer Backbone · p. 4

2) RadarAttention Module:

RadarAttention incorporates the radar-derived transparency prior at the two deepest, spatially compressed encoder stages, while leaving the two higher-resolution stages unchanged. Radar embeddings provide queries and RGB features provide keys and values, with a learnable gate controlling radar influence.

2) RadarAttention Module:

  • RadarAttention (RA) is inserted after each of the two referenced stages to incorporate a radar-derived transparency prior at feature level, where features are semantically rich but spatially compressed.Stages 1 and 2 remain unmodified because computing attention maps at those resolutions costs more than the expected advantage from radar.Source: 2) RadarAttention Module · p. 4
  • The radar mask is downsampled and encoded as queries, while RGB backbone features serve as keys and values in RadarAttention.A learnable spatial gate λ controls how strongly attention modulates RGB features and can help reject radar-induced false positives from depth noise.Source: 2) RadarAttention Module · p. 4, Figure 3 · p. 5
  • The two RA-modulated stages and two unmodified high-resolution stages are concatenated and decoded into a segmentation mask.The decoder uses an MLP followed by two 1 × 1 convolutions.Source: Figure 3 · p. 5

3) Loss Function:

The training objective combines pixel-level binary cross-entropy with Lovász hinge loss, which directly optimizes intersection-over-union. Equal weighting encourages pixel-wise accuracy and region-level overlap.

3) Loss Function:

  • Binary cross-entropy penalizes misclassified pixels and provides stable training gradients.Lovász hinge loss directly optimizes intersection-over-union and is suited to class imbalance or thin structures such as glass surfaces.Source: 3) Loss Function · p. 5
  • The final objective weights binary cross-entropy and Lovász hinge loss equally.The combination supports pixel-wise classification and region-level overlap.Source: 3) Loss Function · p. 5

A. Implementation Details

The section reports real-time inference, describes a synchronized RGB-D-radar dataset, and evaluates the radar-derived mask against learned baselines. Cross-domain evaluation was not performed because existing benchmarks lack synchronized radar data.

A. Implementation Details

B. Dataset

  • The self-collected dataset contains 1,800 synchronized RGB-D-radar frames across indoor and outdoor glass scenes, with varied lighting, distances, and viewing angles.It includes doors, windows, full-length glass walls, mirrors, tinted glass, and partially occluded glass.Source: B. Dataset · p. 6
  • Cross-domain evaluation on benchmarks such as Trans10K and GDD was not performed because they lack synchronized radar data.Source: B. Dataset · pp. 6–7
  • GlassFormer segmentation remains reliable across viewing angles and distances within the radar–RGB-D shared field of view, not only for frontal views.Source: Figure 5 · p. 7

C. Effect of Radar Prior

  • 0.541 mIoU in low light exceeds GDNet, GlassSemNet, and SegFormer (0.462–0.491 mIoU), while Radar Overlay scores 0.469 mIoU in well-lit conditions.The radar-only mask underperforms vision-based methods in well-lit conditions but is competitive with or exceeds several in low light.Source: C. Effect of Radar Prior · p. 7

D. Effect of RadarAttention

RadarAttention improves SegFormer mIoU more in low light than in well-lit conditions. The discussion presents radar as a complementary geometric prior, while noting limits in radar localization and effective sensing range.

D. Effect of RadarAttention

  • 0.590 vs. 0.491 mIoU in low light, with BER improving from 0.282 to 0.241; the well-lit gain is 0.882 vs. 0.880 mIoU.The comparison is between RadarAttention and the fine-tuned SegFormer backbone.Source: D. Effect of RadarAttention · p. 7

VI. DISCUSSION AND CONCLUSION

  • Radar fusion via RadarAttention improves region overlap and boundary refinement without degrading classification stability, while radar alone is insufficient for dense segmentation.The discussion also reports a representative mirror segmentation despite fewer mirror training examples.Source: §VI · p. 7
  • The approach maintains real-time performance with lightweight backbone modifications for resource-constrained robotic platforms.Source: §VI · p. 7
  • A single 1D radar's spatial sparsity and noisy reflections limit precise localization, and the current setup's effective range is about 6 m because of the D455 depth limit.The radar supports ranging up to 13 m; the passage identifies the shorter effective range as a sensor-integration limitation.Source: §VI · p. 7
  • Future work proposes using radar phase information and exploring uncertainty-aware fusion, dynamic confidence weighting, and broader evaluation under extreme illumination and adverse weather.Source: §VI · p. 7

Related papers

Loading 2609.36844v1…