PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxelFix: Post-Hoc Semantic Correction of Completed 3D Voxel Maps

Sunesh Praveen Raja Sundarasami, Taehyoung Kim, Johannes Scherer, Tomaž Cotič, Sivasubiramaniam Subbiah, Andreas Greiner, Paul Spannaus, Sebastian Houben

arXiv:2609.05114v1cs.CV

TL;DR

Automated semantic voxel maps can preserve perception and mapping errors, and existing refinement methods may require unavailable observations or jointly alter occupancy. VoxelFix corrects labels directly from completed-map geometry and semantics while keeping geometry and occupancy fixed. It consistently improves maps from four upstream segmentation models and shows useful transfer to an independently reconstructed scene.

  • Problem

    Completed semantic maps can retain errors, while existing refinement methods do not directly support correction using only a finished map with fixed geometry and occupancy.

  • Method

    VoxelFix uses geometric and semantic graph reasoning with gated fusion, trained on spatially coherent class-dependent corruptions derived from upstream confusion patterns.

  • Results

    4.23–5.00 mIoU percentage points of improvement were obtained across four upstream segmentation models, with further evaluation on an independently reconstructed out-of-distribution scene.

  • Takeaways & Limitations

    Post-hoc semantic correction can complement modular semantic mapping by refining completed maps without revisiting observations or reconstruction.

  • Takeaways & Limitations

    VoxelFix cannot recover missing geometry or correct reconstruction errors, and robustness to geometric errors remains unevaluated because experiments isolate semantic errors with ground-truth depth and camera poses.

Abstract

from arXiv · show

Semantic 3D maps are increasingly constructed automatically for aerial robotics by integrating learned semantic predictions into 3D representations. While this avoids costly manual 3D annotation, errors in the perception and mapping pipeline can persist in the resulting map, reducing its reliability for downstream autonomous tasks. Existing 3D semantic map refinement methods either rely on the original observations, treat occupancy as part of the prediction problem, or apply non-learned local regularization to completed maps. Instead, we study post-hoc semantic correction, asking whether semantic accuracy can be recovered directly from the completed map while keeping its geometry and occupancy fixed. We introduce \method, a graph-based model that corrects voxel labels based on local geometry and neighboring semantic information. To obtain training pairs, we corrupt contiguous regions of annotated OccuFly maps according to class confusions observed in upstream maps. We evaluate \method on completed OccuFly maps generated from predictions of four independently trained 2D segmentation models. \method consistently improves mIoU by 4.23--5.00 percentage points, with gains broadly distributed across the evaluated semantic classes and particularly strong improvements for tree, roof, and wall. Results on an independently reconstructed out-of-distribution aerial scene further suggest that the learned correction can transfer beyond the environments seen during training.

1. Introduction

Semantic voxel maps can retain errors after automated construction, while many existing refinement methods require unavailable mapping information or alter occupancy. VoxelFix addresses this gap by correcting labels directly on completed maps while preserving geometry and occupancy.

  • Motivation: Automated semantic mapping reduces manual annotation effort but can leave perception and mapping errors in completed voxel maps.Semantic labels influence downstream aerial-robotics tasks, including obstacle-aware planning and landing-site assessment.
  • Limitations of existing methods: Existing refinement methods often require observations, fusion state, or multiple views, while semantic scene completion jointly predicts occupancy and semantics.These requirements do not match a setting where only the completed map remains available.
  • Motivation: Local smoothing can fail on spatially coherent errors, such as a roof region whose neighboring voxels share the same incorrect wall label.Completed maps still contain spatial structure and class information that can help identify such errors.
  • Approach: VoxelFix formulates post-hoc semantic correction as voxel relabeling using completed-map geometry and semantic labels while keeping geometry and occupancy fixed.The approach operates after map construction rather than revisiting observations or intermediate mapping state.
  • Approach: VoxelFix combines geometric and semantic graph reasoning through a dual-branch graph attention architecture with learned gated fusion.Its training data use spatially coherent, class-dependent corruptions derived from upstream confusion patterns.
  • Results: 4.23–5.00 mIoU percentage points of improvement were obtained across four upstream segmentation models on the held-out OccuFly test scene.The method was also evaluated for transfer to an independently reconstructed out-of-distribution scene.

3. Method

VoxelFix treats post-hoc semantic correction as relabeling occupied voxels while using only the completed map and keeping geometry and occupancy fixed. It represents occupied voxels as a k-NN graph, combines geometric and semantic context through dual attention branches and gated fusion, and trains with spatially coherent confusion-guided corruptions.

  • Problem formulation: VoxelFix takes the completed map as its only input and changes semantic labels while preserving geometry and occupancy.Its correction function operates after reconstruction, when observations and fusion state are unavailable.
  • Voxel graph representation: The map is represented as a fixed k-NN graph with one node per occupied voxel and edges based on Euclidean distance between voxel centers.The graph is computed once per map because occupancy is fixed.
  • Voxel graph representation: Each voxel combines label-independent local geometry, frozen volumetric geometry embeddings, its current label, and class-conditioned agreement features.The current label is explicitly encoded so the model reasons about correcting the existing labeling rather than predicting semantics from geometry alone.
  • VoxelFix: Parallel geometric and semantic GATv2 branches process the shared graph, with the semantic branch additionally using semantic edge information.A voxel-wise gate varies the relative contribution of geometric and semantic context before correction and error detection.
  • Confusion-guided noise curriculum: Training uses confusion-guided substitutions within contiguous voxel patches to create regional errors resembling upstream map confusions.The corruption rate decreases from 0.50 to 0.05 over training, exposing the model first to larger errors and later to smaller residual errors.

4. Experiments

Experiments evaluate VoxelFix on completed OccuFly maps from multiple upstream models, comparing map-level correction against smoothing, geometry-based, and learned volumetric baselines. Results show consistent in-distribution, upstream-model, and OOD improvements, with ablations supporting both structured corruption and geometric–semantic reasoning.

  • Experimental setup: Experiments use nine OccuFly scenes, four independently trained 2D segmentation models, and a 7/1/1 cross-scene protocol for map correction.Maps are generated by lifting remapped 2D predictions into 3D with ground-truth depth and camera poses, then aggregating them with Radix.
  • Experimental setup: VoxelFix is compared with KNN smoothing, CRF refinement, a geometry heuristic, and MinkUNet using mIoU, voxel accuracy, ECR, and DR.ECR measures initially incorrect voxels corrected, while DR measures initially correct voxels made incorrect.
  • Map-level correction: 30.66±0.55 mIoU is achieved by VoxelFix, with a +4.86 ± 0.55-point gain over Radix and +2.14 points over MinkUNet.VoxelFix corrects 25.36±1.67% of initially erroneous voxels while corrupting 3.55±0.58% of initially correct ones.
  • Effect of upstream quality: +4.23 to +5.00 mIoU percentage points are gained across four upstream segmentation models on the fixed test scene.The gain does not vary monotonically with initial mIoU, indicating dependence on semantic and spatial error structure.
  • OOD evaluation: 32.66 mIoU is reached from 30.95 on an independently reconstructed OOD scene, while ambiguous solar-panel regions remain unresolved.The OOD scene was excluded from training, validation, checkpoint selection, and semantic-prior estimation.
  • Ablation study: Ablations show the largest mIoU drops when the noise curriculum is restricted or reversed, followed by removing auxiliary detection loss or confusion-guided corruption.Removing geometry embeddings or the geometric branch reduces mIoU by approximately 2 points, while ungated fusion and other changes cause smaller decreases.

5. Conclusion

VoxelFix performs post-hoc semantic correction directly on completed voxel maps, refining labels while keeping geometry and occupancy fixed. It improves maps from four upstream models and transfers useful correction to an independently reconstructed scene, but remains bounded by information retained in the map and by reconstruction quality.

  • Conclusion: VoxelFix corrects semantic labels directly on completed voxel maps without revisiting observations or reconstruction, while preserving geometry and occupancy.Its correction combines geometric and semantic graph reasoning with confusion-guided training corruptions.
  • Conclusion: VoxelFix consistently improves maps from four upstream segmentation models and outperforms local, geometry-based, and learned volumetric refinement baselines.This conclusion is supported by the reported cross-model evaluation and baseline comparisons.
  • Limitations: VoxelFix cannot recover missing geometry or correct reconstruction errors, and robustness to geometric errors remains unevaluated because experiments isolate semantic errors with ground-truth depth and camera poses.Correction is also limited when the completed map provides insufficient evidence for the correct class.
  • Conclusion: Useful correction transfers to an independently reconstructed scene, supporting post-hoc correction as a complementary stage in modular semantic mapping pipelines.The OOD evaluation also shows that ambiguous regions remain unresolved when the input map lacks sufficient evidence.

Supplementary Material

The supplementary material identifies the VoxelFix architectural configuration in Table 7.

  • Supplementary material: Table 7 reports the VoxelFix architectural configuration.

6. Additional Implementation Details

The supplementary implementation details specify VoxelFix’s architecture, feature construction, training setup, compute budget, augmentation, validation, and hyperparameter-selection procedures.

  • Architecture: VoxelFix uses geometric and semantic GATv2 branches, voxel-wise gated fusion, a semantic correction head, and an auxiliary error-detection head.The node descriptor combines handcrafted geometry, frozen geometry embeddings, current semantic labels, agreement scores, and prototype deviation.
  • Training: Training uses synthetically corrupted semantic voxel maps, held-out validation scenes, and the checkpoint with the lowest validation loss.Class prototypes are initialized from training scenes and updated with an exponential moving average.
  • Compute: Experiments run on one NVIDIA A100 80 GB GPU, with a two-stage geometry-encoder preparation followed by independent VoxelFix training for each cross-scene split.The geometry encoder is pretrained on STPLS3D, fine-tuned on OccuFly, frozen, and reused across splits.
  • Augmentation: No additional geometric augmentation is applied; training variation comes from spatially contiguous, confusion-guided semantic corruption.
  • Evaluation protocol: Hyperparameters and checkpoint selection use only corresponding training and validation scenes, excluding the fixed test scene from all selection and statistic-estimation steps.

7. Geometry Encoder

VoxelFix learns geometry representations from geometry-only voxel features, using STPLS3D pretraining followed by OccuFly fine-tuning before reuse across splits.

  • 7. Geometry Encoder: The learned geometry representation complements VoxelFix’s handcrafted geometric descriptor while discarding the encoder’s classification head after fine-tuning.The resulting embedding is used as a geometry representation rather than as a standalone classifier.
  • 7. Geometry Encoder: The geometry encoder uses six-dimensional geometric descriptors from occupied 0.5 m voxels, including height, planarity, linearity, sphericity, verticality, and point density.A sparse volumetric U-Net produces a dedicated 16-dimensional embedding.
  • 7. Geometry Encoder: The encoder is pretrained for voxel-wise semantic classification on STPLS3D using geometry-only inputs and partial-scan augmentation.Semantic labels provide supervision, but are not encoder inputs.
  • 7. Geometry Encoder: After pretraining, the geometry encoder is fine-tuned once on OccuFly and reused for all cross-scene splits.Fine-tuning uses seven training scenes, while Scene 8 remains excluded from adaptation and training-derived statistics.

8. Confusion-Guided Noise Curriculum

The noise curriculum synthesizes realistic regional semantic errors from upstream confusion patterns, then anneals corruption while preserving a nonzero training signal.

  • 8.1. Construction of the Class-Transition Mapping: Synthetic errors are guided by voxel-level upstream confusion frequencies and a manually constructed mapping of prominent class misclassifications.The mapping avoids overreliance on long-tail statistical noise and excludes minor classes such as person and bicycle.
  • 8.2. Corruption Rate Schedule: The curriculum decreases the corrupted fraction linearly from ρmax = 0.50 to ρmin = 0.05 during training.Early training presents extensive errors, while later training emphasizes finer distinctions.
  • 8.2. Corruption Rate Schedule: The five-percent floor preserves correction pressure throughout training and prevents an increasingly favorable identity mapping.A schedule decaying to zero would eventually expose the model to an essentially clean label map.
  • 8.1. Construction of the Class-Transition Mapping: Corruption targets contiguous patches rather than independent voxels, with each patch relabeled to one incorrect class.Patch coverage depends on sampled radius and local voxel density, matching spatially correlated upstream errors.
  • 8.3. Synthetic Corruption Examples: Corruptions are resampled every epoch, with the next epoch’s labels precomputed in a background thread to avoid training stalls.This exposes training to different corrupted maps while hiding generation latency.
  • 8.3. Synthetic Corruption Examples: Representative substitutions include roof-to-wall, road-to-grass, and tree-to-grass, producing coherent erroneous surfaces rather than scattered label flips.The corruption mask supervises error detection alongside correction, and the regional structure is intended to address errors local smoothing cannot resolve.

9. Upstream Semantic Segmentation

The upstream semantic models are trained across nine aerial datasets, aligned to a common taxonomy, and evaluated on both validation data and an unseen aerial dataset.

  • 10.1. Upstream Models: The upstream training corpus combines nine aerial datasets containing 5,191 training images and 553 validation images.Two internally curated FPV datasets contribute 2,050 images in total.
  • 10.1. Upstream Models: Annotations are remapped to a unified 26-class taxonomy, while ambiguous, unlabeled, and background regions are excluded through the ignore index.This alignment accounts for differences in class granularity and semantic definitions across datasets.
  • 10.1. Upstream Models: Four independently trained models span SegFormer, UPerNet with Swin-v2-T or ConvNeXt-v2-T, and Mask2Former with Swin-based features.All feature extractors start from pretrained weights.
  • 10.2. Evaluation: The models are evaluated using mIoU on the training-data validation split and an unseen aerial dataset containing 221 images.The unseen dataset is reserved to measure generalization to environments not used during training.
  • 10.3. Taxonomy Alignment: The 26-class predictions are first aligned to the 21-class OccuFly taxonomy and then consolidated into the 12 classes used for correction.Consolidation merges semantically or geometrically close categories, while tree, roof, person, dirt, bicycle, and water remain unchanged.

10. Semantic Voxel Map Generation

Completed semantic voxel maps are built by back-projecting remapped 2D predictions into 3D and aggregating them into sparse 0.5 m occupancy grids before VoxelFix receives only hard labels and fixed geometry.

  • 10. Semantic Voxel Map Generation: Undistorted aerial frames are segmented, remapped to the 12-class correction taxonomy, and back-projected using OccuFly depth maps and camera poses.The resulting per-frame point clouds are aggregated into a semantic voxel map.
  • 10. Semantic Voxel Map Generation: Radix aggregates observations in semantic mode at a voxel size of 0.5 m, maintaining a class distribution for each occupied voxel.The final voxel label is the maximum-probability class after exponential- moving-average updates.
  • 10. Semantic Voxel Map Generation: Ray-based cleaning is disabled because the offline scenes are mostly static and retaining accumulated observations produces a more complete map.The default Radix configuration is otherwise used.
  • 10. Semantic Voxel Map Generation: After map construction, VoxelFix receives only hard semantic labels and fixed voxel geometry, while the CRF baseline additionally uses retained per-voxel class probabilities.Thus, the CRF baseline operates with strictly more information than VoxelFix.

11. Baseline Implementation Details

The baselines enforce the completed-map setting with varying access to geometry, semantic probabilities, structural priors, or learned graph reasoning. Qualitative evaluations compare uncorrected inputs, VoxelFix, and ground truth across held-out scenes.

  • Map-only baselines: KNN applies majority voting among nearest spatial neighbors without learned components or geometric features.It represents purely local smoothing, which can fail when coherent neighborhoods share the same incorrect label.
  • Map-only baselines: The CRF combines upstream class probabilities with dense spatial pairwise regularization, using richer input than VoxelFix.It is the only compared method that consumes class distributions retained during map construction rather than hard labels alone.
  • Structural-prior baseline: Cloth Simulation Filtering tests structural priors by separating ground from above-ground voxels before applying partition-specific correction rules.Ground voxels use restricted KNN voting, while above-ground objects use DBSCAN clustering and per-cluster majority relabeling.
  • Taxonomy: The evaluated taxonomy reduces aligned 26-class upstream predictions to 12 classes for VoxelFix training and evaluation.Classes without corresponding target classes are ignored.
  • Learned baseline: MinkUNet isolates graph-based reasoning with sparse volumetric convolutions, using the label map alone without geometric descriptors or frozen embeddings.It uses the same cross-scene splits and correction and detection objectives as VoxelFix.
  • Qualitative evaluation: VoxelFix qualitative comparisons show uncorrected Radix inputs, corrected outputs, and ground truth for held-out validation scenes.Each scene is corrected by the model from the split in which it was held out.

13. Additional Out-of-Distribution Results

An independently reconstructed aerial scene tests whether VoxelFix transfers beyond the training environments. The OOD results show class-specific gains and qualitative recovery of building regions, while remaining errors occur where the completed map lacks distinguishing evidence.

  • OOD setup: The OOD scene was independently reconstructed from a separate aerial flight covering approximately 50 × 140 m, or ∼7,000 m2.It was captured at 30 m above ground under overcast late-autumn conditions with an RGB camera and INS georeferencing.
  • Quantitative results: OOD gains concentrate in tree (+4.51) and wall (+4.72), matching the classes with the strongest in-distribution improvements.Smoothing baselines remain competitive on dominant classes.
  • Evaluation scope: No variance is reported for the OOD result because evaluation uses a single model rather than the eight cross-validation splits.
  • Qualitative results: Qualitatively, VoxelFix recovers wall and roof regions along building structures and removes spurious water labels from grass.Remaining errors concentrate in structures without distinguishing evidence in the completed map.
  • Evaluation protocol: The cross-scene protocol uses eight independently trained models, with one OccuFly scene fixed as an untouched test scene.Training-derived statistics are estimated only from the corresponding training scenes.
  • Quantitative results: The OOD per-class table reports mIoU over 10 present classes, so it is not directly comparable to in-distribution mIoU values.Only classes present in the scene are shown.
Loading 2609.05114v1…