Source-linked AI summary
LetOccVote: Learning Weakly Supervised 3D Occupancy through Consensus
Chi Zhang, Qi Song, Feifei Li, Jie Li, Rui Huang
TL;DR
Existing weakly supervised occupancy methods use noisy pseudo-labels without explicitly assessing their reliability. LetOccVote uses cross-observation consensus to refine geometric supervision and filter unreliable semantic evidence, achieving state-of-the-art performance among methods trained with 2D pseudo-label supervision.
Problem
Pseudo depth and semantics can contain geometric errors, cross-frame inconsistencies, and confusion between visually similar categories, limiting the reliability of weak supervision.
Method
LetOccVote is a weakly supervised Gaussian-based occupancy framework that uses Depth Vote for geometric refinement or rejection and Semantic Vote to reinforce reliable evidence while masking unreliable segments.
Results
LetOccVote-L achieves 53.27 IoU and 23.11 mIoU∗, consistently outperforming previous methods, while also achieving 20.91 mRayIoU.
Takeaways & Limitations
Multi-observation consensus provides a reliability-aware way to reduce the adverse influence of noisy pseudo-labels and support more robust occupancy learning under imperfect supervision.
Takeaways & Limitations
Challenges remain for ambiguous semantic categories and severely noisy pseudo-labels, motivating stronger visual or vision-language models for more reliable class-feature associations.
Abstract
from arXiv · showhide
Weakly supervised 3D occupancy prediction reduces the reliance on costly 3D annotations by learning from 2D pseudo-labels generated by vision foundation models. However, existing methods typically use these imperfect pseudo-labels directly as supervision, making occupancy learning vulnerable to erroneous geometric and semantic targets. We observe that agreement across repeated observations provides an inexpensive and reliable cue for assessing pseudo-label reliability. Based on this observation, we propose \textbf{LetOccVote}, a weakly supervised Gaussian-based occupancy framework that leverages cross-frame voting to improve both geometric and semantic supervision. For geometry, Depth Vote exploits cross-frame geometric agreement to refine supported pseudo depth and reject contradictory estimates before volumetric lifting and depth supervision. For semantics, Semantic Vote aggregates pseudo-semantic observations in a shared 3D space to identify reliable and contested evidence, strengthening reliable semantic supervision while filtering unreliable pseudo-label segments. The entire framework is trained solely with 2D pseudo-label supervision without requiring 3D occupancy annotations. On Occ3D-nuScenes, LetOccVote achieves 53.27 IoU and 20.39 mIoU, establishing state-of-the-art performance among methods with 2D pseudo-label supervision.
CUHKSZ
LetOccVote addresses noisy 2D pseudo-label supervision by voting across multiple observations to distinguish consistent from conflicting depth and semantic evidence.
- Existing weakly supervised occupancy methods directly learn from noisy 2D pseudo-labels.
- LetOccVote applies multi-view observation voting to depth and semantics.
- The framework strengthens consistent supervision while masking unreliable labels for more accurate occupancy prediction.
1. Introduction
The paper targets unreliable pseudo depth and semantics in weakly supervised occupancy learning. LetOccVote uses repeated-observation consensus to refine or reject geometric evidence and selectively supervise semantic segments, achieving state-of-the-art performance with 2D pseudo-label supervision.
- 3D occupancy annotation is costly, motivating learning from image-space depth and semantic supervision generated by foundation models.
- Existing methods use complementary observations mainly for representation learning or consistency constraints rather than explicitly evaluating pseudo-label reliability.
- Agreement across repeated observations provides direct evidence of pseudo-label reliability because the same physical location should remain consistent across viewpoints and timestamps.
- Depth Vote refines supported pseudo depth and rejects contradictory estimates before volumetric lifting and depth supervision.
- Semantic Vote identifies reliable and contested evidence in shared 3D space, strengthening reliable supervision while filtering unreliable pseudo-label segments.
- LetOccVote achieves state-of-the-art performance among methods trained with 2D pseudo-label supervision on Occ3D-nuScenes.
2. Related Work
Related work spans image-space supervision, foundation-model pseudo-labeling, Gaussian representations, and reliability-aware pseudo-label learning. LetOccVote differs by using repeated-observation agreement to assess supervision reliability before and during occupancy learning.
- Rendering-based methods learn 3D occupancy from 2D depth and semantic labels to reduce dependence on costly 3D annotations.
- Foundation models provide scalable supervision, while methods variously transfer visual knowledge, generate pseudo-labels, or consolidate predictions into 3D pseudo-labels.
- The paper focuses on observation-level evidence for assessing supervision reliability across repeated observations.
- Unlike direct image-to-Gaussian pipelines, LetOccVote retains an intermediate volumetric representation so repeated observations can be spatially aligned before Gaussian sampling.
- The intermediate volume makes geometric correspondence available during feature construction but also increases sensitivity to pseudo-depth errors.
- Prior robust pseudo-label methods select reliable predictions or explicitly treat unreliable supervision, whereas LetOccVote uses repeated-observation agreement for depth and semantic supervision.
3. Method
LetOccVote predicts semantic occupancy with a Gaussian representation learned from 2D pseudo-labels, using cross-frame agreement to refine geometric supervision and filter unreliable semantic supervision. Depth Vote supports volumetric lifting, while Semantic Vote identifies reliable and contested evidence for semantic training.
- Problem formulation: The scene is represented as semantic 3D Gaussians and voxelized into occupancy, using image-level semantic and metric-depth pseudo-labels without 3D occupancy annotations.The Gaussian attributes include centers, covariances, opacities, and semantic predictions; pseudo-labels come from off-the-shelf 2D foundation models.
- Feature lifting with Depth Vote: Depth Vote checks pseudo-depth hypotheses across repeated observations before lifting image features into a shared 3D volume.Features are unprojected using camera geometry, aggregated into voxels, refined by a 3D encoder, and queried by a Gaussian decoder.
- Feature lifting with Depth Vote: A depth sample supported by at least k agreeing frames is corrected and lifted, whereas one contradicted by at least k frames without agreement is excluded.Insufficient or ambiguous evidence leaves the original pseudo-depth unchanged, and current-frame rejection is disabled.
- Refined depth supervision: The same consensus-refined depth supervises Gaussian rendering, with invalidated samples omitted from the depth objective.This aligns the depth used for feature lifting with the depth used for rendering-based optimization.
- Semantic supervision: Semantic Vote projects semantic pseudo-labels into shared voxels, uses majority classes and agreement ratios, and separates reliable from contested feature evidence.Reliable and contested pools support identifying semantic regions whose pseudo-labels should or should not contribute to training.
- Semantic supervision: Tokens are marked contested when their similarity favors competing references, and segments with enough contested tokens are masked rather than reassigned.Retained pseudo-labels keep their original supervision, while class weighting compensates for the fraction of samples removed by masking.
4. Experiments
Experiments on Occ3D-nuScenes evaluate LetOccVote’s performance, component contributions, consensus settings, and robustness to pseudo-label quality. The method consistently improves occupancy and semantic prediction, with gains linked to cross-frame geometric and semantic agreement.
- Main results: LetOccVote-B/L achieve 51.45/53.27 IoU and 22.07/23.11 mIoU∗, outperforming previous weakly supervised methods.The comparison uses methods trained with 2D pseudo-label supervision.
- Main results: LetOccVote-L achieves the best result on 11 of 15 semantic classes, while the two variants together rank first on 14 classes.Gains span foreground objects and large-scale scene structures, including vegetation, construction vehicle, sidewalk, car, and manmade.
- Main results: LetOccVote-B/L achieve 19.86/20.91 mRayIoU, with the Large variant best at every evaluated RayIoU threshold.The results indicate stable geometric improvements under different matching tolerances.
- Effectiveness of Proposed Components: Depth Vote increases IoU from 50.51 to 51.44, while Semantic Vote raises mIoU by 1.02 points with nearly unchanged IoU.The ablation attributes the former mainly to geometric improvement and the latter to stronger semantic discrimination; combining both modules performs best.
- Ablation on k-W Values: Increasing the temporal window improves performance but increases memory use: with k = 2, W from 3 to 5 raises IoU from 49.58 to 51.45 and mIoU from 19.14 to 19.49.A similar improvement occurs for k = 3 when W increases from 5 to 7.
- Ablation on k-W Values: Larger k does not necessarily improve performance, so the memory-efficient default is (k, W) = (2, 5).The stricter consensus criterion reduces the effective coverage of consensus refinement; (3, 7) has the best accuracy but only a small improvement over (2, 5) at substantially higher memory cost.
5. Conclusion
LetOccVote learns 3D occupancy from 2D pseudo-label supervision by using consensus across observations to improve supervision reliability. It combines Depth Vote and Semantic Vote to support more robust occupancy learning, while ambiguous categories and severely noisy pseudo-labels remain challenging.
- LetOccVote is a weakly supervised Gaussian-based framework for learning 3D occupancy from 2D pseudo-label supervision.
- Depth Vote improves geometric consistency and refines depth supervision for more effective feature aggregation and geometric representation.
- Semantic Vote selectively emphasizes reliable semantic patterns and suppresses contested pseudo-label segments during training.
- Extensive experiments demonstrate improved accuracy and robust 3D occupancy learning under 2D pseudo-label supervision.
- Ambiguous semantic categories and severely noisy pseudo-labels remain challenges for the framework.