FM-Occ: Feature-Diverse Upsampling and Memory-Aware Decoding for 3D Occupancy Prediction

1 INTRODUCTION

• FM-Occ targets two issues in sparse occupancy prediction: weak spatial differentiation after upsampling and progressive semantic information loss during iterative decoding.The paper describes its framework as improving feature differentiation during sparse upsampling and semantic preservation across decoder layers.Source: §1 · pp. 1–2
• Spatial CV of voxel-wise RMS responses rises from 0.41% to 4.03% across 50 validation frames with 3S.3S constructs child representations in the channel dimension, jointly transforms and normalizes them, then rearranges them onto a finer sparse grid.Source: §1 · pp. 1–2
• ER-LM with Lcwe improves final mIoU from 12.16% to 12.88% and reduces mean class-wise semantic degradation from 21.00% to 16.50%.The memory retains historical query information through gated updates, with classification entropy regulating propagation to subsequent decoder stages.Source: §1 · p. 2
• 3S reorganizes child-feature construction, normalization, and spatial expansion, while ER-LM uses entropy-regulated memory with class-weighted entropy loss during training.The paper states that ER-LM generalizes across 3D voxel-based and BEV-based frameworks.Source: §1 · p. 2

2.1 REPRESENTATIVE APPROACHES FOR 3D OCCUPANCY PREDICTION

• Sparse occupancy methods avoid maintaining features across the full 3D grid, while dense voxel, BEV, and tri-perspective methods use different scene representations.The paper situates its work within sparse voxel and query-based approaches, examining feature reconstruction and semantic information retention.Source: §2.1 · pp. 2–3

2.2 FEATURE UPSAMPLING

• Dense 2D upsampling methods assume regular spatial grids, whereas sparse 3D upsampling must expand active voxel features and coordinates while preserving their correspondence.3S addresses this with channel-to-space feature rearrangement and matched coordinate expansion.Source: §2.2 · p. 3

2.3 INFORMATION RETENTION IN OCCUPANCY DECODING

• Query-based occupancy decoders refine predictions through successive query updates, which may alter or lose semantic information acquired at earlier stages.This motivates explicit mechanisms for cross-layer information retention.Source: §2.3 · p. 3

3 METHOD

• 3S and ER-LM are integrated into FM-Occ’s sparse feature pyramid and Transformer decoder, with Lcwe additionally applied during training.The framework maps image features to a sparse 3D representation, performs scene completion, and builds a multi-scale sparse feature pyramid.Source: §3 · p. 3

3.1.1 SPARSE TENSOR REPRESENTATION

• The input sparse tensor S = (F, I, s) stores active-voxel features, batch and spatial indices, and the spatial shape.F contains N active-voxel feature vectors; I stores their batch and three-dimensional spatial indices.Source: §3.1.1 · p. 4

3.1.2 CHILD-FEATURE CONSTRUCTION AND JOINT NORMALIZATION

• 3S expands each active voxel’s features to r^3Cout channels, then applies group normalization and ReLU before spatial rearrangement.A submanifold sparse convolution produces the expanded representation at the coarse resolution.Source: §3.1.2 · p. 4

3.1.3 CHANNEL-TO-SPACE REARRANGEMENT

• The normalized representation is reshaped into r^3 location-specific child-feature vectors for each parent voxel.The channel-to-space mapping pairs each child representation with its child index.Source: §3.1.3 · p. 4

3.1.4 COORDINATE EXPANSION AND ALIGNMENT

• Each child feature is paired with its corresponding fine-grid coordinate, and the output spatial shape scales by r in each dimension.The resulting sparse tensor is merged with the existing feature level at the same resolution.Source: §3.1.4 · pp. 4–5

3.2.1 GATED MEMORY ACROSS DECODING STAGES

• ER-LM maintains a separate LSTM memory state for each query across decoder stages and fuses a projected memory feature into incoming queries.The recurrent module is shared across stages; the hidden and cell states propagate between stages.Source: §3.2.1 · p. 5

3.2.2 ENTROPY-REGULATED MEMORY PROPAGATION

• Classification entropy provides a query-specific signal for modulating the cell state transmitted between decoding stages.For stages ℓ≥2, a shared MLP maps normalized entropy to a modulation coefficient; the adjusted cell state proceeds to the next recurrent step.Source: §3.2.2 · pp. 5–6

3.2.3 CLASS-WEIGHTED ENTROPY REGULARIZATION

• Lcwe regularizes entropy for queries matched to ground-truth semantic masks, using their class weights alongside the classification loss.The regularizer applies to initial query predictions and all decoder-layer outputs; λcwe is set to 1.0 in the experiments.Source: §3.2.3 · p. 6

4.1 EXPERIMENTAL SETUP

• FM-Occ is evaluated on SemanticKITTI; the setup trains for 30 epochs on four RTX 4090 GPUs, and reports voxel-wise IoU and mIoU alongside ComplexIoU, TailIoU, and RegularIoU.SemanticKITTI has 21 classes: 19 semantic categories, freespace, and unknown.Source: §4.1 · p. 6

4.2 COMPARISON WITH STATE-OF-THE-ART

• FM-Occ achieves 36.18% IoU and 12.73% mIoU on SemanticKITTI; ER-LM raises DISC from 16.78% to 17.17% mIoU and from 45.54% to 45.67% IoU.DISC + ER-LM reaches 17.17% mIoU, comparable to MRA (17.14%) and higher than SGFormer (16.68%).Source: §4.2 · p. 6, Table 1 · p. 7

4.3 ABLATION STUDIES

• Across five random seeds, 3S retains stable semantic performance, while ER-LM consistently improves mIoU and reduces variation on regular categories.The reported improvements were not dependent on a single favorable training run.Source: Table 3 · p. 7, §4.3 · p. 7

4.4.1 COMPARISON OF UPSAMPLING FEATURE RESPONSES

• 3S achieves a mean spatial coefficient of variation (CV) of voxel-wise RMS responses of 4.03%, versus 0.41% for sparse transposed convolution, a 9.83× increase.Source: §4.4.1 · p. 7, Table 4 · p. 7

4.4.2 EFFECT OF NORMALIZATION AND ACTIVATION PLACEMENT

• Adding ReLU to transposed convolution does not improve mIoU, while applying group normalization after rearrangement gives mIoU similar to original 3S.These comparisons suggest ReLU alone does not account for 3S's gains and normalization order has limited effect on mIoU in these experiments.Source: Table 5 · p. 8, §4.4.2 · p. 8

4.4.3 SUBVOXEL FEATURE ALLOCATION VARIANTS

• Permute–reshape improves mIoU from 12.16% to 12.32%, while 3S reaches 12.42%, supporting its proposed feature allocation.Softmax improves mIoU and RegularIoU but reduces occupancy IoU; sigmoid improves ComplexIoU and RegularIoU but lowers TailIoU, so deterministic rearrangement is retained for balanced performance.Source: Table 6 · p. 8, §4.4.3 · p. 8

4.5 ANALYSIS OF ER-LM

• EMA captures value in historical information but yields limited semantic gains; ER-LM achieves a larger mIoU improvement through learnable memory and forgetting, with more balanced category-group improvements.The analysis also evaluates LayerNorm on the LSTM output before projection.Source: Table 7 · p. 8, §4.5 · p. 8
• From the first to final decoder layer, ER-LM + Lcwe lowers mean class-wise degradation from 21.00% to 16.50% and raises correction from 1.16% to 1.43%.Degradation decreases and correction improves in most adjacent-layer transitions and all intermediate-to-final comparisons.Source: Table 8 · p. 9, §4.5 · p. 9

• FM-Occ addresses feature homogenization during upsampling and semantic degradation across decoder layers in sparse 3D semantic occupancy prediction.Source: §5 · p. 9
• 3S jointly constructs and normalizes child features before spatial rearrangement, preserving greater variation in voxel feature responses.Source: §5 · p. 9
• ER-LM retains historical query information through gated recurrent memory and entropy-guided state modulation, complemented by class-weighted entropy loss Lcwe.Experiments on SemanticKITTI show improved occupancy and semantic prediction, while cross-layer analysis shows better preservation of correct predictions and average error correction across classes.Source: §5 · p. 9

• Groupwise IoU averages category IoU over predefined groups, enabling more fine-grained evaluation beyond overall performance.IoUc denotes the IoU of category c.Source: §A.1 · p. 11
• The 19 non-empty semantic categories are divided by structural complexity, boundary ambiguity, and sample imbalance.Source: §A.1 · p. 11
• ComplexIoU covers categories with extended spatial regions, irregular structures, or ambiguous semantic boundaries, including road, sidewalk, parking, other-ground, terrain, vegetation, and trunk.The listed proportions are road (15.30%), sidewalk (11.13%), parking (1.12%), other-ground (0.56%), terrain (9.17%), vegetation (39.30%), and trunk (0.51%).Source: §A.1 · p. 11
• TailIoU focuses on long-tail categories with limited samples, each below 0.1%, including bicyclist, person, motorcyclist, bicycle, motorcycle, and traffic-sign.Source: §A.1 · p. 11
• RegularIoU evaluates categories with relatively regular geometric structures and well-defined shapes, including car, building, truck, fence, pole, and other-vehicle.Source: §A.1 · p. 11

• For each frame, RMS response is defined for each active voxel feature vector xn in R^C, where C is the number of feature channels.The feature-channel index is j.Source: §A.2.1 · p. 11
• Spatial coefficient of variation uses the mean and standard deviation of the frame's voxel responses {Rn} to quantify their variation.Source: §A.2.1 · p. 11
• Measurements follow group normalization at the second sparse upsampling stage across 50 uniformly sampled SemanticKITTI validation frames, with CV averaged over frames.The Table 4 ratio divides each method's mean CV by that of sparse transposed convolution.Source: §A.2.1 · p. 12

• Fig. 4 compares independently min–max normalized response maps before feature fusion at identical grid resolution and projected sample locations.Region A covers a car and its surroundings, while regions B and C cover vegetation and ground; 3S shows clearer local variation in response magnitudes, not absolute magnitudes or semantic accuracy.Source: §A.2.2 · p. 12

A.3 DEFINITIONS OF SOFTMAX- AND SIGMOID-BASED FEATURE ALLOCATION

• Softmax normalizes allocation scores across candidate child positions, while Sigmoid applies independently and imposes no sum-to-one constraint.The allocation scores come from a two-layer MLP applied to mean-pooled candidate child representations; each resulting weight is applied to its corresponding representation.Source: §A.3 · p. 12

A.4 EFFECT OF ADDITIONAL LAYERNORM IN ER-LM

• With Lcwe, omitting the additional LayerNorm yields higher mIoU, TailIoU, and RegularIoU, at a small cost in ComplexIoU; the authors omit it in the final configuration.Without Lcwe, adding LayerNorm improves occupancy IoU but reduces semantic mIoU.Source: §A.4 · p. 12

A.5 EVALUATION PROTOCOL AND SEMANTIC TRANSITION METRICS

• The evaluation compares both models on 815 SemanticKITTI validation frames using all nine decoder outputs at ground-truth voxel resolution, excluding invalid labels and duplicate distributed samples.Correctness requires an exact semantic-label match, and counts are pooled over the validation set before computing class-wise rates.Source: §A.5 · pp. 12–13
• Degradation is the fraction of initially correct predictions that become incorrect, while correction is the fraction of initially incorrect predictions that become correct.Rates use only non-empty classes with non-zero denominators in both models; undefined rates are excluded, and lower mD and higher mR are more favorable.Source: §A.5 · p. 13