Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation

1 INTRODUCTION

• The question extends classical score- versus decision-level fusion to heterogeneous, independently pretrained frozen sources in generalized few-shot 3D evaluation.The contribution is an empirical and methodological study of information retention, not a new classifier-combination rule.Source: §1 · p. 2
• Top-1 collapse discards secondary semantic support that could agree across heterogeneous dense and sparse sources.The sparse source supplies object-level evidence only for a subset of points, while the dense source can distribute mass across related classes.Source: §1 · p. 2
• On both datasets, the largest improvement follows relaxing top-1 collapse, and moderate top-k retention slightly exceeds unrestricted tails.Top-20 slightly exceeds full retention on ScanNet200; top-10/top-20 slightly exceed it on ScanNet++.Source: §1 · p. 2
• The study varies retained semantic support with a same-input top-k ladder while holding sources, masks, geometry, vocabularies, and fusion rules fixed.The intervention targets information retention rather than comparing unrelated architectures.Source: §1 · pp. 2–3
• Full retention improves HM over matched top-1 by 6.40 points on 156 ScanNet200 scenes and 3.48 points on 50 ScanNet++ scenes.Paired scene-bootstrap confidence intervals exclude zero on both datasets.Source: §1 · p. 3
• A GroundingDINO–SAM2.1 source-replacement diagnostic reproduces the retention effect, while analyses identify both corrections and harms to strong dense predictions.The diagnostic reuses the same spatial detections and masks across retention conditions.Source: §1 · pp. 2–3

2.1 DECISION-LEVEL AND SCORE-LEVEL CLASSIFIER COMBINATION

• Score-level combination is classical, but its relevance to heterogeneous frozen sources with partial coverage and geometric lifting remains an open question.Kittler et al. (1998) studied classifier-combination rules; the paper does not claim score-level fusion itself as novel.Source: §2.1 · p. 3

2.2 GENERALIZED FEW-SHOT 3D SEGMENTATION

• Trained generalized few-shot methods adapt representations with supervision, so they are contextual references rather than matched causal baselines for frozen-source retention.GFS-VL uses pseudo-label selection, adaptive infilling, and novel–base mixing; HOP3D addresses base–novel interference with hierarchical orthogonal prototypes and entropy-based regularization.Source: §2.2 · p. 3

2.3 FROZEN AND TRAINING-FREE 2D-TO-3D COMPOSITION

• Frozen and training-free 2D-to-3D methods produce predictions or instances, whereas this study begins after frozen evidence is produced and isolates the information state at combination.FtD++ uses cross-modal fusion and distillation during adaptation, unlike this study’s no-training, no-distillation approach.Source: §2.3 · pp. 3–4

2.4 CALIBRATION AND RELIABILITY WEIGHTING

• Calibration and reliability weighting are tested as diagnostics of whether added weighting complexity can compensate for early information loss.The paper reports that such complexity generally cannot compensate for the information destroyed by early semantic collapse.Source: §2.4 · p. 4

3 PROBLEM FORMULATION

• The formulation combines a dense distribution with an available normalized sparse semantic vector, retaining the dense distribution when sparse evidence is unavailable.Source availability determines whether sparse evidence contributes to fusion.Source: §3 · p. 4

3.1 ARGMAX AS AN INFORMATION RESTRICTION

• Argmax maps many distinct distributions to one label, so hard labels alone cannot reproduce equal probability fusion for all source distributions.The proposition establishes information loss, not an accuracy claim; empirical usefulness of discarded alternatives is tested separately.Source: §3.1 · pp. 4–5

3.2 TOP-k SEMANTIC RETENTION

• Top-k retention keeps the k largest distribution entries and renormalizes them, ranging from an argmax-equivalent one-hot restriction to the complete distribution.Intermediate k values control the secondary semantic support available before fusion.Source: §3.2 · p. 5

4.1 FROZEN DENSE SOURCE: REGIONPLC

• RegionPLC supplies cached dense class probabilities over the evaluation vocabulary, with cache checks for probability mass and argmax agreement.The source space has 200 classes for ScanNet200 and 30 for ScanNet++; the generalized few-shot metric uses protocol base and novel subsets.Source: §4.1 · p. 5

4.2 PRIMARY SPARSE SOURCE: SAM3 CROSS-VIEW CONCEPT EVIDENCE

• Frozen SAM3 concept masks are lifted across posed views into sparse per-concept support, using binary masks and in-frustum visibility without a depth-based z-buffer.ScanNet++ sparse evidence covers its 18 novel categories and assigns zero sparse mass to the 12 base classes.Source: §4.2 · pp. 5–6

4.3 MATCHED FUSION AND THE RETENTION INTERVENTION

• The matched ladder applies the same top-k operator to both distributions before fusion, changing semantic support while leaving upstream evidence, geometry, and fusion weight fixed.The study also evaluates max and geometric/log pooling as diagnostic alternatives to linear averaging; it separately tests dense- and sparse-favouring tie policies.Source: §4.3 · p. 6

4.4 RELIABILITY-WEIGHTED VARIANTS

• ScanNet200 reliability variants estimate calibration and source or candidate-specific weighting from a fit partition, then test whether weighting complexity can recover information lost to early collapse.The 312 scenes are split by parity into 156 fit and 156 evaluation scenes.Source: §4.4 · p. 6

4.5 ALTERNATIVE SPARSE SOURCE: GROUNDINGDINO–SAM2.1

• A GroundingDINO–SAM2.1 diagnostic retains the dense RegionPLC distribution while varying sparse semantic support across conditions that reuse the same spatial evidence.The diagnostic uses fixed masks, frames, camera projection, dense probabilities, and 0.5/0.5 fusion; its hard control uses the exact argmax of the same semantic vectors.Source: §4.5 · pp. 6–7

5.1 SCANNET200

• ScanNet200 uses a deterministic parity split of 312 scenes, with calibration and learned reliability components fit on 156 scenes and comparisons evaluated on the untouched 156.The parameter-free ladder uses the same held-out scenes for paired comparison with the frozen hard selector.Source: §5.1 · p. 7

5.2 SCANNET++ CROSS-DATASET REPLICATION

• The parameter-free retention experiment is repeated on all 50 ScanNet++ validation scenes across 30 classes and a different sensing and projection regime.No fit split is required because the ladder has no learned parameter or selected threshold.Source: §5.2 · p. 7

5.3 MATCHED CONTROLS

• The matched controls hold model weights, point clouds, vocabularies, masks, geometry, scene lists, outputs, sparse evidence, and 0.5/0.5 fusion fixed.The retention ladder contrasts k = 1 with k = all.Source: §5.3 · p. 7

5.4 CACHE INTEGRITY, STORAGE, AND STREAMING EVALUATION

• The ScanNet200 cache covers all 312 scenes and 49.54 million points, while chunked evaluation keeps peak process memory at 719.52 MiB.The independent ScanNet++ cache contains all 50 scenes with zero integrity failures.Source: §5.4 · pp. 7–8

5.5 STATISTICAL ANALYSIS

• Headline HM comparisons use paired scene-bootstrap confidence intervals against matched top-1, while five-scene source-replacement intervals are labeled diagnostic.Tie policies and alternative operators use 5,000 paired bootstrap replicates; fixed-grid weight and PoE-ϵ sweeps do not select the headline method.Source: §5.5 · p. 8

6.1 MAIN SCANNET200 RESULT: INFORMATION RETENTION DOMINATES WEIGHTING COMPLEXITY

• 34.87 HM with full-distribution fusion exceeds matched top-1’s 28.47 HM by 6.40 points on held-out ScanNet200 scenes.The difference has a 95% CI of [+5.24, +7.64].Source: §6.1 · p. 8, §6.2 · p. 8, Table 1 · p. 9

6.2 SCANNET200 RETENTION CURVE

• On ScanNet200, top-2 raises HM from 28.47 to 31.83, and top-20 reaches 35.49; full retention reaches 34.87 HM.Top-20 exceeds unrestricted retention by 0.62 HM, indicating that retaining every low-probability tail entry is not necessary.Source: §6.2 · p. 8, Table 2 · p. 9

6.3 CROSS-DATASET REPLICATION ON SCANNET++

• On ScanNet++, full retention improves HM from 23.02 to 26.50, while novel mIoU rises and base mIoU falls.The replication changes dataset, capture, projection, geometry, and class-space size; full fusion raises novel mIoU from 17.94 to 25.83 but reduces base mIoU from 32.12 to 27.21.Source: §6.3 · p. 9, Table 3 · p. 10

6.4 SOURCE-INDEPENDENT SPARSE-STACK DIAGNOSTIC

• With GroundingDINO–SAM2.1 replacing SAM3, HM rises from 14.77 at top-1 to 18.75 with full retention.RegionPLC and spatial clusters remain fixed; the result is a five-scene diagnostic, not evidence that the replacement sparse predictor is stronger than the dense-only anchor.Source: §6.4 · p. 9, Table 4 · p. 11

6.5 DISTRIBUTION-LEVEL OPERATOR ROBUSTNESS

• Linear, max, and geometric pooling reach 34.87, 30.41, and 36.16 HM, respectively, exceeding the dense-favouring Top-1 reference at 26.53 HM.The geometric result is exploratory; these comparisons use identical frozen evidence and show the retention result is not unique to the sum rule.Source: §6.5 · p. 10, Table 5 · p. 11

6.6 CALIBRATION, TIE, WEIGHT, AND EFFICIENCY DIAGNOSTICS

• Full-retention advantage persists across tested tie policies, with frozen lowest-index ties yielding the smallest gap; full-distribution HM remains 34.17–34.87 across sparse weights from 0.3 to 0.7.Calibration reduces dense and sparse ECE from 29.20% and 43.29% to 1.54% and 0.69%; calibrated fusion reaches 34.51 HM versus 34.87 for uniform averaging. CPU fusion takes 12.00 s for 25.02 million held-out points.Source: §6.6 · p. 11

6.7 CONTEXTUAL POSITIONING AGAINST TRAINED GFS-PCS SYSTEMS

• The frozen inference-time system does not match trained 1-shot adaptation on ScanNet200; the comparison is contextual because protocols are unmatched.The reported ScanNet++ gap is smaller, and the study’s contribution is a controlled diagnosis within frozen composition.Source: §6.7 · p. 11, Table 6 · p. 12

6.8 WHAT THE THREE RETENTION CURVES SHOW

• Across ScanNet200 and ScanNet++, relaxing retention from top-1 to top-2 yields +3.36 HM and +3.64 HM, respectively; in GroundingDINO–SAM2.1, every successive retention level improves HM.Source: §6.8 · p. 12

7.1 EVALUATION-SCENE SELECTION PROTOCOL

• In three selected scenes, HM improves by +15.49, +13.60, and +12.48, while novel-mIoU improves by +14.70, +14.81, and +16.40.Scenes were selected using positive HM and novel gains and positive net point corrections; their correction counts were +2434, +1717, and +2953.Source: §7.1 · p. 12, Figure 2 · p. 13

7.2 WHERE CORRECTIONS OCCUR

• Figure 3 marks beneficial changes green and harmful changes red; improvements are spatially localized rather than uniform scene relabeling.Light grey marks points correct under both rules, and dark grey marks points wrong under both.Source: §7.2 · p. 12, Figure 3 · p. 13

7.3 FAILURE CASE: WHEN THE TAIL HURTS

• 12.25 HM separates full retention from matched top-1 in scene0490 00, with −12,015 net point corrections.The frozen hard selector, matched top-1 fusion, and full-retention fusion score 18.37, 31.66, and 19.41 HM, respectively.Source: §7.3 · p. 12

8.1 THE NOVELTY IS THE MECHANISTIC DIAGNOSIS, NOT THE SUM RULE

• The contribution is a controlled diagnosis of premature semantic collapse in heterogeneous frozen foundation-model composition, not a novel sum rule.The study asks how much useful semantic information is lost when sources collapse before interaction, using a matched top-k ladder to test that question.Source: §8.1 · p. 13

8.2 INFORMATION REPRESENTATION MATTERS MORE THAN SELECTOR COMPLEXITY

• Calibration improves confidence reliability but does not outperform simple full-distribution averaging; max, sum, and geometric pooling all exceed a common hard Top-1 control.Held-out ECE falls from 29.20% to 1.54% for RegionPLC and from 43.29% to 0.69% for SAM3.Source: §8.2 · p. 14

8.3 A SMALL CANDIDATE SET IS OFTEN ENOUGH

• Across ScanNet200 and ScanNet++, the largest gain comes immediately after top-1 and later gains saturate, suggesting a compact set of plausible alternatives often suffices.A GroundingDINO–SAM2.1 replacement reproduces the trend, though its diagnostic is too small to establish a new benchmark result.Source: §8.3 · p. 14

8.4 BASE–NOVEL TRADE-OFF

• ScanNet++ sparse evidence supports only novel classes, so stronger sparse influence can raise novel mIoU while lowering base mIoU.HM is the protocol-level balance metric, not evidence that both class groups always improve simultaneously.Source: §8.4 · p. 14

8.5 RELATION TO TRAINED GFS-PCS SYSTEMS

• Trained GFS-VL and HOP3D remain stronger in absolute ScanNet200 HM because they adapt representations using labeled few-shot data; they are not matched causal baselines.The contextual comparison is a scale reference, while this study avoids parameter updates to isolate information retention.Source: §8.5 · p. 14

8.6 IMPLICATIONS BEYOND 3D

• The authors suggest early hard decisions may erase useful non-maximal alternatives before composition can exploit them, while cautioning against universal transfer of the numerical findings.They identify 2D–3D perception, multimodal routing, and LLM ensembling as potentially relevant settings.Source: §8.6 · p. 15

• The study diagnoses premature collapse only in the tested frozen 3D foundation-model regime; it does not claim full probability averaging is novel or universally optimal.Classical score-level fusion predates this work.Source: §9 · p. 15
• Moderate top-k retention slightly outperforms unrestricted tails on both principal datasets, and the best amount may depend on source pair, class space, and local ambiguity.Adaptive k and related gating strategies are outside the present causal study.Source: §9 · p. 15
• On ScanNet++, the sparse source supports 18 novel categories but not the 12 base classes, limiting claims of symmetric benefit across class groups.Adding base priors or learned class-conditional sparse calibration would change the sparse evidence and is excluded from the matched intervention.Source: §9 · p. 15
• The primary sparse branch depends on posed imagery, camera calibration, frame sampling, and SAM3 masks, and uses in-frustum projection without a depth-based occlusion filter.These choices affect evidence quality and coverage; the GroundingDINO–SAM2.1 diagnostic reduces but does not eliminate dependence on sparse-source design.Source: §9 · p. 15
• The GroundingDINO–SAM2.1 diagnostic has only five scenes and remains below the dense-only anchor, so it is not a third benchmark or evidence that retention offsets a weak sparse source.Its purpose is a mechanism diagnostic.Source: §9 · p. 15
• Full distributions require substantial storage: the 200-class float16 cache is 400 bytes per point and 18.46 GiB raw, favoring short top-k lists for constrained deployment.Compression reduces the NPZ footprint to 13.31 GiB, while streaming keeps peak process memory below 0.72 GiB.Source: §9 · p. 15
• Geometric/log pooling was tested after the primary linear experiment was frozen, so linear averaging remains the headline method rather than a post-hoc operator change.The paper calls for future comparisons of fusion operators under fully matched top-k interventions.Source: §9 · p. 15
• The study concerns inference-time composition, not training, and does not argue that few-shot support, finetuning, prototype adaptation, or learned alignment are unnecessary.Its claim is limited to how evidence collapse affects what a downstream combiner can use when frozen semantic evidence is available.Source: §9 · p. 15

10 CONCLUSION

• Full retention improves HM over top-1 by 6.40 points on ScanNet200 and 3.48 points on ScanNet++, with paired scene-bootstrap confidence intervals excluding zero.The largest gain on both datasets appears immediately after preserving more than one hypothesis, while moderate top-k retention slightly exceeds the unrestricted tail.Source: §10 · p. 16
• A GroundingDINO–SAM2.1 source replacement increases HM from 14.77 at top-1 to 18.75 with full sparse retention on the same spatial evidence.This diagnostic reproduces the retention effect with a different sparse source stack.Source: §10 · p. 16
• Full-distribution performance varies by less than one HM point across sparse-source weights from 0.3 to 0.7 and beats the hard control under linear, max, and geometric pooling.The retention advantage also survives dense-, sparse-, and lowest-index tie policies; calibration repair does not eliminate it.Source: §10 · p. 16
• The paper frames premature semantic collapse as a repeatable information bottleneck, arguing that compact plausible alternatives can matter more than downstream selector complexity.Its contribution is a controlled modern study of a classical fusion distinction in heterogeneous 3D/2D foundation-model composition.Source: §10 · p. 16

A TIE-POLICY ROBUSTNESS

• 20,402,026 valid points have sparse evidence, and dense and sparse Top-1 agree on 6,542,080 but disagree on 13,859,946.Thus 67.934% of sparse-covered valid points form exact two-class ties under 0.5/0.5 one-hot fusion; the frozen implementation resolves them by lowest class index.Source: A TIE-POLICY ROBUSTNESS · p. 16
• The primary +6.40 HM retention gain is the smallest observed gap because the frozen lowest-index policy is the strongest of the three Top-1 variants.Table 7 compares tie policies on the same 156 held-out ScanNet200 scenes, with full distribution fusion fixed at 34.87 HM.Source: A TIE-POLICY ROBUSTNESS · p. 16, Table 7 · p. 16
• The dense-favouring rule reduces exactly to the dense prediction, while sparse-favouring exposes the lower quality of the sparse Top-1 source.When sources disagree, dense-favouring chooses dense; when they agree, it preserves their shared class.Source: A TIE-POLICY ROBUSTNESS · p. 16

B FUSION-WEIGHT SENSITIVITY

• Full-distribution fusion is stable across the sparse-source weight grid in the ScanNet200 sensitivity analysis.Table 8 reports sensitivity to the sparse-source weight.Source: B FUSION-WEIGHT SENSITIVITY · p. 16, Table 8 · p. 17
• With one-hot sources, sparse weights below 0.5 make dense win every disagreement, while weights above 0.5 make sparse win every disagreement.At the exact 0.5 tie, the Top-1 control resolves in favor of dense to avoid class-index dependence.Source: B FUSION-WEIGHT SENSITIVITY · p. 16
• Retained distributions preserve graded within-class evidence, supporting genuinely graded source weighting rather than a hard switch.The sparse weight is varied as ws while the dense weight is set to 1 − ws.Source: B FUSION-WEIGHT SENSITIVITY · p. 16

C HELD-OUT CALIBRATION DIAGNOSTICS

• The dense source is strongly under-confident before calibration, while the sparse source is strongly over-confident.Calibration changes confidence reliability without changing source accuracy, and does not eliminate the retention advantage.Source: C HELD-OUT CALIBRATION DIAGNOSTICS · p. 17
• Calibrated probability fusion reaches 34.51 HM, below the 34.87 HM of uniform full-distribution averaging.The held-out calibration table reports accuracy, confidence, and ECE as percentages, plus binary correctness Brier score; calibrators are fit on 156 scenes and evaluated on a disjoint 156-scene partition.Source: C HELD-OUT CALIBRATION DIAGNOSTICS · p. 17, Table 9 · p. 17

D COMPUTE AND MEMORY ACCOUNTING

• Materializing the largest scene’s 438,565-point, 200-class distribution as float32 would require 334.60 MiB, motivating a streaming evaluator.Observed saturation around moderate k suggests storage could also be reduced by keeping short candidate lists; Table 10 reports measured cache and CPU evaluation costs.Source: D COMPUTE AND MEMORY ACCOUNTING · p. 17

E POE NUMERICAL-FLOOR SENSITIVITY

• Varying the geometric-pooling numerical floor from 10^-4 to 10^-12 changes HM by only 0.16 points, with effectively identical results for ϵ ≤10^-8.All source evidence and weights remain fixed in this sensitivity test.Source: E POE NUMERICAL-FLOOR SENSITIVITY · p. 17