Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer

Wentao Sun; Yiping Chen; John S. Zelek; Jonathan Li

Source-linked AI summary

Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer

Wentao Sun, Yiping Chen, John S. Zelek, Jonathan Li

arXiv:2608.20659v1Aug 21, 2026cs.CV

TL;DR

2D-to-3D transfer systems are difficult to compare because task and representation labels obscure decisions about support, identity, reconciliation, fusion, and retained state. LAF formalizes those decisions as five operators and a persistent-carrier contract, then audits 161 systems; the resulting traces reveal recurring distinctions in identity, queryability, evaluation, and scoped cost claims.

Problem
Task- and representation-based groupings obscure the decisions determining whether transferred evidence remains coherent across views and useful for later queries or corrections.
Method
LAF models transfer as Generate, Associate, Reconcile, Fuse, and Persist/Query, with a carrier contract covering support, semantics, identity, uncertainty, provenance, and operations.
Results
Decision traces from 161 audited systems show that association and identity differ, carrier design fixes query and correction boundaries, and evaluation and qualifiers require stage-specific interpretation.
Takeaways & Limitations
LAF provides a representation-neutral basis for comparing systems, diagnosing irreversible failures, and specifying revisable 3D perception.
Takeaways & Limitations
Validation depends on faithful reconstruction from published evidence, lacks formal inter-annotator reliability, and includes incomplete runtime reporting.

Abstract

from arXiv · show

Methods that transfer predictions from two-dimensional foundation models into three-dimensional segmentation are commonly grouped by task or representation. Those groupings obscure the decisions that determine whether a system remains coherent across views: where image evidence is grounded, when observations become one identity, how semantic and granularity conflicts are handled, which information is fused, and what state survives for later queries. We introduce \textbf{Lift, Associate, and Fuse (LAF)}, a decision-centric framework that represents a transfer system as five operators: \textbf{Generate, Associate, Reconcile, Fuse, and Persist/Query}. LAF defines an explicit contract for the persistent carrier---its spatial support, semantic state, identity state, uncertainty, provenance, and supported operations---and identifies the first stage at which discarded evidence becomes unrecoverable. We operationalize the framework as a structured audit protocol and apply it to 161 systems available through 7 August 2026, spanning point-, field-, Gaussian-, object-, graph-, and memory-based carriers. Representation, temporal, relational, and feed-forward stress tests required no additional analytical stage after the final confirmation pass. The resulting decision traces expose four recurring properties: association does not establish identity; carrier design fixes both the query interface and correction boundary; rendered-view, native-3D, and proposal-level evaluations are not interchangeable; and qualifiers such as \emph{training-free}, \emph{real-time}, \emph{open-vocabulary}, and \emph{generalizable} are meaningful only when attached to a stage and a complete cost ledger. LAF therefore supplies a representation-neutral method for comparing existing systems, diagnosing irreversible failures, and specifying revisable 3D perception for future agents.

1 Introduction

LAF reframes 2D-to-3D transfer around the decisions that establish support, identity, reconciliation, fusion, and persistent state. Applied to 161 systems, the framework exposes recurring distinctions in evaluation, carrier capabilities, and reporting claims.

Motivation: Existing task- and representation-based classifications are useful for retrieval but weak for diagnosing when systems establish identity, retain evidence, expose queries, and fail.Systems sharing masks and 3D representations can make different decisions, while different representations can implement the same decision sequence.
Decision distinctions: Association locates evidence on 3D support, whereas reconciliation determines identity, semantic labels, and part–whole relations; therefore, correct association does not establish identity.The framework also distinguishes fusion from the later query and correction interface.
Framework: LAF separates Generate, Associate, Reconcile, Fuse, and Persist/Query as decisions governing evidence exposure, 3D support, identity and conflict handling, reusable state, and downstream operations.The operators are analytical decisions rather than mandatory software modules and may be jointly implemented or revisited online.
Persistent state: The persistent-carrier contract records spatial support, semantic state, identity, uncertainty, provenance, and supported operations to determine what later queries and corrections can recover.Carrier outputs may include point labels or language features and are not interchangeable merely because they render similar masks.
Validation and findings: The audit protocol covers 161 systems across six carrier families, and its final confirmation pass required no additional analytical stage.The findings identify distinct roles for association and identity, carrier-dependent query and correction boundaries, non-interchangeable evaluation views, and stage-scoped efficiency or vocabulary claims.

2 The LAF Framework and Validation Protocol

The LAF protocol represents transfer systems as compositional decision traces and audits their persistent carriers, recoverable hypotheses, scope, and evidence. Its validation corpus and stress tests support representation-neutral coverage while identifying limits in evidence reconstruction and runtime reporting.

Operator trace: LAF composes image evidence generation, 3D support assignment, hypothesis reconciliation, state fusion, and querying into an auditable system trace.The trace records supervision, discarded state, and costs even when operators share parameters or are optimized jointly.
Persistent carrier: The carrier contract distinguishes spatial support, semantic state, identity and topology, uncertainty, provenance, and supported operations, including retrieval, correction, updates, and reasoning.The retained tuple, rather than the name of an external model, determines query and correction consequences.
Irreversibility: The first irreversible loss is the earliest stage at which a downstream-relevant hypothesis is absent and discarded evidence cannot be reconstructed by later operators.This loss is query-relative: provenance harmless for one benchmark may be essential for later split, correction, or explanation requests.
Corpus construction: The validation corpus contains 161 contract-satisfying methods within a final snapshot of 166 indexed methods, based on 165 parsed PDFs and 2,277 pages.Eligibility requires material 3D segmentation or persistent labels, broadly pretrained 2D transfer, a substantive decision, and claims checkable against primary evidence.
Stress testing: The final confirmation pass found no missing canonical mechanism or new core operator, supporting analytical saturation at the cutoff rather than guaranteeing future completeness.The protocol included representation, temporal, relational, dynamic, online, feed-forward, and boundary-case checks.
Validity and limitations: The framework depends on faithful reconstruction from published evidence and has incomplete runtime reporting and no formal inter-annotator reliability study.The authors propose a future registered coding study measuring agreement on stage boundaries, identity establishment time, and carrier operations.

3 Operational Definitions and Correspondence Operators

The framework distinguishes evidence, 3D support, reconciliation, fusion, and persistent carrier state, treating these as decisions rather than interchangeable representations. Correspondence mechanisms differ in how they assign support, visibility, semantics, identity, and granularity.

Operational definitions: The reviewed systems transfer colors, masks, embeddings, identities, affinities, and uncertainty, whose valid operations depend on how each is represented.Calling these objects “features” obscures their distinct support, identity, uncertainty, provenance, and query implications.
Correspondence notation: Association weights describe how strongly each image observation supports a 3D element, after which reconciliation and fusion construct persistent state for queries.The persistent state may be memberships, fields, graphs, or other carrier forms.
Hard projection: Explicit projection requires transformed image coordinates, positive depth, bounds, and visibility testing before assigning hard or soft support.Soft variants can incorporate depth residual, view angle, object visibility, or confidence.
Neural fields: Neural fields distribute evidence along camera rays using density and transmittance weights, enabling differentiable rendering and reverse supervision.This association remains dependent on the correctness of learned density.
Gaussian renderers: Gaussian systems use non-equivalent contribution rules, including identity codes, feature accumulation, dominant-primitive assignment, and decoder correlations.Geometric carrier, rendering support, and semantic storage should therefore be recorded separately.
Reconciliation and carrier state: Identity, semantic meaning, and granularity are distinct variables, while persistent state may preserve hierarchies, overlapping memberships, or relation graphs instead of one partition.These choices determine whether later queries can revise a partition.

4 Generate: What Comes from 2D?

Generation sets the information ceiling by determining what 2D models expose, at what scale, with which identity, confidence, vocabulary, and provenance. The surveyed systems use masks, language, self-supervised features, trackers, geometry, and offline corpora as different evidence sources.

Generation scope: Generation determines the pipeline’s information ceiling because 3D processing cannot reliably recover categories, parts, or instances absent from source evidence.The audit therefore records the generated object, granularity, identity, confidence, and vocabulary timing.
Masks and proposals: Mask generators provide object support, but independent masks have arbitrary per-view IDs and proposal topology can constrain later identity and granularity.Systems may use prompts, trackers, or 3D resolution to connect these proposals.
Language supervision: Language supervision enters through dense features, crops, boxes, text-conditioned masks, captions, and generated vocabularies at different spatial units.Generated text can hallucinate entities, omit small objects, and repeat correlated language biases, making provenance and support part of supervision.
Feature sources: Self-supervised features add structural evidence, but their presence does not by itself establish an open-vocabulary interface or determine the evaluation task.HumanCrafter illustrates the distinction by ending with a supervised 28-part classifier despite using DINOv2 features.
Identity and geometry: Trackers assert temporal identity that can bridge weak appearance across views, but identity switches become correlated supervision throughout the scene.Geometry-based reconciliation can reduce dependence on tracking.
Geometric evidence: Removing calibrated sensors can increase deployment flexibility while coupling semantic error with learned correspondence.Uni3R and Ov3R infer geometry from unposed images or video while constructing semantic representations.
Evidence retention: Retaining proposal confidence, provenance, competing masks, tracker confidence, vocabulary source, and explicit absence enables later diagnosis and map revision.Selected-only masks, features, or labels make missing evidence indistinguishable from confident background.
Offline generation: Offline generators and canonical corpora can disappear at deployment while their coverage, naming, spatial assumptions, and biases remain encoded in reusable 3D models.Such systems should report data-engine provenance and canonicalization cost even when inference is fast.

5 Associate: From Image Evidence to 3D Support

Association determines which 3D elements a 2D prediction may supervise and therefore fixes support, uncertainty, and failure correlations for later decisions. The framework distinguishes calibrated, renderer-weighted, and learned or temporary correspondence, including hard and soft variants.

Association families: Association is a substantive support decision, not clerical projection, because it defines the uncertainty and correlated failures inherited by later semantic and instance decisions.The same 2D model can behave differently under different association operators.
Association families: Figure 3 separates hard projection, neural-field association, Gaussian-renderer association, and learned or temporary intermediaries as four realizations of candidate 3D support.None establishes whether observations from different views belong to the same physical entity.
Calibrated lifting: Calibrated lifting projects points into images and applies depth or visibility rules before inheriting pixel or mask evidence.Visibility policies range from binary tolerances to view ranking and co-visible superpoint filtering.
Renderer-weighted lifting: Neural fields replace binary point-pixel relations with density and transmittance distributions along camera rays, supporting differentiable rendering and reverse mask optimization.OpenNeRF can also use feature disagreement to select additional observations.
Gaussian association: Gaussian renderers attach attributes to explicit primitives through alpha-transmittance contributions, while some methods harden association by selecting the maximum-contributing Gaussian.Other systems preserve richer multi-view weights for diagnosis and correction.
Granularity at association: Association can determine the 3D primitive and impose a hard ceiling on the available part inventory before reconciliation begins.PS3 creates texture-aligned superpoints from masks before tracking candidate parts.
Learned and temporary correspondence: Learned or temporary intermediaries make reconstruction, alignment, and view planning part of association rather than neutral preprocessing.Their assumptions determine where support errors enter before reconciliation.

6 Reconcile: Identity, Conflict, and Granularity

Reconciliation decides whether observations share identity, how semantic conflicts are handled, and which granularity persists. Systems may establish identity before learning, through geometry, during optimization, or at query time, with different reversibility and correction boundaries.

Reconciliation decisions: Reconciliation must preserve geometry, identity, and semantic scale because agreement on support does not guarantee shared identity or a single valid granularity.A chair seat and chair can be separate at object scale but related within a part hierarchy.
Identity establishment: Identity may be externally supplied by trackers, geometrically inferred from overlap, or learned through attributes, codebooks, and relation-based carriers.Each timing determines which mistakes can still be corrected.
Identity establishment: Tracker-based carriers preserve identity efficiently but cannot retroactively correct merged objects or identity switches inherited from tracking.This limitation distinguishes identity establishment before 3D learning from geometry-first reconciliation.
Semantic conflict: Semantic codebooks and language alignment can suppress view-dependent conflict, but they cannot recover missed or wrongly tracked objects and may merge rare concepts through compression.Identity and meaning therefore require separate audit dimensions.
Semantic conflict: Contextual MLLM reasoning may disambiguate similar objects but can encourage room-prior shortcuts unsupported by object pixels or geometry.Such reconciliation requires abstention and evidence attribution.
Timing and feedback: Reconciliation may occur before deployment, during carrier optimization, over fixed fields at query time, or through feedback that mutates geometry.These choices create different reversibility and appearance-preservation requirements.
Persistent conflict state: Preserving competing views, graph edges, and provenance is preferable to collapsing semantic disagreement into one confident answer.Proto-SaGa and LUDVIG reduce inconsistency while retaining distinct reconciliation mechanisms.
Scale and hierarchy: PS3 improves medium-scale identity while missing tiny hooks on ScanNet++, motivating evaluation that reports proposal topology and scale-specific recall.Its gains concentrate at AP50 rather than mean AP.

7 Fuse: Combining Evidence and Constructing the Carrier

Fusion converts repeated, conflicting observations attached to 3D support into reusable carrier state, but each rule determines which uncertainty, provenance, and computational cost remain available. Full-pipeline accounting is therefore necessary because hard assignments, averaging, learned fields, and graphs expose different correction and efficiency boundaries.

Fusion design: Fusion converts attached observations into reusable state through averaging, learned fields, discrete assignments, or relational graphs.Fusion should be judged by preserved uncertainty and by costs moved between construction and query time.
Aggregation: Confidence-weighted averaging and prototype consensus are incremental and auditable but can collapse inconsistent observations into a unimodal summary.Without retained weights or uncertainty, contradictory evidence may become unrecoverable.
Aggregation: Robust fusion makes reliability and discriminability explicit, using uncertainty-aware smoothing, instance-centered weighting, or view-reliability estimation.These designs suggest retaining uncertainty or provenance needed to recompute reliability.
Amortization: Amortized fusion reduces deployment work by learning cross-scene carriers, but its construction cost and learned consistency properties remain part of the system contract.PartField’s forward pass is backed by two weeks of training on eight A100 GPUs.
Discrete carriers: Hard point or Gaussian IDs support extraction and editing directly, but suppress alternative correspondences and can change substantially through proposal filtering and merging.Hard assignments should preserve the evidence behind each assignment when later revision matters.
Full cost ledger: A credible efficiency claim separates reconstruction, preprocessing, association, carrier optimization, storage, and query inference rather than reporting only the final operation.LEGaussians reports fast rendering after approximately 30 minutes of dense feature extraction and about one hour of semantic fitting.

8 Persist, Render, and Query

Persistence determines which queries, revisions, and downstream operations remain possible without rerunning upstream 2D models. Carriers that look similar when rendered can nevertheless differ in native support, identity state, correction boundaries, and query-time costs.

Carrier contracts: Persistent carriers differ by retained geometry, semantics, identity, uncertainty, relations, and supported operations despite producing similar colored masks.The natural output and correction boundary define what later queries can recover or revise.
Carrier families: Point carriers support native extraction, while neural fields naturally support novel-view rendering and continuous spatial queries.Hard point masks have low query-time computation but inherit vocabulary and partition choices from upstream masks and tags.
Query surfaces: Index-preserving point carriers and native-scene Gaussian carriers expose different label-return and rendering interfaces, while sparse discrete labels require new upstream detections for new semantic queries.Neither CDSeg nor SAGOnline stores a universal language feature for text lookup against existing state.
Query surfaces: Feed-forward labeled reconstruction can provide a direct approximately 2.3-minute result, whereas its approximately 35-minute LUDVIG variant answers a different interface and cost question.The two outputs should not be treated as interchangeable merely because both originate from the same system family.
Query surfaces: A carrier may expose multiple interacting query surfaces, as COS3D links a compact instance field with a high-dimensional language field through an instance-to-language mapping.Persistence can include interaction rules between fields, not only stored vectors.
Relational carriers: Object and graph carriers support higher-level retrieval, relational reasoning, and planning without returning to dense point clouds for every operation.Octree-Graph combines adaptive occupancy with spatial and semantic relations on graph edges.
Dynamic carriers: Persistence can be an interaction cache or an update rule, supporting repeated static interaction or temporally evolving multimodal perception.WildSeg3D reduces later point-prompt interaction to milliseconds after a reported tens-of-seconds construction stage.
Evaluation interface: Evaluation should match each carrier’s promised interface, including native topology, repeated-query latency, update consistency, or graph retrieval and navigation accuracy.Persistence defines what is reusable, revisable, and evidentially supportable downstream.

9 Evaluation Protocols and the Full Cost Ledger

Evaluation numbers are comparable only when prediction objects, vocabularies, geometries, views, prompts, proposal sources, and oracle access are aligned. The full cost ledger must likewise distinguish work moved upstream from the reported inference operation.

Protocol comparability: Reported accuracy is comparable only when the prediction object, vocabulary, source geometry, and query protocol match.The same mIoU or AP label can refer to materially different rendered, native-3D, oracle, or proposal-based tasks.
Protocol comparability: IQGS rendered-view mIoU and OV3DSeg-VGGT semantic probabilities on predicted geometry are not interchangeable with open-vocabulary AP on fixed ScanNet geometry.IQGS reports 30k A100 iterations without normalized wall time; numerical cost and accuracy claims were withheld for OV3DSeg-VGGT.
Evaluation domains: Rendered-view, native-3D, proposal-level, and downstream evaluations answer different questions about support, topology, instance generation, matching, and action policy.Cross-method comparison requires controlling geometry, views, prompts, conversions, and oracle access.

9.1 The evaluation object comes before the metric

Evaluation must identify the prediction object and protocol before interpreting a metric. Rendered-view, native-3D, and proposal-level results answer different questions and should remain separate without matched conversions or oracles.

Rendered-view evaluation: Rendered-view evaluation compares predicted image masks with 2D annotations, matching methods whose principal interface is differentiable rendering.Examples include LERF, 3D-OVS, LangSplat, and LEGaussians.
Native-3D evaluation: Native-3D evaluation compares labels on points, mesh elements, voxels, or Gaussians rather than rendered images.OpenScene and OpenNeRF use Replica points, while OpenGaussian and COS3D include Gaussian-level protocols.
Protocol boundaries: One dataset can support different evaluation domains, supervision sources, and timing boundaries, so its name alone cannot align reported results.DesktopObjects-360 includes rendered masks, native 3D instances, propagated masks, and an online latency study.
Reporting rule: Table 4 requires conversions, proposal oracles, prompt access, and downstream policies to remain visible when evaluation families are compared.The table keeps families separate unless the same conversion and oracle protocol applies to every method.
Proposal protocols: Instance segmentation results also depend on proposal source, with experiments separating 2D-guided, 3D-network, combined, and oracle-mask settings.Oracle-mask experiments isolate semantic reasoning from proposal quality.
Protocol families: Rendered neural-field metrics, proposal AP, point mIoU, object-part IoU, and prompt-specific segmentation scores should not be placed in one unlabeled ranking.Combining them would erase differences in prediction object, vocabulary, and supervision.

9.2 Vocabulary and supervision define different generalization claims

Vocabulary access and supervision regime describe different generalization dimensions. Evaluation must preserve class partitions, prompt capabilities, training sources, and the distinction between annotation-free and training-free systems.

Vocabulary protocols: Base/novel evaluation must report its class partition because changing seen and unseen category counts changes supervision and semantic difficulty.PLA, RegionPLC, and PGOV3D report harmonic IoU alongside base and novel mIoU across partitions.
Supervision: Annotation-free means no manual target labels, not no training or no supervision.RegionPLC uses generated captions and multi-GPU point-language training, while PGOV3D uses MLLM/SAM pseudo-entities and fine-tuning.
Reporting boundary: A complete cost ledger should separate one-time construction, persistent storage, and marginal query cost, reporting NR when a component is unmeasured.This reporting boundary prevents vocabulary or supervision claims from obscuring deployment costs.
Generalization regimes: Reusable learned representations and per-scene language fields are valid but amortize cost and prior knowledge differently.Cross-scene representations reuse learned parameters, whereas per-scene fields optimize a carrier for the test scene.
Prompt capability: Free-form language evaluation ranges from category labels to attribute, affordance, and referring-expression prompts, requiring dedicated prompt sets and correctness criteria.SOLE and OV3D-CG illustrate increasingly strong language demands beyond fixed label lookup.

9.3 Cost must be reported stage by stage

Cost comparisons are meaningful only when the full pipeline is decomposed by stage and payment boundary. Construction, preprocessing, training, storage, and query costs can shift between one-time, amortized, and marginal workloads.

Cost ledger: A complete ledger includes reconstruction, 2D preprocessing, association and reconciliation, carrier construction or training, persistent storage, and per-query inference.Omitting any stage changes the apparent conclusion about cost.
Query workload: Reports must distinguish first-query latency, subsequent-query latency, and the number of prompts or classes processed together.Cached carriers may reduce marginal latency for repeated prompts, while dense fields may be wasteful for one known category list.
Amortized training: 6.2 seconds of HumanCrafter inference must be read beside seven days of training on eight A800 GPUs and specialized data preparation.Inference latency alone does not describe the full deployment cost.
Construction versus interaction: 30 seconds of WildSeg3D alignment and cache construction precede 5–20 ms interaction, while Rethinking OVRF adds roughly 40 minutes of NeRF fitting and 5 minutes of 3DGS transfer.These examples place substantial work before fast interaction or rendering.
Quality domains: Rendered asset quality, native extraction quality, scene-graph recall, and relationship-guided instance IoU are complementary domains rather than one shared leaderboard.Direct asset protocols expose information hidden by ordinary mask mIoU.
Reconstruction boundary: Changing Gaussian geometry or object-generating anchors creates a different reconstruction pipeline from frozen-carrier transfer and requires reporting that change.COB-GS, ObjectGS, Trace3D, BEA-GS, and FMGS illustrate this distinction.
Stage boundaries: SAGOnline reports 1.47 seconds to aggregate 207 masks and 27 ms refined rendering, whereas CDSeg reports roughly 2.23–3.74 seconds for 300 views while excluding mask inference and reconstruction.The denominators and exclusions differ across the two systems.
Pipeline comparison: PanSt3R takes about 2.3 minutes for direct prediction but about 35 minutes when LUDVIG supplies novel-view 3DGS output.Feed-forward and transferred pipelines expose different amortization boundaries.

9.4 Minimum reporting checklist

The minimum reporting checklist attaches every result to its evaluation object, vocabulary, supervision, geometry, proposal, prompt, runtime, and memory boundaries. Without these denominators, efficiency and accuracy claims are not directly comparable.

Required metadata: Every quantitative row should state dataset version, category split, vocabulary access, output domain, geometry source, proposal source, prompt protocol, and supervision type.Runtime inclusion and memory accounting must also be explicit.
Part methods: Part-method comparisons require separate accounting for pseudo-label pipelines, cross-shape training, prompts, adaptation hardware, and missing wall-clock measurements.Find3D, PatchAlign3D, GeoSAM2, and MV3DIS expose different omissions and boundaries.
Efficiency terminology: Training-free, direct-inference, and end-to-end efficiency are incompatible claims when rendering, detector calls, clustering, or prior corpus construction differ.PartSLIP, COPS, and CoSMo3D illustrate these distinct meanings.
Agentic latency: REALM reports 8.68 seconds per query, while its 354.72 FPS figure measures rendering rather than the complete agentic interface.The query breakdown is 2.53 seconds global reasoning, 2.48 local reasoning, and 3.67 refinement.
Evaluation domain: Rendered-mask IoU and native class-agnostic AP are complementary tests, not interchangeable estimates of accuracy.OpenSplat3D reports 84.0/78.8 rendered mIoU/mBIoU and 24.5/41.7/57.1 native AP/AP50/AP25.
Modality shift: RangeSAM’s 60.9 SemanticKITTI validation mIoU versus RangeFormer’s 73.3 tests modality-shift viability, not open-vocabulary exposure or efficiency.The experiment also uses eight 40-GB A100 GPUs and 60 epochs of pretraining.
Upstream costs: Native point-instance AP can still exclude scene-wide 2D inference and graph construction, so upstream foundation calls and view sampling must accompany the metric.MaskClustering, SAI3D, and PS3 illustrate this missing denominator.
Carrier accounting: A fast lookup is not evidence of a cheap persistent representation when construction, storage, and query units differ.Hierarchical systems report distinct object/part construction, aggregation, embedding, search, storage, or update costs.

10 Failure Propagation and Framework Diagnostics

LAF diagnoses failures by tracing compositional errors to the first stage where information becomes irreversibly lost. The audit distinguishes proposal, geometry, identity, semantic, granularity, compression, and projection failures, along with their recovery boundaries.

Failure propagation: Upstream mistakes alter the evidence available to every downstream stage, so failures are organized by the first irreversible information loss.A final classifier cannot recover an object that was never proposed, and correct association cannot repair an incorrect reconstruction.
Failure propagation: The first irreversible commitment is system- and query-dependent and may occur during Generate, Associate, Reconcile, Fuse, or Persist/Query.The diagnostic map links each failure source to its propagation path, causal test, and required recovery operation.
Geometric correspondence: Association policies depend on accurate poses, depth, and synchronization, while strict or loose depth tolerances respectively drop valid observations or transfer masks to occluded surfaces.Renderer-based assignments soften projection errors but do not eliminate them.
Proposal topology: Proposal ceilings prevent open-vocabulary classifiers from splitting merged masks or recreating omitted objects, although learned 3D proposals can increase coverage.Repair operations such as overlap removal, merging, and containment pruning trade one topology failure against another.
Identity and semantic propagation: External trackers can make identity switches or early merges persistent in 3D, shaping later Gaussian responses and object selections as one.Late semantic attachment localizes teacher errors but does not remove them, and reliable detectors can still bias recall-sensitive downstream tasks.
Other loss channels: Projection, scale, compression, and ontology choices create distinct loss channels that aggregate mIoU may obscure.Diagnostics should separate raster collisions, granularity instability, rare-query loss, and incorrect entity or part inventories from ordinary boundary errors.

11 From Passive Segmentation to Agentic 3D Perception

The paper extends 3D segmentation from static labeling toward agentic perception by treating persistent carriers, active observation, correction, and relational reasoning as a closed loop. This shift requires identities and uncertainty that survive updates, plus explicit operations for revising state.

From passive to agentic perception: Agentic 3D perception must retain identities over time, choose observations, revise beliefs after change, and use entities in relational tasks.The reviewed systems indicate a transition from passive scene labeling to persistent spatial memory rather than a complete agentic stack.
Persistent memory: Online segmentation cannot be evaluated only by frame throughput because merges alter future association, retrieval, and correction.Persistent state should include merge provenance, confidence, timestamps, supporting views, and alternative identities.
Correction and revision: Selective repair can relocalize changed neighborhoods, remove obsolete voxels, fuse new observations, and regenerate affected graph edges without rebuilding the scene.Such local updates require explicit feature invalidation to prevent residual evidence from directing agents toward previous object locations.
Active observation: Active observation makes view selection part of segmentation and its cost, using disagreement or query-conditioned reasoning to acquire additional image evidence.The paper warns that selected views can reinforce reconstruction errors, so active systems need calibrated geometric and semantic uncertainty.
Active observation: REALM shows that excessive selected-view refinement can degrade masks, making held-out evidence necessary for stopping decisions.Increasing refinement from 50 to 500 or 1000 steps overfits selected views.
Relations and correction: Agentic interfaces require relational evaluation and corrective operations, not merely language retrieval or mask updates.Useful operations include revising names, splitting or merging entities with provenance, invalidating observations, and propagating justified consequences.

12 Framework-Derived Research Directions

The framework-derived research agenda centers on preserving uncertainty, identity, hierarchy, relations, provenance, and correction capacity across changing carriers and evaluation settings. It calls for stress-tested benchmarks and complete cost accounting rather than single aggregate scores.

Uncertainty: Future carriers should represent joint uncertainty over geometry, support, teacher reliability, and competing identities rather than estimating one uncertainty in isolation.Existing systems separately measure feature variance, visibility-aware contribution, or uncertainty-modulated smoothing.
Dynamic identity: Dynamic maps need reversible merge histories, identity confidence, bounded-memory strategies, corrective queries, and controlled changes beyond final aggregate-map evaluation.Online association is a sequence of state transitions distinguishing reappearance, motion, appearance change, and earlier association error.
Generalizable carriers: Generalization should be reported separately for scenes, geometry, vocabulary, and tasks because per-scene fields and feed-forward models trade detail, cost, and unusual-geometry coverage.Feed-forward models amortize cost but may collapse unusual geometry or vocabulary, while specialization can preserve detail with closed output vocabularies.
Overlap and relations: Carriers for overlapping affordances, parts, groups, and relations need structured representations without quadratic construction or querying.Promising directions include sparse factor graphs, reversible cluster trees, external semantic indices, and query-dependent materialization.
Evaluation and cost: Benchmarks should separate rendered masks, native surface semantics, point labels, Gaussian assignments, proposal AP, and downstream navigation.They should also report first-query and repeated-query latency, with runtime accounting for reconstruction, preprocessing, association, reconciliation, storage, and querying.
Auditing and hybrid carriers: A unified correspondence audit should hold image evidence and geometry fixed while comparing projection and learned association mechanisms using support, occlusion, boundary, and pose/depth sensitivity metrics.Hybrid carriers could answer automatically from amortized memory while requesting image-space correction where uncertainty warrants interaction.
Design principle: The recurring design principle is to preserve revisable structure so queries, abstention, and local correction remain possible under realistic cost budgets.The empirical findings are scoped to the audited operators, carrier contracts, evaluation domains, and cost components rather than universal laws.

13 Conclusion

LAF provides a representation-neutral audit of 2D-to-3D transfer by tracing decisions, persistent carrier contracts, and complete evaluation costs across systems. Its audit rules emphasize separable correspondence and identity, revision-preserving state, and evidence-based validation limits.

Framework and audit: LAF traces generated evidence, geometric support, identity and conflict decisions, fusion, persistence, and querying through an explicit carrier contract.The contract covers support, semantics, identity, uncertainty, provenance, and operations.
Framework and audit: 161 systems across six carrier families showed that different numerical operators and persistent representations can implement the same five decisions.The audit included point, field, Gaussian, object, graph, and temporal-memory carriers, while representation, temporal, relational, and feed-forward methods retained the same decision structure.
Operational rules: Correspondence and identity must be reported separately because locating evidence on a surface does not determine which observations form an entity.The audit distinguishes association from identity and requires accuracy to be interpreted within a compatible output contract.
Operational rules: Evaluation must include the full cost ledger, while uncertainty and provenance should persist until correction is no longer needed.The ledger includes geometry, 2D preprocessing, association, reconciliation, carrier construction, storage, and first and repeated queries; irreversible errors can cap downstream correction.
Limitations and implications: Validation establishes analytical coverage and saturation on a frozen corpus but lacks prospective inter-annotator reliability and a controlled correspondence benchmark.These evidence-based limitations define the framework’s next tests and motivate evaluating when systems commit, what evidence they retain, which revisions they support, and their complete cost.