PaperlayerView exact revision on arXiv

Source-linked AI summary

Matisse: Evidence-Space Reasoning for Active 3D Reconstruction

Xihang Yu, Kaichen Zhou, Lorenzo Shaikewitz, Clément Jambon, Xiao Zhan, Rajat Talak, Luca Carlone

arXiv:2609.38746v1cs.CVcs.RO

TL;DR

Active reconstruction methods often estimate uncertainty over observed or instantiated geometry, limiting their account of unseen structure, while long-horizon systems must choose which observations to retain. Matisse uses evidence from a pretrained generative 3D model to guide view acquisition and keyframe selection, reducing Chamfer distance across three datasets and using fewer views for long-horizon reconstruction.

  • Problem

    Active reconstruction uncertainty over observed or instantiated geometry can incompletely represent priors about unseen space, while long-horizon reconstruction must retain useful observations with bounded memory and computation.

    Source: §1 · p. 1

  • Method

    Matisse derives uncertainty and information gain from a pretrained generative 3D model’s evidence space to guide active view selection and long-horizon keyframe selection.

    Source: §1 · p. 2

  • Results

    Matisse reduces Chamfer distance versus the best prior active-reconstruction baselines by 12.7%, 3.8%, and 9.2% on GSO30, YCB-V, and Replica, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Experiments show improved reconstruction quality and computational efficiency across object- and scene-level settings; long-horizon generation maintains comparable quality using one-seventh of the input views.

    Source: §7 · p. 9

    Reconstruction quality and resulting decisions depend on the generator prior and Stream3D’s ability to fuse observations accurately; implausible geometry or inconsistent fusion can propagate into those decisions.

    Source: F LIMITATIONS · p. 20

Abstract

from arXiv · show

How can a 3D reconstruction system acquire and retain useful information to understand the geometry of a scene from partial views under a limited computation budget? Existing active view acquisition methods typically estimate uncertainty over observed or instantiated geometry, limiting their ability to reason about unseen structure, while long-horizon reconstruction methods often retain redundant observations. We introduce Matisse, a training-free framework that unifies active reconstruction and keyframe selection by leveraging evidence provided by a pretrained generative 3D model. Matisse estimates Evidential Uncertainty from cross-attention evidence associated with 3D latent tokens and derives an Evidential Information Gain to guide both view acquisition and keyframe selection based on the expected reduction in posterior entropy. Matisse supports multi-object scenes through occlusion-aware, object-balanced aggregation and propagates uncertainty through intermediate latents to avoid full reconstruction during planning. Matisse reduces Chamfer distance by 12.7%, 3.8%, and 9.2% on GSO30, YCB-V, and Replica, respectively, relative to the best baseline on each dataset, and achieves a $1.50\times$ end-to-end speedup over the best active reconstruction baseline on GSO30 with the same reconstruction backend. In the GSO30 keyframe selection experiment for long-horizon reconstruction, Matisse achieves comparable Chamfer distance using 14% of the input views compared with Stream3D.

1 INTRODUCTION

Matisse frames active view acquisition and keyframe retention as shared information-selection problems. It uses evidence from a pretrained generative 3D model to estimate uncertainty and guide both decisions.

Motivation

  • Active reconstruction selects the next view, while long-horizon reconstruction selects observations to retain; both assess information beyond existing evidence under a limited computation budget.Uncertainty over observed or instantiated geometry can incompletely represent priors about unseen space, while retaining views also raises memory and per-update computation concerns.Source: §1 · p. 1
  • Stream3D’s Adaptive Evidential Memory aggregates observation-to-token evidence and retains high-evidence frame indices per token, bounding retained evidence while processing incoming frames.Selecting keyframes remains an opportunity to improve efficiency.Source: §1 · p. 2

Approach

  • Matisse is a training-free framework that represents scenes in a pretrained generative 3D model’s evidence space and derives uncertainty and information gain for view acquisition and keyframe selection.It unifies scene representation, uncertainty, and information gain for deciding which observations to acquire and retain.Source: §1 · p. 2
  • Evidential Uncertainty derives from posterior covariance when Adaptive Evidential Memory is formulated as maximum a posteriori estimation over 3D latent tokens.Because the tokens represent observed and generatively completed geometry, uncertainty can extend beyond currently observed surfaces.Source: §1 · p. 2
  • Evidential Information Gain measures expected uncertainty reduction from additional evidence and evaluates it in evidence space, incorporating generative scene hypotheses for observed and occluded regions.This supports both acquiring and retaining views.Source: §1 · pp. 2–3
  • Evidence-space view scoring avoids expensive mesh generation, and batched GPU evaluation reduces amortized scoring time to 1.26 ms per candidate.Visibility-aware, object-balanced aggregation accounts for occlusion and balances individual objects in multi-object scenes.Source: §1 · p. 3

Results

  • Matisse reduces Chamfer distance versus the best prior active-reconstruction baselines by 12.7%, 3.8%, and 9.2% on GSO30, YCB-V, and Replica, and runs 1.50× faster on GSO30.For long-horizon keyframe selection, it preserves comparable generation quality using 14% as many input views as Stream3D.Source: §1 · p. 3

2 RELATED WORK

Prior active-reconstruction work uses several scene representations, while uncertainty and information-gain approaches remain tied to instantiated geometry. Matisse instead uses generative scene hypotheses to reason about unseen regions.

Scene Representations for Active Reconstruction

  • Active reconstruction uses geometric landmarks and occupancy maps, neural radiance fields, or Gaussian splatting; learned completion methods also predict unobserved geometry.Matisse instead uses a generative 3D foundation model as its reconstruction backbone and maintains an implicit 3D evidence memory.Source: §2 · p. 3

Uncertainty and Information Gain

  • Existing uncertainty measures include predictive variance, rendering coverage, transmittance, and geometric support, but the most uncertain region need not yield the greatest information gain.FisherRF and GauSS-MI remain tied to instantiated scene representations and do not explicitly represent uncertainty or priors over unobserved scene portions.Source: §2 · p. 3
  • Matisse derives uncertainty and information gain from plausible diffusion-model scene completions, reasoning in global scene-hypothesis space rather than only around current model parameters.Its learned geometric completion prior covers occluded and unobserved regions while allowing uncertainty propagation as observations arrive.Source: §2 · p. 3

3 PROBLEM SETUP

The problem setup considers how a mobile agent can recover scene geometry from partial views under a limited computation budget. It must choose where to observe next and which collected observations to retain.

Problem Setup

  • The system’s coupled decisions are where to observe next and which observations to retain for reconstruction.The agent acquires RGB-D observations from camera views and poses, then selects a feasible next viewpoint based on its current scene-geometry estimate.Source: §3 · p. 4
  • Keyframe selection chooses a subset of collected observations to retain for reconstruction.The setup frames acquisition and retention as decisions made over successive time steps.Source: §3 · pp. 3–4

4 PRELIMINARY

Matisse builds on a pretrained 3D generator and Adaptive Evidential Memory, which associates evidence from views with spatially aligned canonical tokens. The selected views’ weighted features are fused for sparse-structure prediction.

3D Generative Model

  • The pretrained 3D generator uses a sparse-structure stage to predict coarse occupancy, followed by a structured-latent stage modeling geometry and appearance.Both stages represent the object with spatially aligned 3D tokens in a canonical coordinate frame.Source: §4 · p. 4

Adaptive Evidential Memory

  • Adaptive Evidential Memory assigns each view an evidence weight per canonical token, measuring how strongly its image features support that spatial location.Low weight can indicate that the region is occluded or outside the view’s field of view.Source: §4 · p. 4

Sparse-Structure Fusion

  • AEM retains up to D high-evidence view indices per token; token votes are aggregated to select K views for reconstruction.Selected views contribute latent features combined using normalized evidence weights; the sparse-structure generator denoises the fused representation and decodes an occupancy prediction.Source: §4 · p. 4

5 METHOD

Matisse derives uncertainty and information gain in evidence space to guide view selection for both active reconstruction and long-horizon keyframe selection. Its system combines multi-object aggregation, efficient candidate scoring, geometric exploration, and occupancy-stability stopping.

5 METHOD

  • Matisse derives evidential uncertainty and information gain from Stream3D’s canonical evidence memory to guide view selection.The framework applies this selection approach to active reconstruction and keyframe selection.Source: §5 · p. 4

5.1 EVIDENTIAL UNCERTAINTY

  • Each view’s support weight is treated as evidence about a canonical token, yielding posterior-covariance uncertainty normalized by its prior value.In the evidence-dominated regime, the fused feature representation approximately equals the MAP estimate.Source: §5.1 · pp. 4–5

5.2 EVIDENTIAL INFORMATION GAIN

  • For occupied tokens, Evidential Information Gain measures expected posterior-entropy reduction from a candidate view, with a visibility-based approximation under stated assumptions.The approximation assumes conditionally independent token latents, small candidate evidence, and binary visibility; visibility is computed by projection and a depth-buffer test.Source: §5.2 · p. 5

5.3 EVIDENCE-SPACE VIEW SELECTION

  • Multi-object scoring normalizes each object’s visible information gain and averages across objects, preventing denser token grids from dominating view selection.The score is computed directly from evidence memory and decreases as evidence accumulates over a fixed set of visible tokens.Source: §5.3 · p. 6
  • Candidate scoring uses sparse-structure decoding and online uncertainty updates, avoiding the SLAT stage; batched GPU calculations take 1.26 ms per candidate on average.The system combines multi-object aggregation, efficient scoring, geometric exploration, and stopping when reconstruction stabilizes.Source: §5.3 · p. 6
  • Acquisition stops when sparse-structure occupancy stabilizes or the view budget is reached; converged objects are frozen while active objects continue.The stability test compares occupied positions across consecutive reconstruction steps and is applied only when a previous reconstruction exists.Source: §5.3 · p. 6
  • An exploration factor downweights candidates near acquired poses, while receding-horizon planning can account for future rewards.The exploration factor approaches one for more distant candidates.Source: §5.3 · p. 6
  • For keyframe selection, unselected frame poses are scored as candidates, and frames are retained until occupancy stabilizes or the view budget is reached.After each selected frame, the evidence memory and uncertainty are updated before the remaining frames are rescored.Source: §5.3 · p. 6

6 EXPERIMENTS

Experiments evaluate Matisse on GSO30, YCB-V, and Replica, spanning single-object, multi-object, and cluttered indoor reconstruction. They report reconstruction quality, runtime, keyframe selection, and analyses of the method’s components and uncertainty.

6.1 EXPERIMENTAL SETUP

  • The experiments cover GSO30 single-object scenes, YCB-V multi-object scenes, and selected objects in cluttered Replica rooms, assessing geometry, appearance, and runtime.All methods start each trial from the same initial observation.Source: §6 · p. 6, §6.1 · p. 7

6.2 SINGLE-OBJECT RECONSTRUCTION

  • On GSO30, Matisse-G leads all six reconstruction metrics, reduces CD by 12.7% versus the strongest baselines, and is 1.50× faster than the fastest prior active-view planner.Figure 2 shows more faithful geometry and appearance reconstruction across four GSO30 objects.Source: §6.2 · p. 8
  • At comparable Chamfer distance on GSO30, Matisse-G converges after 7 selected views, compared with 48 sequential views for Stream3D.Figure 3 compares their convergence trends using exponential fits.Source: Figure 3 · p. 8, §6.2 · p. 8

6.3 MULTI-OBJECT RECONSTRUCTION

  • On Replica, Matisse-G achieves the best results across all six reconstruction metrics, reducing CD by approximately 9.2% and P-FID by 5.2%.It also improves PSNR by 0.39 dB compared with the strongest baseline for that metric.Source: §6.3 · p. 8
  • On YCB-V, Matisse-G achieves the best CD, IoU, SSIM, and LPIPS, while Matisse-RH achieves the best P-FID and PSNR.Both variants outperform active view-selection baselines with 3DGS backends across all six reconstruction metrics.Source: §6.3 · p. 8, Table 2 · p. 9

6.4 ABLATIONS AND ANALYSIS

  • On YCB-V, removing information gain causes the largest overall degradation; the full Matisse-G and Matisse-RH variants lead on different metrics.Without the explorative factor, the model remains competitive and achieves the best P-FID.Source: Table 3 · p. 9, §6.4 · p. 9
  • Among tested pseudo-evidence settings, α = 1 provides the best overall performance and is used in the experiments.The sensitivity analysis evaluates four logarithmically spaced values under the GSO30 protocol.Source: §6.4 · p. 9
  • The uncertainty analysis examines predicted uncertainty in relation to visibility, object symmetry, and model confidence as visual detail is lost.The corresponding analyses are provided in Appendix D.Source: §6.4 · p. 9

7 CONCLUSION

Matisse unifies active reconstruction from sparse views and keyframe selection for long-horizon reconstruction using a generative model. Experiments show improved reconstruction quality and computational efficiency across object-level and scene-level settings, while long-horizon generation retains comparable quality with one-seventh of the input views.

7 CONCLUSION

  • Matisse uses a generative model to unify active reconstruction from sparse views with long-horizon keyframe selection.It formulates Adaptive Evidential Memory as MAP estimation, derives Evidential Uncertainty from posterior covariance, and uses Evidential Information Gain to guide view acquisition and retention.Source: §7 · p. 9
  • Experiments show improved reconstruction quality and computational efficiency across object-level and scene-level settings, with comparable long-horizon generation quality using one-seventh of the input views.The paper attributes the shared basis for acquisition and retention to evidence-space reasoning about each view’s information contribution conditioned on current 3D memory.Source: §7 · p. 9

A PROOFS

The appendix derives the MAP estimate and evidential information gain, describes receding-horizon planning and implementation settings, and reports additional experiments and sensitivity analyses.

A.1 PROOF OF THEOREM 1 (MAP ESTIMATE)

  • Stream3D evidence increases posterior precision additively, and the stationary point is the unique MAP solution.The proof establishes uniqueness because the Hessian is positive definite when α > 0.Source: §A.1 · p. 14

A.2 PROOF OF THEOREM 3 (EVIDENTIAL INFORMATION GAIN)

  • Under conditional independence, a view’s information gain sums token-wise entropy reductions; its visibility-based approximation preserves diminishing returns for well-supported tokens.The approximation substitutes geometric visibility for unknown future evidence association.Source: §A.2 · p. 14

B RECEDING-HORIZON PLANNING

  • Receding-horizon planning scores feasible view sequences by discounted cumulative gain, executes the best branch’s first view, then updates evidence and replans.Hypothetical views do not update the evidence field; only executed observations do.Source: B RECEDING-HORIZON PLANNING · pp. 14–15

C.1 QUANTITATIVE RESULTS ON REPLICA

  • Appendix C.1 reports scene-level Replica results in Table 4.The supplied table passage identifies its scope but gives no metric values or comparisons.Source: Table 4 · p. 15

C.2 DATASET-SPECIFIC IMPLEMENTATION SETTINGS

  • YCB-V candidate views use 30 equal-area-sampled camera positions over a spherical cap; candidate poses point toward the object or scene center.Cartesian camera positions lie on a regular 3D grid within a specified cube.Source: §C.2 · p. 15

C.3 REAL-WORLD EXPERIMENT

  • The real-world robot demonstration captures 19 frames to reconstruct 11 objects, grouping objects sequentially when centers lie within 0.5 m of every group member.The system uses scene-level aggregation because one frame may provide evidence for multiple objects.Source: §C.3 · p. 15

C.4 SENSITIVITY TO PRIOR PSEUDO-EVIDENCE

  • Across four tested α values, performance varies modestly; α = 1 gives the best overall results and is used as the practical default.Table 6 marks the best and second-best settings and highlights the selected α = 1 row.Source: §C.4 · p. 16, Table 6 · p. 16

C.5 VIEW-BUDGET COMPARISON ON YCB-V

  • Figure 4 contrasts Matisse’s 5-frame result with MAGICIAN’s measured Chamfer distance across 5–50 acquired frames on YCB-V scene 48.The plot uses a broken vertical axis and lower Chamfer distance is better; the fit is descriptive.Source: Figure 4 · p. 16

C.6 ADDITIONAL KEYFRAME-SELECTION EXPERIMENT

The additional YCB-V keyframe experiment compares Matisse-G with Stream3D on fixed trajectories, while qualitative figures show reconstructions against ground truth on YCB-V and Replica.

C.6 ADDITIONAL KEYFRAME-SELECTION EXPERIMENT

  • Using 4 views rather than 24, Matisse-G achieves lower mean CD on nine of twelve YCB-V scenes and 6.760 mm versus Stream3D’s 6.994 mm across 55 instances.Both methods start from the first trajectory frame; Table 7 reports scene-wise CD in millimeters and the mean over all instances.Source: §C.6 · p. 16, Table 7 · p. 16

C.7 QUALITATIVE RESULTS ON YCB-V

  • Figure 5 compares ground truth and six reconstruction methods on two YCB-V scenes, with all methods using the Stream3D backend.Rows show scene 50 from the fifth camera and scene 59 from the third; meshes are registered and selected object orientations are corrected for visualization.Source: Figure 5 · p. 17, §C.7 · p. 17

C.8 QUALITATIVE RESULTS ON REPLICA

  • Figure 6 compares registered meshes of selected objects in two Replica office scenes from common viewpoints, against ground truth and six reconstruction methods.The scenes are office_3 and office_4, and all reconstruction methods use the Stream3D backend.Source: Figure 6 · p. 17, §C.8 · p. 17

D ANALYSIS OF PREDICTED UNCERTAINTY

The analysis examines how predicted uncertainty relates to visibility, rotational symmetry, and image detail, alongside related work on uncertainty and information gain for reconstruction.

D ANALYSIS OF PREDICTED UNCERTAINTY

E EXTENDED RELATED WORK

  • A generative 3D foundation model supplies a learned completion prior for occluded and unobserved regions, while evidence memory supports efficient uncertainty propagation as observations arrive.The paper contrasts this with converting predicted unseen-space occupancy into Gaussian primitives for uncertainty rendering.Source: E EXTENDED RELATED WORK · p. 18
  • FisherRF and GauSS-MI remain tied to instantiated scene representations and do not explicitly represent distributions over future geometries.FisherRF uses a local second-order approximation; GauSS-MI models reliability of existing Gaussian primitives.Source: E EXTENDED RELATED WORK · p. 19
  • Matisse derives uncertainty and information gain from plausible diffusion-model scene completions, accounting for structural predictability and model confidence beyond local geometry around current parameters.The paper contrasts this global scene-hypothesis-space formulation with approaches tied to instantiated representations.Source: E EXTENDED RELATED WORK · p. 19
  • Unlike uncertainty methods requiring uncertainty-specific training or repeated image-space evaluations, Matisse fuses training-free uncertainty across views in a shared 3D latent representation.The paper describes this as training-free, multi-view uncertainty estimation for a foundational 3D generative model.Source: E EXTENDED RELATED WORK · pp. 19–20

F LIMITATIONS

Matisse’s reconstruction and selection quality depends on its pretrained generator and Stream3D’s observation association and fusion.

F LIMITATIONS

  • Matisse depends on the pretrained 3D generator and Stream3D fusion; implausible geometry or inconsistent fusion can propagate into uncertainty and subsequent selection decisions.The stated boundary is tied to the generator prior and Stream3D’s ability to associate and fuse observations accurately.Source: F LIMITATIONS · p. 20

Related papers

Loading 2609.38746v1…