PaperlayerView exact revision on arXiv

Source-linked AI summary

GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

Qirui Wu, Stan Birchfield, Hesam Rabeti, Angel X. Chang, Bowen Wen

arXiv:2610.11215v1cs.CVcs.RO

TL;DR

Casual images leave complete object geometry and scene-relative pose to be inferred from sparse, uncertain, and occluded observations. GATOR combines multimodal, text-guided generation with refinement initialized from the generated asset. Across synthetic and real-world settings, it reports strong reconstruction quality and scene-relative pose recovery, with asset-initialized refinement benefiting efficiency and stability.

  • Problem

    Casual images may be sparse, irregular, cluttered, and partly occluded, making complete geometry and scene-relative pose reconstruction a challenge.

    Source: §1 · p. 1

  • Method

    GATOR fuses patch-aligned RGB, pointmap, and target-mask evidence before cross-view reasoning, adds stage-specific text guidance, and refines the generated asset through an observation-guided agent loop.

    Source: §1 · p. 2

  • Results

    GATOR shows strong reconstruction quality across synthetic and real-world settings; on Toys4K, it leads all seven metrics, reducing CD by 19.7% and raising F1 from 0.701 to 0.757 relative to Pixal3D-MV.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Time-budget comparisons and scene-level simulation demonstrate benefits of asset-initialized refinement for efficiency and stability.

    Source: §1 · p. 2, §5 · p. 9

    GATOR relies on target masks and camera and depth estimates; missing segmentation or catastrophically wrong depth may still harm reconstruction, and agentic refinement adds up to ten minutes per object.

    Source: §D.7 · p. 31

Abstract

from arXiv · show

Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/

1 INTRODUCTION

Casual images provide sparse, irregular, and often occluded views of objects embedded in clutter, leaving complete geometry and scene-relative pose to be inferred. GATOR combines observation-grounded generation, semantic conditioning, and asset-initialized refinement; evaluations span synthetic and real-world settings.

1 INTRODUCTION

  • Casual captures can embed partly occluded targets in clutter and provide irregular viewpoints, variable lighting, scale, and visible surface coverage.Single-view methods must infer unobserved backsides from learned priors, while exploiting additional views requires adapting those formulations.Source: §1 · pp. 1–2
  • GATOR fuses patch-aligned RGB, pointmap, and target-mask features before cross-view reasoning, then projects image features for geometry and appearance generation.Pose-aligned training preserves scene-relative orientation, while scale and placement are estimated from observations and refined by differentiable rendering.Source: §1 · pp. 2–3
  • Category names and object captions condition structure, geometry, and appearance generation, complementing patch-level spatial evidence when views are sparse or occluded.Stage-specific text adapters guide category-consistent completion throughout the generative cascade.Source: §1 · pp. 2–3
  • An observation-guided edit-render-review loop refines the generated asset’s structure and texture, including text and logos, without retraining the generator.The scene-aligned asset supplies instance-specific geometry, appearance, and pose for targeted edits.Source: §1 · pp. 2–3
  • GATOR’s evaluations cover diverse casual-image conditions and show strong geometry, appearance, and scene-relative pose recovery, with time-budget comparisons and scene-level simulation demonstrating refinement benefits.The evaluation includes Toys4K and four real-world datasets spanning controlled objects, cluttered tabletops, and indoor scenes.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans feed-forward reconstruction, generative completion, and reconstruction in natural or cluttered scenes. GATOR combines scene-context-preserving, target-specific conditioning with a generative cascade and agentic refinement.

2 RELATED WORK

  • Feed-forward methods recover objects from single or sparse multiview images, while LEAP (Jiang et al., 2024), PF-LRM (Wang et al., 2024b), and FreeSplatter (Xu et al., 2025) address unposed inputs or camera recovery.Geometry estimators including VGGT (Wang et al., 2025) and VGGT-Ω (Wang et al., 2026a) supply camera and depth estimates for downstream reconstruction.Source: §2 · p. 3
  • Generative approaches range from novel-view diffusion to native 3D models; GATOR adopts TRELLIS.2’s (Xiang et al., 2026) structure-geometry-appearance cascade and couples patch-aligned RGB, target-mask, and pointmap evidence before cross-view attention.Prior approaches include Pixal3D (Li et al., 2026b), which projects image features onto 3D structure, and ReconViaGen, which integrates reconstruction features into generation.Source: §2 · p. 3
  • GATOR’s architecture pairs pose-aware generative reconstruction with an agentic edit-render-review loop initialized from a complete, scene-aligned asset.The agent compares original and candidate renders under matched cameras and lighting, retaining or reverting local edits.Source: Figure 2 · p. 4
  • Methods for wild captures address hidden surfaces, pose, and clutter; GATOR retains surrounding scene context while using target masks to reconstruct complete geometry, appearance, and scene-relative pose.Related approaches include O2-Recon, ROODI, and Amodal3R (Wu et al., 2025) for surfaces hidden by occlusion, and ShapeR (Siddiqui et al., 2026) for casual captures and sparse SLAM points.Source: §2 · p. 4

3 METHODOLOGY

GATOR reconstructs complete objects from casual images by combining shared-grid multimodal encoding, cross-view generation, and asset-initialized agentic refinement. Its pipeline preserves scene context and placement while using text and rendered feedback to guide completion and edits.

3.1 PROBLEM FORMULATION

  • The reconstruction input can include any number of casual images, target masks, camera intrinsics and poses, depth, and category text or captions; camera and depth inputs may be estimated from the images.Observation-derived anchor and isotropic scale normalize inputs without rotating the frame; generated points are mapped back to preserve scene-relative orientation, with translation and scale refined after generation.Source: §3.1 · pp. 4–5

3.2 MULTIMODAL OBSERVATION ENCODING

  • RGB, target-mask, and pointmap features share a patch grid, retaining scene context while mask tokens identify the target; attention mixes aligned modality tokens within each patch.The mixer weights cues according to patch content while retaining a token per modality for cross-view aggregation, encoding spatial correspondence directly.Source: §3.2 · p. 5

3.3 ALIGNING AND MIXING MODALITIES FOR STRUCTURE GENERATION

  • Residual attention mixes each patch’s RGB, mask, and pointmap tokens, after which cross-view attention and category-text conditioning guide generation of complete, pose-aware sparse structure.Joint patch cues let the model cross-check estimated depth against appearance and masks; global attention gives structure latents access to all views and context under limited coverage or occlusion.Source: §3.3 · p. 5

3.4 STRUCTURE AND TEXT CONDITIONED GEOMETRY AND APPEARANCE

  • At active sparse-structure locations, projected image features condition geometry and appearance flows, which generate shape and PBR material latents; differentiable rendering and point alignment refine translation and scale.The model averages DINOv3 and NAF-upsampled features from valid target-masked views; predicted sparse structure is used at inference.Source: §3.4 · p. 5
  • Text guides every generation stage, while full-crop context and mask tokens support structure completion and target-masked projections keep occluders and background out of geometry and appearance surface detail.Structure uses a short category prompt; geometry and appearance receive caption tokens, including longer descriptions.Source: §3.4 · p. 6

3.5 AGENT AS RECONSTRUCTION REFINER

  • An agent initialized with the generated mesh uses images, structural priors, and Blender access to make targeted structural or appearance repairs while preserving reliable regions and the asset’s coordinate frame and scale.It compares original and edited renders under matched cameras and lighting, reverting edits that degrade reconstruction and stopping when supported repairs are complete or evidence is insufficient.Source: §3.5 · p. 6

4 EXPERIMENTS

GATOR is evaluated on controlled objects, cluttered tabletops, and indoor scenes, with comparisons covering reconstruction quality, pose, refinement, and runtime. It leads the reported benchmarks, while ablations and time-budget tests examine component contributions and efficiency.

4.1 EXPERIMENTAL SETUP

  • Training uses synthetic scenes built from approximately 980,000 object assets; evaluation covers 525 Toys4K objects and real-world tabletop and indoor captures with varied camera and geometry inputs.Real-world evaluation includes sensor-derived inputs on LM-O and HB, and estimated cameras and depth on HANDAL and ScanNet++.Source: §4.1 · pp. 6–7

4.2 TOYS4K RESULTS

  • 19.7% lower CD and F1 rising from 0.701 to 0.757 versus Pixal3D-MV; GATOR leads all seven Toys4K metrics.All methods use four views; GATOR, Pixal3D, and GPT-6-Astra share Pi3X camera and depth estimates.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 REAL-WORLD RESULTS

  • GATOR leads all nine tabletop metrics among non-proprietary baselines, roughly halving Pixal3D’s CD across LM-O, HB, and HANDAL.Four views are provided per object; RecGen and SimFoundry use fewer views because of their input limits.Source: Table 2 · p. 7, §4.3 · p. 7
  • On HANDAL, GATOR reaches ADD-SB@0.1 of 96.36% and has the lowest mean ADD-SB on both tabletop and ScanNet++ benchmarks.These pose-aware measurements are made before object-to-reference alignment.Source: §4.3 · pp. 7–8
  • On ScanNet++, GATOR leads every metric, cuts CD 23% versus RecGen, raises F1 from 0.540 to 0.603 and IoU from 0.682 to 0.733, and beats GPT-6-Astra on every metric.The evaluation expands to 229 targets across 49 scenes, each with one to ten views; GATOR reaches ADD-SB@0.1 of 95.63%.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 FRAMEWORK ANALYSIS

  • Without refinement, GATOR takes about 0.5 min and already outperforms GPT-6-Astra with a 20-minute budget; GPT-6-Astra returns valid assets for 0/10 objects at one minute.Refinement mainly improves appearance and local topology; GPT-6-Astra returns 5/10 valid assets at two minutes and all ten from five minutes onward.Source: §4.4 · pp. 8–9, Figure 6 · p. 9
  • On 55 HANDAL objects, the modality mixer lowers ADD-SB, text conditioning improves all nine metrics, and agentic refinement improves CD, NC, F1, PSNR, and LPIPS.Refinement edits are local; average geometry changes little on Toys4K, LM-O, and HB, and some HANDAL geometry metrics worsen slightly, although simulation benefits are reported.Source: §4.4 · p. 8, Table 4 · p. 9, Figure 5 · p. 9
  • On LM-O and HB, adding views from two to eight lowers ADD-SB and CD; with two views, GATOR’s CD is about half of Pixal3D’s with eight.The model supports varying numbers of input views.Source: §4.4 · p. 9

5 CONCLUSION

The conclusion summarizes GATOR’s approach to reconstructing complete, scene-aligned 3D objects from casual images and reports reconstruction quality and efficiency across casual capture settings. The accompanying passages also point to supplementary material but provide no further conclusion findings.

5 CONCLUSION

  • GATOR reconstructs complete, scene-aligned 3D object assets from casual images, including observations that are sparse or occluded.The framework recovers scene-relative pose as part of the reconstruction.Source: §5 · p. 9
  • Its local modality mixer combines patch-aligned RGB, pointmap, and target-mask evidence before cross-view reasoning to preserve context while identifying the target.Source: §5 · p. 9
  • Text-guided semantic conditioning supplies category priors across structure, geometry, and appearance generation to guide completion from sparse or occluded observations.Source: §5 · p. 9
  • The generated asset anchors an agentic render-based review loop for targeted structural and texture edits using instance-specific geometry and pose, without generator retraining.Experiments across casually captured settings demonstrate the value of this asset-initialized refinement for efficiency and simulation.Source: §5 · p. 9

SUPPLEMENTARY MATERIAL

A IMPLEMENTATION AND TRAINING

This section describes GATOR’s implementation and training, including coordinate normalization, multimodal and projected conditioning, flow training, placement refinement, and bounded agent workflows for refining or reconstructing posed assets. It also details safeguards for agent edits, verification of geometry, texture, and pose, and publication of the inspected GLB with machine-readable receipts.

A.1 COORDINATES AND OBSERVATION ENCODING

  • Coordinates are normalized by an observation-derived center and isotropic scale without rotation; shared-patch RGB, mask, pointmap, correspondence, and text features condition structure generation.Metric dimensions require metrically scaled input depth, and the variable-view encoder masks padded views without learned view-index embeddings.Source: §A.1 · p. 18

A.2 FLOW OBJECTIVE

  • Structure training minimizes a flow-matching velocity loss, while geometry and appearance flows use the same objective on normalized shape and PBR latents.At inference, the learned velocity is integrated from t = 1 to t = 0 with classifier-free guidance.Source: §A.2 · p. 18

A.3 PROJECTED CONDITIONING AND PLACEMENT

  • Geometry and appearance flows condition on target-masked RGB features projected to active coordinates; placement refinement optimizes translation and isotropic scale while fixing orientation and local geometry.Training with perturbed or estimated cameras improves tolerance to residual misalignment after structure alignment and normalization are recomputed.Source: §A.3 · pp. 18–19

A.4 AGENT REFINEMENT

  • GPT-6-Astra (OpenAI, 2026) refines meshes in Blender using supplied images and geometric observations, preserving reliable regions and frame; candidates are reviewed under matched cameras and lighting.Main experiments allow ten minutes per object, with no fixed revision limit; all delivered assets are scored, including unchanged fallbacks.Source: §A.4 · p. 19
  • The refinement workflow prioritizes evidence-supported repairs, preserves reliable geometry and appearance, and retains an earlier reviewed candidate when a later edit regresses.The agent is instructed to repair defects rather than invent unsupported features, and can report unresolved work as needing review.Source: §A.4 · pp. 19–20
  • From scratch, the agent must build a complete textured object and recover its pose from supplied views and allowed geometric inputs, fitting in the fixed camera world and checking renders against observations.It must handle uncertain or occluded structure using evidence, rather than treating a point cloud, one-view shell, or arbitrary-pose asset as complete.Source: §A.4 · pp. 21–22
  • Checkpoint a valid candidate before risky revisions, retain the best inspected snapshot until a replacement passes review, and avoid sacrificing it for an unfinished edit.Save a complete packed work/candidate.blend early and use the supplied controlled GLB export helper; deterministic serialization/import and final publication are outside the work clock, but modeling, point processing, and pose fitting are not.Source: §A.4 · p. 22
  • Review the final exported GLB under every input camera, and declare reconstruction_complete only after geometry, texture, and pose checks all pass.If checks fail, deliver needs_review with up to three concrete limitations; pose verification is separate from import validity, low point-fit residuals, or an attractive orbit render.Source: §A.4 · p. 22
  • Publish the exact inspected GLB with compact machine-readable receipts, and make finish the last action.No report, gallery, lengthy final response, or unrelated testing is required.Source: §A.4 · p. 22

A.5 IMPLEMENTATION DETAILS

GATOR initializes its structure denoiser from TRELLIS.2 and uses geometry and appearance flows with 512-resolution codecs. Training uses one to four views, and inference uses 12 sampling steps per flow.

Model initialization and inputs

  • The structure denoiser starts from the 1.3B-parameter TRELLIS.2 structure model, while geometry and appearance flows use 512-resolution geometry and PBR codecs.Training uses one to four views, with 384-pixel context crops for structure and 512-pixel target-masked crops for geometry and appearance.Source: §A.5 · p. 22

Optimization

  • Training uses AdamW with weight decay 0.01, bfloat16 mixed precision, adaptive gradient clipping, and an effective batch size of 128.Learning rates are 10−4 for structure and 2 × 10−5 for geometry and appearance; classifier-free guidance conditions are dropped with probability 0.1.Source: §A.5 · p. 22

Sampling

  • Inference uses 12 sampling steps for each flow and a guidance scale of 3.Training timesteps are sampled from a logit-normal distribution for structure and a uniform distribution for geometry and appearance.Source: §A.5 · p. 22

B.1 SYNTHETIC TRAINING DATA

S3 turns a large asset pool into multiview reconstruction examples with clutter, occlusion, and complete object supervision. It varies scene composition, capture conditions, and observation errors while preserving posed object targets.

Corpus design

  • S3 separates asset coverage, scene composition, capture variation, and target construction to scale multiview examples while preserving correspondence between observations and posed objects.The pool contains approximately 980,000 assets, converted to a common representation and sampled for even coverage across sources.Source: §B.1 · p. 23

Cluttered scene composition

  • Scenes combine 8–16 assets with varied scales and rotations, creating relative-scale differences, contacts, partial occlusions, and varied orientations.Some scenes are settled with rigid-body simulation, while others use collision-checked free poses.Source: §B.1 · p. 23

Viewpoints and appearance

  • Orbiting and traversing camera paths vary distance, elevation, roll, and field of view, while randomized lighting and ground appearance diversify rendered views.Authored object materials are retained so rendered appearance matches cached PBR targets.Source: §B.1 · p. 23

Consistent geometric supervision

  • Each view stores RGB, camera parameters, object identities, depth, and visibility, while object-only and opacity-aware rendering preserve consistent masks and geometry supervision.Complete objects are voxelized after placement and cached as latents with placement transforms, including for heavily incomplete observations.Source: §B.1 · p. 23

Training pairs and capture errors

  • Training mixes clean and perturbed observations, image-estimated cameras and depth, and modality dropout while keeping object targets complete.Perturbations affect camera pose and intrinsics, depth, masks, and image quality; view selection retains a small tail of difficult occlusions.Source: §B.1 · p. 23

Scaling and integration

  • S3 is combined with 3D-FRONT, InternScenes, SceneSmith, and SAGE through a common observation-and-target interface, adding organized indoor layouts to procedural clutter.Each rendered scene supplies many object targets and view subsets, amortizing rendering across training examples.Source: §B.1 · p. 23

B.2 EVALUATION DATA

Evaluation includes selected Toys4K objects and real-world datasets with reference models or partial scans. Target selection, masks, and category text rely on annotations for the described real-world data.

Evaluation datasets

  • Toys4K evaluation selects five objects from each of 105 categories before methods are run, spanning within-category geometric complexity.LM-O and HB provide complete reference CAD models; HANDAL targets use four frames selected with visibility annotations and dataset masks, while ScanNet++ contributes 229 targets from 49 scenes with partial-scan references.Source: §B.2 · p. 23

Synthetic data context

  • S3 scene examples show paired viewpoints with variation in occlusion, object scale, background, and lighting while retaining complete object supervision.Table 5 reports synthetic structure-model training-corpus statistics: epoch items are scene–object instances, share is per-source fraction, and reuse is items per asset; assets are counted within each source, so totals are not deduplicated.Source: Figure 7 · p. 24, Table 5 · p. 24

Annotation dependence

  • Target selection, masks, and category text for the described real-world datasets rely on annotations.ScanNet++ masks are prompted by annotations, and HANDAL frame selection uses visibility annotations.Source: §B.2 · pp. 23–24

C EVALUATION METRICS

The evaluation measures pose and shape, aligned geometry, and rendered appearance across controlled objects, cluttered tabletops, and indoor scans. It reports dataset-specific comparisons, nested-view results, and qualitative reconstructions under shared inputs and evaluation conditions.

C.1 JOINT POSE AND SHAPE

  • ADD-SB evaluates posed placement without fitting; ADD-SB@0.1 is the percentage of objects below one tenth of the reference diameter.When cameras are estimated, a single scene-wide similarity transform from the cameras registers the reconstruction frame.Source: §C.1 · p. 25

C.2 ALIGNED GEOMETRY

  • For aligned geometry, ICP fits a similarity transform; F1 uses a 1% diameter threshold on Toys4K and 2% on real data.Silhouette IoU is averaged over matched cameras per object and then across objects.Source: §C.2 · p. 25

C.3 APPEARANCE

  • Appearance scores compare aligned renders under shared cameras and studio lighting, without exposure or color fitting; ScanNet++ is evaluated on geometry only.Reference CAD models use rough diffuse materials, so the scores measure agreement with rendered references rather than calibrated material recovery.Source: §C.3 · p. 25

D.1 TOYS4K PROTOCOL

  • Toys4K uses four casually rendered input views and six held-out views for evaluation; all methods receive the same images and masks.GATOR also receives the category name, while the qualitative comparisons show outputs before and after agentic refinement.Source: §D.1 · p. 25, Figure 8 · p. 26

D.2 TABLETOP PROTOCOL

  • The generative model alone outperforms every baseline on every geometry metric, while refinement improves PSNR and LPIPS on every tabletop dataset.Table 6 reports results separately for LM-O, HB, and HANDAL.Source: Table 6 · p. 27
  • Tabletop tests use RGB and target masks across LM-O, HB, and HANDAL, with depth and camera inputs varying by dataset; ShapeR produces untextured meshes only.LM-O and HB use sensor depth; HANDAL uses estimated cameras and depth, and GATOR receives dataset-specific text prompts.Source: §D.2 · p. 27, Figure 9 · p. 28

D.3 VIEW COUNTS ON LM-O AND HB

  • From two to eight views, ADD-SB falls from 0.0148 to 0.0134 and CD from 0.0109 to 0.0096, while ADD-SB@0.1 remains 98.03%.On pooled LM-O and HB, GATOR leads every joint and geometry metric at each view count.Source: §D.3 · p. 29, Table 7 · p. 29

D.4 SCANNET++ PROTOCOL

  • The ScanNet++ benchmark adds 160 targets from 43 scenes to a prior 69-object, six-scene protocol and evaluates completion against partial scans.Alignment uses observed surfaces, while symmetric distances penalize completed surfaces absent from the scan.Source: §D.4 · p. 29

D.5 ABLATION PROTOCOL

The ablation and simulation evaluations examine agentic refinement’s local edits and their effects on reconstructed geometry and physical stability.

D.5 ABLATION PROTOCOL

  • Refinement adds or repairs local details, removes fused geometry, and corrects textures, while changing average geometry little on most benchmarks.HANDAL is an exception: CD, F1, and IoU worsen slightly; the ablation models are trained separately, so their scores are not directly comparable to Tables 2 and 6.Source: §D.5 · p. 29

D.6 SCENE-LEVEL RIGID-BODY SIMULATION

  • In simulation, three generated chairs lean or topple by 40–79°, while their refined counterparts remain within 1.3° of upright.The 15 reconstructed ScanNet++ meeting-room objects are simulated with shared placements, mass, physics settings, and convex collision approximations.Source: §D.6 · p. 31
  • The agent straightens the conference table’s uneven, slanted legs before simulation.The generated table is described as standing on uneven, slanted legs.Source: §D.6 · p. 31

D.7 LIMITATIONS AND FUTURE WORK

GATOR depends on target masks and camera and depth estimates, and the authors identify gross input-estimation failures as a remaining risk.

D.7 LIMITATIONS AND FUTURE WORK

  • Missing segmentation or catastrophically wrong depth may harm reconstruction, and agentic refinement adds up to ten minutes per object.The authors suggest tighter integration with segmentation and geometry estimation, open agents, and automatic geometric verification as future directions.Source: §D.7 · p. 31

Related papers

Loading 2610.11215v1…