PaperlayerView exact revision on arXiv

Source-linked AI summary

PocketSplat: Mobile Gaussian Reconstruction via World-Space Latent Allocatio

Wenzhi Guo, Xianda Chen, Dongxuan Chen, Guangchi Fang, Bing Wang

arXiv:2610.03192v1cs.CV

TL;DR

Mobile Gaussian reconstruction needs both device-feasible execution and assets sized for downstream use, while dense image-aligned predictors tie Gaussian population to input resolution and view count. PocketSplat allocates an exact budget across world-space latent cells and decodes retained candidates; its allocation improves PSNR over count-matched random selection and remains useful under changed scene content and capture conditions.

  • Problem

    Dense image-aligned Gaussian prediction commonly ties final representation size to image resolution and input views, despite mobile reconstruction needing assets sized for downstream use.

    Source: §1 · pp. 1–2

  • Method

    PocketSplat groups geometry-aware candidates in predicted world-space latent cells, allocates exact integer capacity, fuses cell evidence, and decodes complete Gaussian attributes only for retained candidates.

    Source: §1 · p. 2

  • Results

    World-space allocation improves PSNR over count-matched random selection at 113K, 226K, and 338K, and remains useful when scene content and capture conditions differ from training.

    Source: §4.2 · p. 7, §4.4 · p. 8

  • Takeaways & Limitations

    An explicit output budget exposes controllable Gaussian asset cardinality for downstream mobile use.

    Source: §1 · p. 2

Abstract

from arXiv · show

Mobile Gaussian reconstruction must satisfy two requirements: the reconstruction model must execute within a device resource envelope, and the resulting Gaussian asset must expose a representation size suited to downstream mobile use. Existing feed-forward Gaussian reconstructors commonly decode dense, image-aligned candidates whose final cardinality is implicitly determined by the input resolution and number of views. We present PocketSplat, a feed-forward framework for budgeted mobile Gaussian asset construction. Given a prescribed output budget, PocketSplat organizes dense geometry-aware latent candidates in predicted world space, allocates exact integer capacity across local latent cells, and decodes complete Gaussian attributes only for retained candidates. Cell-conditioned latent fusion aggregates repeated multi-view evidence before decoding, while spatial responsibility decoding adapts Gaussian support after local sparsification. Experiments on DL3DV and out-of-distribution benchmarks establish a strong quality--budget trade-off against feed-forward Gaussian reconstruction baselines. On Mip-NeRF 360, PocketSplat executes directly on a target iPhone and constructs compact, higher-quality Gaussian assets substantially faster than a deployable streamed MVSplat variant; native MVSplat and DepthSplat exceed the device memory budget.

1 INTRODUCTION

Mobile Gaussian reconstruction must fit both device execution limits and downstream asset-size needs. PocketSplat makes representation capacity explicit by allocating an exact Gaussian budget across predicted world-space latent cells before decoding retained candidates.

1 INTRODUCTION

  • Mobile reconstruction must fit device resources while producing an asset usable after construction, because asset cardinality affects storage, transmission, rasterization, and interactive processing.An explicit budget lets an application select a representation-size operating point instead of inheriting a size fixed by reconstruction inputs.Source: §1 · pp. 1–2
  • Dense image-aligned predictors commonly tie Gaussian population to image resolution and view count, while post-hoc pruning leaves budget allocation secondary.Under a prescribed budget, candidates must be associated across views, fuse repeated observations, and adapt support as local density falls.Source: §1 · p. 2
  • PocketSplat constructs a feed-forward Gaussian asset at an explicit output budget, separating delivered cardinality from the dense candidate field used for geometric reasoning.The allocator assigns exact integer capacity across world-space cells before complete Gaussian attributes are decoded for retained candidates.Source: §1 · p. 2
  • The proposed allocator combines local detail, cross-view redundancy weighting, zero-detail support, and capacity completion, followed by hidden fusion and scale-only spatial responsibility decoding.These mechanisms allocate capacity before attribute decoding and adapt retained primitives to local sparsification.Source: §1 · p. 2

2 RELATED WORK

Related work progresses from optimized Gaussian representations and feed-forward scene reconstruction to sparse or controllable-count methods. Geometry priors and mobile-efficiency techniques provide context for PocketSplat’s exact allocation over world-space latent candidates.

2 RELATED WORK

  • Novel-view synthesis moved from neural radiance fields and grid or hash encodings to 3D Gaussian Splatting for real-time rasterization after scene-specific optimization (Mildenhall et al., 2021; Müller et al., 2022; Kerbl et al., 2023).Other work improves Gaussian fidelity or compacts optimized scenes through pruning, quantization, coding, and constrained optimization.Source: §2 · p. 3
  • Generalizable reconstruction predicts renderable Gaussians without per-scene optimization, while sparse and controllable-count methods adapt primitive placement, alignment, anchors, or density (Charatan et al., 2024; Chen et al., 2024c; Zhang et al., 2026b; Park et al., 2025; Kim et al., 2026).PocketSplat complements these approaches by allocating an exact output population over grouped world-space latent candidates before attribute decoding.Source: §2 · p. 3
  • Geometry-prior and efficiency research spans depth, pointmap, matching, and camera priors, plus mobile rasterization, optimized Gaussian rendering, pruning, and localization.PocketSplat uses frozen geometry priors for budgeted construction and directly controls downstream representation cardinality for device-specific needs.Source: §2 · p. 3

3 METHOD

PocketSplat turns multi-view geometric evidence into a Gaussian asset with a prescribed cardinality. It groups dense latent candidates in world space, allocates an exact budget, and decodes and adapts only selected candidates.

3 METHOD

  • The method maps multi-view images and a feasible budget B to exactly B world-space Gaussians, each represented by a mean, covariance, degree-2 RGB spherical-harmonic coefficients, and opacity logit.The latent field consolidates dense multi-view geometric evidence before Gaussian attributes are materialized.Source: §3 · p. 3

3.1 WORLD-SPACE LATENT FIELD

  • Candidates lifted to predicted world space are grouped into cells, allowing observations at distant image coordinates but in a common local region to share representation decisions.Cell width is set relative to median finite positive multi-view depth; the dense field supports every requested budget.Source: §3.1 · pp. 3–4
  • Cell capacity reflects local 7 × 7 multi-view RGB neighborhood entropy, discounts detail repeated across views, and retains a representative for occupied zero-detail cells.The support floor protects zero-detail cells, while view-mean weighting assigns capacity according to scene content rather than repeated image-plane evidence alone.Source: §3.1 · p. 4

3.2 BUDGET-CONDITIONED GAUSSIAN ASSET DECODING

  • For feasible budgets, active-set water filling and capacity completion distribute slots, then corrected stochastic rounding produces integer quotas and the exact requested asset cardinality.At the dense operating point, all candidates are selected; otherwise reproducible keys select distinct candidates within each cell.Source: §3.2 · p. 5
  • Before decoding, reliability- and support-weighted cell fusion updates retained candidate latents; complete Gaussian attributes are projected only for selected candidates.Spatial responsibility decoding adapts covariance after selection, increasing each retained Gaussian’s local area responsibility as its cell becomes sparser.Source: §3.2 · p. 5

3.3 TRAINING

  • Training uses 50,000 DL3DV steps with a frozen geometry backbone and optimizes the GS head, allocation-related modules, fusion module, and SRD head.Training begins with the Full budget for 10,000 steps, then cycles through descending budgets with 100-step holds.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate exact-budget reconstruction quality, transfer beyond DL3DV, direct mobile deployment, and the contributions of key construction components. Results cover compactness, fidelity, and execution under a target device’s resource limits.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses four context images per scene, fixed input preprocessing, several benchmark datasets, and a shared FP32 hybrid execution path for mobile results.DL3DV uses 50 evaluation frames per scene; Tanks & Temples and MegaDepth use frozen checkpoints, while Mip-NeRF 360 is evaluated on-device.Source: §4.1 · p. 6, §4 · p. 6, §4.3 · p. 7

4.2 RECONSTRUCTION QUALITY ACROSS DATASETS

  • At 226K, PocketSplat uses 16.4% of pixelSplat’s reported Gaussian count and improves PSNR by 1.559 dB; at 338K, it exceeds pixelSplat by 1.891 dB.The comparison also reports advantages over MVSplat, DepthSplat, TranSplat, and F4Splat at similar populations; Figure 3 visualizes the DL3DV reconstructions.Source: §4.2 · p. 6, Table 1 · p. 7, Figure 3 · p. 7
  • At 226K, PocketSplat improves PSNR over the strongest listed baseline by 2.126 dB on Tanks & Temples and 0.965 dB on MegaDepth.The frozen checkpoint retains budget-controlled behavior on both out-of-distribution datasets; Figure 4 shows Tanks & Temples scenes.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Figure 4 · p. 8

4.3 MOBILE EVALUATION

  • PocketSplat Light uses 75.4% fewer Gaussians than MVSplat-Stream while improving PSNR by 2.143 dB; native MVSplat and DepthSplat exceed the target iPhone’s memory budget.PocketSplat constructs assets in 10.20–10.95 seconds across budgets, while Dense reduces construction and inference time by 75.0% and 76.9% relative to MVSplat-Stream.Source: §4.3 · pp. 7–8, Table 2 · p. 8, Figure 5 · p. 9

4.4 ABLATION STUDIES

  • World-space allocation improves PSNR over count-matched random selection by 3.0049 dB at 113K, 3.6915 dB at 226K, and 2.1916 dB at 338K.Each ablation holds the requested budget fixed; Table 3 reports the quality results.Source: §4.4 · p. 8, Table 3 · p. 9
  • Spatial responsibility decoding improves PSNR by 3.3567 dB at 113K, 0.8290 dB at 226K, and 0.1707 dB at 338K.It improves all 140 scenes at 113K and 226K, and paired 95% bootstrap intervals remain positive across reported budgets.Source: §4.4 · p. 9, Table 3 · p. 9
  • At 113K, budget-before-decoding reduces attribute projection from 1.308 to 0.411 ms and construction from 166.089 to 163.216 ms versus decode-before-allocate.Attribute-projection reductions are 68.5%, 44.9%, and 21.3% at 113K, 226K, and 338K, respectively; at Full, the methods take 96.75 and 96.65 ms.Source: §4.4 · p. 9

CONCLUSION

PocketSplat constructs feed-forward Gaussian assets at explicit representation budgets through world-space latent allocation, cross-view fusion, selective attribute decoding, and support adaptation. Evaluations across in-domain, out-of-domain, and mobile settings demonstrate quality–asset-size operating points and direct mobile execution.

CONCLUSION

  • PocketSplat’s world-space latent allocation constructs Gaussian assets at explicit budgets, with results demonstrating quality–asset-size operating points and direct mobile execution.The budget-conditioned decoder fuses cell information into retained latents, projects complete attributes only for selected support, and adapts covariance under local sparsification.Source: CONCLUSION · p. 9

AI USE STATEMENT

The statement describes the use of generative AI tools and specifies export artifacts and parity requirements.

AI USE STATEMENT

  • Generative AI tools were primarily used to polish manuscript language and presentation; the authors are responsible for the technical content, results, citations, and conclusions.Source: AI USE STATEMENT · p. 13

A MOBILE EXPORT AND PARITY REQUIREMENTS

  • The mobile export path includes preprocessing and post-allocation ONNX graphs, an FP32 pre-allocator Core ML package, and six DA3 Core ML stage packages.Export manifests should retain model and configuration identifiers, precision, input shapes, hashes, and interface names; numerical behavior needs checking near voxel boundaries.Source: A MOBILE EXPORT AND PARITY REQUIREMENTS · p. 13

B REPORT CONSISTENCY AND ADDITIONAL EXPERIMENTAL DETAILS

The appendix provides supplementary experimental details and implementation information.

B REPORT CONSISTENCY AND ADDITIONAL EXPERIMENTAL DETAILS

C ALLOCATION IMPLEMENTATION DETAILS

PocketSplat restores context RGB before computing quantized luma detail, then uses deterministic pseudorandom keys for within-cell selection and reliability–support weighted pooling.

C ALLOCATION IMPLEMENTATION DETAILS

D HYBRID IOS DEPLOYMENT AND IMPLEMENTATION DETAILS

PocketSplat splits mobile execution across Core ML, Swift, ONNX Runtime, and Metal, with a camera-coordinate contract for consistent Gaussian export.

D HYBRID IOS DEPLOYMENT AND IMPLEMENTATION DETAILS

  • PocketSplat uses six Core ML backbone stages, Core ML FP32 for depth, cameras, and pre-allocation, Swift for grouping, allocation, and hidden fusion, ONNX Runtime for preprocessing and post-allocation decoding, and Metal for viewing.The described deployment uses this hybrid runtime partition for its on-device pipeline.Source: D HYBRID IOS DEPLOYMENT AND… · p. 13
  • The direct hybrid path exhausts memory for cost-volume baselines; streamed MVSplat becomes deployable, while DepthSplat remains out of memory.MVSplat streams 32 depth candidates in chunks of eight and places its Core ML backbone on CPU.Source: D HYBRID IOS DEPLOYMENT AND… · p. 13
  • The post graph retains N rows, assigns unselected entries opacity logit −20, and the PLY writer filters values ≤−19, leaving exactly B selected candidates.Current UI budgets are 451,584, 338,000, 225,792, and 112,896.Source: D HYBRID IOS DEPLOYMENT AND… · pp. 13–14
  • ARKit OpenGL camera-to-world poses are converted with C2Wcv = C2Wgl diag(1, −1, −1, 1), and predicted context cameras are similarity-aligned before export.Images and intrinsics follow the GPU resize/crop transform, and translation is normalized using the median multi-view relative translation scale.Source: D HYBRID IOS DEPLOYMENT AND… · p. 14

E CROSS-DATASET RESULTS AND PROTOCOL

Across Tanks & Temples and MegaDepth, PocketSplat’s reported quality varies by metric and output budget, with the intermediate 338K budget remaining a useful out-of-distribution choice.

E CROSS-DATASET RESULTS AND PROTOCOL

F EFFICIENCY PROTOCOLS AND COMPLETE MEASUREMENTS

The efficiency measurements report construction time and peak memory under a fixed GPU profiling protocol, while the archived OOD panels are illustrative rather than matched-budget evidence.

F EFFICIENCY PROTOCOLS AND COMPLETE MEASUREMENTS

  • Eight fixed scenes receive three warm-ups and ten measured repeats per scene; TINFER50 excludes loading, disk I/O, preprocessing, copies, metric computation, and saving.Peak memory is measured after load and warm-up in a separate process.Source: F EFFICIENCY PROTOCOLS AND COMPLETE… · p. 14
  • 322.3 ms and 2.705 GB are PocketSplat’s Full-budget median TINFER50 and peak memory, respectively, close in time to MVSplat’s 322.8 ms.PocketSplat’s reported peak is lower than the other methods in Table 6.Source: F EFFICIENCY PROTOCOLS AND COMPLETE… · p. 14, Table 6 · p. 15
  • Reducing PocketSplat’s output from Full to 113K raises TINFER50 from 322.3 to 601.8 ms while peak memory stays at 2.705 GB.Dense features and allocation overhead dominate the savings from fewer rendered primitives.Source: F EFFICIENCY PROTOCOLS AND COMPLETE… · p. 14, Table 6 · p. 15
  • At 113K, budget-before-decoding reduces projection time by 68.5% and construction time by 1.73% versus decode-before-allocate.Allocation takes approximately 127 ms, versus 1.308 ms for dense projection; the experiment uses identical allocation and selected candidates.Source: F EFFICIENCY PROTOCOLS AND COMPLETE… · pp. 14–15, Table 7 · p. 15

G ARCHIVED OOD QUALITATIVE PANELS

  • The MegaDepth panels show context inputs, four reconstruction methods including PocketSplat at 226K, 338K, and Full, and ground truth, but are not controlled matched-budget evidence.Their image-level provenance and budgets are not specified in the supplied summaries.Source: Figure 6 · p. 15, G ARCHIVED OOD QUALITATIVE PANELS · p. 15

Related papers

Loading 2610.03192v1…