PaperlayerView exact revision on arXiv

Source-linked AI summary

CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage

Jiale Liu, Jungang Li, Jieming Yu, Xinglin Yu, Zihao Dongfang, Zongjian Ding, Kaifeng Ding, Yi Yang, Lidong Chen, Yang Zou, Shunwen Bai, Jiahuan Zhang, Haoran Huang, Shan Huang, Yudong Gao, Mingjun Cheng

arXiv:2605.15597v1cs.CVcs.GRcs.LGcs.RO

TL;DR

Existing 3D assets lack a sparse, comparable, and geometry-consistent panoramic training interface. COVER addresses this gap with training-free conflict-aware ERP view selection, and CM-EVS packages the results into a compact, provenance-tracked RGB-D-pose dataset with improved coverage–conflict trade-offs.

  • Problem

    Existing 3D assets and sampling policies do not directly provide sparse, comparable, geometry-consistent panoramic RGB-D-pose observations with auditable provenance.

  • Method

    COVER greedily selects ERP viewpoints by projecting accumulated range-depth geometry into candidate probes, scoring incremental coverage, and penalizing depth conflicts.

  • Results

    CM-EVS contains 36,373 curated ERP frames from 1,275 indoor scenes, with a median of ∼25 frames per scene, coverage of all 13 unified room types, and improved coverage–conflict trade-offs.

  • Takeaways & Limitations

    CM-EVS provides a sparse, compact, and auditable panoramic RGB-D-pose resource for geometry-consistent 3D learning.

  • Takeaways & Limitations

    The evaluation targets curator-layer coverage and depth-conflict statistics rather than downstream task accuracy.

Abstract

from arXiv · show

Modern 3D visual learning relies on observations sampled from metric 3D assets, yet existing scans, meshes, point clouds, simulations, and reconstructions do not directly provide a sparse, comparable, and geometry-consistent panoramic training interface. Dense trajectories duplicate nearby views, source-specific rendering policies yield heterogeneous annotations, and sparse heuristics may miss important regions or introduce depth-inconsistent observations. We study how to convert 3D assets into sparse panoramic RGB-D-pose data that preserves complete scene coverage with low redundancy and auditable provenance. We propose COVER (Coverage-Oriented Viewpoint curation with ERP Range-depth warping), a training-free ERP viewpoint curator that projects geometry observed from selected views into candidate ERP probes, scores incremental coverage, and penalizes depth conflicts. Under bounded proxy error, its greedy coverage proxy preserves the standard coverage-style approximation behavior up to an additive error term. Using COVER, we build CM-EVS (Coverage-curated Metric ERP View Set), a panoramic RGB-D-pose dataset with 36,373 curated ERP frames from 1,275 indoor scenes across Blender indoor, HM3D, and ScanNet++, complemented by outdoor panoramas from TartanGround and OB3D re-encoded into the same schema. Each frame provides full-sphere RGB, metric range depth, calibrated pose; COVER-produced indoor frames include per-step provenance logs. With a median of only 25 frames per indoor scene, CM-EVS covers all 13 unified room types while maintaining compact scene-level coverage. Experiments show that COVER improves the coverage-conflict trade-off, making CM-EVS a sparse, compact, and auditable RGB-D-pose resource for geometry-consistent panoramic 3D learning.

1 Introduction

CM-EVS addresses the under-specified observation layer between metric 3D assets and panoramic model training by selecting compact, geometry-consistent, auditable RGB-D-pose views. COVER provides the conflict-aware curation policy, while CM-EVS packages its outputs into a provenance-tracked dataset.

  • Motivation: Panoramic RGB-D-pose data provides a compact interface between scene-scale geometry and model training by combining full-sphere context, appearance, metric depth, and calibrated pose.A single ERP frame records a full 4π solid angle under a shared spherical ray parameterization.
  • Motivation: Existing assets and sampling policies do not directly provide compact, comparable, geometry-consistent panoramic training observations.Dense trajectories duplicate nearby views, sparse heuristics may miss important regions, and source-specific rendering policies create heterogeneous evidence.
  • COVER: COVER is a training-free greedy ERP curator that selects views by accumulating observed geometry, scoring incremental coverage, and penalizing range-depth conflicts.Its coverage proxy has a bounded-error analysis.
  • CM-EVS: 36,373 ERP frames from 1,275 indoor scenes form CM-EVS’s curated core, complemented by schema-compatible outdoor panoramas from TartanGround and OB3D.Samples provide full-sphere RGB, metric range depth, calibrated pose, and provenance metadata for COVER-produced frames.
  • Results: A median of ∼25 ERP frames per indoor scene covers all 13 unified room types, while COVER improves the coverage–conflict trade-off over four baselines.The baselines are random, single-view-probe, coverage-only, and low-conflict-only selection.
  • Implication: CM-EVS offers an auditable observation layer for evaluating and training geometry-consistent panoramic 3D models.The dataset emphasizes compactness, geometry awareness, and reproducibility.

2 Related Work

Related work establishes panoramic RGB-D-pose data as a compact interface for 3D learning and distinguishes COVER from online view-planning methods. COVER instead performs offline, training-free curation from existing 3D assets with released provenance metadata.

  • Panoramic Data for 3D Learning: Panoramic RGB-D-pose observations capture a full 4π field of view in one ERP frame under a unified spherical parameterization.This representation supports 3D perception, reconstruction, and generation.
  • Panoramic Data for 3D Learning: Existing scene resources provide geometry, annotations, or simulation environments, while panoramic datasets and reconstruction methods demonstrate the value of full-sphere observations.The related resources span multiple scene and observation formats.
  • View Selection for Data Curation: COVER occupies an offline, training-free, fixed-budget regime that selects panoramic training views from existing 3D assets using coverage and depth-conflict criteria.This complements online camera-pose selection for active reconstruction or exploration.
  • View Selection for Data Curation: CM-EVS releases candidate pools, coverage gains, conflict ratios, selection scores, and provenance logs to support reproducible view-policy auditing.The release follows Datasheets for Datasets and Croissant practices.

3 Method

COVER selects panoramic views through a conflict-aware, budgeted greedy pipeline that uses low-resolution warping proxies before rendering chosen candidates at full resolution. The pipeline normalizes assets, generates valid candidates, updates an accumulated point cloud, and records returned ERP RGB, range depth, pose, and provenance information.

  • Problem Formulation: COVER formulates fixed-budget panoramic viewpoint selection as conflict-aware coverage maximization over feasible candidate poses.Candidates are filtered using a geometric-validity predicate before selection.
  • Guarantee: Low-resolution proxy scoring avoids rendering every candidate at full resolution, while bounded per-step proxy error contributes an additive penalty to the coverage guarantee.The exact greedy oracle would cost 10^2–10^3 times the final K-frame rendering cost.
  • Conflict-aware Warping Oracle: The warping oracle projects accumulated range-depth geometry into each candidate ERP and compares it with a low-resolution probe to identify explained, new, and conflicting pixels.The depth tolerance is δ = 0.5% of the AABB diagonal, clamped per source.
  • Conflict-aware Warping Oracle: Candidates are scored using normalized coverage gain and conflict penalty, with st(v) = Gt(v) − λLt(v) and λ=0.35 throughout the paper.Because new and conflicting pixels are disjoint, λ re-ranks candidates rather than rescaling them.
  • Algorithm: Algorithm 1 starts from an interior seed, repeatedly warps the accumulated cloud, selects the highest-scoring remaining candidate, renders it, and updates the cloud until the budget is reached.The selected set is returned with per-frame ERP RGB, range depth, and pose.
  • Per-scene Pipeline: Outdoor re-encoded trajectories do not carry the per-step provenance log because they are not curator-selected subsets.This scope distinction is documented alongside the adapter pipeline.
  • Per-scene Pipeline: The per-scene pipeline normalizes source assets, generates and filters candidates through source-specific adapters, then applies the budgeted greedy selection phase.Curator adapters serve Blender indoor, HM3D, and ScanNet++, while re-encoding adapters convert TartanGround and OB3D trajectories into the unified schema without COVER.

4 The CM-EVS Dataset

CM-EVS is a provenance-tracked panoramic RGB-D-pose dataset built from multiple indoor sources and schema-compatible outdoor panoramas. It emphasizes full-sphere coverage, unified modalities, room-type diversity, and low redundancy.

  • Dataset composition: CM-EVS applies COVER across Blender indoor, HM3D, and ScanNet++, while re-encoding TartanGround and OB3D into a unified ERP and pose schema.The schema uses a right-handed world frame, OpenCV camera coordinates, scalar-first world-to-camera quaternions, and relative camera positions.
  • Coverage: Selected ERP viewpoints span scene space through multi-view 4π coverage, with each viewpoint contributing a full sphere rather than a slice.The dataset characterizes this property alongside its unified schema, scene-type diversity, and low redundancy.
  • Unified representation: Each frame provides RGB, ERP range depth, and calibrated pose under one representation, with source-dependent depth ranges and resolutions.Blender indoor depth spans 0.3–30+ m, while HM3D and ScanNet++ concentrate around 1.4–1.9 m; outdoor sources extend to tens of metres.
  • Scene-type diversity: CM-EVS covers all 13 unified room-type buckets, with Shannon entropy 3.10 bits and Gini concentration 0.49.Blender indoor contributes commercial, attic, basement, and library categories, while HM3D and ScanNet++ supply residential rooms.
  • Low redundancy: A median of ∼25 frames per indoor scene provides compact scene-level coverage, roughly 4–7× fewer frames than fixed-budget baselines.The gain-gradient early stop uses τ = 1% for m = 2 steps; frame counts range from 1–54 across curator sources.

5 Curator analysis

The curator analysis evaluates COVER against alternative selection rules, its conflict-weight sensitivity, and consistency across indoor sources. Results indicate that iterative re-ranking preserves coverage while reducing conflicts and remains stable under a shared default.

  • Analysis scope: The analysis examines fixed-budget baselines, λ sensitivity, and cross-source behavior, with the noisy-oracle bound aligning with the observed λ plateau.Experiments use shared feasible candidate pools and common seed viewpoints.
  • Fixed-budget comparison: At K = 4, CM-EVS matches Greedy coverage while shifting selection toward lower-conflict viewpoints; non-iterative baselines collapse and Low-conflict only is overly conservative.The shared candidate pool and seed isolate selection-rule effects.
  • Conflict-weight sensitivity: λ∈[0.1, 0.5] forms a stable plateau at K = 30, while λ = 0 collapses onto a high-conflict mode and larger values trade coverage for conflict reduction.The paper adopts λ = 0.35 as a conservative default.
  • Cross-source consistency: At λ = 0.35 and K = 30, coverage is 0.413 on Blender indoor, 0.393 on HM3D, and 0.735 on ScanNet++.The higher ScanNet++ coverage is attributed to smaller, cleaner room-scale scans.
  • Cross-source consistency: Despite a 7× spread in conflict statistics, the same selection rule produces a stable operating point across Blender indoor, HM3D, and ScanNet++.HM3D has conflict prior 0.0713 versus 0.0175 for Blender indoor and 0.0103 for ScanNet++.

6 Conclusion

The paper presents COVER and CM-EVS as a principled approach to sparse panoramic observation construction, balancing coverage, depth consistency, and provenance. Its evaluation is limited to curator-layer statistics rather than downstream task accuracy.

  • Conclusion: COVER selects sparse panoramic views by balancing incremental scene coverage against range-depth conflicts.It is described as a training-free ERP viewpoint curator.
  • Conclusion: CM-EVS provides provenance-tracked panoramic RGB-D-pose data with sparse yet comprehensive coverage, unified conventions, and auditable selection metadata.The paper positions these properties as criteria beyond frame count or source scale.
  • Implications: The paper argues that panoramic datasets should be judged by coverage efficiency, redundancy, geometric consistency, and reproducibility, not only frame count or source scale.This conclusion follows the dataset construction and curator analysis presented in the paper.
  • Limitations: The evaluation targets coverage and depth-conflict statistics on shared candidate pools, not downstream task accuracy.Benchmarking ERP depth estimation, novel-view synthesis, reconstruction, and world-model pretraining is deferred to future work.

A Datasheet for the dataset

The datasheet specifies CM-EVS instances, release counts, sampling procedures, schemas, filtering, licensing, and intended use. It distinguishes COVER-curated indoor data from outdoor trajectory re-encodings and documents reproducibility boundaries.

  • Instances and release: Each instance is an ERP triple containing RGB, range depth, and camera pose, with scene metadata declaring the coordinate convention.HM3D and ScanNet++ provide scene identifiers, viewpoint metadata, and regeneration scripts rather than redistributed raw assets.
  • Composition: The full v1.0 release totals 822,717 frames across 1,350 units, including 13,631 Blender frames, 783,944 TartanGround frames, 2,400 OB3D frames, 14,475 HM3D frames, and 8,267 ScanNet++ frames.Release status differs by source and redistribution permission.
  • Sampling and adapters: Outdoor frames carry unified schema fields but not per-step provenance logs because COVER is not run on TartanGround or OB3D in v1.0.The outdoor portion releases full re-encoded source trajectories rather than a COVER-selected subset.
  • Splits and reproducibility: The default split is scene-level 70 / 15 / 15, keeping frames from each scene or space unit in one split.Downstream task evaluation is deferred, while provenance logs support rerunning alternative viewpoint policies on the same candidate set.
  • Sampling and adapters: COVER generates indoor frames through candidate filtering, conflict-aware greedy selection, high-resolution ERP rendering, and unified-schema export.Outdoor data is re-encoded from full source trajectories without running COVER.
  • Quality control: The 26-direction geometric sanity filter rejects embedded, wall-flush, out-of-AABB, and degenerate candidates before selection, while post-render checks reject frames exceeding a 90% invalid-depth ratio.An automated audit script reapplies these checks across the public release.
  • Labeling and quality: The 13 room-type buckets are assigned through a deterministic hand-authored mapping of source labels, without per-frame human labeling.Invalid depth values follow source conventions and are documented in metadata.
  • Licensing: Redistribution follows source licenses: Blender frames are released under CC-BY 4.0, while HM3D and ScanNet++ provide metadata and regeneration scripts under upstream access terms.The curator code and documentation are released under MIT.

B.4 Greedy parameters

COVER uses fixed probe and conflict parameters alongside gain-gradient stopping to spread selections across candidate space while adapting to scene saturation and depth distributions.

  • Parameters: COVER uses 128 × 256 low-resolution probes, 32 center-near candidates, λ = 0.35 conflict weighting, and δ = 0.5% of the AABB diagonal.Evaluation budgets are K = 4 and 30; production stopping uses τ = 1% for m = 2 consecutive sub-threshold steps.
  • Selection geometry: Default λ = 0.35 spreads selected viewpoints across candidate space rather than producing localized clusters in Blender indoor, HM3D, and ScanNet++ examples.The comparison uses feasible candidate pools and selected viewpoints across three curator-source examples.
  • Early stopping: Production selection stops when Gt < τ for two consecutive steps, allowing scene-specific termination instead of wasting frames after marginal gains become negligible.Fixed-budget evaluation disables this early stop; production uses τ = 1% and m = 2.
  • Depth distributions: Released range depths span 0.3–30+ m for Blender indoor, concentrate around 1.4–1.9 m for HM3D and ScanNet++, and reach tens of metres outdoors.The Blender long tail is associated with atria and large open-plan spaces.
  • Examples: Six selected viewpoints span entryway, living area, and bedroom alcove in a Blender residential example, while an open-plan office covers four zones by approximately t≈6.The office reaches the τ = 1% marginal-gain threshold around t ≈22 under K = 30.

C.5 Per-source bad-case rate

The bad-case audit measures manual-review rates by source and links source-specific failure profiles to the corresponding depth and rendering evidence.

  • Audit definition: The audited bad-case rate is the fraction of source units entering manual review, and it upper-bounds potential public-release leakage without the post-render validity gate.Every flagged case is excluded from the release.
  • Per-source profile: HM3D concentrates F1 and F2 failures, ScanNet++ is dominated by F3, and F4 appears only in the synthetic Blender path.The disjoint profile reflects adapter-specific failure responsibilities and upstream asset or point-cloud limitations.
  • Visual evidence: Each failure cell pairs ERP RGB with range depth, because some unusable RGB renders retain valid depth while other failures create holes in both modalities.Border colours and F1–F5 tags identify the failure class.

C.7 Resolution status and v1.1 roadmap

The release excludes audited failures, documents their diagnostic taxonomy, and proposes v1.1 changes targeted at residual geometry, point-cloud, and rendering issues.

  • Release status: The v1.0 release excludes every audited bad case, while the audit records failures intended for earlier detection in the curator.The taxonomy separates responsibility between existing diagnostics and future tightening.
  • F1/F5: The v1.1 roadmap tightens wall-proximity filtering and adds outdoor sky-visibility checks for residual F1 and F5 cases.These changes target failures that slip past the existing 26-direction filter.
  • F2: F2 cases are caught by a post-render finite-depth-ratio threshold, with v1.1 adding per-frame invalid-pixel ratios to the step logs.The Blender examples demonstrate asset-side mesh holes flagged at render time.
  • F3: F3 is structural to the ScanNet++ point-cloud adapter; an optional mesh-fallback path is planned at the cost of one extra preprocessing step per scan.The adapter mode remains exposed so downstream users can opt out of point-cloud-derived frames.
  • F4: F4 is an asset-import-time failure, and v1.1 adds a post-render colour-histogram check rejecting renders with more than 20% pure-magenta or pure-black pixels.The existing import-time NaN/luminance check did not catch every unusable Blender render.
  • Reproducibility: The 50 audited bad cases ship with full ERP sequences, range-depth maps, and pose JSONs so users can rerun the curator on the same assets.The cases are retained as inspectable evidence rather than removed from the diagnostic record.
  • Positive audit: A random audit of 50 public Blender frames passes the automated checks, which also hold across the full Fpub release.The audit samples only frames that passed the pre-sampling failure checks.

D Warping oracle empirical validation

The warping proxy approximates the exact pre-render-all oracle with measurable error but substantially lower rendering cost, while conflict-aware reranking preserves useful final coverage.

  • Approximation behavior: Conflict-aware reranking absorbs oracle noise within the bounded λ plateau, matching the regime predicted by Lemma 1.The empirical result connects the observed coverage gap to the paper’s additive-error guarantee.
  • Accuracy: The warping oracle has mean per-step proxy error ¯ϵ = 0.4254 ± 0.2223 and final coverage gap 8.10 ± 5.50 percentage points relative to pre-render-all.Strict top-1 agreement is 0.023 ± 0.150, with Pearson r = 0.148 and Spearman ρ = 0.366.
  • Compute cost: 133.4 ± 17.2× wall-clock speed-up results from replacing 1.74 GPU-hours of full-resolution rendering with 0.014 GPU-hours of probe rendering.The comparison uses the same hardware and motivates the noisy-oracle design.

E Proof of Lemma 1

Lemma 1 extends the standard greedy coverage guarantee to COVER’s noisy, conflict-penalized oracle. The resulting bound retains the (1 − 1/e) behavior up to proxy-error and conflict terms.

  • COVER’s warping proxy estimates true marginal coverage with error bounded by ϵ_t + ηL_t(v).
  • The telescoped guarantee preserves (1 − 1/e) coverage up to additive proxy-error and conflict penalties.The tighter form weights each step’s penalty by (1−1/K)^(K−t).
  • When oracle-best candidates have low conflict, γ_t ≈ 0 and the bound reduces to the coverage-only noisy-oracle result.For conflicted oracle-best candidates, 2λγ_t is the worst-case coverage cost of avoiding unstable proxy regions.
  • COVER is positioned as an offline, training-free, fixed-budget curator rather than an online reconstruction or exploration policy.
  • CM-EVS exposes candidate pools, coverage gains, conflict ratios, selection scores, and provenance logs for reproducible viewpoint-policy auditing.

NeurIPS Paper Checklist

The checklist materials identify three substantiated contributions, explicit assumptions and a complete proof, detailed reproducibility information, and documented limitations. The main experimental tables report point estimates without error bars, while licensed real-scan data remain access-controlled.

  • Claims: The paper substantiates COVER, CM-EVS, and coverage–conflict experiments across shared candidate pools and sources.The contributions include a noisy-oracle approximation guarantee and 36,373 frames across 1,275 indoor scenes.
  • Limitations: Evaluation is limited to curator-layer coverage and conflict statistics rather than downstream task accuracy, and HM3D and ScanNet++ frames are not redistributed.Users must regenerate licensed-source frames locally under the original access terms via released adapters.
  • Theory assumptions and proofs: Lemma 1 assumes monotone submodular coverage, bounded proxy noise, λ ≥ η, and bounded oracle-best conflict γ_t.The complete proof includes the per-step inequality, telescoping argument, and tighter telescoped form.
  • Experimental result reproducibility: The curator pipeline specifies Algorithm 1, geometric sanity predicates, hyperparameters, evaluation budgets, probe resolution, and seed-pool size.
  • Experimental result reproducibility: The fixed-budget experiments report point estimates without error bars, while multiseed intervals and downstream-task error bars are deferred.
Loading 2605.15597v1…