Less Is More: Genetic Frame Selection for Efficient Novel View Synthesis

1 INTRODUCTION

• Choosing views is difficult because their value depends on the target views and the other selected frames, while redundant or poorly posed inputs can waste resources or degrade quality.The introduction notes that predetermined input views leave the selection question largely unexplored.Source: §1 · p. 2
• A compact scorer uses target relevance, redundancy, appearance, and pose relationships to greedily select frames, with reconstruction-aware genetic-search choices providing training supervision.Selection requires no rendering, reconstruction fitting, or per-scene optimization at inference time.Source: §1 · p. 2
• Across outdoor, unbounded, and handheld indoor scenes, the selector outperforms geometric and reconstruction-aware baselines; selected subsets can also outperform full inputs.The frozen selector transfers to other reconstructors without retraining and extends to object-targeted selection.Source: §1 · p. 2

2 RELATED WORK

• Per-scene NeRF and 3D Gaussian Splatting methods optimize each capture, while feed-forward methods predict scene representations directly from multiple views.The paper evaluates whether its selector transfers across reconstructors and notes that reconstruction quality depends on the views provided.Source: §2 · p. 3
• Geometric view-selection methods emphasize camera placement or sparse geometry, whereas reconstruction-aware methods score uncertainty, information gain, or estimated quality.For already captured frames and feed-forward reconstruction, iterative fitting during selection can cost more than the downstream task; this paper instead selects retrospectively for known target viewpoints.Source: §2 · p. 3
• Unlike prior distillation examples that imitate optimization decisions or policies, this method turns reconstruction-optimized frame subsets into ordered, selection-conditioned supervision for a sequential selector.The cited examples include imitation of branch-and-bound decisions and ranking-based policy distillation.Source: §2 · p. 4

3.1 PROBLEM SETUP

• Target poses identify optimization views O, which serve as observable proxies; target-view RGB images are unavailable during selection, training, and subset search.The candidate pool P supplies K input views to the reconstructor, which renders the held-out targets H.Source: §3.1 · p. 4

3 METHOD

• The selector uses target-related, selection-conditioned, and intrinsic features to score candidate frames, then greedily selects K frames without rendering or per-scene optimization at inference.Target-related features use optimization views as proxies; selection-conditioned features change as the subset grows, and intrinsic features include image sharpness.Source: §3.2 · p. 5, §3.3 · p. 5

3.2 FEATURES

• Candidate features capture geometry and appearance relative to optimization views, complementarity with selected frames, and image sharpness.Target-relational features remain fixed; selection-conditioned distances, similarities, and angular novelty are recomputed as frames are selected.Source: §3.2 · pp. 4–5

3.3 SCORER AND INFERENCE

• A shared-weight scorer evaluates candidates one at a time, recomputing selection-conditioned features at each greedy step and appending the highest-scoring remaining frame.Each of K steps uses one scoring-network pass, with no rendering, reconstruction, or per-scene optimization.Source: §3.3 · p. 5

3.4 SUPERVISION BY SUBSET SEARCH

• An offline genetic algorithm searches candidate subsets by reconstructing and rendering optimization views, using normalized average PSNR, SSIM, and negated LPIPS as fitness.The search retains the top four subsets per scene for supervision because different subsets can be similarly effective; the search is too costly for deployment.Source: §3.4 · pp. 5–6, §3.5 · p. 7

3.5 TRAINING

• Training orders each retained subset by marginal reduction in optimization-view distance, then uses its sequence as per-step supervision for the scorer.The model predicts the next frame with a softmax cross-entropy objective; fitness-weighted trajectories from multiple subsets train it on a local distribution of high-quality solutions.Source: §3.5 · pp. 6–7

4.1 SELECTION QUALITY AND COMPUTATIONAL EFFICIENCY

• The selector is trained on DL3DV labels only, with feature selection on disjoint ScanNet++ validation, and evaluated across six corpora spanning outdoor, unbounded, and handheld indoor captures.Source: §4 · p. 7
• At K=30 and K=40 on DL3DV, selected subsets improve PSNR by 1.33 and 1.68 dB while reducing reconstruction time by factors of 7.3 and 5.1.Peak memory falls from 17.8 GB for the full sequence to 4.6 GB at K=30 and 5.9 GB at K=40.Source: §4.1 · p. 7, Table 5 · p. 9

4.2 GENERALIZATION ACROSS RECONSTRUCTORS AND PARADIGMS

• The same frozen selector, supervised with labels from one reconstructor, supports generalization across different reconstruction paradigms.Table 3 applies the same weights to Nerfstudio and 3DGS under the same scenes, candidate pools, and held-out targets, changing only the reconstructor.Source: §4.2 · p. 7, Table 3 · p. 8
• For object-centric synthesis at K=30, the method improves PSNR over the strongest baseline by 0.50 dB on ScanNet v2 and 0.82 dB on CO3D.ScanNet v2 scenes contain objects in cluttered rooms, while CO3D depicts individual objects; both provide ground-truth object masks.Source: §4.2 · p. 8, Table 2 · p. 8
• In the 7-Scenes cross-capture protocol, the method is ahead of every baseline at both frame budgets when targets come from a separate capture.No target-capture frame is selectable; the real-world example likewise uses observations from a separate trajectory.Source: §4.2 · p. 8, Table 1 · p. 8, Figure 5 · p. 9

4.3 ABLATIONS

• Estimated VGGT-Ω poses reduce performance relative to COLMAP poses, with a more visible gap at larger input budgets, but the selector remains effective across budgets.The selector and reconstruction pipeline are unchanged in this DL3DV comparison.Source: §4.3 · p. 8, Table 4 · p. 9
• Removing appearance features substantially reduces selection time with only a small quality reduction, while also removing sharpness produces a clearer, consistently larger degradation.The ablation attributes most performance to inexpensive geometric and selection-context features; sharpness adds information not captured by geometry alone.Source: §4.3 · p. 8, Table 5 · p. 9

5 CONCLUSION

• Frame selection directly determines reconstruction quality in feed-forward novel view synthesis.The method distills an expensive reconstruction-aware genetic search into a lightweight selector that requires no rendering, reconstruction, or per-scene optimization at inference.Source: §5 · p. 9
• Across diverse datasets and budgets, the selector outperforms geometric, retrieval-based, and set-aware baselines, transfers without retraining to Nerfstudio and 3DGS, and extends to object-targeted selection.Ablations identify geometry and selection context as the main signals, with appearance offering dataset-dependent gains at additional computational cost.Source: §5 · p. 9
• Optimization views provide a dataset- and geometry-dependent approximation to the evaluation objective, and both selector and reconstructor require camera poses.Poses must be estimated beforehand when they are unavailable.Source: §5 · p. 9
• The paper identifies future directions in direct geometric features for requested target poses, dynamic scenes, learned candidate features, and reinforcement-learning subset policies.Direct target-pose features are proposed while retaining proxy views only for appearance.Source: §5 · p. 9

AI USE STATEMENT

• Generative AI tools were used to implement the selector, feature extraction, genetic-algorithm label generation, baselines, and evaluation harnesses.The tools also supported methodology and experiment feedback, data cleaning and reformatting, capture conversion, anomaly diagnosis, and hypothesis refinement.Source: AI USE STATEMENT · p. 10
• The authors report no AI use for synthetic-dataset generation, theoretical models or conceptual frameworks, mathematical claims, or translation.Qualitative and thematic data-analysis support was not applicable.Source: AI USE STATEMENT · p. 10
• The authors additionally used AI for code editing, drafting parts of the paper, literature analysis, reference formatting, paper structure and title suggestions, experimental parameters, brainstorming, and information search.All experimental code was executed and outputs inspected; numbers and paired-scene comparisons were regenerated or recomputed, and baseline implementations checked against original sources.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Appendix A.1–A.3 specify the feature representation, genetic search, and training data, splits, and hyperparameters; Appendix C.1 gives complete per-dataset, per-budget results.Appendix A.4 reports the reconstructor choice.Source: REPRODUCIBILITY STATEMENT · p. 10
• Benchmark experiments use public datasets with the splits described in Section 4, and selector performance is reported as the mean across three random seeds.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 FEATURE REPRESENTATION

• The deployed representation uses 18 scalar features: 13 relate candidates to proxy views, four to selected frames, and one measures sharpness.The four selected-set features are recomputed after each selection, so candidate scores reflect changing redundancy.Source: §A.1 · p. 14
• Target-view features summarize camera distance, viewing direction, DINO appearance similarity, and triangulation geometry relative to optimization views.The geometry group also includes visibility indicators that do not account for occlusion.Source: §A.1 · pp. 14–15
• Selected-frame features measure candidate distance, directional and appearance similarity, and angular novelty relative to the current selection.The separate sharpness feature captures image blur alongside pose and appearance relationships.Source: §A.1 · p. 15

A.2 GENETIC-SEARCH SUPERVISION

• Genetic-search individuals are subsets of K distinct frames; fitness averages independently normalized PSNR, SSIM, and negated LPIPS with equal weights.The search begins with 40 individuals, initialized from spatial farthest-view, uniformly spaced, and random subsets.Source: §A.2 · p. 16
• The search preserves its two fittest individuals, then fills the remaining slots through tournament-selected crossover and mutation, which may both affect a child.Crossover occurs with probability 0.8; mutation occurs independently with probability 0.3 and replaces max(2, ⌊K/5⌋) frames.Source: §A.2 · p. 16

A.1 FEATURE REPRESENTATION

• None of the tested additional feature groups improves on the deployed 18-feature representation; the set-aware trio is −0.295 dB.The table compares alternative configurations against the deployed representation at K=30 on ScanNet++ validation scenes.Source: §A.1 · p. 16, Table 6 · p. 17

A.3 TRAINING DETAILS

• Training uses 70 DL3DV scenes, with 21 held out for testing and 20 for validation; the selector is evaluated zero-shot on six other datasets.Thirty scenes are excluded because they were used to tune and validate the genetic search; training labels cover five target-region spread levels.Source: §A.3 · p. 17
• For supervision, genetic search is run at K=20; per item, the top 4 of 30 retained highest-fitness subsets become K per-step ordering targets for autoregressive per-step cross-entropy training.Source: §A.3 · p. 17
• The selector’s three-seed variability stays below the reported margins over the strongest baseline, with paired scenes and held-out target views across methods.Reported dataset-mean standard deviations are summarized in the seed-variability table.Source: §A.3 · p. 18, Table 7 · p. 18
• The fixed-length feature representation keeps scoring cost linear in candidate-pool size, avoiding pool-wide attention over captures with thousands of frames.The benchmarks include pools up to 2,781 frames on ScanNet++ and over 10,000 on 7-Scenes.Source: §A.3 · p. 18

A.4 RECONSTRUCTOR DETAILS

• LongLRM is 6–8 dB ahead of AnySplat and Wild3R at every studied budget and can process full captures within 17.8 GB peak memory.Feeding all approximately 130 frames exhausts a 40 GB A100 GPU for AnySplat and Wild3R on the 21 held-out scenes.Source: §A.4 · p. 18

B ADDITIONAL ABLATIONS

• No tested scorer architecture improves beyond experimental resolution over the fixed MLP; the largest positive difference is +0.004 dB.The comparison includes wider and deeper MLPs, attention variants, recurrent decoders, and selected-set conditioning.Source: B ADDITIONAL ABLATIONS · p. 18, Table 9 · p. 19
• The final feature configuration achieves the highest mean PSNR among the tested representations under the standard K=30 protocol.Each configuration uses the same network and training procedure; the comparison averages three training seeds.Source: B ADDITIONAL ABLATIONS · p. 19

B.1 COMPARISON WITH THE GENETIC SEARCH

• 20.97 dB is the selector’s average PSNR, versus 20.99 dB for the best genetic-search subset and 20.37 dB for uniform random search.The search evaluates all reconstructed subsets at an average of 20.07 dB; the selector beats the genetic-search result on 12 of 21 scenes.Source: §B.1 · p. 19
• 0.48 dB average PSNR improvement: the learned scorer beats untrained feature aggregation in 29 of 30 dataset-budget combinations.The table reports the PSNR difference between learned scoring and a control using the same representation and greedy loop without a learned scorer.Source: Table 10 · p. 20, §B.1 · p. 20

C.1 PERFORMANCE ACROSS INPUT BUDGETS

• Across every dataset and budget K ∈ 10, 20, 30, 40, 50, the selector achieves higher PSNR than the strongest external baseline.Figure 6 provides a qualitative comparison on Tanks and Temples at K=40.Source: §C.1 · p. 20, Figure 6 · p. 20

• At K=40, qualitative comparisons span Tanks and Temples, ScanNet++, 7-Scenes, ScanNet-iPhone, and Mip-NeRF 360.Source: §C.2 · p. 20, Figure 7 · p. 21, Figure 8 · p. 22
• The examples illustrate reconstruction quality from subsets selected by different methods across diverse indoor and outdoor scenes.Source: §C.2 · p. 20
• Table 11 sweeps input budgets across datasets, reporting PSNR, SSIM, and LPIPS for listed baselines and selector variants with or without appearance features.Source: Table 11 · p. 21