Source-linked AI summary
CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels
Jirong Li, Satoshi Ikehata, Shuhei Kurita, Ikuro Sato
TL;DR
3DGS viewpoint selection remains manual, while existing language-guided 3D methods rarely control compositional framing. CapFrame introduces TIVG and solves it with Retrieve–Translate–Refine pose optimization using MLLM-derived geometric pseudo labels, achieving stronger text-frame alignment across diverse evaluations than stated baselines. Its reliance on MLLM prompting remains a sensitivity and scope limitation.
Problem
Existing language-guided 3D methods mainly localize objects rather than controlling viewpoint attributes such as subject orientation and frame layout.
Method
CapFrame retrieves and ranks views, translates instructions into orientation and layout pseudo labels, and refines 6-DoF poses through differentiable 3DGS optimization.
Results
CapFrame produces better text-aligned viewpoints than heuristic viewpoint search and adapted trajectory-generation methods across 38 real-world scenes and 135 instructions, supported by VLM metrics, MLLM judges, and user studies.
Takeaways & Limitations
The framework supports language-driven exploration of diverse 3D Gaussian environments by grounding compositional instructions in camera placement.
Takeaways & Limitations
CapFrame depends heavily on MLLM retrieval and translation, making pseudo labels and camera placement sensitive to prompting.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) enables photorealistic real-time novel view synthesis, yet placing a virtual camera to capture a desired frame remains largely manual. Existing language-guided approaches in 3D scenes mainly focus on object-centric grounding, determining what to observe but rarely controlling how it should appear in a single frame, such as subject orientation or frame layout. To address this limitation, we introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), which aims to identify a 6-DoF camera pose in a 3D Gaussian scene whose rendered frame aligns with a text instruction. To solve this task, we propose CapFrame, a partially differentiable framework that converts language into geometric pseudo labels for camera pose optimization. CapFrame follows a Retrieve-Translate-Refine pipeline: it retrieves relevant views and ranks them through a Question-Evaluation process with MLLMs, translates the instruction into orientation and layout pseudo labels, and refines the camera pose via differentiable optimization with layout and orientation losses in 3DGS. Experiments on 38 real-world scenes with 135 instructions indicate that CapFrame produces viewpoints better aligned with texts than heuristic viewpoint search and adapted trajectory generation baselines, validated by VLM metrics, MLLM judges, and user studies. Code is available at: https://github.com/jirongli/CapFrame
1 Introduction
3DGS makes novel-view synthesis efficient and photorealistic, but selecting an intent-aligned viewpoint remains manual. CapFrame addresses this gap by grounding compositional text instructions in camera poses and reports stronger alignment than prior baselines.
- Motivation: 3DGS supports photorealistic real-time novel-view synthesis, but desirable viewpoint selection remains tedious and scales poorly with scene complexity.Automatic viewpoint recommendation is therefore important for intuitive scene interaction.
- Motivation: Existing language-guided 3D methods localize what to observe but generally do not control how the subject should appear through orientation and framing.Compositional instructions can specify spatial relationships and photographic attributes such as a close-up front view.
- Approach: CapFrame introduces Text-Instructed Viewpoint Grounding and uses Retrieve–Translate–Refine stages to optimize a 6-DoF camera pose.The framework retrieves and ranks views, converts language into geometric pseudo labels, then backpropagates layout and orientation losses through 3DGS.
- Results: 38 real-world scenes and 135 curated instructions show that CapFrame produces viewpoints better satisfying compositional requirements than heuristic viewpoint search and adapted trajectory-generation methods.Evaluation uses text-image similarity metrics, MLLM judges, and a user study.
2 Related Work
Prior work advances 3D representation, vision-language understanding, and camera control, but language-guided 3D methods remain focused mainly on object-centric attention rather than compositional framing.
- 3D Gaussian Representation: 3DGS represents scenes with anisotropic Gaussian primitives, combining real-time rasterization and high visual fidelity for novel-view synthesis.The representation has also been applied to tasks such as 3D segmentation and scene editing.
- Vision-Language Understanding: Vision-language methods extend image-text semantics into 3D Gaussian primitives for open-vocabulary querying and amodal reasoning under occlusion.These approaches remain largely object-centric and focus on what to attend to rather than how to frame it.
- Camera Control: Camera-control research spans mathematical and rule-based cinematic systems as well as learning-based language-conditioned camera motion and trajectory generation.Examples include conversational camera navigation and synthesized cinematic trajectories.
3 Preliminaries: 3D Gaussian Splatting
3DGS models scenes with anisotropic Gaussians, projects them differentiably into image space, and composites their colors for rendering. This differentiability enables gradient-based optimization of 6-DoF camera poses.
- 3D Gaussian Representation: Each 3DGS primitive is an anisotropic Gaussian defined by a 3D mean and covariance.The Gaussian representation provides the scene primitives used for rendering.
- Differentiable Rendering: A 6-DoF camera pose projects each Gaussian into image space using perspective projection, camera rotation, and the projection Jacobian.The projected mean and covariance determine the Gaussian’s image-plane footprint.
- Differentiable Rendering: Pixel colors are rendered by alpha compositing the colors and opacities of overlapping projected Gaussians.The compositing order accumulates each Gaussian’s contribution while attenuating later contributions by preceding opacity.
- Pose Optimization: Differentiable rasterization propagates gradients to the camera pose, enabling gradient-based optimization over 6-DoF camera configurations.The pose-dependent derivative is defined on the SE(3) manifold.
4 Text-Instructed Viewpoint Grounding in 3DGS
TIVG asks for a camera pose whose rendered 3DGS image satisfies a natural-language viewpoint description. Because descriptions can admit multiple valid viewpoints, CapFrame combines retrieved views, geometric pseudo labels, and differentiable refinement.
- Task Definition: TIVG formalizes viewpoint grounding as identifying a camera pose that renders an image aligned with a natural-language instruction.The pose comprises rotation and translation and determines differentiable scene rasterization.
- Task Definition: The target pose must satisfy both subject orientation and frame layout, not merely localize the described object.A teddy-bear example requires a left-facing view and right-side placement.
- CapFrame Pipeline: CapFrame retrieves and ranks relevant training views, translates language into orientation and layout pseudo labels, and refines the pose through differentiable optimization.The pipeline uses an MLLM for Question-Evaluation ranking and pseudo-label construction before optimizing in the 3D Gaussian scene.
- Evaluation: TIVG generally has no unique solution, so evaluation uses external VLM similarity, MLLM alignment ratings, and perceptual user studies.Multiple viewpoints may satisfy the same compositional description.
5 Method
CapFrame uses Retrieve–Translate–Refine to turn compositional language into geometric camera constraints and optimize a 6-DoF viewpoint in 3DGS. It combines MLLM-ranked initialization, subject-centric pseudo labels, and differentiable layout and orientation losses.
- Retrieve: CapFrame retrieves and ranks semantic anchor views, then uses their poses to initialize continuous camera optimization.Global FG-CLIP filtering reduces candidates, while MLLM-based Question-Evaluation ranking reflects compositional alignment and provides a high-quality starting viewpoint.
- Retrieve: Question-Evaluation decomposes instructions into attribute-specific questions and averages MLLM scores across candidate views.Questions target requirements such as subject visibility and frame position, addressing failures of holistic VLM similarity in cluttered scenes.
- Translate: CapFrame localizes mentioned and visually dominant subjects to Gaussian subsets, enabling subject-centric reasoning and differentiable mask construction.Segmentation masks from retrieved views are back-projected with 3DGS depth, then matched to nearby Gaussian means using KNN search.
- Translate: Orientation pseudo labels combine text-derived azimuth, elevation, and roll offsets with estimated subject orientation for asymmetric subjects.Symmetric objects rely on layout constraints, while the resulting world-coordinate targets act as differentiable orientation regularizers.
- Translate: Layout pseudo labels represent photographic intent as normalized image-coordinate bounding boxes, including placement and close-up constraints.If roll is specified, bounding-box corners are rotated around their center to form the final target.
- Refine: Refinement updates the camera pose in se(3) by minimizing layout and orientation losses computed from differentiable subject masks.The layout loss uses soft centroids and additionally penalizes insufficient coverage inside, or leakage outside, target regions.
6 Results
Across 38 real-world scenes and 135 instructions, CapFrame aligns rendered viewpoints with compositional text requirements using retrieval, MLLM evaluation, and camera-pose refinement. Its outputs outperform heuristic searches and adapted trajectory methods across VLM, MLLM, and user-study evaluations, while component analyses explain the roles of retrieval, geometric losses, and refinement.
- Evaluation Setup: CapFrame is evaluated on 38 indoor and outdoor scenes with 135 manually curated text descriptions.The scenes come from five real-world 3D reconstruction datasets, with 3–4 instructions per scene.
- Evaluation Setup: The evaluation combines CLIP and SigLIP2 text-image similarity, MLLM alignment scores and win rates, and user-study ratings and preferences.Two VLMs and two blind MLLM judges are used because the task lacks a unique ground-truth viewpoint.
- Main Results: CapFrame produces frames better aligned with subjects, composition, and viewpoint than IVS, SVS, and adapted trajectory-generation baselines.Qualitative comparisons show that competing methods often localize visible subjects but miss specified placement or orientation.
- Main Results: CapFrame achieves the highest VLM-based text-image alignment, leading both MLLM judges in alignment score and win rate and receiving stronger user preference than IVS and SVS.Table 1 reports VLM similarity, MLLM alignment and win-rate measures, and user-study outcomes.
- Component Analysis: Question-Evaluation retrieval improves candidate-view initialization by assessing multiple semantic aspects instead of relying on coarse similarity or a single holistic score.This is especially useful in cluttered scenes, where coarse retrieval may return semantically related views lacking the target subject.
- Component Analysis: Layout and orientation losses jointly enforce subject placement and viewing direction, while refinement adjusts poses continuously in SE(3) beyond the retrieved views.Removing either loss degrades the corresponding requirement, and refined views consistently score higher than retrieved initial views on SigLIP2.
- Sensitivity Analysis: CapFrame remains close to its unperturbed output under moderate pseudo-label noise, while lower-ranked initialization gradually reduces alignment scores.Removing layout labels increases translation deviation, whereas removing orientation labels increases rotation deviation.
7 Conclusion
CapFrame translates natural-language instructions into geometric pseudo labels that guide differentiable camera-pose refinement in 3DGS, producing viewpoints aligned with compositional requirements. Its effectiveness depends on retrieval initialization and MLLM prompting, which can affect pseudo labels and camera placement.
- CapFrame translates natural-language instructions into orientation and layout pseudo labels that guide camera-pose refinement through differentiable optimization in 3DGS.
- The framework addresses compositional viewpoint grounding by specifying both how subjects should appear and where they should be placed in the rendered frame.
- Ablations show that removing orientation loss prevents the bear from achieving a front view, while removing layout loss prevents right-side placement.
- CapFrame relies heavily on MLLMs for retrieval and translation, making pseudo labels and camera placement sensitive to the prompts used.
Gaussian Scenes via Geometric Pseudo Labels
The supplementary material documents implementation details, pseudo-label visualizations, runtime analysis, extended comparisons, failure cases, MLLM prompts, and the user-study questionnaire.
- Section A covers additional implementation details, including IVS, SVS, and gravity-direction estimation.
- Section B visualizes pseudo labels produced during the Translate stage.
- Section C analyzes runtime for the Retrieve, Translate, and Refine stages and presents an acceleration strategy.
- Sections D and E provide extended viewpoint-grounding results, component analysis, comparisons beyond source-view initialization, and failure-case analysis.
- Sections F and G document the MLLM prompts and the user-study questionnaire.
A.1 Details of IVS and SVS
IVS and SVS adapt interpolation and bounded sampling strategies for text-conditioned viewpoint search, while CapFrame translates instructions into geometric constraints for camera optimization.
- IVS (Interpolation-based Viewpoint Search): IVS selects the top-L training images aligned with the instruction using FG-CLIP, then interpolates their camera poses to generate candidates.L = min(13, N), where N is the number of training images.
- IVS (Interpolation-based Viewpoint Search): IVS generates three intermediate poses for every pair among the top-L views, interpolating rotations with SLERP and translations linearly.
- SVS (Sampling-based Viewpoint Search): SVS samples camera directions around a sphere enclosing instruction-relevant subject Gaussians, with distances randomized within [1.1d, 1.6d].Elevations are restricted to [−70°, 90°] to avoid viewpoints from below the scene.
- Candidate selection: For both baselines, the final pose is the candidate with the highest VLM text-image alignment score.
- Comparison: IVS is effective when aligned training viewpoints exist, whereas SVS better handles coarse layouts without specific orientations or detailed compositions.
- CapFrame pseudo labels: CapFrame’s Translate stage extracts subjects, associates them with 3D Gaussians, and produces normalized 2D layout boxes and orientation cues for refinement.Orientation labels are used for asymmetric subjects when the instruction specifies a viewing direction.
C Computational Time
CapFrame’s computational cost is distributed across retrieval, translation, and refinement, with runtime varying according to scene and optimization conditions. Across the evaluated scenes, the average query time is 102.5 seconds.
- Retrieve: The Retrieve stage is dominated by similarity search over training images because the number of top-K views is fixed at K = 2.
- Translate: Translate runtime scales mainly with the number of parsed subjects because extraction and Gaussian association are performed per subject.
- Refine: Refine runtime depends largely on image resolution and the number of optimization iterations.
- Runtime interpretation: Reported runtimes are indicative estimates because early stopping and randomly selected initial views produce variable iteration counts.
- Overall runtime: 102.5 s average wall-clock time per query comprises Retrieve (33.7 s), Translate (25.3 s), and Refine (43.5 s) across 38 scenes.
D.1 More Qualitative Results
CapFrame grounds free-form instructions to viewpoints satisfying composition and viewing-angle constraints, outperforming heuristic baselines in challenging and weakly covered viewpoints. Component analyses show that MLLM-based question evaluation, joint losses, and continuous pose refinement support this behavior, while subject-processing errors remain failure sources.
- CapFrame produces viewpoints satisfying both composition and viewing-angle constraints from free-form text instructions.
- CapFrame more reliably satisfies joint layout and orientation constraints than IVS and SVS, particularly for viewpoints weakly covered by training cameras.
- Initialization ablation: Question-Evaluation improves initialization by distinguishing visually similar subjects and directing attention to photographic attributes.It aggregates MLLM scores over questions addressing visibility, layout, orientation, and text-image alignment.
- Loss ablation: Using both layout and orientation losses is necessary to jointly control subject placement, scale, and viewing direction.Removing either loss causes failures in position or orientation, including incorrect profiles and viewing angles.
- Beyond source-view initialization: Continuous SE(3) optimization discovers viewpoints absent from the input images, including a refined pose with ∆R = 57.76° and ∆t = 4.285.CapFrame only requires an initialization where the target subject is visible, not a final camera near the source view.
- Failure cases: CapFrame fails when subject localization selects the wrong object or subject parsing adds irrelevant objects that impose noisy layout constraints.These errors produce inaccurate supervision and can lead to suboptimal viewpoints.