Source-linked AI summary
VISTA: Differentiable Viewpoint Planning for Long-Horizon Task Visualization
Donggang Jia, Da Li, Zainab Alsuwaykit, Sebastian Hartwig, Dominik Engel, Timo Ropinski, Ivan Viola
TL;DR
Existing camera methods often target static objects, while long-horizon tasks require viewers to follow changing parts and relationships over time. VISTA optimizes adaptive viewpoints for task-conditioned targets and supports both narrative visualization and agent observation. Across evaluated appliances and state-based tasks, it improves perception and goal decisions, with higher closed-loop control success.
Problem
Existing camera-control methods commonly target static objects and may not maintain legibility of task-relevant components and evolving relationships through long-horizon executions.
Source: §1 · p. 1, §2.3 · p. 4Method
VISTA optimizes camera parameters for action-conditioned object–part or relational targets, balancing visibility, structural information, functional alignment, contextual focus, and temporal continuity.
Source: §1 · p. 2, §9 · p. 13Results
Across diverse appliances and state-based tasks, VISTA improves perception accuracy and goal decisions, with up to 2.1× higher success rates in closed-loop control.
Source: §1 · p. 2Takeaways & Limitations
The same planner supports human-facing narrative visualization and agent-facing active observation in embodied tasks.
Source: §1 · p. 2, §9 · p. 13VISTA does not yet model richer action semantics such as motion direction, contact geometry, affordances, or causal state transitions.
Source: §8 · p. 13
Abstract
from arXiv · showhide
Long-horizon object-centered tasks in interactive 3D environments involve heterogeneous events that unfold across time and space, including fine-grained manipulations, larger structural motions, and subtle state transitions. Effectively visualizing such processes is essential for helping users understand how actions lead to outcomes over time. However, existing camera control methods in 3D systems are commonly designed for static objects, and often fail to maintain the legibility of task-relevant components and their evolving relationships throughout long-horizon executions. We present VISTA, a differentiable viewpoint planning framework for long-horizon task visualization. We formulate camera control as a visualization policy that optimizes camera parameters under a unified rendering-based quality objective and we propose five quality metrics to satisfy our five design goals. To support both local part inspection and interaction-centric visualization centered on a primary object, our formulation handles targets defined either as object--part pairs or as relational tuples involving another interacting object. This allows the planner to reveal not only what part is important, but also how it functionally relates to surrounding structures during task execution. We demonstrate that the same planner supports two complementary use cases: presentation-driven narrative visualization for communicating how long-horizon tasks unfold to human viewers, and agent-driven active observation within an observe-act loop. Together, these results position differentiable viewpoint planning as a general mechanism for ``how-things-work'' visualization of long-horizon object-centered interactions in 3D environments.
1 INTRODUCTION
Long-horizon task visualization must communicate how actions change object states and relationships over time, not merely show clear views of a scene. VISTA plans adaptive viewpoints around task-relevant parts and interactions, and is evaluated for both human-facing visualization and agent observation.
1 INTRODUCTION
- Long-horizon object-centered tasks require following state changes, action dependencies, and spatially distributed interactions, while existing systems often rely on fixed, manually authored, or geometry-driven camera choices.The paper focuses on interactive 3D object-centered tasks rather than whole-scene navigation, social interaction, or full human-activity understanding.Source: §1 · p. 1
- VISTA formulates camera control as a visualization policy that organizes a coherent visual narrative and preserves task-relevant states, context, and action–outcome relationships.The framework optimizes camera parameters under a unified projection-based objective and considers the target part, parent structure, and action semantics.Source: §1 · p. 2
- Across diverse appliances and state-based tasks, the approach improves perception accuracy, goal decision, and closed-loop control, achieving up to 2.1× higher success rates.In a user study with 21 participants, the approach was rated significantly higher than fixed-camera and entropy-based baselines across all evaluation criteria and ranked first in 96% of tasks.Source: §1 · p. 2
- Action-conditioned targets use object–part pairs or relational tuples so different steps can emphasize manipulated parts, interacting entities, and contextual structures.This formulation supports local inspection while retaining interaction context.Source: §1 · p. 2
- Five differentiable image-space metrics jointly account for visibility, semantic relevance, and temporal coherence, optimizing task-relevant criteria over continuous camera parameters.The metrics translate task-conditioned visualization requirements into interpretable objectives.Source: §1 · p. 2
2 RELATED WORK
Prior viewpoint-selection work provides geometric, information-theoretic, learned, and dynamic-camera approaches, but generally does not plan task-conditioned viewpoints as a continuous policy for long-horizon explanations. VISTA instead unifies task-dependent visualization goals with temporally coherent viewpoint planning.
2.1 Heuristic View Quality Measures
- Heuristic measures use geometric cues or projected visibility distributions, but often optimize a single view or dominant cue and do not explicitly model action phases, interaction semantics, or temporal continuity.Viewpoint entropy favors balanced projected polygon visibility, while VMI compares current visibility with a reference distribution; neither is a long-horizon visualization policy.Source: §2.1 · p. 3
2.2 Learning-Based Viewpoint Selection
- Learning-based methods predict high-quality or preferred views and can improve view prediction, but remain focused on static best-view estimation, preference modeling, or system-level prefetching.The cited methods include model-based prediction and human-preference learning, while RmdnCache predicts viewpoints for data prefetching rather than informative viewpoint planning.Source: §2.2 · p. 3
2.3 Dynamic Camera Planning and Visibility-Aware Control
- Dynamic camera approaches optimize paths for time-varying data or visual explanations, yet typically target short sequences, predefined animations, or fixed objectives rather than a continuous differentiable policy over long-horizon tasks.VISTA formulates a unified optimization problem for task-dependent goals and temporally coherent viewpoints.Source: §2.3 · p. 4
3 DESIGN GOALS AND UNIFIED CAMERA QUALITY OBJECTIVE
VISTA plans a camera sequence around the changing explanatory needs of long-horizon object-centered tasks, using action-conditioned targets and explicit quality objectives. Its five metrics balance target visibility, structural and functional information, contextual emphasis, and continuity.
3.1 Action-Conditioned Visualization Targets
- Object–part targets support local manipulation inspection, while relational targets represent actions whose interpretation depends on another object's spatial or functional relation.These are the primary target cases considered, not an exhaustive taxonomy.Source: §3.1 · p. 4
- Existing whole-object viewpoint methods are poorly suited to task demands that shift between close-up part views and wider relational views across action phases.The paper also notes that discrete candidate search and training-data dependence limit prior approaches in different ways.Source: §3.1 · p. 4
- VISTA optimizes continuous camera parameters for explicit task-dependent criteria rather than selecting from a densely enumerated view set or relying on explanatory behavior learned from data.The camera policy generates a sequence whose targets and required context evolve with the task.Source: §3.1 · p. 4, §3 · p. 4
3.2 Design Goals and Viewpoint-Quality Metrics
- Five metrics operationalize viewpoint quality: visible coverage, patch entropy, functional alignment, context-aware focus, and temporal continuity.They respectively address target legibility, structural informativeness, meaningful camera orientation, task-weighted context, and compact transitions.Source: §3.2 · pp. 5–7
- Patch entropy uses six semantic directional surface groups, while functional alignment favors the target's functional front and an upright image orientation.The shared semantic frame supports both structural-diversity measurement and camera-alignment evaluation.Source: §3.2 · pp. 5–6
- Context-aware focus balances target and surrounding-entity prominence against task importance, while temporal continuity favors shorter transitions between consecutive camera positions.Temporal continuity is a path-length proxy and does not directly model velocity, acceleration, screen-space displacement, optical flow, or visual correspondence.Source: §3.2 · pp. 6–7
TIABLE RASTERIZATION
VISTA makes image-space viewpoint objectives differentiable with respect to camera parameters by replacing hard rendering quantities with soft rasterized attributes. It combines differentiable metric realizations into a motion-aware objective evaluated across sampled action states.
TIABLE RASTERIZATION
- Hard visibility masks and projected areas can remain unchanged under small camera perturbations and change abruptly at occlusion boundaries, preventing direct differentiation for three rendering-based metrics.Functional Alignment is analytic in camera orientation; Temporal Continuity depends on the transition path.Source: TIABLE RASTERIZATION · p. 7
4.1 Differentiable Camera-to-Image Mapping
- The framework replaces hard rendering quantities with soft image-space attributes so gradients from viewpoint-quality objectives propagate to continuous camera parameters.It builds on a standard differentiable rasterization pipeline rather than introducing a new renderer.Source: TIABLE RASTERIZATION · p. 7, §4.1 · p. 7
4.2 Soft Rasterized Attributes
- Differentiable antialiasing assigns fractional attributes near silhouette and visibility boundaries while preserving interpolated attributes inside triangles.Front-most-surface rasterization accounts for occlusion, and categorical attributes use one-hot vectors for soft per-pixel membership.Source: §4.2 · p. 7
4.3 Differentiable Realization of the Metrics
- Soft rasterized quantities realize visible coverage, patch entropy, and context-aware focus continuously, while functional alignment remains analytic and continuity uses a differentiable path-length approximation.Coverage is saturated to avoid overly zoomed-in views, and patch entropy favors balanced visibility of multiple target patches.Source: §4.3 · p. 8
4.4 Unified Objective and Optimization
- The unified objective multiplies the five metric terms, so severe failure on one objective cannot be fully offset by gains on unrelated terms.For motion-aware planning, four per-view metrics are averaged across sampled states within an action, while continuity is evaluated once between consecutive camera configurations.Source: §4.4 · p. 8
5 IMPLEMENTATION DETAILS
VISTA is implemented as an Isaac Sim extension and evaluated on articulated assets from appliance, vehicle, and simulator libraries. Its configurable optimization uses semantic directions, multi-start camera search, and motion sampling, with per-action runtime varying by asset.
5 IMPLEMENTATION DETAILS
- The Isaac Sim 4.5 extension uses mesh assets from RealAppliance, a Chevrolet C8 model, and the built-in Forklift, with level-of-detail meshes for selected complex target parts.The extension integrates with the simulator interface, rendering pipeline, and articulated-object control system.Source: §5 · p. 8
5.1 Semantic Direction Extraction
- A fixed GPT-5.4-2026-03-05 model identifies functionally meaningful front and top directions, which are cached for alignment and semantic patch objectives.Differentiable rasterization runs at 64 × 64 pixels using PyTorch and nvdiffrast.Source: §5.1 · p. 9
5.2 Camera Configuration
- Camera anchors are fixed per action: manipulated-part centers for object–part actions and parent-object centers for relational actions; independent panning and look-at offsets are not optimized.The manipulated object and, where applicable, destination remain separate visual targets.Source: §5.2 · p. 9
5.3 Differentiable Optimization
- Optimization evaluates 15 initial positions, refines the three lowest-loss candidates with Adam for at most 80 iterations, and samples three motion states including start and end.The default learning rate is 0.08, with early termination after five iterations without improvement greater than 10−6.Source: §5.3 · p. 9
5.4 Objective Configuration
- All five objective weights default to 1.0, and users can adjust weights, target importance, coverage, and optimization settings through the interface or configuration files.The default visible-coverage target is 0.2, described as an empirical balance between legibility and surrounding context.Source: §5.4 · p. 9
5.5 Runtime
- Per-action optimization takes approximately 5–10 seconds for RealAppliance and approximately 20 seconds for the Chevrolet C8 and Forklift models.Reported wall-clock times include multi-start evaluation and refinement but exclude offline LOD construction and cached semantic-face inference.Source: §5.5 · p. 9
6 USE CASE
The air-fryer example shows VISTA selecting views for different action-conditioned targets across a long action sequence. Local part manipulation and multi-object interaction require distinct visual compositions.
6 USE CASE
- For opening the air-fryer basket, the object–part target is the basket, and the view should show it clearly with essential surrounding context.The example sequence also includes cooking actions, knob rotations, and repeated basket and chicken interactions.Source: §6 · p. 9
- For moving chicken into the basket, the relational target includes both meshes, and the view should present the food, basket, and their relation.The air fryer remains the parent object providing context.Source: §6 · p. 9
- The example's metric visualizations distinguish target visibility, target-surface patch diversity, and context allocation across the target, parent, and surrounding parts.Patch entropy excludes parent occlusion, whereas context-aware focus compares observed pixel allocation with desired context importance using Jensen–Shannon divergence.Source: §6 · p. 9
7 EVALUATION
VISTA is evaluated through semantic-direction extraction, objective ablation, a user study, and a VLM perceptual benchmark. Results assess both human-facing trajectory quality and agent-facing observation across long-horizon appliance tasks.
7 EVALUATION
- Evaluation also demonstrates adaptive views on cooking actions, shifting between global appliance framing for coarse interactions and close views for fine operations.The four evaluation stages assess semantic-direction reliability, objective contributions, human judgments of trajectories, and VLM performance under different camera strategies.Source: §7 · pp. 9–10, Figure 4 · p. 10
7.1 VLM Semantic Direction Extraction Evaluation
- GPT-5.4-2026-03-05 achieved 92.65% front, 95.59% top, and 92.65% joint accuracy on 68 annotated instances; GPT-4o-2024-08-06 had lower front and joint accuracy.GPT-4o-2024-08-06 achieved 55.88% front, 91.18% top, and 55.88% joint accuracy; joint accuracy requires both directions to be correct for an instance.Source: §7.1 · p. 10
7.2 Qualitative Objective Ablation
- Removing visible coverage shrinks the basket in view; removing context-aware focus loses the balance between the part and appliance, while patch entropy and functional alignment shape surface exposure and orientation.The complete objective gives the most balanced composition; the ablation is a qualitative illustration, not a statistical test of metric importance or weight sensitivity.Source: §7.2 · p. 10
7.3 User Study
- In a 21-participant study on RealAppliance datasets, our method rated significantly higher than Fixed and Entropy across six criteria, scoring 4.53 (std = 0.88) versus 2.24 and 2.02, respectively.The corresponding baseline scores were 2.24 (std = 1.20) for Fixed and 2.02 (std = 1.13) for Entropy; participants ranked our method first in 96% of tasks (101 out of 105).Source: §7.3 · p. 11
7.4 VLM Perceptual Benchmark
- Across VLMs, the adaptive camera leads all three tasks, reaching 65.7% SR and 0.74 StateErr on T3, a 2.1× improvement in closed-loop success rate over static baselines.The benchmark reports T1 state-recognition accuracy and T2 goal-decision accuracy alongside Task 3 closed-loop control.Source: Table 1 · p. 12, §7.4 · pp. 12–13
- The benchmark isolates viewpoint-related perceptual difficulty from task reasoning using three single-image appliance-state tasks with minimal contextual prompts.This is motivated by closed-loop benchmarks that conflate perceptual quality with task reasoning and can be influenced by semantic priors.Source: §7.4 · pp. 11–12
- The adaptive camera averages 69.3% on T1 and 79.3% on T2, exceeding the listed alternatives on both tasks.On T1, the listed alternatives are Learning (54.7%), Fixed (50.7%), and Entropy (48.0%); on T2, Fixed and Learning tie at 54.7%, ahead of Entropy at 44.0%.Source: §7.4 · p. 12, Table 1 · p. 12
8 DISCUSSION
The discussion treats the five view-quality objectives as one operationalization among alternatives and identifies optimization, semantic, annotation, and sampling boundaries that qualify VISTA’s use.
8 DISCUSSION
- Alternative formulations can define visible coverage, structural information, functional alignment, and context using different supervision and optimization choices.Examples include image-space framing or task-recognition confidence, viewpoint or surface-information entropy, interaction geometry or affordances, and saliency alignment or composition rules.Source: §8 · p. 13
- Multi-start initialization and Adam refinement mitigate non-convex optimization, but do not guarantee a global optimum or recover targets occluded throughout an action.Narrow visible regions may be missed, and fully hidden targets can produce locally uninformative visibility gradients.Source: §8 · p. 13
- VISTA does not yet model richer action semantics such as motion direction, contact geometry, affordances, or causal state transitions.Some transformations may require views revealing an interaction axis, receiving region, or before-and-after state.Source: §8 · p. 13
- Geometric and semantic annotation errors can misdirect functional alignment or distort coverage and contextual focus, while an overly high coverage target can remove useful context or clip a moving target.Incorrect front or top directions are especially problematic for symmetric objects or parts without a clear functional front.Source: §8 · p. 13
- Sparse motion sampling can miss brief occlusions or contacts, and patch entropy may not reflect perceptual informativeness for uniform objects or fine structures.Thin structures, small functional features, and fine occlusion relationships may also be poorly represented by simplified geometry or finite rasterization resolution.Source: §8 · p. 13
9 CONCLUSION
VISTA plans differentiable viewpoints for object-centered long-horizon task visualization. It uses action-conditioned targets and balances five view-quality objectives, supporting both human-facing narratives and agent-facing active observation.
9 CONCLUSION
- VISTA plans differentiable viewpoints for object-centered long-horizon task visualization.Source: §9 · p. 13
- Action-conditioned object–part or relational targets are optimized to balance visible coverage, structural information, functional alignment, contextual focus, and temporal continuity.Source: §9 · p. 13
- The framework supports both human-facing narrative visualization and agent-facing active observation.Source: §9 · p. 13