Unifying Video Tasks via Spatiotemporal Analogy

1 Introduction

• Visual analogy infers a mapping from a source-target image pair and applies it to a query, but this capability has remained largely confined to still images.Video tasks add demands for spatiotemporal consistency and complex motion understanding.Source: §1 · p. 1
• VIGEO treats video analogy as spatiotemporal canvas completion, arranging the demonstration input, target, and query in one grid and generating the missing query-target quadrant from visual context.The formulation avoids auxiliary task heads and textual descriptions.Source: §1 · pp. 1, 3
• The evaluation uses a structured taxonomy and strict task-level train-test split to measure transfer to unseen tasks rather than memorization of trained task formats.Camera control is evaluated against dedicated baselines, while modal-guided generation and video manipulation use held-out tasks.Source: §1 · pp. 2–3
• VIGEO matches or is competitive with dedicated baselines on camera control and unseen modalities, and correctly performs the demonstrated transformation on 7 of 12 held-out video manipulation tasks.The held-out modality results include event-camera data.Source: §1 · p. 3
• Task internalization occurs when a familiar query format overrides the demonstration; a small amount of task-unrelated data removes the shortcut and restores demonstration-following in both held-out directions.The shortcut arose because a training-corpus format paired a frozen RGB query video with an RGB target.Source: §1 · p. 3

2 Related Work

• Visual (image) analogies infer a transformation from a source-target image pair and apply it to a query; Image Analogies introduced this framing.Visual Prompting later reformulated the idea as an inpainting/completion pretext task [Bar et al., 2022].Source: §2 · p. 3
• Prior visual-prompting methods are described as limited by object-centric or low-level scope, auxiliary text prompts, or evaluation on training tasks.The passage notes exceptions to the training-task evaluation pattern, including [Li et al., 2025].Source: §2 · p. 3
• Video one-shot learning has targeted specific domains such as camera pose, motion, and editing, often with task-specific architectural changes and objectives [Luo et al., 2025, Mitchel et al., 2025, Sun et al., 2024, Zhang et al., 2025b].VIGEO is presented as a generic framework based on visual analogies.Source: §2 · p. 4

3.1 Canvas Setup

• ViGeo arranges demonstration input, demonstration target, query input, and query target in a 2 × 2 video canvas, with the query target as the completion.The demonstration occupies the top row and the query the bottom row.Source: §3.1 · p. 4

3.2 Task Design

• The pixel-video canvas unifies camera control, modality-guided generation, and video manipulation, including diverse visual representations and task families.Training spans over 20 task families from complementary data sources, without task-specific text.Source: §3.2 · pp. 4–5
• A strict task-level split holds evaluation tasks and their source videos out of training to test in-context transfer across unseen modalities, operations, and environments.The held-out axes include event-camera guidance, structurally distinct manipulations, and camera views in unseen buildings and rooms.Source: §3.2 · p. 5

3.3 Learning Objective

• The model predicts the target region’s velocity from a composite latent containing the canvas context and a noisy state for the masked query target.The target data is the encoded query target; training uses the mean squared error between predicted and true velocity.Source: §3.3 · p. 5

3.4 Training Setup

• Training fully fine-tunes Wan2.2-TI2V-5B for 10K steps at a learning rate of 0.00005 and batch size 512.Videos are resized to 480 × 480 with length 81; training uses 32 NVIDIA A100 nodes.Source: §3.4 · p. 5

4.1 Modal Guided Generation

• On unseen flow-guided generation, VIGEO achieves the lowest optical flow EPE and highest PSNR, SSIM, and LPIPS, while point-guided generation remains competitive with task-specific baselines.For event-guided interpolation, VIGEO achieves the best LPIPS, but trails specialized methods on some pixel-fidelity and motion-accuracy metrics.Source: §4.1 · p. 6, Table 2 · p. 6

4.3 Video Manipulation

• 7 of 12 held-out video manipulation tasks generalize, while deblurring and geometric flipping remain challenging and default to copying the query video.VIGEO handles spatial permutations and temporal interpolation, and improves over the copy baseline on 3D outpainting despite lower accuracy than inpainting.Source: Figure 2 · p. 7, §4.3 · pp. 7–8

4.2 Camera Control

• On camera_motion_transfer, VIGEO is competitive with task-specific baselines and outperforms CamCloneMaster on rotation and overall camera error; on camera_view_transfer, it has lower camera errors than XFactor on held-out environments.The camera_view_transfer comparison may be unfair because VIGEO training includes the dual-camera setup, unlike XFactor training.Source: Table 3 · p. 6, §4.2 · p. 7, Table 4 · p. 7

5.1 Task Internalization: The Interplay Between Familiar Queries and Novel Demonstrations

• Repeating the first RGB frame across the query makes outputs copy RGB, but a single RGB hint or no hint yields the demanded modality.For depth-to-flow, repeated-frame correlation with RGB is 0.85, compared with near-zero correlations under the other prompts.Source: §5.1 · p. 8, Figure 4 · p. 9
• The authors term this task internalization: a familiar 2×3 canvas with a frozen RGB query predicts an RGB target in training, and the matching query template can override the demonstration.They hypothesize that the format–target shortcut accounts for the behavior.Source: §5.1 · p. 9

5.2 Mitigating Task Internalization

• Finetuning with a normal-to-edge dataset comprising 1.1% of the original corpus for 750 steps removes the repeated-RGB-hint correlation spike in both held-out tasks.Under that prompt, correlations fall from 0.85 to 0.04 for depth-to-flow and from 0.81 to −0.06 for flow-to-depth.Source: §5.2 · p. 9, Figure 4 · p. 9

6 Conclusion

• VIGEO uses one spatiotemporal canvas completion objective across video operations, and the conclusion reports competitive or matching performance on unseen event-camera modalities and success on 7 of 12 held-out manipulations.The paper frames task internalization, where familiar query formats override demonstrations, as an insight for designing spatiotemporal prompting.Source: §6 · p. 9

A.1 Visualization

• Qualitative rollouts illustrate depth-flow dual estimation, flow-guided generation, and camera view transfer, with the authors noting spatiotemporal consistency across modalities.Source: §A.1 · p. 14

A.2 Camera Control

• For camera view transfer, VIGEO obtains superior performance to XFactor in the reported reconstruction and covisible metrics.The covisible metrics evaluate pixels visible from both query-input and query-target poses.Source: §A.2 · p. 15, Table 5 · p. 15
• When demo and query videos come from different environments, performance worsens; the authors link this boundary to training that lacks such scenarios and may encourage reliance on both environments' context.They suggest synthetic demos with matching relative pose differences as a future step toward pose and context disentanglement.Source: §A.2 · p. 15, Table 6 · p. 15

• Flow-to-depth translation shows a task-internalization pattern similar to the depth-to-flow experiment.Source: §A.3 · p. 15
• Query formats resembling training inputs trigger task internalization, in which format overrides the demonstration.Source: Figure 6 · p. 15

A.4 Failure Mode Analysis of XFactor

• The cross-domain test varies the source of the demo transformation and the domain of the query to distinguish motion effects from scene appearance.Source: §A.4 · p. 16
• The authors attribute failure to training-test rotation differences; the robotic demo combines roughly 80° yaw, 75° pitch, and 30° roll.Source: §A.4 · p. 16
• XFactor transfers outdoor demo motions across outdoor and indoor queries, but collapses on robotic-camera demo motions.Source: §A.4 · p. 16, Figure 7 · p. 17

B.1 Additional result for task internalization

• 62% and 68% error reductions bring depth-to-flow and flow-to-depth errors to 0.142 and 0.108, respectively, after finetuning.The repeated first-frame hint is used; the finetuning data contains neither depth nor optical flow, and both directions are held out.Source: Table 7 · p. 17, §B.1 · p. 17

B.2 Seeing through the style: Perceptual Invariance

• ViGeo maintains robust generation with unseen Inferno or Turbo depth maps, black-background flow, and white-background edge maps.Source: §B.2 · p. 18
• The authors hypothesize that the model separates high-frequency spatiotemporal structure from low-frequency color styling when interpreting guidance.Source: §B.2 · p. 18

C.1.1 Training tasks.

• Camera-motion transfer teaches a demonstrated trajectory across scenes, while camera reangling transfers an A-to-B view correspondence to another scene.Source: §C.1.1 · p. 19

C.1.2 Evaluation tasks.

• Camera-view-transfer evaluation holds out physical environments, while camera-motion-transfer evaluation uses DL3DV subsets.Source: §C.1.2 · p. 19

C.2 Multimodal

• The multimodal task taxonomy covers 20 training tasks and 3 testing tasks.Source: §C.2 · p. 19

• SpatialVID tasks estimate depth, surface normals, and optical flow from RGB videos, with demo and query drawn from different scenes.Source: §C.2.1 · p. 19
• Guided-generation tasks synthesize RGB video from a frozen first frame and modality video, including depth-, normal-, and edge-guided variants.Source: §C.2.1 · p. 20
• The multimodal quad task jointly reproduces RGB, depth, normal, and flow within a nested grid for each scene.Source: §C.2.1 · p. 20
• Point estimation tracks 5, 10, 20, or 40 selected points across RGB video using semantic-aware filtering.Source: §C.2.1 · p. 20
• Video-manipulation tasks use independently sampled demo and query degradation parameters; held-out tasks test new operation families or degradations.Source: Table 9 · p. 21
• Hypersim supplies indoor-scene modalities for estimation and completion tasks, including reconstructing video from objects visible in only one keyframe.Source: §C.2.1 · pp. 20–21
• SA-V training uses masklet annotations and varied first-frame mask hints to predict mask overlays across video frames.Source: §C.2.1 · pp. 19, 21

C.2.2 Evaluation tasks.

• Held-out RGB-generation tasks use optical flow, moving points, and event-camera visualizations as unseen guidance; comparisons include flow-generation and event-interpolation baselines.The event-camera setup provides two boundary frames and blanks the 79 intermediate frames.Source: §C.2.2 · pp. 21–22

C.3 Video Manipulation

• The video-manipulation taxonomy includes three task categories and nine tasks, spanning training and testing tasks.The taxonomy is summarized in Table 9.Source: §C.3 · p. 22, §C.3.1 · p. 22

C.3.1 Training tasks.

• Zoom training tasks cross zoom-in and zoom-out with static and animated variants, using paired wide and close-up views to convey the transformation.Animated variants synthesize a dolly-zoom by sweeping the crop between the nested regions.Source: §C.3.1 · p. 22
• Other training tasks use shared time-varying image parameters or spatial and temporal permutations that the model must infer, restore, or apply from the demonstration.The permutation tasks include grid warping, strip shuffling, and temporal shuffling.Source: §C.3.1 · pp. 22–23

C.3.2 Evaluation tasks.

• Manipulation evaluation uses 55 DL3DV scenes and 15 tasks, with independently sampled degradation parameters in paired degraded-input and clean-target rows.The tasks include 3D inpainting and outpainting, plus spatial, motion, and temporal deblurring.Source: §C.3.2 · p. 23
• Further evaluation tasks test interpolation from 8 or 16 visible frames, 2D-tile jigsaw restoration or application, low-light enhancement, and temporal or horizontal flipping.Jigsaw tasks provide an out-of-distribution counterpart to the training strip-shuffle tasks.Source: §C.3.2 · p. 23

C.4 Camera Pose Metrics

• Camera-pose evaluation reports rotation error as summed geodesic distances between predicted and ground-truth rotations, translation error as summed ℓ2 distances after scale normalization, and camera-motion error as summed Frobenius norms of 3×4 pose-matrix differences.Normalization is applied independently to the ground-truth and predicted trajectories, making the metrics scale-invariant.Source: §C.4 · p. 24
• Camera-view transfer evaluates a two-frame relative pose, whereas camera-motion transfer evaluates an 81-frame trajectory; their summed metric values are not directly comparable.VGGT extracts predicted and ground-truth extrinsics for these evaluations.Source: §C.4 · p. 24

D Limitations

• The model struggles on some test tasks, including flipping and deblurring, and task internalization bounds its generalization.The discussion characterizes these as current limitations of the video-analogy framework.Source: D Limitations · p. 24
• The current model relies entirely on visual prompts; text-based task and video prompts remain a possible extension.The authors suggest investigating whether text conditioning could improve performance and generative quality.Source: D Limitations · p. 24
• Training-task diversity remains bounded, and the authors identify broader task variety and visual appearances as steps toward greater robustness and generalizability.Source: D Limitations · p. 24
• The approach has not yet been extended to other complex data structures, such as spectrograms, point clouds, or volumetric images.The authors list these modalities as directions for future research.Source: D Limitations · p. 24