Source-linked AI summary
In-Video Instructions: Visual Signals as Generative Control
Gongfan Fang, Xinyin Ma, Xinchao Wang
TL;DR
The paper asks whether video models’ visual capabilities can also provide control for image-to-video generation, rather than relying only on coarse textual prompts. It embeds commands and spatial markers in the input frame, and reports that models reliably interpret these instructions, particularly in complex multi-object scenes.
Problem
Text prompts provide coarse descriptions, motivating investigation of whether visual signals can control image-to-video generation with spatially grounded guidance.
Source: §1 · p. 2Method
The method places short commands and arrows in the input frame to associate intended behaviors with targets, without retraining.
Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3Results
Experiments across several generators report more reliable instruction binding to the correct subjects, especially in multi-object or cluttered scenes.
Source: §1 · p. 2, §4.3 · p. 8Takeaways & Limitations
The results support using spatially placed visual signals as a flexible way to guide distinct object behaviors in complex scenes.
Source: §1 · p. 2, §4.3 · p. 8The analysis is largely qualitative, instructions are manually constructed, and whether models interpret natural signals such as traffic lights remains open.
Source: §5 · pp. 8–9
Abstract
from arXiv · showhide
Large-scale video generative models have recently demonstrated strong visual capabilities, enabling the prediction of future frames that adhere to the logical and physical cues in the current observation. In this work, we investigate whether such capabilities can be harnessed for controllable image-to-video generation by interpreting visual signals embedded within the frames as instructions, a paradigm we term In-Video Instruction. In contrast to prompt-based control, which provides textual descriptions that are inherently global and coarse, In-Video Instruction encodes user guidance directly into the visual domain through elements such as overlaid text, arrows, or trajectories. This enables explicit, spatial-aware, and unambiguous correspondences between visual subjects and their intended actions by assigning distinct instructions to different objects. Extensive experiments on three state-of-the-art generators, including Veo 3.1, Kling 2.5, and Wan 2.2, show that video models can reliably interpret and execute such visually embedded instructions, particularly in complex multi-object scenarios.
1. Introduction
The paper asks whether video models’ visual understanding can also control image-to-video generation, and introduces instructions embedded in the first frame as a spatially grounded alternative to coarse text prompts. Across several models, these instructions are reported to improve spatial grounding, particularly when binding commands to subjects in complex scenes.
1. Introduction
- Large-scale video models can generate temporally coherent, logically consistent video from current-frame context, prompting the question of whether visual signals can also control generation zero-shot.Prior work has explored their visual abilities in perception, manipulation, puzzle solving, and mathematical reasoning.Source: §1 · p. 2
- In-Video Instruction embeds short commands and arrows in the visual input, using arrows to localize targets or indicate interaction direction.The approach is described as flexible and zero-shot, without retraining, and supports independent or sequential instructions for different objects.Source: §1 · p. 2
- In experiments across several video generators, In-Video Instructions offer a clear spatial-grounding advantage, especially for binding commands to the correct subjects in multi-object or cluttered scenes.The evaluation examines text comprehension, subject localization, object and camera motion, and multiple sequential or independent instructions.Source: §1 · p. 2
- The paper presents the paradigm as a direct, flexible interface for expressing intent visually and enabling spatially aligned control.It frames instruction interpretation as part of the video model’s handling of the visual input.Source: §1 · p. 2
2. Related Works
Related work describes video models’ emerging reasoning abilities and the development of controllable video generation. Prior control methods range from text-only conditioning to visual and multimodal conditions, including frameworks that combine multiple conditions.
Video Models as Zero-shot Reasoner
- Video-generation models have been studied as zero-shot reasoners for visual puzzles, mathematical reasoning, manipulation, autonomous driving, and medical applications.Related work also reports that models can understand textual and symbolic information embedded in videos [29] [36].Source: §2 · p. 2
Controllable Video Generation
- Controllable video generation has moved from text-only conditioning toward visual conditions such as initial frames, depth maps, edges, boxes, trajectories, 3D conditions, sketches, and motion.The cited work describes text-only conditioning as often unable to convey complex spatiotemporal semantics.Source: §2 · pp. 2–3
- Emerging multi-condition frameworks jointly use visual, spatial, and temporal information for compositional reasoning and creative flexibility.These approaches extend beyond control by a single condition.Source: §2 · p. 3
3. In-Video Instructions
In-Video Instruction places commands and spatial markers in the initial frame so a pretrained video model can interpret them alongside the scene. The method uses a standard image-to-video inference pipeline without retraining or architectural changes.
3. In-Video Instructions
- In-Video Instruction places commands inside the frame to establish explicit correspondences between visual subjects and their instructions.The pretrained model interprets the annotations together with the scene for spatially aware motion and interaction control.Source: §3 · p. 3
3.1. Embedding Instructions into Video Frames
- The method uses short textual commands to specify behavior and arrows to localize targets or indicate direction or region of influence.Multiple commands can correspond to one or more subjects.Source: §3.1 · p. 3
- Commands may sit inside a target region or externally, with arrows strengthening the spatial link to a subject or region.The instruction placement is flexible.Source: §3.1 · p. 3
- Instructions on the first frame provide spatial grounding for their scope and support independent or sequential commands for different targets.The figure caption describes the controls as less ambiguous and identifies in-frame visual signals as the control source.Source: Figure 2 · p. 3
- Explicit ordering, such as numbered labels, supports multi-step instructions.The example labels commands as “1. Instruction #1” and “2. Instruction #2.”Source: §3.1 · p. 3
3.2. Generation Procedure
- Generation follows a pretrained model’s standard image-to-video pipeline, with the annotated frame as the initial conditioning frame.A single global prompt reinforces adherence to the visual instructions.Source: §3.2 · p. 3
- The prompt is optional, no fine-tuning or architectural modification is used, and only the first frame contains instructions.Subsequent frames are synthesized freely as the model propagates intended motion, pose change, or interaction over time.Source: §3.2 · p. 3
4. Experimental Results
The experiments assess In-Video Instruction across commercial and open-source generators, progressing from text comprehension and localization to motion and complex multi-instruction control.
4.1. Text Understanding and Locating
- On VBench, In-Video Instruction performs slightly below but remains close to direct text inputs when models interpret and follow embedded commands.The commands are embedded above the image, and the caption notes that following them remains more challenging than responding to text prompts.Source: §4.1 · p. 4, Table 1 · p. 5
- In multi-object scenes, spatially placed commands and visual markers align actions with specific subjects, while text-only prompts often struggle with spatial ambiguity.Figure 3 compares In-Video Instruction with text prompts using ChatGPT-generated descriptions such as “the N-th object from the left.”Source: Figure 3 · p. 4, §4.1 · p. 5
4.2. Motion Control
- For translation and rotation, arrows indicate motion direction and magnitude or rotational direction and pivot, supporting object-specific motion control.The elephant example assigns different rotation instructions to different objects, producing independent yet coordinated rotations.Source: §4.2 · p. 6
- For trajectories and pose, models follow sketched paths with temporal consistency and generate smooth, coherent posture adjustments from localized visual and textual cues.The motion categories include trajectories and pose as well as translation and rotation.Source: §4.2 · p. 6
- Embedded text cues guide seven camera motions, although camera control is global and can also be handled well by conventional text prompts.The examples include static, pan left or right, tilt down or up, and zoom in or out.Source: §4.2 · p. 6, Figure 5 · p. 6
4.3. Multiple Objects and Instructions
- Across Figure 6 settings, visual grounding supports selective shared commands for multiple objects and ordered actions for one object, while text and visual instructions perform comparably on simple tasks.The examples include two birds following “fly away” while a third remains still, and a seal following three ordered commands.Source: §4.3 · pp. 7–8, Figure 6 · p. 7
- With multiple seed frames, the model combines spatial and temporal cues into coherent, continuous videos and maintains consistency across distinct visual contexts.The multiple-frame examples use Kling 2.5.Source: §4.3 · p. 8, Figure 7 · p. 8
- In the multiple-objects, multiple-instructions setting, embedded visual instructions achieve higher success rates than text-only prompts across diverse motion patterns.The human assessment generated 24 videos per method; for backing up the white car, success rates were 20.8% with In-Video Instruction and 8.3% with text prompting.Source: Table 2 · p. 8, §4.3 · p. 8
5. Limitations
The limitations include visible annotations that often need removal, largely qualitative analysis, and manually constructed instructions; interpreting natural visual signals remains future work.
5. Limitations
- Visible instructions persist in generated videos and often require post-processing for removal.The authors also observe that markers may become occluded during synthesis.Source: §5 · p. 8
- The analysis remains largely qualitative, and all instructions examined are manually constructed rather than naturally occurring signals.Whether models interpret and react to signals such as traffic lights or signboards remains open.Source: §5 · pp. 8–9
- Explicitly prompting models to remove visible annotations may further improve results.Source: §5 · p. 8
6. Conclusion
In-Video Instruction is a simple, training-free approach that embeds human intent in visual input for controllable video generation. Experiments on multiple models show that visual signals can be reliably interpreted as actionable guidance, supporting fine-grained, spatially grounded control across diverse tasks.
6. Conclusion
- Experiments on multiple models demonstrate reliable interpretation of visual signals as actionable guidance for controllable generation.Source: §6 · p. 9
- The training-free approach embeds human intent directly into visual input.Source: §6 · p. 9
- The method supports fine-grained, spatially grounded control across diverse tasks and complex scenes.Source: §6 · p. 9