PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual prompt engineering for video models

Robert Geirhos, Yuxuan Li, Thaddäus Wiedemer, Neha Kalibhat, Zi Wang, Mani Malek, Oyvind Tafjord, Kevin Swersky, Been Kim, Priyank Jaini

arXiv:2607.25537v1cs.CVcs.AI

TL;DR

Video models use visual prompts for reasoning, but prior work largely treated those inputs as fixed. This paper introduces VIPE, which edits task images while preserving task logic, and finds systematic reasoning improvements across tasks, including benefits from realism and potential advantages over text optimization and test-time scaling.

  • Problem

    The paper asks whether transforming video models’ visual task inputs can improve visual reasoning, addressing the prior treatment of visual prompts as fixed.

  • Method

    VIPE uses ideation and image editing to replace an original task image with a task-preserving visual variant, optionally selecting among variants with quality filtering.

  • Results

    VIPE systematically improves video reasoning across tasks and can be more effective than text-prompt optimization or test-time scaling in the reported comparisons.

  • Takeaways & Limitations

    Photorealistic visual contexts can better match video-model representations, so abstract benchmarks may underestimate underlying reasoning performance.

  • Takeaways & Limitations

    VIPE quality depends strongly on the image editor, which may alter the task or introduce artifacts; the pipeline can also add cost overhead.

Abstract

from arXiv · show

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.

1. Introduction

Prompt engineering improves language-model performance by optimizing inputs, motivating whether the same principle extends to visual inputs for video-based visual reasoning. VIPE addresses this question by transforming task images while preserving the underlying task.

  • Prompt engineering systematically improves language-model outputs by optimizing text inputs without modifying the model.
  • Video models use both text prompts and visual prompts, with the image providing visual context, spatial layout, objects, and constraints.
  • VIPE transforms a task image into a variant, such as a photorealistic scene, while keeping the underlying task identical.
  • The pipeline uses an ideator, an image editor, and an optional quality-control filter before video generation and pass/fail scoring.
  • The paper reports that VIPE consistently improves video reasoning, can be automated, and may outperform text optimization and test-time scaling.

2. Visual prompt engineering (VIPE)

VIPE replaces an original visual prompt with a task-preserving variant selected through ideation, image editing, and optional quality filtering. The stochastic pipeline can also generate multiple variants for ensembling.

  • VIPE defines each task sample as an image-text pair and replaces the original image with a variant without altering the task’s underlying logic.
  • Step 1: Ideator: An ideator uses an ideation prompt and task samples to describe a visual edit in natural language.
  • Step 1: Ideator: The ideation prompt constrains edits so core task elements remain unchanged.
  • Step 2: Editor: An image editor applies the edit instructions to each original image and samples m candidate variants.
  • Step 3: Filter (optional): An optional scoring function evaluates candidate quality and faithfulness, and the highest-scoring variant is selected.
  • Because the pipeline is stochastic, it can be repeated to obtain n variants for the same visual prompt, including for ensembling.

3. Text-based prompt engineering helps language models for language tasks. Does visual prompt engineering help video models for visual tasks?

The VPCT experiment tests whether changing a ball-and-ramp image from an abstract sketch to a realistic scene improves video-model physics reasoning. Across models, visual realism improves performance, while 3D appearance alone is insufficient.

  • VPCT asks models to predict which bucket a white ball reaches after rolling through ramps and can represent the same task as either a sketch or photorealistic scene.
  • The experiment uses image editing to transform VPCT sketches into realistic photographs while preserving camera perspective and task elements.
  • VIPE improves both open- and closed-source video-model performance across the evaluated conditions.
  • 41.3% to 59.3%: Veo 3.1 performance increases under visual prompt engineering.
  • Ablation: unnatural textures: Most models show no significant difference between sketches and unnatural-texture 3D inputs, indicating that 3D appearance alone is insufficient.

4. How can visual prompt engineering be automated?

VIPE can be automated through both freeform VLM ideation and structured step-by-step image edits, producing improved video reasoning across tasks. The approaches preserve task structure while changing visual presentation, and structured edits reveal recurring beneficial transformations.

  • How can visual prompt engineering be automated?: Automating VIPE addresses the vast search space of visual prompts when effective edits are not known in advance.The paper compares freeform ideation with step-by-step Atomic Concept Editing.
  • How can visual prompt engineering be automated?: The evaluation covers six visual reasoning tasks, including mazes, Connect the Dots, Sort 3 Numbers, Conjunctive Search, and RushHour.Experiments consistently use Veo 3.1 because video generation is expensive.
  • 4.1. Freeform ideation by a VLM: Freeform VLM ideation transfers tasks into new visual domains while preserving spatial layout, object correspondence, and task structure.The open-loop pipeline uses an ideator, image editor, and filtering model without downstream evaluation feedback.
  • 4.1. Freeform ideation by a VLM: Over 75% error reduction is achieved by some freeform VIPE variants compared with the baseline without visual prompt engineering.The variants improve Veo 3.1 performance simply by altering the visual context of reasoning.
  • 4.2. Step-by-step edits: Atomic Concept Editing (ACE): ACE explores a tree of atomic image edits, applying generated changes to child images and branching until enough successful nodes are found.Unlike freeform ideation, ACE operates in a closed loop using autorater feedback to adapt subsequent edits.
  • 4.2. Step-by-step edits: Atomic Concept Editing (ACE): 100% error reduction is reported for Sort 3 Numbers with ACE, while ACE otherwise produces systematic improvements comparable to freeform VIPE.ACE was evaluated only on Connect the Dots, Conjunctive Search, and Sort 3 Numbers because it is expensive.
  • 4.2. Step-by-step edits: Atomic Concept Editing (ACE): ACE edits commonly improve realism of material properties for temporal object permanence and isolate targets with high contrast.Examples include matte clay or textured chalkboard elements and solid dark backgrounds.
  • 4.2. Step-by-step edits: Atomic Concept Editing (ACE): VIPE automation does not require human intuition and can use either freeform VLM ideation or structured step-by-step concept edits.Both approaches are presented as successful instantiations built from standard components.

5. Can visual prompt engineering be used for test-time scaling?

VIPE functions as an effective test-time scaling strategy for video reasoning and can be combined with self-consistency. Across the tested settings, exploring visual variants is generally more cost-effective than generating more attempts from one prompt.

  • Can visual prompt engineering be used for test-time scaling?: Test-time scaling for video models has received limited attention compared with pre-training and post-training scaling.Prior approaches include scaling generated frames or selecting among multiple early-decoded videos.
  • Can visual prompt engineering be used for test-time scaling?: 59.3% accuracy is achieved with one engineered visual prompt on VPCT, compared with 50.0% from 20 self-consistency samples on the baseline prompt.The corresponding improvements are +18 percentage points and +8.7 percentage points from the 41.3% baseline.
  • Can visual prompt engineering be used for test-time scaling?: 68.0% accuracy is reached by combining engineered visual prompts with 20 test-time samples on VPCT.Self-consistency scales on engineered prompts as well as on original prompts, producing a compounded +27.7 percentage-point improvement.
  • Can visual prompt engineering be used for test-time scaling?: Automated VIPE can itself provide test-time scaling by combining multiple generated videos with multiple visual variants under a fixed budget.A VIPE variant currently costs only 1/8 of a video, even accounting for initial proposals and filtering.
  • Can visual prompt engineering be used for test-time scaling?: For a given budget, allocating resources to more VIPE variants is generally more cost-effective than generating more videos per variant.This pattern is reported for VPCT, with similar results for Maze and RushHour in the appendix.
  • Can visual prompt engineering be used for test-time scaling?: VIPE and additional video generation can be combined, with VIPE offering a more cost-effective test-time scaling strategy than simply generating more videos.The paper presents the two approaches as complementary rather than mutually exclusive.

6. Is language or visual prompt engineering more effective for video models?

The paper compares text and visual prompt engineering as two ways to influence video-model reasoning while keeping the task structure fixed. Both can improve performance, but visual prompting is more powerful on some tasks and can produce larger gains on average for others.

  • Is language or visual prompt engineering more effective for video models?: Both text and visual prompt engineering substantially improve video reasoning over the baseline across tasks.Each modality is treated as a viable lever for influencing model behavior.
  • Is language or visual prompt engineering more effective for video models?: On Sort 3 Numbers, the best text and image variants achieve 86% and 76% accuracy, versus 4% without prompt engineering.These correspond to roughly 20-fold improvements for each modality.
  • Is language or visual prompt engineering more effective for video models?: Image variants produce substantially larger average improvements than text variants on Conjunctive Search and Sort 3 Numbers.The single best text variant can nevertheless exceed the single best image variant on some tasks.
  • Is language or visual prompt engineering more effective for video models?: On VPCT, a notable image edit raises Veo 3.1 accuracy from 57% to 73%, although freeform text and image variants generally show little improvement.Connect the Dots remains challenging under diverse text and image variant proposals.
  • Is language or visual prompt engineering more effective for video models?: Depending on the task, visual prompt engineering can be more powerful than text prompt engineering.The comparison supports a task-dependent advantage rather than universal superiority of one modality.

7. Does visual prompt engineering help native image generation models, too?

VIPE does not systematically improve native image-generation model reasoning, unlike its reported benefits for video models. The authors hypothesize that VIPE helps when it bridges a representation gap that may be larger for video models.

  • VIPE does not systematically improve image model reasoning performance on VPCT.Native image-generation models were evaluated on both sketch and photorealistic VIPE datasets across multiple configurations.
  • The authors hypothesize that VIPE works by bridging a representation gap between the input format and the model’s internal representation.They suggest this gap may differ between video and image-generation models.
  • Video models may have a larger abstract-to-realistic representation gap than image-generation models.The paper proposes that native image models may be trained on more diverse styles, including sketches and realistic images.
  • Veo 3.1 achieves 41.3% sketch accuracy and 59.3% VIPE accuracy on VPCT, while Gemini 3.1 Pro achieves 96% on the sketch dataset.These values provide comparisons with video and text-only vision-language models.

8. Why does visual prompt engineering help?

VIPE appears to help primarily by moving abstract inputs toward realistic representations. Experiments show that increasing realism improves video scene consistency, reducing changes that can disrupt downstream reasoning.

  • VIPE’s effectiveness primarily stems from bridging the domain gap between abstract and realistic input.The paper links stronger performance to transformations that add both 3D structure and realistic textures.
  • Realistic scenes reduce random object or scene-element changes that can alter the task setup and impede downstream reasoning.The paper describes objects appearing, disappearing, changing shape, or morphing in abstract settings.
  • Each step toward increased realism makes video-model generations more consistent.The figure evaluates 100 videos per condition using human pass/fail judgments of scene consistency.

9. Discussion

The discussion presents VIPE as a cheap way to improve video reasoning, while showing that realism bias both enables gains and exposes a weakness in current models.

  • 9. Discussion: VIPE improves video reasoning across tasks and can outperform self-consistency and text-prompt optimization at comparable test-time budgets.The authors characterize visual prompt engineering as an effective, compute-efficient test-time strategy.
  • Realism bias: Photorealistic transformations improve accuracy, suggesting that abstract or synthetic benchmarks can underestimate video models’ underlying competence.The discussion frames this as a performance-versus-competence distinction and recommends realistic task presentations when feasible.
  • Realism bias: Realism bias is suboptimal because models should ideally solve the same task across different visual presentations.The authors relate this brittleness to unwanted shortcuts or dataset bias and leave the distributional explanation unresolved.
  • When does VIPE help?: VIPE helps video models by translating unfamiliar representations into familiar photorealistic scenes, whereas image-generation models covering both spaces gain little.This pattern suggests representation mismatch with the model’s training distribution rather than a universal property of visual models.
  • Limitations: A major limitation is that VIPE quality depends on the image editor preserving the task and avoiding artifacts.The authors mitigate this risk through filtering or quality control of image edits.
  • Outlook: The outlook treats visual instructions as an optimizable space and anticipates extending VIPE to multiple reference images or videos.This follows the analogy with optimized text prompts as models accept richer multimodal context.

A. Datasets

The evaluation draws on VPCT, several manually created visual reasoning tasks, maze and RushHour benchmarks, and inference through commercial model APIs.

  • VPCT: VPCT contains 100 samples; the study uses all samples in Section 3 and the first ten in Section 4.The dataset is distributed under an MIT license on HuggingFace.
  • Additional tasks: Conjunctive Search, Sort 3 Numbers, and Connect the Dots contain 5, 5, and 6 manually generated image samples, respectively.The tasks were drawn with permission from Wiedemer et al., and their distractors or colored circles vary by task.
  • Maze: Maze includes four splits: 5 × 5, 7 × 7, and 9 × 9 square grids with 50 samples each, plus 40 curved-path mazes.Section 4 reports results on the first ten samples of each split.
  • RushHour: RushHour has five 50-sample difficulty splits, but the analysis focuses on Levels 2 and 3 because Level 1 is easy and Levels 4–5 are too difficult.Difficulty corresponds to the number of cars required for a minimum-length solution.
  • Inference setup: Gemini, Veo, and Omni models are evaluated through the Vertex AI API with images and videos generated at 720p.Videos use an 8-second duration, while image-editing and video-generation costs differ substantially.

C. VPCT: evaluation details

VPCT evaluation combines trajectory-specific video evaluators, controls for shortcut behavior, and test-time scaling analyses to probe physics reasoning and VIPE effects.

  • Evaluation details: The original-sketch evaluator partitions each frame into three bottom regions and uses regional mean squared error over sliding five-value windows.A 12-frame warmup initializes the sliding-window averages.
  • Evaluation details: VIPE variants are evaluated by tracking the red ball through HSV color thresholding, contour extraction, and geometric filtering.This evaluator is used because visual prompt engineering introduces more variation than the original MSE setup.
  • Evaluation details: Removing or replacing buckets in VIPE images prevents a prompt rewriter from spelling out the solution and preserves the need to simulate the trajectory.Buckets are retained for human interpretability despite making the task somewhat harder to understand.
  • Test-time scaling: Test-time scaling estimates majority-vote accuracy for k = 1 through 20 independent runs on 100 three-class samples, with 33.3% chance accuracy.All subsets are enumerated when feasible; otherwise 5,000 random subsets are sampled, and ties are resolved uniformly.
  • Physics controls: VLM accuracy on original VPCT rose from below 50% in April 2025 to 91% for Gemini 3 Pro Preview and 96% for Gemini 3.1 Pro in the authors’ experiments.The resulting control experiments ask whether high original-task accuracy reflects robust physics reasoning.
  • Physics controls: Turning buckets upside down reduces model accuracy to 3–4%, indicating superficial physics reasoning despite high original-task performance.The models appear to use a shortcut rather than represent that balls cannot fall into upside-down buckets.
  • Physics controls: Gemini 3.1 Pro identifies whether buckets are upside down with 100% accuracy, showing that the failure is not merely visual misperception.The model accesses the relevant visual information but fails to combine it correctly for the physics task.
  • Prompt ablation: With the Wan prompt rewriter enabled, VIPE VPCT accuracy is 37.7% for TI2V, below 42.3% for TI2V English and 43.3% for TI2V Chinese.For this task, the model performs better without rewritten prompts.

E.1. Freeform ideation

Freeform VIPE automatically explores diverse visual variants, and these open-loop edits often improve video reasoning, though one variant may not generalize across task splits.

  • Automated freeform ideation: Sequential ideation appends prior edit instructions to later prompts, encouraging diverse visual variants.The automatic visual prompter iterates in sequence, carrying each iteration’s edits into the next ideation prompt.
  • Visual edits: Detailed base descriptions can constrain edits to subtle visual changes, yet even a low-contrast pixel texture can improve performance.The cited maze example changes only the scene’s screen-like pixel texture.
  • Limitations: Freeform ideation can struggle to find one variant effective across different maze sizes or RushHour levels.
  • Model scope: Freeform ideation with Wan2.2 TI2V also demonstrates effectiveness beyond Veo 3.1.
  • Ineffective edits: Several edits were unreliable: reflective textures degraded temporal consistency, while lighting, dark backgrounds, and distractor shrinking disrupted spatial perception or correspondence.
  • Successful edits: Individual saturated color coding achieved a 50% pass rate (3/6 videos) in one run, the only mutation to reach the success threshold.
  • Successful edits: Bold outlines reached a 25% pass rate in multiple runs, apparently sharpening circle boundaries for connectable endpoints.
  • Evaluation: Pass rate is used as a proxy for self-consistency, and VIPE can therefore provide a cost-effective test-time scaling strategy under a sequence-aggregation heuristic.

G. Image model comparison

Image-generation models do not uniformly benefit from VIPE: gains are modest or negative, while text mode reveals reasoning ability otherwise obscured by rendering demands.

  • VIPE effects: VIPE improves NB Pro image-mode Pass@1 by +5 pp over sketches, from 45% to 50%.
  • VIPE effects: VIPE decreases NB Flash image-mode Pass@1 by −16 pp, from 37% on sketches to 21%.Majority vote accuracy also falls from 48% to 32%.
  • Image versus text mode: +29 pp raises NB Flash Pass@1 from 21% to 50% when switching from VIPE image mode to text mode.The underlying model is the same; image mode additionally requires rendering the predicted final state.
  • Rendering bottleneck: Gemini 3.1 Pro reaches 96% accuracy on VPCT sketches, compared with 50% for NB Pro image mode.The passage attributes the gap to the additional rendering burden of expressing answers through generated images.
  • Best configuration: NB Flash in text mode on sketches achieves 58% Pass@1 and 80% majority vote accuracy, the best overall configuration.This outperforms NB Pro in image mode on VIPE inputs, which reaches 50% Pass@1 and 54% majority vote accuracy.
  • Autorater validation: The rubric-based autorater agrees with human ratings on 85.6% of 216 videos, rising to 93.8% after excluding debatable cases.Agreement is Cohen’s κ=0.711 across all videos and κ=0.874 for clear-cut cases.
  • Autorater validation: A static grid improves Maze autorater alignment from 0.25 to 0.58 Cohen’s Kappa but reduces RushHour alignment by −0.12 Cohen’s Kappa.The grid’s benefit is therefore task-dependent.

H.2. VPCT container-choice autoraters

The VPCT container-choice autorater combines frame captions with video and input-image reasoning to identify the first ball-container contact, with conservative human agreement.

  • Container-choice autorater: The autorater samples frames at 6 FPS, captions ball position and container interactions, then uses the input image, captions, video, and rubric to score the outcome.
  • Frame captioning: Frame captioning tracks ball location, container contact, container visibility, camera movement, and scene changes relative to the starting frame.
  • Decision rule: The target is the first moment the ball touches or enters any container; nearby floating without contact does not count.
  • Decision rule: The autorater reports left, center, right, or unknown when no contact occurs or the original scene becomes unavailable.Unknown also applies when the ball never reaches a container by the video’s end.
  • Human validation: Validation covers 100 videos drawn from 20 image variants and 20 text variants.
  • Human validation: The autorater is more conservative than human raters, marking unknown more often because of scene distortion.
Loading 2607.25537v1…