Source-linked AI summary
Video models are zero-shot learners and reasoners
Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, Robert Geirhos
TL;DR
The paper asks whether generative video models can become general-purpose vision models rather than remain task-specific systems. It probes Veo 3 by prompting it on diverse visual tasks and finds broad zero-shot abilities, including early visual reasoning, while performance remains below specialized models and depends on prompting.
Problem
Machine vision lacks a model that can solve arbitrary novel tasks through prompting, despite progress from task-specific models toward unified systems.
Method
The paper prompts Veo 3 with input images and text instructions, evaluating qualitative and quantitative tasks across perception, manipulation, and visual reasoning.
Results
Veo 3 solves a wide range of untrained tasks spanning perception, modeling, manipulation, and early chain-of-frames visual reasoning.
Takeaways & Limitations
The findings indicate that video models are developing toward unified, general-purpose foundation models for vision.
Takeaways & Limitations
Veo 3 often performs below specialized models, and measured performance is a lower bound affected by prompt and starting-frame choices.
Abstract
from arXiv · showhide
The remarkable zero-shot capabilities of Large Language Models (LLMs) have propelled natural language processing from task-specific models to unified, generalist foundation models. This transformation emerged from simple primitives: large, generative models trained on web-scale data. Curiously, the same primitives apply to today's generative video models. Could video models be on a trajectory towards general-purpose vision understanding, much like LLMs developed general-purpose language understanding? We demonstrate that Veo 3 can solve a broad variety of tasks it wasn't explicitly trained for: segmenting objects, detecting edges, editing images, understanding physical properties, recognizing object affordances, simulating tool use, and more. These abilities to perceive, model, and manipulate the visual world enable early forms of visual reasoning like maze and symmetry solving. Veo's emergent zero-shot capabilities indicate that video models are on a path to becoming unified, generalist vision foundation models.
1. Introduction
The paper asks whether generative video models can develop general-purpose vision understanding like LLMs developed general-purpose language understanding. It reports that Veo 3 solves diverse untrained tasks and shows early visual reasoning, while still trailing specialized models.
- Machine vision remains dominated by task-specific models, and no existing model can solve arbitrary problems through prompting.
- Large-scale generative training on web-scale data provides video models with primitives analogous to those behind zero-shot learning in NLP.
- Veo 3 was evaluated across 18,384 generated videos spanning 62 qualitative and 7 quantitative tasks it was neither trained nor adapted for.
- Veo 3 combines visual perception, world modeling, manipulation, and early chain-of-frames reasoning for tasks such as mazes and symmetry solving.
- Task-specific bespoke models still outperform Veo 3, but performance improves substantially and consistently from Veo 2 to Veo 3.
2. Methods
The method is intentionally minimalist: prompt Veo with an input image and text instruction, then inspect generated videos across tasks. The study compares Veo 3 with Veo 2 and uses qualitative success estimates plus a broad task overview.
- The core method is prompting Veo rather than task-specific training or adaptation.
- Figure 1 summarizes success rates across 12 samples as rough estimates over 62 tasks spanning the vision stack.
- Each task uses an initial image as the first frame and a text instruction, producing an 8-second 16:9 video at 720p and 24 FPS.
- The study compares Veo 3 with Veo 2 to assess how rapidly video-model performance is improving.
3. Qualitative results: Sparks of visual intelligence?
Veo 3 exhibits zero-shot abilities across perception, modeling, manipulation, and reasoning over space and time. The qualitative investigation finds broad but imperfect competence on tasks outside explicit training.
- Capability hierarchy: The qualitative hierarchy organizes capabilities as perception, modeling, manipulation, and reasoning across space and time.
- Evaluation: Each qualitative task is prompted 12 times, with success rate defined as the fraction of generated videos judged by the authors to solve it.
- Perception: Perception tasks include edge detection, segmentation, keypoint localization, superresolution, deblurring, denoising, enhancement, search, and ambiguous-image interpretation.
- Modeling: intuitive physics & world models: Veo models physical properties and relationships, including buoyancy, material dynamics, air resistance, optical phenomena, object fit, and plausible object removal.
- Manipulation: editing & imagination: Veo performs image editing and 3D visual transformations, including background removal, style transfer, inpainting, outpainting, novel views, and perspective changes.
- Manipulation: editing & imagination: Veo simulates interactions and object manipulation, interprets affordances, and demonstrates procedures such as drawing a shape or rolling a burrito.
- Visual reasoning across time and space: Chain-of-frames reasoning appears in graph traversal, visual search, sequence completion, puzzle solving, tool use, Sudoku, maze navigation, and rule extrapolation.
- Summary: Across the examples, Veo 3 consistently solves many tasks without explicit task-specific training, although results are not always perfect.
4. Quantitative results
The quantitative assessment covers seven tasks spanning perception, manipulation, and visual reasoning, with performance evaluated across best and last frames where applicable. Veo 3 generally improves substantially over Veo 2, sometimes matching or exceeding Nano Banana, while remaining imperfect and sensitive to prompting.
- Evaluation setup: The evaluation measures edge detection, segmentation, image editing, object extraction, maze solving, visual symmetry, and visual analogies across best and last frames where applicable.Best-frame results estimate a performance ceiling, whereas last-frame results are predetermined but can be reduced by continued animation after task completion.
- Overall trends: Veo 3 shows a large performance increase from Veo 2 to Veo 3, often matching or exceeding Nano Banana, and performance improves substantially from k=1 to k=10 attempts.The paper notes that multiple attempts can find a good solution in a reasonable number of tries.
- Perception: 0.77 pass@10 OIS for Veo 3 edge detection remains below task-specific SOTA at 0.90, although the model often produces more detailed edge maps than ground truth.The authors suggest that omitted foliage and tire-profile edges in the dataset can hurt the measured score.
- Perception: Veo 3 achieves 0.74 mIoU best frame pass@10 for class-agnostic instance segmentation, comparable to Nano Banana’s 0.73, with green backgrounds outperforming white backgrounds at 0.74 versus 0.66.The task segments all objects without specifying a category or location.
- Manipulation: Veo 3 reaches up to 93% pass@10 on extracting one to nine animals into a row, while Veo 2 performs around chance and a perfect model would reach 100%.The evaluation counts connected components in the last frame to assess whether the number of extracted animals is correct.
- Reasoning: On 5×5 mazes, Veo 3 achieves 78% pass@10 versus Veo 2’s 14%, while visual analogy performance remains below chance on reflect and rotate transformations.Veo 3 correctly completes color and resize analogies, but both models show a systematic bias on reflect and rotate analogies.
5. Discussion
Veo 3 solves a wide range of vision tasks zero-shot, spanning perception, world modeling, manipulation, and early visual reasoning. Its consistent improvement over Veo 2 supports the possibility that video models could become general-purpose vision foundation models, despite current performance and cost limitations.
- Veo 3 solves diverse zero-shot tasks across perception, modeling, manipulation, and early visual reasoning.The reported task range includes visual understanding and manipulation as well as maze and symmetry solving.
- Veo 3’s performance is not yet perfect, and specialized models outperform it on many tasks.
- Prompt-dependent results should be treated as a lower bound on the model’s underlying capabilities because task representations and prompts vary widely.The paper distinguishes measured task performance from competence and notes that visual and textual prompt design matters.
- Inference costs for video generation are currently higher than for bespoke task-specific models, although the paper argues that such costs tend to fall.The discussion points to declining inference costs for general-purpose models as a relevant economic trend.
- The consistent performance improvement from Veo 2 to Veo 3 and higher pass@10 than pass@1 suggest room for further scaling and optimization.
- The paper presents video models as potentially enabling a machine-vision shift toward generalist foundation models analogous to NLP’s transformation.
A.1. Perception
Veo 3 demonstrates zero-shot perception abilities across low-level vision, localization, segmentation, visual search, and visual illusion understanding, with highly variable reliability across tasks.
- Edge detection succeeds at 0.92, while instance segmentation succeeds at 0.33 and keypoint localization at 0.58.
- Super-resolution succeeds at 0.75, while blind deblurring and blind denoising each succeed at 1.0.
- Low-light enhancement succeeds at 0.92, using an image and ground-truth source from the LOLv2 dataset.
- Conjunctive search and binding succeeds at 0.75, indicating selective visual modification of the blue ball.
- Dalmatian illusion understanding, cue-conflict shape understanding, and Rorschach blot prompt following each succeed at 1.0.The Rorschach result is reported as 1.0 for prompt following.
A.2. Modeling
Veo 3 models physical properties and object relations in several zero-shot tasks, but reliability varies substantially across materials, dynamics, categorization, and symbolic manipulation.
- Material properties succeed at 0.25, and gravity with air resistance succeeds at 0.5 on both Earth and the Moon.
- Rigid-body transformation succeeds at 1.0, while soft-body draping succeeds at 0.67.
- Buoyancy succeeds at 0.58 for a bottle cap and 0.83 for a rock.
- Object packing succeeds at 0.75, whereas visual Jenga succeeds at 0.5 based on removing at least three objects.
- Material optics succeeds at 0.92 for glass and 1.0 for a mirror, while additive color mixing succeeds at 0.92 and subtractive mixing at 0.75.
- Categorizing objects succeeds at 0.33, while character recognition, generation, and parsing and memory of world states are also evaluated as modeling tasks.
A.3. Manipulation
Veo 3 can zero-shot manipulate and transform visual scenes, simulate actions and tool use, and attempt structured reasoning tasks, but success rates range from 0.08 to 1.0.
- Background removal, inpainting, outpainting, and image editing with doodles each succeed at 0.83 or 1.0.
- Style transfer succeeds at 0.75, colorization at 0.08, and scene composition at 0.75.
- Single-image novel view synthesis succeeds at 0.92 and 3D-aware reposing at 0.83, while transfiguration succeeds at 0.17.
- Dexterous manipulation succeeds at 1.0 for jar opening and throwing and catching, but at 0.08 for rotating Baoding balls.
- Visual instruction generation, graph traversal, connecting colors, shape fitting, and numeric sorting succeed at 0.25 or below.
- Tool retrieval succeeds at 0.92 with a tool and 0.08 when retrieval must avoid intersecting the aquarium glass.
- Structured reasoning tasks show mixed success, including 0.67 for 4x4 Sudoku, 0.17 for maze solving, and 0.08 for rule extrapolation.
B. Quantitative results: experimental details
The quantitative experiments span multiple visual tasks, with task-specific datasets, prompts, sampling procedures, and evaluation protocols. These protocols include automated image comparisons, mask matching, connected-component counting, and human ratings.
- Evaluation: Edge evaluation applies non-maximum suppression, thresholding, thinning, and local-shift matching within 0.75% of the image diagonal.The evaluation uses 16 evenly spaced thresholds and an approximately 11-pixel matching radius.
- Datasets: The study evaluates segmentation, edge detection, counting, and image editing using task-specific datasets and prompts.The supplied procedures include BIPEDv2 for edges, LVIS for instance segmentation, a 54-image animal-counting dataset, and Emu-edit images.
- Models & prompts: Veo 3 and Veo 2 are tested through the Vertex AI API, while Nano Banana is tested through Google AI Studio.
- Sampling: Sampling commonly uses 10 videos per sample with a fixed prompt, while image editing uses two samples and rates the first.
- Evaluation: Segmentation evaluation identifies predicted masks from hue differences and pairs each ground-truth mask with the prediction having the highest IoU.
- Evaluation: Image-editing fidelity and precision are evaluated by three human raters, with precision requiring no unintended changes.
B.5. Reasoning: Maze solving
Maze solving is evaluated as a path-following task across video, image-editing, and text-based inputs. Success requires reaching the goal without illegal moves or wall intersections.
- Evaluation: Veo maze solving is scored frame by frame by checking whether the moving agent reaches the goal without invalid moves.Invalid moves include jumping walls, crossing boundaries, or altering the goal position.
- Evaluation: Nano Banana is evaluated on whether its drawn path connects the start and end points without intersecting walls or leaving the valid path.
- Evaluation: For Gemini maze inputs, evaluation checks whether the reported grid positions form an uninterrupted path from start to goal.
- Dataset: The maze dataset contains random rectangular mazes and 40 unique samples derived from 10 hand-drawn irregular mazes.
- Video prompting: The Veo prompt specifies a static top-down animation in which a red circle slides along white paths to a green goal without entering black walls.
- Text and image prompting: For image and ASCII inputs, the requested solution is a coordinate list describing one-step, non-diagonal movement from start to goal.
B.6. Reasoning: Visual symmetry solving
Visual symmetry solving asks Veo to complete the missing half of a grid pattern by reflecting the existing half across the central vertical axis. Performance is measured against ground-truth symmetry using frame-level cell-color errors.
- Task: The task uses a 10×16 grid whose left-half pattern must be completed symmetrically on the empty right half.
- Evaluation: Evaluation compares Veo’s best-frame and last-frame grids with ground truth using incorrectly-colored cell counts.Cell differences are computed from average colors in CIELAB space with a threshold of 15.0.
- Dataset: The dataset contains 25 samples with common symmetrical shapes and 25 samples with randomly-colored cells.
- Results reporting: Figure 61 evaluates majority-vote performance over 10 attempts rather than best performance, so performance need not increase with more attempts.
- Prompting: Veo is prompted to reflect the pattern along the central vertical axis while preserving the existing colored pattern.
B.7. Reasoning: Visual analogy completion
Visual analogy completion evaluates whether Veo can generate a missing object that satisfies a reference-object relationship. The broader qualitative set also documents failures on depth, surface normals, motion, physical manipulation, and visual puzzles.
- Visual analogy evaluation: Veo is evaluated on visual analogies by generating the missing lower-right object and comparing it with three candidate choices using an autorater.
- Dataset and metric: The analogy benchmark uses KiVA test trials and choice images, with pass@1 reported for Veo 2 and Veo 3 at k=1.
- Autorating: The Gemini-assisted autorater receives condition information and achieved above 88% agreement with expert human ratings in pilot experiments.
- Prompting: The analogy prompt asks Veo to generate the missing object while keeping the original three objects still in a static shot.
- Qualitative results: The qualitative examples report failures in depth coloring, surface-normal consistency, force and motion following, knot tying, and path tracing.
- Qualitative results: Additional puzzle examples show hallucinated path highlights, incorrect word-search or equation edits, missed differences, and incorrect visual-IQ patterns.