PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Visual Intelligence: Insights from Video Pretraining

Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro

arXiv:2510.24448v2cs.CVcs.AI

TL;DR

Visual models lag LLMs in compositional skills, sample efficiency, and versatile problem-solving, motivating a test of video pretraining as a source of visual inductive biases. The paper compares adapted VDMs and LLMs on structured visual tasks and reports that VDMs adapt with fewer examples. It concludes that modality-aligned pretraining is a promising direction for visual intelligence.

  • Problem

    Visual models remain behind LLMs in compositional understanding, sample efficiency, and versatile problem-solving.

    Source: §1 · p. 2

  • Method

    The paper adapts pretrained VDMs and LLMs with lightweight approaches and compares them on tasks in their natural modalities.

    Source: §1 · p. 2

  • Results

    VDMs pretrained on spatiotemporal data adapt to structured visual tasks with fewer training examples than comparable LLMs.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Modality-aligned pretraining is a promising direction for advancing visual intelligence.

    Source: §5 · p. 9

    The study focuses on grid-based benchmarks and synthetic puzzles, which do not capture the full diversity of real-world challenges.

    Source: LIMITATIONS AND FUTURE WORK · p. 13

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models.

1 INTRODUCTION

Visual models still lag LLMs in compositional skills, sample efficiency, and versatility; the paper investigates whether video pretraining can help close this gap. It introduces a VDM adaptation framework and a controlled comparison with LLMs.

1 INTRODUCTION

  • Figure 1 presents a ConceptARC competency radar comparing VDMs and LLMs, with GPT-4 [IC] as an additional reference.Source: Figure 1 · p. 1
  • VDMs are adapted to image-to-image visual tasks by reframing examples as temporal sequences.Source: §1 · p. 2
  • The evaluation compares VDMs and LLMs using the same lightweight LoRA-based adaptation approach.Source: §1 · p. 2
  • The paper reports empirical evidence that video-pretrained VDMs benefit visual intelligence, pointing toward visual foundation models with generative and problem-solving strengths.The introduction frames this as a potential path, not a claim that such foundation models have already been built.Source: §1 · p. 2

2 RELATED WORK

The related work situates the study among adaptable language models, video diffusion, and generative visual foundation models. The paper investigates whether spatiotemporal pretraining can contribute to visual problem-solving.

Language Foundation Models

  • LLMs’ broad pretraining supports few-shot and in-context learning, while LoRA enables specialization with the backbone frozen; this paper uses that adaptability as a comparison point.Source: §2 · p. 2

Video Diffusion Models

  • Video diffusion research emphasizes realistic synthesis and controllable motion, dynamics, and future-state prediction.Source: §2 · p. 2

Visual Foundation Models

  • Generative vision work spans prompting, in-context learning, instruction following, and sequential modeling; Lin et al. (2025) study temporal prompts but target broad computer-vision benchmarks and require extensive training.Source: §2 · p. 3
  • Rather than training a foundation model from scratch, the paper tests whether a pretrained next-frame-prediction VDM’s spatiotemporal inductive biases support visual foundation-model-like properties.Source: §2 · p. 3

3 METHODOLOGY

The methodology compares models on visually grounded tasks through parallel text and image representations, measuring both exact-match accuracy and skill acquisition under limited supervision. It adapts VDMs by turning input-output examples into transition videos and adapts LLMs through JSON-to-JSON prediction.

3.1 SETUP AND COMPARISON PROTOCOL

  • Intelligence is evaluated as skill-acquisition efficiency across tasks relative to priors, experience, and generalization difficulty.The evaluation therefore considers how quickly models acquire capabilities under limited supervision, not only endpoint accuracy.Source: §3.1 · p. 3
  • Each sample is represented as an image pair for VDMs or a JSON pair for LLMs, with neutral serialization and identical train and evaluation splits.Source: §3.1 · p. 3
  • Accuracy requires an exact match to the ground-truth grid; datasets for puzzles with multiple valid solutions are filtered for unique solutions.Shortest Path instead uses complementary metrics because unique solutions cannot easily be guaranteed.Source: §3.1 · p. 3
  • The benchmark includes ARC-AGI tasks with 2–5 demonstrations and structured tasks where the number of training examples per task is varied.Source: §3.1 · pp. 3–4

3.2 ADAPTING VIDEO DIFFUSION MODELS FOR IMAGE-TO -IMAGE

  • Each VDM example becomes a transition video from input image to output image, with intermediate frames interpolated; experiments use a discrete input-then-output transition to avoid introducing bias.Source: §3.2 · p. 4
  • VDMs are conditioned on the first frame and a fixed neutral text embedding, trained by updating LoRA weights while the pretrained model remains frozen, and predict through reverse diffusion.The final video frame supplies the output prediction.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 ADAPTING LARGE LANGUAGE MODELS

  • LLM examples are framed as JSON-to-JSON translation and trained with a sequence-to-sequence objective that maximizes target likelihood under teacher forcing.Source: §3.3 · p. 4
  • LLMs use lightweight LoRA modules with most backbone parameters frozen, and generate predictions autoregressively at test time.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments compare video diffusion and language models across ARC-family benchmarks and structured visual tasks. VDMs show advantages in games, low-sample route planning, and two-dimensional cellular automata, while their one-dimensional cellular automata performance is broadly competitive.

4.1 ARC FAMILY

  • In the cited ARC evaluation protocol, up to two attempts per test input are allowed, and a question is solved only if all predictions match ground truth; VDM/LLM overlap indicates complementary strengths, while results highlight strong visual priors.ConceptARC has 16 categories of 10 tasks each, with 3 test inputs per task; its protocol allows 3 attempts per input and counts it solved if any attempt is correct.Source: §4.1 · pp. 5–6

4.2 STRUCTURED VISUAL TASKS

  • For structured visual tasks, the study pairs CogVideoX1.5-5B with Qwen3-4B-Instruct-2507, aligning model scale while contrasting pretraining modalities.Source: §4.2 · p. 6

4.2.1 VISUAL GAMES

  • CogVideoX1.5-5B surpasses Qwen3-4B-Instruct-2507 in four of five games, especially Sudoku and Hitori, while chess favors the LLM.The authors suggest chess’s exception may reflect abundant chess material in textual corpora that LLMs can partially internalize during pretraining (Kuo et al., 2023).Source: §4.2.1 · pp. 6–7, Figure 4 · p. 7

4.2.2 ROUTE PLANNING

  • VDMs construct valid paths with up to a tenfold reduction in low-sample data requirements and generalize faster from smaller to larger mazes.Maze generalization trains on 13 × 13 grids and tests on 21 × 21 grids. Relative Path Length can rise as better models produce valid paths on harder cases, even as overall performance improves.Source: §4.2.2 · p. 7, Figure 5 · p. 7, Table 2 · p. 8

4.2.3 CELLULAR AUTOMATA

  • Across 1D ECA rules, VDM performance is broadly competitive; on 2D Life-like automata and Langton’s ant it needs fewer examples, with the ant gap widening for longer predictions.Source: §4.2.3 · p. 8, Figure 7 · p. 8, Figure 9 · p. 9

5 CONCLUSIONS

VDMs pretrained on spatiotemporal data adapt to structured visual tasks with fewer examples than comparable LLMs. The conclusions frame modality-aligned pretraining as promising while identifying benchmark scope, computational cost, and interpretability as unresolved concerns.

5 CONCLUSIONS

  • VDMs pretrained on spatiotemporal data adapt to structured visual tasks with fewer training examples than comparable LLMs.The conclusions distinguish VDM strengths on spatial structure and temporal transformation from LLM strengths in symbol-rich domains.Source: §5 · p. 9
  • Navigation-style tasks such as mazes and route planning suggest potential downstream relevance to planning, simulation, or robotics, but embodied validation remains future work.The authors call for validation in more realistic, embodied environments.Source: §5 · p. 9
  • Modality-aligned pretraining is presented as a promising venue for advancing visual intelligence.The paper links this direction to large-scale pretraining on spatiotemporal data with representations aligned to visual structure.Source: §5 · p. 9

LIMITATIONS AND FUTURE WORK

  • The evaluation focuses on grid-based benchmarks, which do not capture the full diversity of real-world challenges.The controlled setup is described as valuable for isolating and analyzing modality-aligned pretraining.Source: LIMITATIONS AND FUTURE WORK · p. 13
  • Iterative diffusion sampling adds significant computational overhead, and complex tasks such as ARC-AGI often require longer schedules for structural consistency and coherence.Some tasks can perform well with only a few sampling steps.Source: LIMITATIONS AND FUTURE WORK · p. 14
  • VDM fine-tuning remains computationally demanding despite LoRAs; modular, composable adapters and in-context adaptation are proposed directions.The authors suggest these approaches could reduce retraining costs while enhancing cross-task generalization.Source: LIMITATIONS AND FUTURE WORK · p. 14
  • Understanding the mechanisms behind VDM visual intelligence remains an important research direction.Future work could investigate how VDMs internally represent and manipulate concepts.Source: LIMITATIONS AND FUTURE WORK · p. 14

A EXPERIMENTAL DETAILS

The experimental details report hardware-dependent GPU-hour costs and provide LoRA and training configurations for both model families, with a note on rank differences and performance sensitivity.

A EXPERIMENTAL DETAILS

  • Tables 3 and 4 report wall-clock GPU hours for ConceptARC, ARC-AGI, and structured visual tasks, with costs dependent on hardware.The passages identify the tables’ scopes but do not provide the hour values.Source: A EXPERIMENTAL DETAILS · p. 14, Table 3 · p. 14, Table 4 · p. 14
  • Tables 5 and 6 provide the LoRA, training, and optimizer configurations used for VDMs and LLMs.The configurations are included to support reproducibility.Source: A EXPERIMENTAL DETAILS · p. 14
  • VDMs use LoRA rank 64 and LLMs rank 32, but reported results are described as largely insensitive to this difference.Qwen3 rank 64 performs comparably to rank 32, while CogVideoX1.5-5B rank 32 matches reported rank 64 results; the paper reports the initially stronger configuration.Source: A EXPERIMENTAL DETAILS · p. 14

B TASK DETAILS

The appendix details task rules and modality encodings and lists LoRA finetuning configurations for both model families; qualitative ARC-AGI examples show cases where either model family, or both, succeed.

B.1.1 HITORI 5X5

  • Hitori requires shading cells to eliminate repeated numbers while keeping shaded cells nonadjacent and unshaded cells connected.The objective is to leave each number at most once per row and column.Source: §B.1.1 · p. 15

B.1.2 SUDOKU

  • Sudoku tasks fill grids under row, column, and subgrid constraints, with evaluated variants of 4×4 and 9×9.Mini Sudoku uses 2×2 subgrids; standard Sudoku uses 3×3 subgrids.Source: §B.1.2 · pp. 15–16

B.1.3 CONNECT 4; B.1.4 CHESS MATE-IN-1

  • Connect4 is restricted to single-move wins, while Chess Mate-in-1 asks for an immediate checkmate on a white move.Connect4 tokens fall to the lowest available cell; chess scenarios are filtered to white moves.Source: §B.1.3 · p. 17, §B.1.4 · p. 18

B.2 ROUTE PLANNING

  • Route planning covers corner-to-corner mazes and shortest paths between arbitrary grid points, using orthogonal movement through open cells.Maze training and evaluation use 21×21 grids in the base setting, while generalization trains on 13×13 and tests on 21×21 grids.Source: §B.2 · p. 18, §B.2.1 · pp. 18–19, §B.2.2 · p. 20

B.3.1 ELEMENTARY CELLULAR AUTOMATA (ECA)

  • ECA tasks use one-dimensional binary cells updated from local neighborhoods and test four representative rules from each Wolfram class.The rule number ranges from 0 to 255.Source: §B.3.1 · p. 20

B.3.2 LIFE-LIKE CELLULAR AUTOMATA

  • Life-like cellular automata vary birth and survival rules; standard Game of Life births a dead cell at three live neighbors and preserves live cells at two or three.The evaluation includes variants with diverse behaviors.Source: §B.3.2 · p. 21

B.3.3 LANGTON’S ANT

  • Langton’s ant moves on a binary grid, turning right on a 0 cell and left on a 1 before moving forward; experiments fix its initial direction and cell state.The task predicts the complete grid after n steps; the described dynamics eventually form a repeating highway after chaotic behavior.Source: §B.3.3 · p. 21

C.1 ARC-AGI

  • ARC-AGI examples show three patterns: only the LLM succeeds, both models succeed, or only the VDM succeeds.The shared successes are attributed to structure capturable through symbolic reasoning or visual pattern learning; VDM-only successes are linked to visual inductive biases.Source: §C.1 · p. 22

C.2 STRUCTURED VISUAL TASKS

This section adds qualitative examples for structured visual tasks and expands the ConceptARC model comparison; it also notes a possible compression-related tradeoff for LTX.

C.2 STRUCTURED VISUAL TASKS

  • Four ARC-AGI figures provide further qualitative examples, covering problems 60a26a3e through 8ee62060.The listed problem identifiers are 60a26a3e, 62b74c02, 8a371977, 2072aba6, 4aab4007, 5207a7b5, ca8de6ea, d37a1ef5, e95e3d8e, 575b1a71, 68b67ca3, and 8ee62060.Source: Figure 22 · p. 23, Figure 23 · p. 24, Figure 24 · p. 25
  • Additional examples show predictions for Shortest Path (n=300), Maze (n=300), Sudoku (n=1000), Hitori (n=100), and Langton Ant at horizon 10 (n=1000).The figures place model predictions alongside inputs and ground-truth outputs.Source: Figure 26 · p. 27, Figure 27 · p. 28, Figure 28 · p. 29

E ARC FAMILY

  • The ConceptARC appendix compares finetuned LLMs and VDMs; it suggests LTX’s lower performance may stem from aggressive VAE compression, which can discard structure in exchange for faster generation.Table 9 presents concept-wise overall accuracy across models, but the supplied caption does not give its cell values.Source: E ARC FAMILY · p. 32, Table 9 · p. 32

E.1 PITFALLS OF VISION LANGUAGE MODELS

The Sudoku experiments find no measurable benefit from adding image input to the tested VLM, and a reconstruction probe indicates difficulty extracting grid structure from images.

E.1 PITFALLS OF VISION LANGUAGE MODELS

  • With 1,000 samples, adding image input to Gemma-4B produced no measurable Sudoku improvement, and the image-only variant performed worse than a trivial baseline.The same model was tested in text-only, image-only, and combined image–text configurations with other settings fixed.Source: §E.1 · p. 33, Table 10 · p. 33
  • On grid reconstruction with n=3, 5, or 10 samples, the image-only model reproduced training examples verbatim regardless of input rather than interpreting the grid.The authors report that it learned little about the visual input’s underlying structure.Source: §E.1 · p. 33, Table 11 · p. 33
  • Because tested VLMs offered no measurable advantage on structured visual tasks, the authors use LLMs as the primary baseline for comparison with VDMs.They characterize VLMs as struggling to extract structured information from images despite multimodal pretraining.Source: §E.1 · p. 33

F RESULTS - FULL TABLES

The appendix supplies full experimental results underlying the main-paper figures, with tables covering structured games and cellular automata.

F RESULTS - FULL TABLES

  • The full results are identified as the underlying data for figures in the main paper.Table 12 reports accuracy comparisons for CogVideoX1.5-5B and Qwen3-4B-Instruct-2507 on structured games.Source: F RESULTS - FULL TABLES · p. 34, Table 12 · p. 34
  • Table 13 compares CogVideoX1.5-5B and Qwen3-4B-Instruct-2507 accuracy across Life-like cellular automata variants.Source: Table 13 · p. 34
  • Table 14 compares the two models’ Langton’s Ant accuracy by the number of future steps predicted.Missing values are shown as dashes.Source: Table 14 · p. 34

G EXPLORING GENERALIZATION OF I2I-TUNED VDMS

The section tests whether pretrained CogVideoX1.5-5B generalizes to classical image-to-image tasks through few-shot adaptation. Qualitative examples show geometric transformations, one-shot style transfer, and selected vision tasks after fine-tuning with up to 30 paired examples.

G EXPLORING GENERALIZATION OF I2I-TUNED VDMS

  • Few-shot adaptation serves both as a tuning strategy and as a probe of whether task abilities were already internalized during pretraining.The section frames success with very few paired examples as an indication of such pretraining competence.Source: G EXPLORING GENERALIZATION OF… · p. 34
  • CogVideoX1.5-5B is fine-tuned with one to 30 paired examples using the main experiments’ architecture, schedule, and hyperparameters, without auxiliary losses or task-specific modifications.This setup is intended to isolate the contribution of pretrained knowledge.Source: G EXPLORING GENERALIZATION OF… · p. 34
  • The section also explores established datasets spanning scene understanding and generative image transformation, including NYUv2, ADE20K, ML-Hypersim, COCO 2017, and DreamBooth.The cited passage associates the datasets with references Nathan Silberman & Fergus (2012), Zhou et al. (2017; 2019), Roberts et al. (2021), Lin et al. (2014), and Ruiz et al. (2022).Source: G EXPLORING GENERALIZATION OF… · p. 36
  • Few-shot examples show geometric transformations and one-shot style transfer, while 30-sample fine-tuning yields qualitative binary-segmentation and pose predictions.The examples include distinct artistic styles and dog segmentation and human pose estimation.Source: G EXPLORING GENERALIZATION OF… · p. 36, Figure 31 · p. 36, Figure 32 · p. 37
  • The qualitative task examples include inpainting, colorization, and jigsaw at different training-example counts.The figure caption identifies these tasks but does not report comparative outcomes.Source: Figure 33 · p. 37
  • One-shot image-to-segmentation examples are shown for Chamber, while segmentation-to-image examples test transfer to other environments within Chamber, Coast, and Badlands.Each environment is a separate one-shot training; Chamber trains on one chamber and tests on others, with the same within-category protocol for Coast and Badlands.Source: Figure 36 · p. 39, Figure 37 · p. 40
Loading 2510.24448v2…