PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelWorld: How Far Are We from Perceiving Everything as Pixels?

Zhiheng Lyu, Xueguang Ma, Wenhu Chen

arXiv:2501.19339v3cs.CVcs.CL

TL;DR

Existing multimodal systems use separate pixel and token pipelines, creating representation mismatch and preprocessing overhead for intertwined visual–textual environments. The paper introduces PEAP and PixelWorld to evaluate shared pixel-space perception, finding comparable semantic understanding but weaker mathematics and code reasoning, with partial mitigation from Chain-of-Thought prompting.

  • Problem

    Separate image-pixel and text-token pipelines create representation mismatch, preprocessing overhead, information loss, and layout inconsistencies in unified multimodal interaction.

    Source: §1 · p. 1

  • Method

    The paper introduces PEAP and PixelWorld, rendering text, structural, and multimodal inputs into pixel space for paired evaluation across vision–language models and benchmarks.

    Source: §1 · p. 1, §2 · p. 4

  • Results

    PEAP performs on par with token-based approaches on semantic understanding, while performance drops on mathematics, logic, and code tasks; Chain-of-Thought prompting mitigates but does not close the gap.

    Source: §1 · p. 3, §3.1 · p. 6, §6 · p. 11

  • Takeaways & Limitations

    Pixel-based perception can transfer some global language-modeling behavior and benefits closely integrated visual–textual tasks, while explicit reasoning structure remains useful for complex tasks.

    Source: §1 · p. 3, §6 · p. 11

    Complex reasoning remains a bottleneck, and image-based inference can require substantially more computation than text-based inference.

    Source: §3.3 · p. 7, §4.2 · p. 8

Abstract

from arXiv · show

Recent agentic language models increasingly need to interact with real-world environments that contain tightly intertwined visual and textual information, often through raw camera pixels rather than separately processed images and tokenized text. This shift highlights the need for a unified perception paradigm. To investigate this idea, we explore Perceive Everything as Pixels (PEAP) and introduce PixelWorld, a benchmark that renders natural-language, tabular, mathematical, and diagrammatic inputs into a shared pixel space. Experiments across multiple benchmarks show that PEAP achieves comparable performance to token-based approaches on semantic understanding tasks, suggesting that vision transformers can partially capture global textual semantics without explicit tokenization. In contrast, reasoning-intensive tasks such as mathematics and code show notable performance degradation, although Chain-of-Thought prompting helps mitigate this gap by compensating for missing symbolic structure. We further find that when visual and textual information are closely integrated, representing everything as pixels simplifies preprocessing and avoids cross-modal misalignment. PixelWorld thus provides a systematic and practical framework for evaluating unified vision--language models and facilitates further exploration of pixel-based multimodal learning.

1 Introduction

PixelWorld investigates PEAP, which represents text, structural, and visual inputs in one pixel space to evaluate unified vision–language perception. Its experiments show benefits for multimodal and layout-heavy tasks, trade-offs on complex reasoning, and partial transfer of language-modeling behavior into visual representations.

Benchmark and setup

  • PixelWorld evaluates PEAP across diverse modalities and task scenarios, comparing token-based and pixel-based inputs for vision–language models of varying scales.The benchmark transforms datasets into counterpart input formats and evaluates models from Qwen2VL-2B to GPT-4o.Source: §1 · pp. 2–3

Task–scale insights

  • PEAP loses accuracy on reasoning- and code-centric tasks, but the performance gap narrows as model scale increases.The findings identify model capacity as important for cross-modal transfer.Source: §1 · pp. 2–3

Representation analysis

  • Attention visualizations show partially shared global structures between token- and pixel-based inputs, suggesting limited transfer of language-modeling behavior into visual space.The evidence does not imply full equivalence between the representations.Source: §1 · pp. 2–3

Prompt sensitivity

  • Chain-of-Thought prompting produces more consistent gains than direct prompting under pixel-based representation, indicating a connection between reasoning supervision and visual encoding.This analysis addresses prompt sensitivity rather than establishing equivalence between pixel and token processing.Source: §1 · p. 3

Task–scale insights

  • PEAP improves layout-heavy and intrinsically multimodal tasks such as website, slide, and document understanding by reducing OCR-related problems.These gains are reported alongside lower accuracy on reasoning- and code-centric tasks.Source: §1 · pp. 2–3

Efficiency and interpretability

  • PEAP-Fast prunes blank pixel patches while preserving positional embeddings, achieving up to 3× lower latency with minimal accuracy loss.The strategy reduces redundant visual computation without discarding spatial layout information.Source: §1 · p. 3

2 Datasets

The benchmark constructs paired text and image inputs across text-only, structured, and multimodal datasets. Text and structured data are rendered into images, while multimodal inputs use OCR or dataset-provided textual components, with CoT prompting reserved for selected complex tasks.

Prompting protocol

  • The study primarily uses dataset-provided prompts, applies Direct Prompting by default, and uses Chain-of-Thought for MBPP, MMLU-Pro, and MathVerse.All evaluations are conducted zero-shot to reduce sensitivity to few-shot prompting.Source: §2 · pp. 3–4

Dataset construction

  • The evaluation uses paired text-input and image-input prompts to compare token-based and pixel-based methods across transferred modalities.Modality transfer reduces dependence on the original information modality of each dataset.Source: §2 · p. 4
  • Text-only and structured datasets are rendered as images, whereas multimodal datasets use OCR-extracted text or textual components supplied by the original data.MathVerse includes textual descriptions of image-based information as a text-only modality.Source: §2 · p. 4

Image synthesis

  • Synthetic images use adaptive widths, fixed height, varied font sizes and padding, and multiple noise types to simulate camera-related distortions.Structured data such as tables is rendered with dataframe_image.Source: §2 · p. 4

3 Experiments

Experiments compare token-based and pixel-based inputs across text, structured, and multimodal tasks, showing that PEAP’s performance depends strongly on model scale, task complexity, and modality integration.

3.1 Text Input

  • PEAP performance varies by task: larger models transfer better across text and image inputs, while advanced reasoning and coding tasks show the largest degradation.On ARC, GPT-4o drops 0.59 points from text to synthetic images, whereas Qwen2-VL-2B drops 21.73 points; GPT-4o’s MMLU-Pro decline exceeds 25 points while GLUE and SuperGLUE declines remain under 5 points.Source: §3.1 · pp. 5–6

3.2 Structured Input

  • Structured tasks show smaller text–image gaps than text-only tasks, with semantic subsets more robust than numerical reasoning and visualization.Qwen2VL-2B outperforms its text-based result on Fact Checking, while Numerical Reasoning and Visualization exhibit larger declines with synthetic images.Source: Figure 4 · p. 5, §3.2 · p. 6
  • Mixed-modality inputs underperform both fully text-based and fully image-based formats on structured data.The semi format keeps questions as text and tables as images, suggesting a bottleneck for conventional approaches that process modalities separately.Source: Figure 4 · p. 5, §3.2 · p. 6

3.3 Multimodal Input

  • Images improve multimodal disambiguation, with all models outperforming text-only baselines on SlidesVQA and larger models gaining visual-context benefits on WikiSS-QA and MMLU-Pro.These gains are strongest where visual context clarifies otherwise ambiguous text.Source: §3.3 · p. 7, Figure 6 · p. 7
  • Complex reasoning remains a multimodal bottleneck: visual cues do not support multi-step deductions reliably, while long-context WikiSS-QA challenges smaller models and GPT-4o.Even Gemini-Flash loses accuracy on intricate MathVerse reasoning tasks, and GPT-4o underperforms in token-based WikiSS-QA tasks because it tends to generate longer responses.Source: §3.3 · p. 7, Figure 6 · p. 7
  • Larger models benefit more from multimodal inputs, with Gemini_Flash improving 34.24 points on SlidesVQA versus 23.55 points for Qwen2-VL-7B.The experiments use VLMs spanning Qwen2VL-2B through GPT-4o and evaluate them with task-specific metrics.Source: §3 · p. 5, §3.3 · p. 7

4 Discussion

The analyses find broadly comparable attention patterns across token- and pixel-based inputs, while PEAP incurs substantial latency costs that PEAP-Fast reduces with little accuracy loss; CoT improves pixel-based performance on some textual tasks.

4.1 Q1: Does PEAP have the same attention?

  • Comparable attention patterns focus on task-relevant prompts, keywords, and answer formats across textual and visual inputs, although blank image patches can attract redundant activation.The heatmap analysis averages final-layer attention across heads during greedy decoding.Source: §4.1 · pp. 7–8

4.2 Q2: How to make PEAP more efficient?

  • 150% to 250% higher inference latency makes image-based inputs substantially more expensive than text-based inference.The added cost is attributed partly to the ViT backbone and redundancy in image tokens.Source: §4.2 · p. 8
  • 82.98% lower average overhead results from PEAP-Fast pruning empty patches while preserving positional embeddings, with only a 1.17% accuracy drop.The reported overhead decreases from 205.27% to 123.19%; the supplied table caption reports an 82.08% reduction, whereas the prose reports 82.98%.Source: §4.2 · p. 9, Table 3 · p. 9, Table 2 · p. 9

4.3 Q3: Is PEAP sensitive to the prompting method?

  • CoT prompts improve image-input performance by 2.58% on SuperGLUE, exceeding the 0.3% improvement for text inputs.Qwen2VL-7B improves notably on image-input tasks such as CB and RTE where image inputs initially underperform text inputs.Source: §4.3 · p. 9, Table 4 · p. 10

5 Related Work

Related work spans multimodal benchmarks, screenshot-based language modeling, and alternatives to conventional tokenization. PixelWorld focuses these directions on comparing language understanding from visual inputs with traditional token inputs.

Multimodal Large Language Models and Benchmarks

  • Multimodal benchmarks have expanded from task-specific VQA and DocVQA datasets to broader evaluations, while relatively few address language understanding within images.PixelWorld shifts attention toward comparing visual-input language understanding with traditional token-based input.Source: §5 · p. 10

Screenshot LMs

  • Synthetic screenshot pretraining can support language modeling and document retrieval without OCR, with some vision-language models matching or exceeding text-only systems in selected settings.The cited studies also report performance gaps when relevant pretraining is absent.Source: §5 · p. 10

Language Tokenization

  • Recent tokenization alternatives motivate testing whether adaptive image patches can represent textual meaning in a shared text-image space.The related approaches include fixed-length, entropy-based, and higher-level semantic processing.Source: §5 · p. 10

6 Conclusion

PixelWorld evaluates Perceive Everything as Pixels across semantic, reasoning, and multimodal tasks, revealing both promising transfer from language modeling and persistent costs in reasoning and computation.

6 Conclusion

  • PEAP matches token-based baselines on sentence- and paragraph-level semantic tasks, while pixel attention shows similar global structures to token attention.This suggests partial transfer of language-modeling behavior into the visual domain.Source: §6 · p. 11
  • PEAP performance drops on math, logic, and program-repair benchmarks, while Chain-of-Thought prompting mitigates but does not close the gap.The result highlights the continued need for explicit reasoning structure.Source: §6 · p. 11
  • Pixel-based inputs outperform OCR pipelines on websites, slides, and documents by preserving spatial context and avoiding recognition noise.These findings concern intrinsically multimodal perception tasks.Source: §6 · p. 11
  • PEAP-Fast prunes blank patches and achieves up to a 3× inference speedup with minimal accuracy loss.The method addresses the higher computational cost of pixel inputs.Source: §6 · p. 11

A Example Input

The section provides examples of vision inputs through Figures 8 and 9.

A Example Input

B Attention Heatmap before and after ImageFast Method

The heatmap comparison shows that PEAP-Fast reduces redundant patches while preserving attention on key regions.

C Attention Heatmap Comparison Between Datasets

The section illustrates attention patterns across representative datasets and shows a TableBench example rendered with a Direct Prompt.

C Attention Heatmap Comparison Between Datasets

  • Attention visualizations compare patterns across program synthesis, reasoning, and STEM tasks represented by MBPP, MMLU-Pro, and MathVerse.The supplied figures provide examples from each dataset, while the accompanying text frames them as representative task categories.Source: C Attention Heatmap Comparison Between… · p. 14
  • A TableBench example is presented using a Direct Prompt.Source: Figure 9 · p. 15

D Broader Impact Statement

PixelWorld offers a reproducible basis for evaluating unified multimodal perception while highlighting computational, environmental, privacy, and ethical constraints of pixel-based models.

D Broader Impact Statement

  • Pixel-based models retain significantly higher computational costs than text-based counterparts, potentially limiting accessibility and increasing large-scale training and deployment carbon footprints.The statement identifies computational cost as a practical constraint on adoption and sustainability.Source: D Broader Impact Statement · p. 15
  • Pixel inputs may expose sensitive visual information, requiring privacy- and ethics-compliant data synthesis and collection.Source: D Broader Impact Statement · p. 15
  • PixelWorld provides a transparent, reproducible foundation for fair comparisons across models and modalities.The benchmark is positioned as a basis for assessing multimodal understanding and examining where pixel-based representations succeed or fail.Source: D Broader Impact Statement · p. 15
  • By revealing success and failure patterns, the benchmark aims to guide more efficient and interpretable multimodal systems.The stated goal is to encourage broader exploration of unified perception while considering ethical and environmental concerns.Source: D Broader Impact Statement · p. 15

Related papers

Loading 2501.19339v3…