PaperlayerView exact revision on arXiv

Source-linked AI summary

Stateful Visual Encoders for Vision-Language Models

Zirui Wang, Junwei Yu, Adam Yala, David M. Chan, Joseph E. Gonzalez, Trevor Darrell

arXiv:2606.04433v1cs.CVcs.CLcs.LG

TL;DR

Existing open-weight VLMs encode images independently, so subtle changes may be weakened before the language model compares them. This paper introduces a Stateful Visual Encoder that conditions current visual features on prior-image features, achieving consistent gains across controlled and real-world visual comparison tasks. The approach remains compatible with pretrained VLM interfaces but is bounded by indirect access to long-range history and evaluation concentrated on comparison domains.

  • Problem

    Open-weight VLM visual encoders are stateless, encoding images independently and potentially suppressing subtle task-critical differences before language-model comparison.

  • Method

    The Stateful Visual Encoder injects cross-image interaction inside the visual encoder by conditioning current visual representations on previous-image features.

  • Results

    SVEs consistently improve controlled and real-world visual comparison tasks across resolutions, model sizes, and VLM families, sometimes matching or surpassing specialized models.

  • Takeaways & Limitations

    State-aware visual encoding offers a practical way to improve multi-image reasoning while preserving the pretrained VLM interface.

  • Takeaways & Limitations

    The formulation accesses long-range evidence only indirectly through intermediate states and is evaluated mainly on image-pair or image-sequence comparison domains.

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly used in multi-image, multi-turn agentic settings where decisions depend on visual changes. However, in existing open-weight VLMs, visual comparisons happen only inside the language model, while the visual encoder itself remains stateless: each image is encoded independently, without access to the prior visual context. As a result, small but task-critical changes may be attenuated before the language model has a chance to compare them, especially when those changes do not affect the high-level semantics of the scene. We introduce a Stateful Visual Encoder, which conditions each visual representation on prior visual features. Under supervised finetuning, VLMs equipped with stateful encoders achieve consistent improvements on controlled tasks involving cross-image spatial aggregation, multi-object visual differencing, and visual trajectory behavior cloning. These improvements are consistent across input resolutions, language model sizes, and VLM backbones. Finally, we validate our model on real-world tasks, including longitudinal radiology, fine-grained image comparison, and remote sensing, where stateful encoders consistently improve generalist VLM baselines and can match or surpass specialized models in selected domains. Project page: https://statefulvisualencoders.github.io/

1. Introduction

Open-weight VLMs typically encode images independently, delaying visual comparison until the language model and risking loss of subtle but task-critical differences. The Stateful Visual Encoder adds cross-image interaction inside the visual encoder and improves comparison across controlled and real-world tasks.

  • Open-weight VLMs usually encode each image independently, leaving visual comparison to the language model and making the visual encoder stateless.
  • Subtle localized changes can be critical for radiology, satellite imagery, and image editing, yet independent per-image encoding may suppress these differences.
  • The Stateful Visual Encoder conditions current visual representations on previous-image features before passing tokens to the language model.
  • SVE variants explore self-attention extension, cross-attention, FFN augmentation, and adaptive normalization, with initialization and gradient choices designed to preserve pretrained behavior.
  • SVEs consistently improve synthetic and real-world comparison tasks, including radiology, fine-grained image comparison, and remote sensing, beyond naive finetuning.
  • The method adds cross-image interactions without replacing the visual backbone or retraining the full VLM from scratch.

2. Related work

Related work includes specialized image-difference encoders, spatiotemporal video encoders, and multi-image VLMs. These approaches provide context for positioning stateful visual encoding among existing comparison and sequence-modeling strategies.

  • Specialized image-difference encoders compare images inside the visual encoder but target specific change-detection tasks rather than general-purpose visual processing.
  • Video visual encoders learn spatiotemporal representations from frame sequences, spanning architectures such as I3D, MViT, Video Swin, TimeSformer, and ViViT.
  • Recent multi-image VLMs show strong multimodal reasoning abilities, but multi-image state reasoning remains challenging and most adopt late fusion.

3. Stateful Visual Encoders

Stateful Visual Encoders add cross-image interaction inside the visual encoder, conditioning current visual tokens on prior-image features for fine-grained comparative tasks. The Cross+FFN design performs best in controlled spatial aggregation, visual differencing, and trajectory cloning, with ablations supporting its stateful and initialization choices.

  • Architecture: SVE designs condition current visual tokens on prior-image tokens through self-attention extension, adaptive normalization, cross-attention, or Cross+FFN.Cross uses current-image queries and previous-image keys and values; Cross+FFN additionally transforms cross-attended features with an FFN.
  • Task Setup: The controlled evaluation covers spatial aggregation, multi-object visual differencing, and visual trajectory behavioral cloning.Tasks measure geometric prediction, categorical and language-based change descriptions, and next-action prediction from visual interaction histories.
  • Results: Cross+FFN performs best across spatial aggregation tasks and improves over Cross, while Self-Ext underperforms the stateless baseline and AdaLN-Zero remains near it.The results suggest token-level retrieval and the added FFN help transform prior-image information for precise cross-image localization.
  • Results: On CLEVR, Cross+FFN improves CIDEr from 529.5 to 543.9 and accuracy from 91.1 to 92.7, while also improving all four VisGym tasks.It improves across CLEVR perplexity, change accuracy, and language-generation metrics; other variants are less consistent or less effective.
  • Ablations: A capacity-controlled Self+FFN baseline remains below Cross+FFN on all tasks except Patch Reassembly, indicating gains are not explained solely by added parameters or FLOPs.Patch Reassembly is the only task where visual comparison is not strictly required.
  • Ablations: The Cross+FFN recipe benefits from cloned input-side attention weights, zero-initialized output projections, stop-gradient on prior keys and values, positional embeddings, and real visual features.Removing zero initialization causes the largest degradation; removing stop-gradient weakens spatial aggregation, while removing positional embeddings degrades spatial aggregation and visual differencing.

4. Feature Analysis of Stateful Representations

Stateful visual encoders produce context-dependent representations by conditioning current-image features on prior visual features. Their changes are useful for visual change understanding and are spatially selective, concentrating updates on localized tokens.

  • Stateful visual encoding reads previous-image features, whereas the capacity-controlled stateless baseline uses current-image features and therefore tests statefulness rather than added capacity.Both models share architecture, parameters, training data, and optimization setup; only the temporal cross-attention source differs.
  • SVE representations depend on the preceding visual state, while the stateless control remains invariant when predecessor images are swapped.SVE produces substantially lower minimum token similarity under predecessor swaps.
  • SVE wins substantially more often among non-tied examples, indicating that state-dependent representation changes predict improved visual change understanding.The comparison uses per-example Change-Acc scores and excludes tied examples.
  • SVE feature analysis compares context-dependent features with a stateless baseline and reports a large SVE advantage when the models disagree on CLEVR-Change.
  • Cross-image updates are spatially sparse: most positions change little, while a small number of tokens absorb most of the update.This pattern supports selective cross-image reading that preserves most pretrained visual features while updating localized features relevant to state comparison.

5. Validating SVE in Real-world Tasks

SVEs improve visual comparison across longitudinal radiology, fine-grained image editing, and remote-sensing tasks. The evaluations use structured clinical metrics, judged edit descriptions, and captioning comparisons with stateless and specialist baselines.

  • 5.1. Longitudinal Radiology: 16,347 paired chest X-rays in Medical-Diff-VQA support longitudinal comparison, where SVE improves subtle-change capture and standard captioning metrics.The pairs come from the same patients and include annotations describing medical changes between studies.
  • 5.1. Longitudinal Radiology: SVEs outperform the stateless baseline on Micro F1, Macro F1, and any-change detection across 27 chest-related finding types.RATE evaluates predicted added or resolved findings against reference checklists.
  • 5.2. Fine-grained Image Comparison: ImgEdit evaluates whether models can infer the transformation between source and edited images across seven change categories.The evaluation excludes shortcut-prone categories such as style change.
  • 5. Real-world qualitative results: Figure 6 shows qualitative SVE-versus-stateless comparisons for radiology, fine-grained image comparison, and remote sensing, with green and red text marking correct and incorrect descriptions.The figure organizes radiology examples at top, fine-grained comparisons bottom-left, and remote sensing bottom-right.
  • 5.2. Fine-grained Image Comparison: An MLLM judge prefers SVE outputs over both the stateless baseline and the original reference instruction.Pairwise preferences are used because reference instructions may not match the actual visual transformation.
  • 5.3. Remote Sensing: Remote-sensing change captioning compares before–after images of the same region to describe newly constructed buildings, removed infrastructure, or altered land use.LEVIR-CC is evaluated with standard captioning metrics and S* following prior work.
  • 5.3. Remote Sensing: SVEs improve over the stateless remote-sensing baseline and outperform all prior specialist models and architectures.Results are reported in the LEVIR-CC evaluation.

6. Conclusion

The paper concludes that Stateful Visual Encoders provide a practical way to add cross-image interaction inside VLM visual encoders. Across synthetic and real-world tasks, they improve multi-image reasoning while preserving the pretrained VLM interface.

  • 6. Conclusion: SVEs consistently outperform stateless baselines across synthetic datasets and real-world applications from longitudinal radiology to remote sensing.The reported robustness extends across resolutions, model sizes, and architectures.
  • 6. Conclusion: The method preserves the pretrained VLM interface while making visual encoders state-aware for tracking, comparing, and reasoning over dynamic visual contexts.The conclusion presents this as a practical path toward improved multi-image reasoning.

A. Limitations

The paper identifies limits in long-range visual comparison, domain coverage, and computational scaling. These boundaries matter especially for long histories and broader multimodal state-tracking settings.

  • Boundary of visual comparison: The current encoder conditions each image only on its immediate predecessor, so evidence from many earlier observations is accessed indirectly.This formulation is suitable for before–after comparison and short visual trajectories but may be insufficient for distributed long-range evidence.
  • Domains that benefit from capturing changes: Real-world evaluations cover radiology, image editing, and remote sensing but do not fully represent multimodal state tracking in embodied agents, robotics, tactile interaction, audio-visual perception, or long-running computer use.The limitation concerns the breadth of evaluated domains rather than the diversity of visual changes within those domains.
  • Computational overhead: SVE adds cross-image computation inside the visual encoder, with overhead that can become nontrivial as image resolution, sequence length, or visual-state count increases.The paper notes that this overhead is usually modest compared with scaling the language backbone.
  • Computational overhead: Scaling stateful visual encoding to very long visual histories will require more efficient memory, retrieval, or sparse-attention mechanisms.The paper identifies these mechanisms as potential responses to the scaling boundary.

B. Training Data Formatting

The appendix standardizes data formatting, task prompts, supervision masking, and evaluation inputs across six task families. It documents both shared conventions and task-specific conversation structures.

  • Common formatting conventions: All non-VisGym datasets use LF’s ShareGPT-style JSONL with messages and images, while VisGym uses an older conversation schema with embedded image paths.The k-th <image> tag binds to images[k] in the non-VisGym format.
  • B.3. Visual Trajectory Behavioral Cloning: Visual Trajectory Behavioral Cloning uses VisGym’s older schema, where later human turns contain environment feedback and new visual observations.Its appendix entry identifies the task as one of the six SVE task families.
  • B.4. Longitudinal Radiology: Longitudinal Radiology presents reference and current chest X-rays from the same patient and supervises templated descriptions of added, missing, or unchanged findings.Medical-Diff-VQA images are converted from DICOM and resized during training for memory reasons.
  • B.5. Fine-grained Image Comparison: Fine-grained Image Comparison repurposes ImgEdit for image-pair and image-sequence descriptions, including identification of underlying editing rules.Only the final edit description is supervised, and images are downsized during training.
  • B.6. Remote Sensing: Remote Sensing uses LEVIR-CC bitemporal satellite images from the same location and prompts models to describe changes between before and after views.The dataset images have fixed resolution 256 × 256.
  • Common formatting conventions: Image tags begin user messages, and history masking supervises only the final answer for single-shot datasets; all tasks use LF’s qwen3 5 template with thinking disabled.Intermediate filler responses are masked out when mask history=True.
  • B.1. Cross-image Spatial Aggregation: Cross-image Spatial Aggregation uses red-dot screenshots to estimate normalized distance or polygon area, with outputs rounded to four decimal places.The task spans two to five images and uses rich AgentNet backgrounds downsampled to 384 × 216.
  • B.5. Fine-grained Image Comparison: Reference captions in ImgEdit come from editing instructions and may describe changes that are not visually present when the editing model fails to follow instructions.This creates a boundary for interpreting instruction-derived reference labels.

C. Evaluation metric conventions

Evaluation uses task-specific protocols aligned with each output type and prior benchmark conventions, while test examples are held out without image-pair or answer-key overlap.

  • All test sets are held out from training, with no overlap in image-pair and target-answer keys.
  • Numeric tasks use regression, agentic tasks use action likelihood, and captioning tasks use standard image-captioning metrics.The protocol is matched to the output type and dominant convention in prior published benchmark work.

C.2. Caption-metric conventions

Captioning evaluation distinguishes a lightweight sanity-check protocol from a paper-aligned protocol used for comparisons with prior work, while multi-object differencing uses order-invariant matching.

  • Captioning results use either lightweight Python metric implementations or a paper-aligned image-captioning protocol.The paper-aligned protocol is the primary basis for comparisons with prior published results.
  • The paper-aligned protocol standardizes caption tokenization before computing BLEU, METEOR, ROUGE-L, CIDEr, and SPICE.Tokenizer choice and METEOR implementation can shift absolute values, especially for short or templated text.
  • Perplexity is token-weighted under the training supervision mask, covering final answer tokens for single-shot captioning and all action-generating turns for behavioral cloning.The computation uses n_i, the number of supervised tokens for sample i.
  • Multi-object visual differencing receives permutation-invariant scores because the order of simultaneously described changes has no semantic content.Predictions and references are split into changes, scored against valid lexical variants, and matched one-to-one to maximize total similarity.

D. Training Configuration, Environment, and Infrastructure

Experiments use a fixed, fully sharded full-finetuning setup that trains the visual encoder end-to-end alongside the language model and projector, with task-specific training choices.

  • All experiments use a single Python 3.12 environment with PyTorch, Transformers, Accelerate, FlashAttention, LlamaFactory, and bf16 precision.The software environment is summarized in Table 15.
  • Full-finetuning runs use single-node, 8-GPU FSDP with full sharding across language-model decoder and vision-encoder blocks.Gradient checkpointing uses a non-reentrant implementation for FSDP compatibility.
  • The language model, visual encoder, and multimodal projector remain trainable because the method modifies the visual encoder and trains it end-to-end.
  • Baseline and SVE runs reuse identical cached text-image inputs when data-formatting settings match.Cache keys include the chat template, cutoff length, history-masking setting, and dataset identity.
  • Design-table tasks use short fixed-step training, whereas real-world tasks use epoch-based training and task-specific history masking.Real-world multi-image captioning masks filler assistant turns; synthetic and agentic tasks use their available assistant turns as signal.
  • Real-world tasks initialize the SVE output projection with σ = 10−4, compared with standard Transformer-scale σ ≈2 × 10−2.Controlled design tasks use zero initialization, while the nonzero real-world initialization followed ablations showing better optimization.

F. Finding-level Medical-Diff-VQA Evaluation Details

Medical-Diff-VQA evaluation parses reference and model outputs into finding-level change labels, then reports label-level F1 and pair-level change accuracy with explicit handling of no-change cases.

  • The test split contains 16,347 paired chest X-rays evaluated with a reference-versus-current-image change prompt.
  • References and free-form outputs are parsed into added, missing, and no-change finding tuples using a regular-expression pipeline.Outputs that do not follow the template account for less than 2% of cases and are treated as no change.
  • Each test pair is represented by a 54-dimensional binary vector covering 27 finding categories and two directions.Per-label F1 is computed from accumulated true positives, false positives, and false negatives.
  • Finding-level F1 is calculated on the 14,030 pairs whose references identify at least one specific change.No-change pairs use an all-zero finding vector rather than finding-level annotations.
  • Change Accuracy measures whether the model correctly recognizes that the patient is stable at the pair level.
  • Per-anatomy analysis decomposes Micro F1 by the anatomical groupings used in the finding-category evaluation.The breakdown compares SVE with the stateless baseline.
Loading 2606.04433v1…