PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvCue: Complementary Visual Inductive Biases for Vision-Language Models

Zixuan Lan, Shichu Sun

arXiv:2609.34196v1cs.CV

TL;DR

Visual representations may miss distinctions required by downstream questions, motivating ways to supplement existing VLMs. ConvCue adds adapted features from a frozen pretrained CNN through gated cross-attention, and it achieves higher average performance than original models and matched fine-tuning controls across three backbones.

  • Problem

    Visual representations may not capture all distinctions needed for downstream questions, prompting the question of whether supplementary representations can improve existing VLMs.

    Source: §1 · p. 1

  • Method

    ConvCue adapts features from a parallel frozen pretrained CNN and fuses them with native visual tokens through gated cross-attention before the original projector.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Across three backbones and 13 benchmarks, ConvCue achieves higher average performance than both the original models and matched Direct SFT controls.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pretrained convolutional representations, integrated through learned adaptation and fusion, can improve existing VLMs without replacing their original visual encoders.

    Source: §1 · p. 3

Abstract

from arXiv · show

Modern vision-language models (VLMs) achieve strong performance across a broad range of multimodal tasks, yet still struggle with visual questions that require fine-grained discrimination and spatial understanding. These limitations motivate investigating whether supplementary visual representations can improve existing VLMs without replacing their native visual encoders. Pretrained convolutional networks offer a candidate feature source, motivated by their local connectivity and spatial weight sharing. We introduce CONVCUE, which augments the native visual representations of a pretrained VLM with final-stage features from a parallel, frozen pretrained CNN. A learnable adapter maps convolutional features to the native visual feature dimension, while gated cross-attention allows the original visual tokens to retrieve information from the CNN features. The enhanced tokens are passed through the original visual-to-language projector, and the model is adapted through a two-stage training procedure. We evaluate CONVCUE on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B across 13 multimodal benchmarks covering visual question answering, document and chart understanding, and multimodal reasoning. CONVCUE improves average benchmark performance over both the original models and matched two-stage fine-tuning controls on all three backbones. On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks and raises the average score from 75.00 to 78.82 relative to the matched fine-tuning control. These results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

1 Introduction

Visual representations can miss distinctions needed for downstream questions, motivating the study of supplementary pretrained CNN features. ConvCue integrates them into existing VLMs, and its evaluations report higher average performance than original models and matched fine-tuning controls.

1 Introduction

  • Visual representations may support their training task without capturing all distinctions needed for downstream questions, such as object attributes and relationships.This motivates investigating whether pretrained CNN representations can provide useful supplementary information for existing VLMs.Source: §1 · p. 1
  • 6.14 percentage points: cross-attention outperforms weighted fusion in the full visual-side adaptation setting of the Spatial QA pilot.The pilot figure also compares accuracy by position and trainable-component configuration.Source: Figure 1 · p. 2
  • ConvCue retains the original visual pathway while adapting and fusing features from a separate frozen pretrained CNN.The enhanced representations pass through the original visual-to-language projection module.Source: §1 · pp. 2–3
  • Across three backbones, ConvCue achieves higher average performance than both original models and matched Direct SFT controls across 13 benchmarks.On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks.Source: §1 · p. 2

2 Pilot Experiments

Pilot experiments compare fusion strategies and visual-side adaptation configurations on LLaVA-1.5-7B using Spatial QA. Cross-attention and training original visual components alongside the added modules guide ConvCue’s subsequent design.

Pilot setup

  • The pilots use LLaVA-1.5-7B and COCO-derived Spatial QA, with the language model and pretrained CNN frozen.The experiments guide initial design choices but do not establish their effects on other models or tasks.Source: §2 · p. 3

An initial comparison of fusion strategies

  • 87.08% overall accuracy with cross-attention versus 80.94% with weighted fusion; cross-attention is also higher in all five position categories.The comparison trains the original visual encoder and projector together with the added learnable components.Source: §2 · p. 3

Exploring the scope of visual-side adaptation

  • In pilot cross-attention configurations, training only the added adapter and fusion module gives 84.90% overall accuracy; including the projector raises it to 86.57%, and additionally training the visual encoder raises it to 87.08%.Source: §2 · p. 3

3 Method

ConvCue augments a VLM’s visual tokens with adapted features from a frozen pretrained CNN, then passes the enhanced tokens through the original projector. Two-stage training adapts the visual pathway and language model while keeping the CNN frozen.

3.1 Preliminaries

  • A VLM’s visual encoder produces visual tokens, which its projector maps into representations compatible with the language model; ConvCue augments those tokens before projection.In this paper, visual tokens refer to features before projection.Source: §3.1 · p. 3

3.2 Convolutional Feature Extraction and Adaptation

  • A frozen CNN processes the same image; a trainable adapter projects its final-stage feature map to the visual-token dimension while preserving its spatial grid.The adapter uses a 1 × 1 convolution, spatial flattening, LayerNorm, and a two-layer MLP; CNN and original token counts need not match.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Gated Cross-Attention Fusion

  • Original visual tokens query adapted CNN tokens through cross-attention, then a shared channel-wise gate scales the residual update before the original projector.The fusion preserves the original visual-token shape and does not require one-to-one spatial correspondence.Source: §3.3 · p. 4

3.4 Training

  • Training has an alignment stage with the language model frozen, followed by supervised fine-tuning that also unfreezes the language model; the CNN stays frozen in both.The first stage trains the visual encoder, adapter, fusion module, and projector; the second additionally updates the language model.Source: §3.4 · p. 4

4 Experiments

Experiments compare ConvCue with original models, matched Direct SFT controls, and other visual-enhancement methods across three backbones and multiple visual-understanding benchmarks. ConvCue achieves stronger benchmark results than the original models and matched controls, with gains across task categories.

4.1 Experimental Setup

  • The evaluation covers 13 visual-understanding benchmarks and compares three backbones, using shared training data and greedy decoding for the models tested.The benchmarks include visual question answering, OCR, multimodal reasoning, document understanding, and chart understanding tasks.Source: §4.1 · p. 5

4.2 Main Results

  • ConvCue-4B improves over Qwen3-VL-4B on all 13 evaluated benchmarks, spanning visual question answering, multimodal reasoning, and document and chart understanding.The results also report that ConvCue-4B outperforms Qwen3-VL-8B on all 13 benchmarks.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • Across Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B, ConvCue outperforms matched Direct SFT controls using the same data, training procedure, unfreezing settings, and hyperparameters.Direct SFT provides modest average gains on the Qwen3-VL backbones and slightly reduces LLaVA-OneVision-7B’s average score.Source: §4.2 · pp. 6–7
  • Figure 4 contrasts visual-branch sensitivities, token contributions for the baseline, Direct SFT, and ConvCue, and layer-wise answer readouts on a red-car-location question.Its signed maps encode positive or negative contributions toward the centered answer versus foreground-left, while brighter reading weights indicate stronger attention.Source: Figure 4 · p. 7
  • ConvCue-4B achieves the highest average score and outperforms four visual-enhancement methods on eight of nine shared benchmarks, with ScienceQA the exception.The comparisons include general visual understanding and text, chart, and document understanding.Source: Table 2 · p. 6, §4.2 · p. 7

5 Analysis

The analysis finds that ConvCue adds a substantial residual to the original visual features. In one case study, its visual contributions and later-layer answer preference differ from the comparison models, which ultimately choose the wrong answer.

5.1 Magnitude of Visual Representation Updates

  • 18.40% mean and 18.16% median update-to-feature ratios show that ConvCue’s added residual is substantial on the analyzed ERQA subset.The 10th–90th percentile range is 16.15%–20.51%.Source: §5.1 · p. 8

5.2 Case Study of Visual Responses and Answer Preferences

  • ConvCue retrieves CNN features from surrounding regions near positive CNN sensitivity responses, and the two visual branches show different spatial sensitivity patterns.The attention weights depend on image features, not the question, and do not indicate support for either answer.Source: §5.2 · p. 8
  • The original and matched-training models show direct visual-token contributions toward the correct option in parts of the sky and grass yet select the wrong option; ConvCue selects A with a different contribution pattern.The maps account for attention weights and value content but exclude indirect effects through subsequent computation, so they do not by themselves explain the final answer choice.Source: §5.2 · p. 8
  • All three models favor the correct option in some intermediate layers, but only ConvCue recovers that preference after a decline around layer 25 and selects A at the final layer.The comparison models’ margins fall and the correct option’s vocabulary ranking worsens around layer 25, and both ultimately select B.Source: §5.2 · p. 9

6 Related Work

Prior work improves VLM visual representations through image processing, multi-scale features, and combinations of encoders or feature depths. ConvCue instead adds a supplementary convolutional branch while retaining the existing VLM’s visual encoder.

6 Related Work

  • Prior approaches include image slicing and token compression, multi-scale features, combining visual encoders, and combining features from different depths and prompts.Examples include LLaVA-UHD, S2, Cambrian-1, Eagle, and Florence-VL.Source: §6 · p. 9
  • Unlike approaches that replace or integrate CNN features into a ViT pathway, ConvCue retains the existing encoder and uses native visual tokens to retrieve frozen-CNN features through gated cross-attention.This avoids requiring one-to-one spatial correspondence or increasing the number of visual tokens passed to the language model.Source: §6 · p. 9

7 Conclusion and Limitations

ConvCue adds pretrained convolutional features to existing VLMs through a frozen CNN branch, a learnable adapter, gated cross-attention, and two-stage adaptation. Across 13 benchmarks it improves average performance over original models and matched fine-tuning controls, while its evaluation scope leaves other CNN settings and video for future study.

7 Conclusion and Limitations

  • ConvCue integrates frozen pretrained CNN features into existing VLMs through a learnable adapter and gated cross-attention, retaining the original visual pathway and projected-token count.The framework is adapted in two training stages.Source: §7 · p. 9
  • Across 13 benchmarks, ConvCue has higher average performance than both original models and matched fine-tuning controls on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B.On Qwen3-VL-4B, it improves over the original model on all 13 benchmarks, spanning visual question answering, document and chart understanding, and multimodal reasoning.Source: §7 · p. 9
  • The experiments cover image-based tasks with one pretrained CNN backbone at a fixed input resolution; other architectures, resolutions, and video remain unevaluated.Video extension also requires investigating how supplementary features should be integrated across frames.Source: §7 · p. 9

A Experimental Settings

The experiments use a two-stage training setup and matched Direct SFT controls, with benchmark-specific evaluation splits and scoring. The appendix also describes a pilot Spatial QA dataset for coarse absolute object localization.

A Experimental Settings

  • ConvCue uses a frozen pretrained ConvNeXt-Base branch and its final-stage feature map, with CNN inputs center-cropped to 336 × 336 and ImageNet-normalized.The models retain each backbone’s image processor and tokenizer.Source: A Experimental Settings · p. 13
  • The two training stages use image–text pairs for alignment and visual instruction examples for supervised fine-tuning, with the same stage-specific data used for ConvCue and Direct SFT.The second stage initializes from the first-stage checkpoint and additionally unfreezes the language model.Source: A Experimental Settings · p. 13
  • Direct SFT controls follow ConvCue’s two-stage procedure, data, duration, batch configuration, and optimization settings, but omit ConvCue; original-model baselines receive no additional training.The original visual encoder and visual-to-language mapping modules are trained in both Direct SFT stages.Source: A Experimental Settings · p. 13
  • Evaluation uses specified test, development, validation, or test-dev splits and task-specific scoring; the reported overall average is the unweighted mean across 13 benchmarks.Reported scores use a 0–100 scale.Source: A Experimental Settings · p. 13

B Spatial QA Construction

  • Spatial QA questions are generated from COCO object categories and bounding boxes, labeling object centers in four corner cells or the center of a 3 × 3 grid.Images with multiple instances of the same category and objects smaller than 0.5% of image area are excluded; test and validation splits are balanced across five labels and share no images.Source: B Spatial QA Construction · p. 14

C.1 Additional Details of Pilot Experiments

Across three backbones, ConvCue outperforms matched Direct SFT controls, and its average-score gains extend across model sizes and families. The pilot experiments compare fusion configurations and trainable visual-side components under controlled settings.

C.1 Additional Details of Pilot Experiments

  • Pilot fusion experiments use LLaVA-1.5-7B and a shared COCO-derived Spatial QA dataset, with the language model and pretrained CNN frozen.They report overall and per-position answer accuracy; data splits are described in Appendix B.Source: §C.1 · p. 14
  • The pilot compares weighted fusion, which requires spatial grid alignment, with gated cross-attention, which uses original visual tokens as queries without one-to-one spatial correspondence.Both settings train the original visual encoder, projector, CNN adapter, and fusion module under the same data splits and hyperparameters.Source: §C.1 · pp. 14–15
  • Weighted fusion projects CNN features to the ViT dimension, pools them to 24 × 24, and mixes aligned features using channel-wise weights initialized to 0.9 for ViT features.The mixing weights are shared across positions and images.Source: §C.1 · p. 14
  • Cross-attention retains the CNN grid and adds retrieved features through a channel-wise gated residual connection followed by LayerNorm.Original visual tokens are queries; adapted CNN tokens are keys and values, allowing attention across CNN positions without equal token counts.Source: §C.1 · p. 14
  • With cross-attention fixed, pilot settings train the adapter and fusion module alone, add the original projector, or also add the original visual encoder; the language model and CNN remain frozen.These configurations are labeled A, A+P, and A+P+E, respectively.Source: §C.1 · p. 15

C.2 Generalization Across Model Backbones

  • ConvCue raises average scores from 69.65 to 72.64 on Qwen3-VL-2B, 74.12 to 78.82 on Qwen3-VL-4B, and 70.91 to 80.66 on LLaVA-OneVision-7B.The results span two Qwen3-VL sizes and the LLaVA-OneVision model family, while improvement magnitude varies across backbones.Source: §C.2 · p. 15

C.3 Comparison with Direct Supervised Fine-Tuning

  • ConvCue scores 72.64, 78.82, and 80.66 on Qwen3-VL-2B, Qwen3-VL-4B, and LLaVA-OneVision-7B, respectively, outperforming matched Direct SFT controls.Direct SFT averages 69.81 on Qwen3-VL-2B and 75.00 on Qwen3-VL-4B; its 70.52 average on LLaVA-OneVision-7B is slightly below the original baseline.Source: §C.3 · p. 15
  • Compared with Direct SFT, ConvCue scores higher on 10 benchmarks and ties on one for Qwen3-VL-2B, improves on 12 for Qwen3-VL-4B, and wins all 13 for LLaVA-OneVision-7B.GQA, RealWorldQA, and HallusionBench improve across all three backbones, though gains are not universal on every benchmark.Source: §C.3 · p. 15

D Inference Efficiency

Under the evaluated synthetic-input settings, ConvCue adds modest first-token latency and memory overhead. The latency addition remains approximately constant as text context grows while image resolution is fixed.

D Inference Efficiency

  • Measurements use synthetic 896 × 896 images, a 336 × 336 CNN input, batch size one, and greedy generation of one token on a single NVIDIA A100.The Qwen backbone uses BF16, while the CNN branch, adapter, and fusion module use FP32.Source: D Inference Efficiency · p. 15
  • Time-to-first-token measurements include visual encoding, fusion, language-model prefill, and token selection, while excluding preprocessing, data transfer, and model loading.After 10 warm-up calls, 30 measurements per configuration provide median latency and maximum peak allocated memory.Source: D Inference Efficiency · p. 16
  • ConvCue adds 7.4 to 9.6 ms latency and approximately 0.41 GiB peak memory across both evaluated model sizes and input lengths.With image resolution fixed, added latency remains approximately constant as text context grows, reducing relative overhead at longer inputs.Source: D Inference Efficiency · p. 16, Table 4 · p. 16

Related papers

Loading 2609.34196v1…