CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion

1. Introduction

• Token insertion interleaves visual and text embeddings, but its memory and compute costs grow with image-token counts, constraining high-resolution images, multi-image conversations, and streaming video.Recent cross-attention models lag insertion-based models on some tasks, particularly document and chart understanding, and the causes of that gap remain poorly understood.Source: §1 · p. 1
• The paper analyzes efficiency trade-offs and tests cross-attention both from a text-only LLM and by adapting a pretrained insertion-based VLM.In the matched training setting, vanilla cross-attention narrows the gap to token insertion to only a few percent on most benchmarks.Source: §1 · p. 1
• Cross-attention offers memory and speed advantages at training and inference, and maintains near-constant memory and latency for real-time video captioning.Token-insertion models quickly exhaust their memory budget in this streaming-video setting.Source: §1 · pp. 1–2
• The adapted model retains most of Qwen2.5-VL 3B’s performance and outperforms prior cross-attention VLMs, especially on ChartQA and DocVQA.The comparison spans prior cross-attention model scales of 2B–14B; controlled comparisons find the gap to token insertion is only a few percent on most benchmarks.Source: §1 · p. 1, Figure 1 · p. 2

2. Related work

• Token insertion places visual embeddings in the language model’s input sequence, where self-attention enables visual-text interaction, but visual-token growth raises memory and compute costs.Proposed cost mitigations include token compression, insertion in only some layers, reduced attention costs, KV-cache compression, and positional-embedding changes; these techniques are largely orthogonal to fusion choice.Source: §2 · p. 2
• Flamingo popularized gated cross-attention for conditioning a frozen LLM on visual inputs; later work revisited cross-attention for long-context and streaming applications [6] [23].Prior approaches include updating visual representations [23], adding register tokens [6], and bespoke architectural changes.Source: §2 · p. 2
• Recent cross-attention VLMs underperform token-insertion models on fine-grained chart and document tasks, but identical-pipeline comparisons show a smaller gap even with vanilla cross-attention.The text attributes the prior variants’ efforts to improving cross-attention performance, but does not establish the causes of the performance gap.Source: §2 · p. 2
• Streaming approaches constrain token-insertion history through KV-cache compression, frame pruning, or visual memories, whereas cross-attention naturally handles dense image streams.StreamChat [23] updates visual keys and values at each decoding step and uses dedicated FFNs for visual tokens; ablations find those updates give a small performance boost but incur high memory and compute costs, especially during training.Source: §2 · p. 2

3. To Cross-Attend or Self-Attend?

• Cross-attention avoids adding image tokens to the KV cache or updating them through FFNs, but image representations remain unrefined and past images are not directly accessible.These are central efficiency and performance trade-offs of the mechanism.Source: §3 · pp. 2–3

3.1. Preliminaries

• CA layers let text tokens attend to the corresponding image within a local window; their output is combined with text-only causal self-attention, which preserves text-to-text interaction.In the studied design, CA and text-only SA run in parallel with separate projections on the same inputs.Source: §3.1 · p. 3

3.2. From Cross-Attention to Token Insertion

• Sharing projections, adding local text-to-text attention, and replacing some SA layers with CA are design choices that progressively bring CA closer to token insertion.Replacing every second SA layer with CA yields efficiency gains with only a minor performance drop compared to CA; not every layer can be replaced because text-to-text attention must remain.Source: §3.2 · p. 3
• For video, each window contains the latest frame and gist tokens after prior frames, so the model accesses the last frame plus compressed visual history.In the described video QA setting, the authors report strong performance even with this compressed history.Source: §3.2 · pp. 4–5
• D1–D3 can use lightweight modifications, whereas updating images through FFNs adds substantial training memory and retaining past images in the KV cache conflicts with streaming efficiency.The latter trade-off arises because accumulated image tokens add memory and latency overhead.Source: §3.2 · pp. 4–5

3.3. Scalable Cross-attention Training

• Training packs multimodal sequences and uses block-wise FlashAttention-2 windows delimited by image insertion points to match CA's corresponding-image inference behavior.For CAt+v, the image must begin each window to preserve text-token causality under the implementation's bottom-right-aligned attention mask.Source: §3.3 · p. 5

4.1. Experimental Setting

• The experiments train a CA model from text-only Helium1-2B and adapt a frozen Qwen2.5-VL-3B model, using Qwen2.5-VL’s vision encoder in both settings.Evaluation covers image benchmarks including document and chart understanding, OCR, and general QA; video benchmarks cover understanding, temporal reasoning, and long-video tasks.Source: §4.1 · p. 5

4.3. Adapting an Insertion VLM to Cross-Attention

• Adapting Qwen2.5-VL with CA layers causes a 6.8% average performance drop, while retaining most base-model capabilities after 25k training steps.Only the CA layers and last four visual-encoder blocks are trained; other parameters remain frozen.Source: §4.3 · p. 6, Table 3 · p. 7

4.4. Ablation Experiments

• Fewer CA layers tune the efficiency–performance trade-off; FFN-updating image tokens adds ≈2% average performance but substantially raises training memory.The layer-frequency comparison varies replacement intervals; the image-token update ablation freezes the image encoder to accommodate the higher memory use.Source: §4.4 · p. 6, Table 6 · p. 8

4.2. From LLM to VLM with Cross-Attention

• Vanilla CA and its variants trail matched-condition token insertion by an average 1.5 percent, with the largest gaps on ChartQA and InfographicVQA.Table 2 compares the fusion approaches; the remaining gap is concentrated on complex graphics and figures.Source: §4.2 · p. 6, Table 2 · p. 7

4.4. Ablation Experiments

• Token compression has limited impact on general VQA but significantly hurts HRES and OCR tasks, and does not remove streaming memory bottlenecks.The bottleneck remains because the KV cache continues to grow during streaming inference.Source: §4.4 · p. 7, Table 5 · p. 8

4.5. Application to Live Video Captioning

• Despite its smaller size (3B versus 7B), the CA model scores similarly to LiveCC on LiveSports captioning and sustains low-latency inference over longer streams.Insertion-based methods become progressively slower and use memory more rapidly as frames accumulate; the CA model maintains high inference speed over longer horizons.Source: §4.5 · pp. 8–9, Figure 4 · p. 9

• The authors identify five design differences that progressively bridge cross-attention and token insertion, then assess their efficiency and performance impacts.Source: §5 · p. 9
• Simple cross-attention performs close to token insertion on most image and video benchmarks, even without prior architectural modifications.Source: §5 · p. 9
• The adapted model outperforms prior, larger cross-attention VLMs and supports live captioning with near-constant memory and latency.Source: §5 · p. 9

• Image models use FineVision and selected LLaVA-OneVision-1.5 datasets, together covering 24M images across diverse vision-language tasks.Source: §A.1 · p. 13
• Images are processed at native resolution up to 9522 pixels, with pixel unshuffling reducing visual tokens by a factor of four.Source: §A.1 · p. 13
• Video training uses LLaVA-Video-178K, sampling frames at 2 fps except clips under 10 seconds, which use 20 uniformly sampled frames.Source: §A.1 · p. 13
• Training uses 64 H100 GPUs, with image and video samples mixed at a 3:1 ratio and model-specific sequence and step schedules.Source: §A.1 · p. 13
• Optimization uses answer-token-only next-token loss and AdamW, with learning rates of 10−4 for new parameters and 10−5 for adapted parameters.Source: §A.1 · p. 13

• Packed token-insertion samples use block-wise attention so each question-answer sample attends only to itself, avoiding padding while matching batched training.Source: §A.2 · p. 13
• Cross-attention windows pair images with associated text; video settings use per-frame windows or, for comparison, one window spanning all frames.Source: §A.2 · p. 13
• Qwen2.5-VL retains its chat template but omits image-token placeholders for cross-attention, while Helium uses a minimal turn-delimited template.Source: §A.2 · pp. 13–14
• Training configurations distinguish newly introduced from pretrained parameters, and the appendix evaluates CAt+v-Helium-2B and InsertionHe-2B at different image resolutions.Source: Table 7 · p. 14, Table 8 · p. 14
• Images use Qwen2.5-VL preprocessing, videos are processed frame by frame, and CAt+v applies the language backbone’s RoPE with image temporal positions frozen.Source: §A.2 · p. 14

• Most performance gains occur within the first 20k training steps for both the Helium-trained and Qwen2.5-VL-adapted models.Source: §B.1 · p. 14
• Figure 5 reports training-time performance for the Helium-trained cross-attention model, while Figure 6 reports the adapted Qwen2.5-VL model.Source: §B.1 · p. 14
• Final results are reported at 40k training steps for the Helium model and 25k steps for the adapted model.Source: §B.1 · p. 14

B.2. Impact of image resolution

• Reducing inference resolution causes little to no performance loss on general VQA tasks, but noticeably lowers performance on DocVQA and InfoVQA.The section reports results at a maximum resolution of 952^2 and additional results across 448^2, 672^2, 896^2, and 1344^2.Source: §B.2 · p. 14
• Table 2 reports VLM benchmark results for CAHe-2B and InsertionHe-2B evaluated at a maximum resolution of 952^2.Source: §B.2 · p. 14
• Table 8 reports results for inference with maximum resolutions of 448^2, 672^2, 896^2, and 1344^2.Source: §B.2 · p. 14

C. Live Video Captioning

• The CAtv, Q2.5-VL model is finetuned on Live-Whisper-526K for up to 20,000 steps for streaming video captioning.Source: C. Live Video Captioning · p. 14
• Caption subtitles are displayed when generated, making the qualitative videos mimic livestreaming conditions.Static excerpts are shown in Figures 3 and 7; full samples are on the project page.Source: C. Live Video Captioning · p. 14, Figure 7 · p. 17
• Figures 5 and 6 show performance across training on all benchmarks for CA models Helium1-2B and Qwen2.5-VL 3B, respectively.Source: Figure 5 · p. 15, Figure 6 · p. 16