PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering

Anoop Senthil

arXiv:2608.28707v1cs.CLcs.CV

TL;DR

ReVA asks whether visual-question answering becomes more reliable when a language model receives both whole-image and explicit region tokens. It combines multi-level region-aware visual representations with automatic detector proposals, improving POPE object-hallucination performance while preserving or modestly improving several broader VQA results.

  • Problem

    ReVA addresses whether visual questions can be answered more reliably with both image tokens and explicit region tokens for object-level grounding.

  • Method

    ReVA uses image and region bridges to connect frozen CLIP ViT-L/14 features with Qwen2.5-7B-Instruct, using automatic RAM++, spaCy, and Grounding DINO proposals and multi-level cropped ViT features.

  • Results

    POPE mean F1 improves by +1.71 over the image-token baseline, while VQAv2 soft accuracy rises on both test-dev and test-standard.

  • Takeaways & Limitations

    Explicit region tokens improve object-level grounding and reduce hallucination while largely preserving general VQA capability relative to whole-image encoding alone.

  • Takeaways & Limitations

    Residual overlap with evaluation images remains possible despite decontamination of the Stage 2–3 training data.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.

I. INTRODUCTION

ReVA addresses fine-grained grounding limitations in VQA by combining whole-image and region-level evidence for language generation. Its automatic detector stack supplies both question-agnostic and question-dependent regions without inserting boxes as prompt text.

  • Motivation and contribution: ReVA links a frozen CLIP ViT-L/14 encoder and a LoRA-adapted Qwen2.5-7B-Instruct model through image and region bridges.The image bridge produces 576 image tokens, while the region bridge produces local region tokens.
  • Region-aware grounding: RAM++ tags image objects, spaCy adds nouns from the question, and Grounding DINO localises the resulting tags into boxes.The resulting proposals are both question-agnostic and question-dependent.
  • Motivation and contribution: Whole-image encoding can leave spatial relations and attribute–instance binding ambiguous even with a dense 576-token grid.This motivates adding explicit local evidence rather than relying only on image-wide representations.
  • Positioning: The approach combines established frozen-encoder-to-LLM bridge designs with explicit regional representations for finer visual grounding.Related work includes Q-Former, linear or MLP bridges, and text-based location representations.
  • Region-aware grounding: ReVA follows region-token approaches while automatically constructing regions from open-vocabulary detectors for ordinary natural-language VQA inputs.Boxes determine feature crops and do not enter the prompt as text.

III. SYSTEM ARCHITECTURE

ReVA uses a dual-bridge architecture: final CLIP features provide whole-image tokens, while enriched intermediate features are cropped and pooled into region tokens. These representations are concatenated for Qwen to jointly use scene context and local evidence.

  • Image bridge: The image bridge maps final CLIP transformer features through an Image Feature Projector into 576 image tokens.Images are encoded at 336×336 into a 24×24 grid of patch tokens.
  • Region bridge: The region bridge enriches intermediate ViT features with coordinate channels, per-level projections, and attention before RoI Align cropping.This combines shallow texture information with deeper object cues while preserving spatial information.
  • Region bridge: ReVA selects earlier and penultimate ViT representations because the final block is more image–text aligned but less localisation-preserving.The region bridge is therefore designed to retain finer spatial and object-level detail.
  • Region bridge: RoI Align converts each box into a fixed 14×14 feature window, and learnable queries pool each cropped level into K=16 tokens.The 24/336 scale maps image-grid box coordinates to patch-grid coordinates.
  • Region bridge: The Region Feature Projector concatenates four level-specific pooled vectors for each query and maps them into Qwen’s embedding space.Each box contributes K region tokens alongside the 576 image tokens during Stage 2 alignment.

C. Region Detector

ReVA automatically generates regional crops from image tags and question nouns, then supplies their embeddings alongside image tokens for answering.

  • Proposal generation: RAM++ produces up to 20 image tags, which are united with spaCy nouns extracted from the question.This combines image-present objects with question-mentioned nouns.
  • Proposal generation: Grounding DINO localises the combined tag set into zero-shot bounding boxes for regional feature extraction.The boxes are used to crop features rather than appearing as prompt text.
  • Prompt construction: The prompt contains 576 image token embeddings and N × K region token embeddings, followed by an answer-format instruction.The model is instructed to answer using a single word or phrase.

IV. TRAINING REGIME

Training uses three sequential stages so the image and region bridges align independently before Qwen is adapted to combine their outputs.

  • Three-stage training: Stages 1–2 train the image and region bridges separately while CLIP and Qwen remain frozen.This establishes each visual-to-language alignment before joint answering.
  • Three-stage training: Stage 3 freezes both bridges and updates only LoRA adapters in Qwen.The LLM then learns to answer from the concatenated image and region tokens.

A. Stage 1: Image Alignment

Stage 1 aligns the image bridge to Qwen using whole-image data while keeping CLIP and Qwen frozen. The resulting bridge is trained independently of the region bridge and LoRA.

  • A. Stage 1: Image Alignment: The image-bridge training pool contains 558,128 LLaVA-Pretrain images retained after pHash checks against COCO and Visual Genome.Flagged pairs were treated as false positives.
  • A. Stage 1: Image Alignment: The backbone choice is independent of the region bridge and LoRA because Stage 1 trains the image bridge alone.
  • A. Stage 1: Image Alignment: Stage 1 trains only the image bridge on whole-image data, while CLIP and Qwen remain frozen.The image bridge contains approximately 16.5 million parameters.

B. Stage 2: Region Alignment

Stage 2 teaches the region bridge to represent box contents through region–text supervision, while Stage 3 trains Qwen to use image and region tokens for VQA. Training uses decontaminated, source-balanced data and keeps both bridges frozen during LoRA adaptation.

  • B. Stage 2: Region Alignment: Stage 2 trains the region bridge on decontaminated image–box–text pairs from COCO, RefCOCO variants, Visual Genome, and GRIT-20M.Stage 2 uses ground-truth box proposals, while inference later uses Grounding DINO proposals.
  • B. Stage 2: Region Alignment: The region objective combines fluent box-caption training with contrastive alignment that pulls matching region–text pairs together and pushes mismatches apart.The contrastive term uses InfoNCE with τ=0.07 and λ_itc=0.5; only description positions receive loss.
  • B. Stage 2: Region Alignment: Stage 3 trains Qwen to use image and region tokens for VQA with up to 20 boxes per image from five annotated datasets.VQAv2 and A-OKVQA use COCO train2014 boxes, whereas GQA, Visual7W, and VCR use their own annotations.
  • B. Stage 2: Region Alignment: The Stage 3 sampler targets Visual7W, GQA, VQAv2, VCR, and A-OKVQA at 35%, 30%, 15%, 10%, and 10%, respectively.The mix is intended to expose training to pointing, spatial awareness, open-ended perception, commonsense reasoning, and multiple-choice answering.
  • B. Stage 2: Region Alignment: LoRA updates Qwen attention and MLP layers, while the image and region bridges remain frozen during 10,000 optimizer steps.

V. EXPERIMENTAL SETUP

The evaluation covers complementary VQA, perception, reasoning, hallucination, and multimodal-comprehension behaviors using established benchmark protocols. Training and evaluation data are decontaminated, and matched-question statistical tests are reported where predictions are available.

  • V. EXPERIMENTAL SETUP: The study evaluates four primary suites selected to stress complementary failure modes of whole-image MLLMs.The suites are VQAv2, MMBench, POPE, and SEED-Bench.
  • V. EXPERIMENTAL SETUP: VQAv2 measures open-ended short-answer accuracy with official soft scoring over ten human answers per question.The evaluation follows official punctuation and digit/article normalization.
  • V. EXPERIMENTAL SETUP: MMBench evaluates English perception and reasoning through multiple-choice accuracy under CircularEval with shuffled options.This reduces lucky guessing and tests intended-choice following.
  • V. EXPERIMENTAL SETUP: POPE measures object hallucination on random, popular, and adversarial COCO splits using per-split and mean F1.Precision and recall matter because over-answering “yes” can appear accurate while hallucinating.
  • V. EXPERIMENTAL SETUP: SEED-Image reports image-only accuracy across evaluation dimensions 1–9 rather than SEED All.Video questions are excluded because ReVA is an image VQA system.
  • V. EXPERIMENTAL SETUP: Training pools undergo pHash decontamination, with Stage 3 also removing exact question/image ID overlaps.Greedy decoding is used for all reported evaluations, and paired t-tests and McNemar tests are reported when matched predictions exist.

B. Implementation

Implementation uses standard transformer and parameter-efficient fine-tuning software with detector inference on NVIDIA A100 GPUs. The selected CLIP ViT-L/14 resolution improves Stage 1 validation performance over 224 px.

  • B. Implementation: Experiments use PyTorch, Hugging Face Transformers, and PEFT LoRA, with Grounding DINO and RAM++ inference on NVIDIA A100 GPUs in bf16.
  • B. Implementation: 336 px CLIP ViT-L/14 improves GQA validation by 3.90 points and VQAv2 validation by 3.75 points over 224 px.The higher resolution corresponds to a 24×24 grid with 576 tokens rather than a 16×16 grid with 256 tokens.
  • B. Implementation: DINOv2-ViT-L/14 trails CLIP-ViT-L/14-336 under the Stage 1 frozen-encoder comparison.The paper therefore fixes CLIP ViT-L/14 at 336 px.

B. Image-Token Baseline vs. Full System

The full ReVA system adds region tokens to the image-token baseline, improving VQAv2 and POPE performance while leaving broad multiple-choice gains mixed. Matched statistical tests identify a significant benefit on POPE but not on MMBench or SEED-Image.

  • System comparison: The comparison uses ReVA with image and region tokens against an image-token baseline that omits region tokens.The baseline retains the Stage 1 image bridge and Stage 3 LoRA, while the full system adds the Stage 2 region bridge and detector boxes at inference.
  • Benchmark comparison: +1.71 mean F1 points on POPE separates full ReVA from the image-token baseline.The full system improves random, popular, and adversarial POPE F1 by +2.11, +1.45, and +1.57 points, respectively.
  • Benchmark comparison: +0.39 and +0.65 VQAv2 soft-accuracy points appear on test-dev and test-standard, respectively.Full ReVA rises from 72.89 to 73.28 on test-dev and from 73.14 to 73.79 on test-standard.
  • Benchmark comparison: MMBench drops by 1.1 points while SEED-Image rises by 0.18, indicating mixed effects on broad multiple-choice competence.Localisation and attribute comparison improve, whereas coarse perception and relation reasoning fall on MMBench.
  • Statistical significance: VQAv2 is excluded from the paired tests because labeled per-question JSONL dumps were unavailable.Table VI therefore reports EvalAI overall soft scores only.
  • Statistical significance: +1.18 POPE accuracy points are statistically significant under both paired t-test and McNemar test at α=0.01.The paired tests report pt=3.165×10−6 and McNemar p=3.961×10−6; MMBench and SEED-Image show no significant change.

VII. DISCUSSION

ReVA’s region-aware design redistributes MMBench skills toward local object evidence, while residual training–evaluation overlap remains a limitation. The overall MMBench decrease is not statistically significant, so it is better interpreted as skill redistribution than reliable regression.

  • Decontamination and data scale: Residual overlap with evaluation images remains possible despite decontamination, so the retained training pools are not guaranteed to be fully clean.Table VIII reports removals rather than proof that all remaining pools are contamination-free.
  • MMBench skill trade-off: The CircularEval Overall drop of 1.1 points on MMBench is not uniform across skills.The full system gains on object localisation (+8.65), attribute comparison (+6.82), and identity reasoning (+4.45), but declines on several coarse, relational, OCR, recognition, and image-quality skills.
  • Decontamination and data scale: Table VIII presents training-pool sizes after decontamination.
  • MMBench skill trade-off: The Overall drop is not significant under the paired t-test or McNemar test, supporting its interpretation as a redistribution of skill rather than a reliable regression.

VIII. FUTURE WORK

Future work targets detector mismatch, region-token cost, broader grounded tasks, and training-run stability. These directions follow reported constraints in inference-time box proposals, computational overhead, task scope, and single-checkpoint evaluation.

  • Train–test box gap: Training with Grounding DINO proposals, or mixing clean and noisy boxes, could reduce the train–test mismatch at inference.
  • Train–test box gap: A question-relevance reranker could retain at most N regions without forcing the LLM to attend to every detector proposal.
  • Architecture and compute: The region bridge adds up to 20×K tokens beyond 576 image tokens and incurs RAM++/Grounding DINO latency, motivating pruning, smaller K, or distillation.
  • Broader grounded tasks: ReVA is currently trained on short-phrase VQA, leaving open-ended answers, referring dialogue, grounded captioning, and sequential multi-clue reasoning for extension.
  • Multi-seed replication: Tables VI–VII compare one checkpoint pair, measuring question-level reliability rather than stability across training runs.Several seeds would test whether the POPE gain and non-significant MMBench and SEED-Image differences generalize.
  • Results motivating future work: ReVA improves mean F1 on every POPE split and raises VQAv2 soft accuracy on both test-dev and test-standard relative to the image-token baseline.
Loading 2608.28707v1…