PaperlayerView exact revision on arXiv

Source-linked AI summary

FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding

Amitangshu Mukherjee, Kaushik Roy

arXiv:2609.04392v1cs.CV

TL;DR

Fine-grained visual understanding needs local detail, but full-image high resolution is costly and compact global encoding can weaken that evidence. FAVE adaptively encodes externally selected regions while retaining global context, improving results across small-object recognition, TextVQA, and GQA attributes.

  • Problem

    Visual encoders must preserve both global scene context and fine local evidence, although full-image high resolution is costly and downsampling can discard task-relevant details.

    Source: §1 · p. 1

  • Method

    FAVE preserves native region geometry and adapts local resolution and token count for selected crops, complementing FastVLM’s global tokens while its global pathway and language model remain frozen.

    Source: §1 · pp. 2–3

  • Results

    FAVE improves ImageNet small-object Top-1 by 9.4 points, TextVQA by 1.60 points, and FastVLM-1.5B GQA attribute accuracy by 1.31 points.

    Source: §1 · p. 3, §3.1 · p. 6, §4.2 · p. 8

  • Takeaways & Limitations

    Targeted local visual capacity provides an efficient complement to broader global representations and model scaling for fine-grained evidence.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Fine-grained visual understanding depends on local detail, yet visual encoders face a trade-off between costly full-image high-resolution processing and compact global encoding that can weaken such evidence. Inspired by human active vision, we separate where to look from what to encode. We focus on the latter and introduce FAVE (Foveated Adaptive Visual Encoding), a lightweight variable-resolution ViT that encodes externally selected regions at high acuity while preserving native geometry. We first isolate this encoding problem using oracle ground-truth crops in a controlled small-object regime. On ImageNet objects with a native maximum side of 96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7 times lower FLOPs. Increasing global resolution or backbone capacity does not recover the same operating point. We then integrate FAVE as a complementary local branch in FastVLM. Its local tokens are combined with FastVLM's global visual tokens, while the original global pathway and language model remain frozen. With at most 16 additional local tokens, FAVE improves TextVQA by 1.60 points and achieves a 3.3 times controlled TTFT speedup over SmolVLM2-2.2B. On GQA attribute questions, it improves FastVLM-1.5B by 1.31 points, extending the benefit beyond text while narrowing the gap to FastVLM-7B. Together, these results show that selectively allocating high-acuity local capacity provides an efficient complement to broader global representations and model scaling for fine-grained understanding of small objects, text, and attributes.

1. Introduction

FAVE allocates high-acuity processing to selected regions while retaining global context, addressing the trade-off between costly full-image detail and efficient but potentially lossy downsampling. The paper tests this encoding approach on ImageNet and as a local complement to FastVLM.

1. Introduction

  • Global scene structure supplies context, while fine local details distinguish small objects, attributes, and text; useful perception must preserve both without uniformly high resolution.Doubling both ViT image dimensions yields four times as many visual tokens, whereas global downsampling can discard small task-relevant evidence.Source: §1 · p. 1
  • FAVE separates where to look from what to encode: an external selector chooses regions, and an adaptive local encoder preserves evidence within them.The ImageNet study uses oracle ground-truth boxes to isolate encoding once a region is selected; the VLM setting tests complementarity with a compact global pathway.Source: §1 · pp. 1–2
  • FAVE preserves each selected region’s native geometry and adapts its resolution and token count to region scale, rather than forcing every region to a common resolution.Small regions remain computationally small, while larger regions can use more visual computation within the budget.Source: §1 · p. 2
  • On ImageNet objects with native maximum side ≤96 pixels, FAVE improves Top-1 by 9.4 points over a fixed-resolution ViT on the same crop window with 12.7× lower FLOPs.FAVE also improves TextVQA by 1.60 points with at most 16 additional local tokens and FastVLM-1.5B attribute accuracy on GQA by 1.31 points.Source: §1 · p. 3
  • In FastVLM, FAVE adds sparse local tokens while the original global pathway and language model remain frozen.This setup isolates the local visual pathway’s contribution while retaining FastVLM’s global scene context.Source: §1 · pp. 2–3

2. Related Work

Related work spans biologically inspired foveation, efficient and adaptive visual encoding, and compact multimodal perception. FAVE differs by applying adaptive variable-resolution encoding to externally selected regions while retaining a separate global pathway.

2.1. Foveated vision

  • Foveated vision models peripheral processing with eccentricity-dependent blur, pooling, or sampling, while FocL uses fixed-resolution object crops and Segment This Thing uses a fixed point-centered pattern.FAVE instead adapts local resolution to the native geometry of externally selected regions.Source: §2.1 · p. 3

2.2. Efficient and Adaptive Visual Encoding

  • Efficient visual encoders use multiscale attention, efficient inference, variable-resolution inputs, or token reduction; NaViT preserves native resolution and aspect ratio with Patch-n-Pack batching.FAVE applies variable-resolution processing to selected regions, adapting their spatial budget without token pruning or learned selection.Source: §2.2 · p. 3

2.3. Efficient and Adaptive Multimodal Perception

  • Compact VLMs reduce cost through smaller language backbones, efficient visual encoders, or compact visual-token sequences, while other methods refine or revisit informative regions.Multi-encoder systems combine visual representations; FAVE instead retains FastVLM’s global representation and re-encodes externally selected regions with a lightweight local pathway.Source: §2.3 · p. 3

2. Related Work

  • FAVE’s object-recognition pipeline preserves crop aspect ratio, forms variable-resolution patch grids, and produces one CLS representation per crop for recognition.The sequences can be padded or packed under a token budget with block-diagonal attention to prevent cross-sample interaction.Source: Figure 2 · p. 4

3. Foveated Adaptive Visual Encoding

FAVE encodes selected crops at adaptive resolution while preserving their geometry, then evaluates the approach with oracle spatial selection on ImageNet. Its small-object results outperform fixed-resolution controls, and targeted compute allocation and backbone-scaling comparisons test the source of that advantage.

3. Foveated Adaptive Visual Encoding

  • FAVE crops retain native dimensions and aspect ratio, align boundaries to a 16-pixel patch grid, and use variable patch grids with factorized positional encoding.Patch counts vary with crop geometry, allowing one ViT to process different grid shapes without fixed-grid positional embeddings.Source: §3 · p. 4
  • Variable-length crop sequences are either padded to the longest sequence or packed under a token budget with block-diagonal attention that prevents cross-sample interaction.Both batching schemes preserve each sample’s token sequence and can run the same FAVE checkpoint.Source: §3 · pp. 4–5

3.1. Adaptive Object Recognition

  • Ground-truth boxes isolate how a selected region is encoded, while the main comparison uses Tiny-scale ViTs and contrasts object-crop training with full-image controls.The evaluation uses ≤96, ≤224, and unrestricted General-2K cohorts; larger fixed-resolution backbones are evaluated separately.Source: §3.1 · p. 5
  • On the native-bbox ≤96 cohort, FAVE reaches 50.45 Top-1, 9.4 points above fixed-resolution ViT-T/16 on the same crop window, with 11.1× fewer tokens and 12.7× lower analytical FLOPs.Table 1 compares full-image and GT-selected object regions across native object-scale cohorts; ground-truth boxes fix spatial selection.Source: Table 1 · p. 5, §3.1 · p. 6
  • Scale-dependent magnification improves FAVE by 3.55 Top-1 points on the ≤96 cohort and 1.65 points on the ≤224 cohort, while higher full-image NaViT resolution adds cost without improving Top-1.The comparison tests additional resolution directed to a GT-selected region against additional resolution spread across the full image.Source: Table 2 · p. 5, §3.1 · p. 6
  • On the ≤96 cohort, GT-crop ViT-B/16 reaches 47.35 Top-1, below FAVE-T at 50.45; FAVE-T uses 5.70M parameters and 0.101 GFLOPs versus ViT-B/16’s 86.57M and 17.68 GFLOPs.The comparison evaluates fixed-resolution ViTs on full images and GT-selected crops against FAVE-T’s adaptive encoding of the same selected region.Source: Table 3 · p. 5, §3.1 · p. 6

4. FAVE for Multimodal Visual Understanding

FAVE adds compact, high-acuity encodings of selected regions to FastVLM’s global representation while keeping its global pathway and language model frozen. Evaluations cover text understanding and fine-grained object attributes.

4. FAVE for Multimodal Visual Understanding

  • FAVE independently re-encodes externally selected regions, compresses each into a local token, and combines local tokens with FastVLM’s 256 global tokens for the frozen Qwen2 model.The local encoder and reducer are trained jointly in Stage 1; Stage 2 freezes the FAVE ViT and FastVLM global pathway while training the reducer and local projector.Source: §4 · p. 7

4.1. Text-Rich Visual Understanding

  • 1.60 percentage points: FAVE improves reproduced FastVLM-1.5B on TextVQA with at most 16 additional local tokens.It adds 7.04M vision-side parameters without modifying the global pathway or language model.Source: §4.1 · p. 7, Table 4 · p. 7
  • Without OCRBench-specific training, the TextVQA-trained model gains 1.3 points overall and 13.5 points on Scene Text-centric VQA.The strongest improvements are on ESTVQA and STVQA.Source: §4.1 · pp. 7–8, Table 6 · p. 8

4.2. Fine-Grained Attribute Understanding

  • 1.31 percentage points: FAVE improves FastVLM-1.5B on GQA attributes, matches Mini-Gemini-HD-8B, and comes within 0.68 points of FastVLM-7B.The comparison leaves the global FastVLM pathway and 1.5B language model unchanged.Source: §4.2 · p. 8, Table 7 · p. 8

5. Conclusion

The conclusion presents FAVE as a way to preserve fine-grained evidence through adaptive local encoding rather than uniformly scaling image resolution or model capacity. The related-work discussion distinguishes this encoding role from existing foveation, token-efficiency, and multimodal approaches.

5. Conclusion

  • FAVE encodes selected small-object regions more accurately and efficiently than fixed- and variable-resolution controls, while adding sparse local capacity to FastVLM for text and attribute gains.TextVQA improves by 1.60 points and GQA attributes by 1.31 points; on GQA, FAVE comes within 0.68 points of FastVLM-7B.Source: §5 · pp. 8–9

A.1. Foveated Vision

  • Unlike fixed-resolution or fixation-centered foveation, FAVE adapts the selected region’s spatial budget to its native geometry while keeping spatial selection separate from encoding.Resolution, patch grid, token count, and visual compute vary with region scale.Source: §A.1 · p. 9

A.2. Efficient and Adaptive Visual Encoding

  • FAVE applies native-resolution processing to externally selected regions, adapting their patch grids rather than pruning, sampling, or merging an already-built visual representation.Patch-n-Pack batches variable-length sequences under a shared token budget.Source: §A.2 · p. 9

A.3. Efficient and Adaptive Multimodal Perception

  • In multimodal use, FAVE re-encodes selected source-image regions as a bounded local complement to FastVLM’s global pathway, which and the language model remain frozen.This separates where and when to acquire detail from how each selected region is encoded.Source: §A.3 · pp. 9–10

B. Additional FAVE Implementation Details

The appendix describes crop construction, the variable-resolution encoder, variable-length batching, and optimization for FAVE and its controls.

B.1. Object-Centric Crop Construction

  • Training crops receive box-scale and center jitter, while evaluation uses deterministic ground-truth crops; crops up to the task-specific 224-pixel ImageNet cap retain native scale.Larger crops are isotropically downsampled to the cap, with dimensions aligned to the 16-pixel patch grid without anisotropic warping.Source: §B.1 · p. 10

B.2. Variable-Resolution FAVE Encoder

  • FAVE-T uses a 12-block, 192-dimensional ViT with three attention heads and 16×16 patches, retaining all valid patch tokens and classifying from a normalized CLS representation.Adaptive crop dimensions yield sample-dependent patch-token counts.Source: §B.2 · p. 10

B.3. Variable-Length Batching and Patch-n-Pack

  • Patch-n-Pack schedules complete image sequences together under a token budget, using block-diagonal attention to prevent cross-image interaction.It changes batching rather than per-image computation, token counts, model weights, or the learned prediction function.Source: §B.3 · pp. 10–11

B.4. Optimization

  • FAVE-T trains for 300 epochs with AdamW, warmup, cosine decay, label smoothing, augmentation, and an exponential moving average; larger fixed-resolution controls use scale-appropriate training recipes.The matched Adaptive ViT-T control shares FAVE-T’s optimization, regularization, EMA, and appearance-augmentation settings.Source: §B.4 · p. 11

C.1. Dataset Split and Annotation Protocol

The ImageNet experiments use a fixed annotated-data split and a consistent rule for choosing object crops.

C.1. Dataset Split and Annotation Protocol

  • The annotated ImageNet subset is divided into a fixed 90/10 train–validation partition shared by FAVE and all reported controls.This keeps training-data differences from accounting for method comparisons.Source: §C.1 · p. 11
  • Only images with at least one valid ground-truth box are retained, and the largest valid box is selected when multiple boxes are present.The same largest-box convention defines object-centric crops and native object scale.Source: §C.1 · p. 11
  • The selected-box convention is applied consistently to crop construction and to defining native object scale.This anchors both operations to the same object region.Source: §C.1 · p. 11

C.2. Evaluation Cohorts

The evaluation uses three independently sampled 2,000-image cohorts, including two filtered by object size and one unrestricted cohort.

C.2. Evaluation Cohorts

  • Each cohort contains 2,000 independently sampled validation images: S ≤96 pixels, S ≤224 pixels, or no object-size restriction.S is the native maximum side of the largest ground-truth bounding box.Source: §C.2 · p. 11
  • The ≤96 cohort combines the S ≤48 and 49 ≤S ≤96 scale bins; cohorts are filtered samples, not disjoint or nested subsets.The ≤224 cohort aligns with the study’s ImageNet operating resolution.Source: §C.2 · p. 11

C.3. Evaluation and Resolution Policies

The evaluation policies define how selected crops and full images are processed, how compute is counted, and how local FAVE features join FastVLM’s global representation.

C.3. Evaluation and Resolution Policies

  • Fixed-Res ViT-T/16 resizes the same selected crop to 224 × 224, while Adaptive ViT-T (Full) and FAVE preserve adaptive crop geometry.For full-image evaluation, Adaptive ViT-T (Full) preserves aspect ratio and downsamples only when the longest side exceeds 224 pixels.Source: §C.3 · p. 11
  • Adaptive zoom magnifies selected regions according to their native size before a 224-pixel cap; NaViT-T either caps the full image’s long side or targets an area near 224^2 pixels.The magnification factors are 1.8×, 1.5×, 1.4×, or 1.2×, depending on the bounding-box size.Source: §C.3 · p. 11

C.4. Token and Analytical FLOP Accounting

  • FAVE averages 17.7 visual tokens and 0.101 analytical GFLOPs per image on the ≤96 cohort, versus 197 tokens and 1.283 GFLOPs for fixed 224 × 224 ViT-T/16.FLOPs use each image’s actual valid sequence length; Patch-n-Pack changes scheduling, not per-image token or FLOP accounting.Source: §C.4 · pp. 11–12

D.1. Common Multimodal FAVE Architecture

  • The multimodal setup processes the full image through frozen FastVLM to produce 256 global tokens, while FAVE encodes original-image crops and a shared reducer makes one token per crop.The FAVE ViT and reducer are trained first; later the ViT is frozen and the reducer fine-tuned while the global pathway and Qwen2 remain frozen.Source: §D.1 · p. 12

D.2. TextVQA Training and Evaluation Details

TextVQA training combines local visual representation learning with question-conditioned relevance supervision, then adapts compact crop tokens to a frozen FastVLM system.

D.2. TextVQA Training and Evaluation Details

  • DBNet supplies the local regions used at Stage 2 and inference, while cleaner Rosetta boxes provide weak question-conditioned relevance supervision during Stage 1.Rosetta and DBNet regions are aligned to transfer supervision toward the downstream proposal domain.Source: §D.2 · pp. 12–13
  • Stage 1 combines two-view visual consistency, question-conditioned region relevance, and cross-source alignment to train the FAVE ViT and reducer.The two-view InfoNCE term treats augmented views of the same region as positives and views from other regions as negatives.Source: §D.2 · pp. 12–13
  • A shared learned query cross-attends to variable-length crop features to produce one compact 192-D representation per region, rather than equally averaging patches.The ViT and reducer are optimized jointly during Stage 1.Source: §D.2 · p. 13
  • In Stage 2, the FAVE ViT is frozen while the reducer, local projector, and auxiliary region-relevance head are trained; the reported setup adds at most 16 local tokens.The global FastVLM components and language model remain frozen.Source: §D.2 · p. 13
  • The reproduced baseline and FAVE use identical Rosetta OCR prompt input, and the selected checkpoint is evaluated on the official 5,000-example TextVQA validation set.Stage 2 trains for three epochs; checkpoint selection uses development-set sharpened multi-answer cross-entropy.Source: §D.2 · p. 14
  • 73.55 TextVQA score improves on the reproduced FastVLM-1.5B baseline of 71.95.The final configuration uses K = 16 regions and M = 1, adding at most 16 local visual tokens.Source: §D.2 · p. 14

D.3. OCRBench Transfer Protocol

The TextVQA-trained local pathway is evaluated on OCRBench without OCRBench-specific training or parameter updates, with scene-text VQA as the closest transfer test.

D.3. OCRBench Transfer Protocol

  • The evaluation retains the TextVQA model’s DBNet local pathway, K = 16, M = 1 reducer, and 384-pixel crop preprocessing.Scene Text-centric VQA is the closest match to the localized natural-scene text targeted by the TextVQA training setup; other OCRBench categories are less directly targeted.Source: §D.3 · p. 14
  • Overall OCRBench improves from 54.5% to 55.8%, while Scene Text-centric VQA rises from 66.5% to 80.0% without task-specific adaptation.The largest subset gains are on ESTVQA (58.0% →80.0%) and STVQA (40.0% →66.0%).Source: §D.3 · p. 14

D.4. TextVQA Efficiency and Latency Accounting

The appendix specifies the end-to-end TTFT timing boundary and separately accounts for visual-encoder latency. It also checks that the controlled crop cap does not materially alter the reported latency conclusions.

TTFT protocol

  • TTFT is measured from visual encoding through multimodal token construction, language-model prefill, and the first output token, after external preprocessing.The timed region is one synchronized generate(max new tokens=1) call; image loading, OCR prompt construction, and DBNet proposal lookup and crop extraction are excluded.Source: §D.4 · p. 14
  • The TTFT experiment times 500 ordered TextVQA examples on an NVIDIA A40 at batch size 1, with three warm-ups and five synchronized greedy one-token calls per example.FastVLM and FAVE use the same OCR-conditioned prompt and input order; reported latency is the arithmetic mean of the timed calls.Source: §D.4 · p. 14

Vision-encoder accounting

  • On the 224-pixel crop cap, total visual-encoder cost is 38.792 ms with mean pooling and 38.716 ms with the learned K = 16, M = 1 reducer.These totals combine the frozen FastViTHD global encoder at 29.540 ms/image with local FAVE encoding.Source: §D.4 · p. 15

Crop-cap sensitivity

  • Changing the crop cap from 224 to 384 pixels shifts mean-pooling TTFT from 68.026 to 67.928 ms and learned-reducer TTFT from 74.510 to 74.759 ms.The corresponding local FAVE latencies change from 9.252 to 9.195 ms for mean pooling and from 9.176 to 9.231 ms for the learned reducer.Source: §D.4 · p. 15

Compact-VLM reference

  • SmolVLM2-2.2B is evaluated in BF16 with SDPA on the same 500 question–image pairs and one-token timing boundary, using its native chat template and tokenization.SDPA is used for this comparison because the FastVLM implementation does not use FlashAttention-2.Source: §D.4 · p. 15

D.5. GQA Training and Evaluation Details

The GQA experiment evaluates attribute accuracy on a targeted subset and combines locally encoded proposals with FastVLM's global representation. The appendix details the training and inference setup and illustrates cases where local evidence helps recover attributes, states, and object–attribute bindings.

Evaluation scope

  • Evaluation targets 5,186 attribute-type questions from GQA testdev-balanced, rather than aggregate GQA reasoning.The balanced splits contain 942,848 training questions and 132,062 validation questions.Source: §D.5 · p. 15

Stage-1 supervision

  • Stage 1 trains on GQA object crops with masked object-identity and grouped attribute supervision, plus two-view same-object consistency; the downstream system uses one 192-D FAVE token per region.Attributes span seven groups, and missing groups are treated as unknown; the active group losses are averaged equally.Source: §D.5 · p. 15

Deferred long-tail balancing

  • Long-tailed object and attribute labels receive training-only logit adjustment, with adjustment strengths ramped linearly from epochs 11–20.The strengths reach τobj = 1.0 and τattr = 0.5 at epoch 20; validation and downstream logits remain unadjusted.Source: §D.5 · p. 15

Stage-2 proposal composition and training

  • Stage 2 selects up to 32 question-independent YOLOE proposals, freezes the global FastVLM pathway and language model, and trains a reducer and local projector to append local tokens.Training may insert unmatched relevant ground-truth objects as fallbacks, but the crop budget remains 32 and validation and test use proposals alone.Source: §D.5 · pp. 16–18

Inference and attribute evaluation

  • At inference, the reported setting retains 16 proposals and M = 1, yielding at most 16 additional local tokens without ground-truth information.The model weights and global pathway remain unchanged from the Stage-2 checkpoint.Source: §D.5 · p. 19

Targeted local capacity versus model scaling

  • FAVE raises FastVLM-1.5B attribute accuracy from 70.73 to 72.04 while increasing parameters from 1.675B to 1.682B (0.42%), matching Mini-Gemini-HD-8B at 72.02 and staying within 0.68 points of FastVLM-7B at 72.72.The comparison is restricted to the GQA attribute subset.Source: §D.5 · p. 19

Qualitative interpretation

  • Qualitative cases show local evidence distinguishing a pot's full or empty state and assigning white appearance to players rather than the queried field.The traffic-light example also recovers the middle light's green color where the global pathway predicts incorrectly; the global branch supplies scene context while the local branch supplies object-centric evidence.Source: §D.5 · p. 19

Related papers

Loading 2609.04392v1…