PaperlayerView exact revision on arXiv

Source-linked AI summary

EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception

Yaoxin Niu, Zhangquan Chen, Yang Zhang, Xiang An, Zhumei Wang, Chih-Ting Liao, Hongkun Cao, Ruqi Huang

arXiv:2609.37123v1cs.CV

TL;DR

High-resolution vision-language perception must locate subtle evidence without spending tokens uniformly across irrelevant image regions or losing context in isolated crops. EviViT uses human-search-supervised evidence allocation and coordinate-aware fusion to acquire detail in context. Across nine hosts it improves fine-grained accuracy and outperforms global-only processing at matched token ceilings while using fewer visual tokens.

  • Problem

    High-resolution scenes can contain small answer-relevant evidence that fixed token budgets erase, while uniformly increasing resolution spends computation on irrelevant regions and isolated crops can lose context.

    Source: §1 · p. 2

  • Method

    EviViT uses human visual-search traces to train a lightweight attachment that predicts question-conditioned evidence, rereads selected regions from original pixels, and fuses them with the global scene.

    Source: §1 · pp. 2–3

  • Results

    Across nine hosts EviViT improves average fine-grained accuracy, and at every tested token ceiling it outperforms global-only processing while using fewer visual tokens.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    Evidence allocation transfers to compatible post-trained hosts without refitting and remains useful during language-side adaptation.

    Source: §5 · p. 10

    EviViT currently learns from 1K+ annotated searches; scaling the data to more objects and scenes could extend its reach to small-object and embodied perception.

    Source: §5 · p. 10

Abstract

from arXiv · show

Fine-grained visual perception enables vision-language models to distinguish subtle attributes and ground their answers in visual evidence. In high-resolution scenes, processing the whole image at greater resolution spends visual tokens on irrelevant content, while isolated crops can lose the context needed to interpret the selected evidence. We introduce EviViT, a lightweight attachment that learns where a pretrained vision transformer should acquire detail. Human visual-search traces supervise a question-conditioned evidence density, which guides regional re-reading from the original pixels and the allocation of visual tokens. A sparse, coordinate-aware bridge then connects the regional features to the global scene, allowing the host to interpret precise evidence in context. Learned with the host backbone frozen, the attachment serves both the base model and compatible post-trained descendants without refitting. Experiments across nine hosts show consistent gains in average fine-grained accuracy. Matched-budget comparisons further show that EviViT outperforms global-only processing at every tested token ceiling while using fewer visual tokens.

1 INTRODUCTION

EviViT addresses the allocation of visual resolution in high-resolution scenes by using human visual-search traces to guide question-conditioned detail acquisition and linking local evidence to global context. Its lightweight attachment is designed to be reusable across compatible hosts, and experiments report gains in fine-grained accuracy and token efficiency.

1 INTRODUCTION

  • In high-resolution scenes, fixed token budgets can erase small, answer-relevant evidence, while uniformly increasing resolution spends computation on irrelevant regions.The central question is where the model should spend visual tokens, not simply how many it receives.Source: §1 · p. 2
  • Prior crop-and-inspect methods add inference overhead and tie evidence acquisition to the answering policy, while pruning encoded tokens cannot recover detail lost during resizing.These limitations motivate learning which original-image regions need additional resolution before answer generation.Source: §1 · p. 2
  • EviViT distills full human visual-search trajectories into question-conditioned evidence density that supervises where a vision transformer allocates resolution.The supervision uses the broader exploration process rather than only the terminal evidence box.Source: §1 · pp. 2–3
  • A lightweight attachment predicts evidence, selects non-redundant decisive and context regions, rereads original pixels, and sparsely fuses local features with coordinate-matched global tokens.This design addresses the need to retain scene context while acquiring local detail.Source: §1 · pp. 2–3
  • Trained with the host ViT, merger, and LLM frozen, EviViT can serve compatible foundation models and post-trained descendants without refitting its evidence allocator.Inference needs only the original image and question, not a human trace or explicit crop instruction.Source: §1 · pp. 2–3
  • All nine foundation and post-trained hosts improve in average fine-grained accuracy; Qwen3-VL-4B/8B gain 5.60/6.09 points on the seven-benchmark average.At matched 1K–20K token ceilings, EviViT outperforms global-only processing while using 90–97% as many visual tokens; gains persist after language-side LoRA adaptation.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses fine-grained perception through high-resolution views, adaptive computation, and human spatial guidance. EviViT differs by acquiring detail from original pixels and connecting regional features to the global scene within a frozen vision encoder.

Vision Encoders

  • Dynamic-resolution encoders make more detail available but spend tokens across the whole image, whereas EviViT learns which regions to reread for a given question and fuses them with the global grid.Vision Transformers, CLIP, SigLIP, and Qwen-VL-family encoders are discussed as relevant encoder approaches.Source: §2 · p. 3

Visual Perception

  • High-resolution perception methods use tiling, selective cropping, or interactive search, while training-time distillation transfers detailed observations into the answering model; EviViT instead trains a visual attachment with that model frozen.EviViT acquires detail before answer generation without language-model crop commands and retains the global view alongside local observations.Source: §2 · p. 3

Adaptive Attention

  • DynamicViT and token merging prune or combine encoded tokens, while query-aware methods retain selected local and global features; these approaches cannot acquire detail absent at input resolution.EviViT allocates fresh tokens from original pixels while preserving a global context stream.Source: §2 · p. 3

3 METHOD

EviViT predicts question-conditioned evidence, allocates native-resolution regional views, and connects them to the global visual grid inside a frozen host. Human-search traces supervise the evidence and context targets, while staged training fits the attachment components without updating the host.

3 METHOD

  • EviViT separates deciding where more detail is needed from interpreting that detail in context by predicting evidence density, allocating regional resolution, and connecting reread observations to the global grid.Both components operate inside the frozen host vision transformer before answer generation.Source: §3 · p. 4, Figure 2 · p. 4

3.1 HUMAN VISUAL-SEARCH DATA

  • Visual-search records from 1K+ VisualProbe training questions log cursor positions, zoom windows, resets, and final evidence in source-image coordinates to supervise evidence and context prediction.The records capture broader search as well as selected evidence; inference takes only an image and question.Source: §3.1 · p. 4

3.2 LEARNING A SPATIAL EVIDENCE DENSITY

  • Search events become a soft evidence target by mixing maps of all visited regions, the successful branch, the last completed zoom, and the final box; PTEA predicts it from question-aligned visual features.A separate context branch is trained from broader inspected regions, and abandoned regions form a weak negative map.Source: §3.2 · pp. 4–5

3.3 NON-REDUNDANT NATIVE-DETAIL ALLOCATION

  • The planner retains up to two distinct decisive regions and one complementary context region, reserving 10–25% of local visual tokens for context and dividing the remainder by evidence mass.H-Safe adjusts each selected region’s center and size using visual features, query representation, and box geometry.Source: §3.3 · p. 5

3.4 NATIVE RE-READING AND SPARSE FUSION

  • Selected regions are sampled from original pixels at allocated resolution, then local tokens attend to coordinate-matched global neighborhoods and send updates to global tokens with local children.The bridge lets local features access surroundings while the global grid incorporates detailed observations of matching locations.Source: §3.4 · p. 5

3.5 TRAINING AND PORTABILITY

  • Training fits PTEA and its context branch first, then ContextNeed and H-Safe, and finally the Sparse Bridge, while the host ViT, merger, and LLM remain frozen.Transfer to post-trained descendants is supported when they share the host’s visual blocks, channel width, merger, and coordinate system.Source: §3.5 · p. 6

4 EXPERIMENTS

Across fine-grained and broader-capability benchmarks, the experiments test host transfer, token efficiency, continued language-side adaptation, and the contribution of individual design choices.

4.1 FINE-GRAINED PERCEPTION AND HOST TRANSFER

  • Across nine hosts, the seven-benchmark average improved for all; VisualProbe Medium/Hard rose 8.2/8.6 points, while EviViT-equipped leaders scored 63.69 VP-All and 66.04 ZoomBench.Figure 3 includes a SeProD VP-All bar derived from author-reported scores under a different protocol, so it is contextual rather than a matched comparison.Source: Table 1 · p. 6, §4.1 · p. 7, Figure 3 · p. 7

4.2 ACCURACY–EFFICIENCY TRADE-OFF

  • At every 1K–20K ceiling, accuracy was higher with 90–97% of global-only tokens; at 8B/4K, EviViT scored 68.87 with 3.62K tokens versus 62.83 with 3.83K globally.At 8B/4K and 8B/16K, gains were 6.04/6.49 points for 18/3 ms more median end-to-end time, with nearly unchanged peak memory.Source: §4.2 · pp. 7–8, Table 2 · p. 8, Figure 4 · p. 8

4.3 PRESERVING GENERAL CAPABILITIES

  • MMBench, MMStar, and Visual-CoT results were broadly stable across four foundation hosts, while Qwen3.5-4B improved on all three.The authors interpret this stability as support for sharpening selected evidence without discarding the broader scene representation.Source: Table 4 · p. 9, §4.3 · p. 9

4.4 CONTINUED TRAINABILITY

  • With visual weights frozen, EviViT+SFT retained 6.80-point VP-All and 7.34-point ZoomBench leads over Base+SFT under the same 10K-example language-LoRA schedule.Both setups improved on Visual-CoT, and EviViT+SFT's five-metric mean also rose.Source: §4.4 · p. 9

4.5 ABLATION STUDY

  • Shuffling PTEA's question reduced VP-All by 4.47/3.31 points; random location changes reduced pooled accuracy from 66.57 to 57.08/56.94, and full-trace supervision beat final-box on all three metrics.Full-trace supervision reached 56.60 versus 52.83 on VP-Hard, indicating added value beyond terminal evidence-box labels.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION AND LIMITATIONS

EviViT connects regional evidence to the global scene and improves fine-grained perception across hosts. Its current training uses 1K+ annotated searches, and the attachment transfers to compatible post-trained descendants without refitting.

5 CONCLUSION AND LIMITATIONS

  • EviViT improves average fine-grained perception across nine hosts and uses visual tokens more effectively at every tested budget.It connects detail acquired from human search annotations with the global scene.Source: §5 · p. 10
  • EviViT currently learns from 1K+ annotated searches; scaling them to more objects and scenes could extend its reach to small-object and embodied perception.Those settings require focused detail to remain grounded in the wider scene.Source: §5 · p. 10
  • The public repository provides preparation, training, evaluation, and judging scripts, but excludes model weights, benchmark images, human-search traces, and QA training data.Experiments use public visual benchmarks and search annotations produced by the research team.Source: REPRODUCIBILITY STATEMENT · p. 10, ETHICS STATEMENT · p. 10

A.1 MODEL AND TRAINING INSTANTIATION

  • Qwen3-VL hosts use separate attachments after different visual blocks, while each Qwen3.5 attachment is reused on its corresponding Vision-OPD descendant without refitting.The vision backbone, merger, and LLM remain frozen during all four stages.Source: §A.1 · p. 13

A.2 ATTACHMENT ARCHITECTURE

The attachment predicts question-conditioned evidence and context maps, selects complementary regions, and connects their features to global tokens. Its staged fitting uses trace targets for allocation and reference-answer loss for the bridge.

A.2 ATTACHMENT ARCHITECTURE

  • PTEA aligns question representations with intermediate visual features to predict evidence logits, while a complementary branch predicts context from exploration and pre-reset inspection maps.The context target mixes those maps 0.70/0.30, or uses the full-trace map alone when the latter is empty.Source: §A.2 · p. 13
  • The planner retains up to two non-redundant decisive anchors and masks context at those anchors to select one context region, using two or three regions under a three-slot cap.H-Safe predicts bounded residual corrections to anchor geometry.Source: §A.2 · p. 13
  • ContextNeed predicts a target context share of 0.10–0.25 from map statistics; region size, available capacity, and patch-grid alignment also determine realized token counts.H-Safe starts as an identity transform because its output is zero-initialized.Source: §A.2 · p. 13
  • The sparse bridge uses coordinate offsets and log scale as attention bias, with zero-initialized output projections and residual updates clipped to 20% of receiving-token RMS.Reverse messages are aggregated by coordinate parent and written only to global tokens with local children.Source: §A.2 · p. 13
  • The four-stage fit caches frozen features, trains PTEA and the context branch from trace targets, fits routing components, then trains the bridge with answer-token cross-entropy and an identity penalty.The reported checkpoint is saved after the bridge's first complete pass.Source: §A.2 · p. 13
  • Held-out evidence score peaks at epoch three although both training losses continue falling, so PTEA duration is selected from trace targets rather than external QA results.The final PTEA is then fit for three epochs on all 1,144 traces.Source: §A.2 · p. 14, Figure 5 · p. 14
  • The image-dependent F(N) capacity is a soft envelope, not a guaranteed realized-token minimum; matched-budget ceilings instead cap samples at their paired global-only baseline counts.Reported token counts are measured after grid alignment.Source: §A.2 · p. 14

A.3 COMPONENT COMPARISONS

Component comparisons attribute gains to source-pixel re-reading, question-aligned routing, and learned fusion, while paired controls test location and answer changes. Evaluations compare attached and base hosts under matched inputs and scoring procedures.

A.3 COMPONENT COMPARISONS

  • The full model gains 5.04 points at 4B and 5.31 at 8B over global B16, while component effects vary across tasks rather than adding uniform gains.Table 6 compares fixed-budget re-reading, continuous boxes, and full input-dependent capacity allocation across eight benchmarks.Source: §A.3 · p. 15, Table 6 · p. 15
  • Shuffling PTEA's question reduces VisualProbe accuracy, and replacing the learned bridge with identity lowers the five-dataset mean on both hosts.The question-shuffle effect on other benchmarks is smaller and less uniform.Source: §A.3 · p. 15, Table 7 · p. 15
  • Randomly translating or cross-image shuffling crop centers reduces pooled accuracy by about 9.5 points with view count, size, and token budget fixed.Table 8 changes crop positions alone under native multi-image input.Source: §A.3 · p. 15, Table 8 · p. 16
  • On a disjoint 186-image audit, PTEA reaches 0.9294 pixel AUROC and top-eight proposals cover at least 90% of every official target in 70.43% of cases.Centered-kernel alignment is 0.948 at insertion, 0.895 at the final ViT block, and 0.926 after merger.Source: §A.3 · p. 16
  • Rescues outnumber losses in every host–tier pair; Medium-tier gains are 8.96–11.19 points, with retention of 86.73–92.44% and exact McNemar p ≤7.2×10−4 across four hosts.The paired pattern spans Qwen3-VL and Qwen3.5 model families and scales.Source: §A.3 · p. 16, Table 9 · p. 16

A.4 PROMPTS, DECODING, AND SCORING

  • Paired evaluations use identical images, questions, prompts, and deterministic decoding; a frozen semantic judge sees questions and answers but not images or method names.Runs are reported only when every example has a prediction and valid judgment.Source: §A.4 · p. 16

A.5 MATCHED RECOVERY SFT

Matched language-side LoRA adaptation improves both Qwen3-VL-8B paths while EviViT remains frozen. EviViT+SFT retains advantages on VisualProbe and ZoomBench and improves the Protect-5 macro.

A.5 MATCHED RECOVERY SFT

  • The shared 10,000-example QA manifest contains 6,694 document/OCR examples, 2,900 Visual-CoT examples, and 406 replay examples; neither arm receives evidence boxes as policy inputs.Both arms use the same image, question, and answer inputs.Source: §A.5 · p. 17
  • Both arms train for three epochs in the same seeded order, updating language LoRA while the vision tower and all EviViT modules remain frozen.The reported pair uses the common final endpoint rather than separate task-specific best checkpoints.Source: §A.5 · p. 17
  • During three matched epochs, answer loss falls and answer-token accuracy rises for both paths, with EviViT+SFT generally maintaining lower loss.Figure 6 plots 400-example moving means; only language LoRA is updated.Source: §A.5 · p. 17, Figure 6 · p. 17
  • Under matched 10K-data, order, LoRA, epoch-3 recovery SFT with EviViT frozen, both paths improve on Visual-CoT, while EviViT+SFT retains its VisualProbe and ZoomBench advantages.Source: Table 10 · p. 17, §A.5 · p. 18

B ADDITIONAL EVALUATION RESULTS

The additional evaluation examines perceptual skills and broader multimodal benchmarks; across four foundation hosts, EviViT improves most reported host–benchmark pairs.

B ADDITIONAL EVALUATION RESULTS

  • The evaluation compares each host before and after attachment on individual perceptual skills and broader multimodal capabilities.The broader benchmarks are MMBench, MMStar, and Visual-CoT.Source: B ADDITIONAL EVALUATION RESULTS · p. 18
  • PB-1img comprises all 2,652 single-image questions in PerceptionBench; the remaining 348 questions require a multi-image setting outside this comparison.PerceptionBench’s full suite contains 3,000 questions.Source: B ADDITIONAL EVALUATION RESULTS · p. 18
  • EviViT improves 10 of 12 host–benchmark pairs across four foundation hosts on MMBench, MMStar, and Visual-CoT.Table 11 covers all nine hosts; the reported 10-of-12 pattern is for the four foundation hosts.Source: B ADDITIONAL EVALUATION RESULTS · p. 18, Table 11 · p. 18

C.1 REUSING EVIDENCE VIEWS ACROSS INPUT INTERFACES

The interface tests find that selected evidence remains useful when fused internally or supplied as separate images, while controlled comparisons show that question-relevant locations matter more than equally sized alternatives.

C.1 REUSING EVIDENCE VIEWS ACROSS INPUT INTERFACES

  • Internal fusion and native multi-image input have identical pooled accuracy across the 706 questions, with small differences in opposite directions on the two benchmarks.The same regions, order, resize dimensions, and per-view token counts are used for both interfaces.Source: §C.1 · p. 19, Table 13 · p. 19
  • The internal route connects local observations to the global grid through source coordinates and gives the host one visual sequence without requiring a separate crop-input workflow.It keeps local detail linked to its scene location while avoiding dependence on native multi-image support.Source: §C.1 · p. 19
  • With views held fixed, the learned bridge improves over identity on both benchmarks, separating the benefit of integration from the usefulness of the views themselves.Native multi-image input shows that the selected views are informative without the learned bridge.Source: §C.1 · p. 19

C.2 WHY THE SELECTED LOCATIONS MATTER

  • The location test keeps the global view, crop count, order, dimensions, token allocations, host, prompt, decoding, and judge fixed while changing crop locations.Random translations and cross-image shuffling preserve crop sizes; shuffling breaks alignment with the current image–question pair.Source: §C.2 · p. 20
  • Selected locations outperform random translations and cross-image shuffling by pooled margins of 9.49 points on VisualProbe and 9.63 points on V∗.The controls preserve equally sized, similarly detailed views but lose much of the answering benefit.Source: §C.2 · p. 20

C.3 COVERAGE BY THE REGIONS ACTUALLY DEPLOYED

Deployed crops concentrate detail on target regions, while context can supply missing relational evidence; a matched supervision comparison favors full search traces on the displayed fine-grained metrics.

C.3 COVERAGE BY THE REGIONS ACTUALLY DEPLOYED

  • Selected crops occupy 12.39% of image area and achieve 73.69% mean target-area coverage, versus 18.47% for joint-random placement.The layouts preserve crop sizes and total union area; centering also leaves a substantial coverage gap.Source: §C.3 · p. 20, Table 14 · p. 20
  • Every target reaches at least 90% coverage in 120 of 186 questions with context, compared with 98 without it.Of 22 additional questions covered with context, six have two targets; in five, context supplies the missing side of the relation.Source: §C.3 · p. 21
  • The crops focus detail on relevant objects while the global stream preserves surrounding scene information, and coordinate fusion reconnects the two.This lets the host interpret a local attribute in its broader spatial setting rather than relying on isolated evidence boxes.Source: §C.3 · p. 21

C.4 THE VALUE OF FULL-TRACE SUPERVISION

  • The matched supervision runs use the same 1,144-example training set, architecture, schedule, seed, and three-epoch bridge endpoint.Full-trace uses human-event evidence and recorded exploration for context; Final-box uses terminal boxes and geometrically expanded context targets.Source: §C.4 · p. 21
  • Full-trace supervision improves all three displayed metrics, with its largest gain on VP-Hard, while the two trained attachments tie on native ZoomBench.The comparison tests the full supervision recipe, not any one trace cue.Source: §C.4 · p. 21

D QUALITATIVE EVIDENCE-LOCALIZATION CASES

The examples illustrate evidence localization across small-text recognition, relative-position reasoning, and evidence-grounded explanation.

D QUALITATIVE EVIDENCE-LOCALIZATION CASES

Related papers

Loading 2609.37123v1…