PaperlayerView exact revision on arXiv

Source-linked AI summary

Variable-Granularity Tokenization for High-Resolution Object Detection

Khayrul Islam

arXiv:2608.28706v1cs.CV

TL;DR

Aerial ViT detectors face a trade-off between resolving few-pixel objects and the cost of a uniform pre-encoder token grid. VGTok uses training-free surround separability and structure geometry to set a per-image, variable-granularity token partition before the encoder. It improves published results across VisDrone and AI-TOD-v2 budgets, including frozen-checkpoint transfer, while reducing encoder computation and memory.

  • Problem

    Uniform pre-encoder patch grids make fine resolution for tiny aerial objects expensive across full frames.

  • Method

    VGTok uses a multi-scale morphological top-hat and a λmin structure-tensor gate to select per-region granularity and token budgets from pixels before learned stages.

  • Results

    VGTok beats every published VisDrone-val AP and APS result from 40% to 100% retention and transfers to AI-TOD-v2 state-of-the-art results, including 48.38 AP dense VisDrone and 37.27 AP on AI-TOD-v2.

  • Takeaways & Limitations

    A fixed pre-backbone budget based on local separability and structure geometry preserves tiny-object accuracy while enabling shorter packed sequences and lower encoder cost.

  • Takeaways & Limitations

    The evaluation uses a Co-DETR detector with an EVA-02 ViT-L encoder and inherits its pretraining stages, so the demonstrated scope is tied to this detector setting.

Abstract

from arXiv · show

ViT detectors fix a uniform token grid before any learned stage. A native-resolution aerial detector must then choose between resolving few-pixel objects and staying inside compute and memory limits. We introduce VGTok, a training-free tokenizer that sets patch granularity per region from pixels, ahead of the encoder. VGTok scores each region by multi-scale morphological top-hat separability from its surround, then thresholds those scores at a per-image percentile, which fixes the token budget. A structure-tensor gate ($λ_{\min}$) refines only where two-dimensional object structure supports it, leaving one-dimensional clutter coarse. The resulting token set is a strict partition of the image. In a Co-DETR detector with an EVA-02 ViT-L encoder, VGTok clears every published VisDrone-val AP and AP$_S$ at every budget from 40\% to 100\% of tokens. At 40\% it records 44.22 AP with three fifths of the sequence discarded before the first transformer block; dense, it reaches 48.38 AP, $6.08$ above the strongest published entry. VGTok transfers to AI-TOD-v2 untouched, same scorer and same rank, and sets a new state of the art at 37.27 AP and 19.51 AP$_{vt}$. As a pure drop-in into a frozen checkpoint it reaches 36.29 AP at 78.5\% of tokens, above every published entry, where our 376.3M-parameter detector clears a 3.0B multi-expert model. We show that a token budget fixed before the backbone, from local separability and structure geometry alone, holds accuracy on the tiny-object regimes that dominate aerial detection, at $3.1\times$ less encoder compute and $1.9\times$ less encoder memory. Code and models are available at \href{https://github.com/khayrulbuet13/vgtok}{\texttt{github.com/khayrulbuet13/vgtok}} and \href{https://huggingface.co/khayrulbuet13/vgtok}{\texttt{huggingface.co/khayrulbuet13/vgtok}}.

6.08 above the strongest published entry. VGTok transfers to AI-TOD-v2 untouched, same scorer and same rank, and sets a new state of the art at 37.27 AP and 19.51

VGTok addresses the conflict between fine resolution for tiny aerial objects and the cost of a uniform patch grid by allocating tokens from pixel-level separability and structure geometry. It achieves strong VisDrone and AI-TOD-v2 results across budgets, including competitive frozen-checkpoint transfer.

  • Results: 36.29 AP at 78.5% of tokens exceeds every published entry in a frozen-checkpoint drop-in setting.The reported detector has 376.3M parameters and clears a 3.0B multi-expert model.
  • Motivation: VGTok must allocate resolution selectively because aerial objects occupy few pixels while uniform grids spend computation across entire frames.Later token-pruning, sparse-head, and tiling methods inherit a token budget fixed before their learned stages.
  • Method: VGTok scores regions by separability from their surround and refines only where λmin indicates two-dimensional structure, keeping one-dimensional clutter coarse.The tokenizer separates refinement decisions from evidence strength and preserves a strict image partition.
  • Results: 44.22 AP and 35.56 APS remain at 40% of tokens, while dense VGTok reaches 48.38 AP, 6.08 above the strongest published VisDrone entry.At 40% retention, three fifths of the sequence are discarded before the first transformer block.

2. Related Work

Related work establishes adaptive granularity, but VGTok distinguishes itself by using surround separability, per-image percentile budget control, and a structure-aware stopping gate before patch projection. Its packed variable-length execution turns removed tokens into removed computation while retaining a strict partition.

  • 2. Related Work: Aerial detection methods recover resolution through chips, pyramids, warping, sparse heads, or token pruning, but operate within budgets settled before execution.These approaches either spend resolution uniformly or work within a previously fixed sequence length.
  • 2. Related Work: A per-image percentile sets the threshold and lets competing scorers use identical token budgets, unlike absolute per-scale constants.The same rank set can transfer from VisDrone to AI-TOD-v2 unchanged.
  • 2. Related Work: The λmin gate stops refinement where gradients agree on one direction, preventing four-token refinement of edges or painted lines lacking two-dimensional structure.The retained tokens remain a strict partition, with no overlapping descriptions of pixels.
  • 2. Related Work: The multi-scale top-hat measures separability from a region’s surround, using white and black responses at widths 5, 9, and 17.The six responses are reduced by a pixelwise maximum into the top-hat score.
  • 2. Related Work: Packed variable-length execution is necessary for shorter sequences to reduce actual work, since padding, masking, and dispatch can erase pruning benefits.The detector uses block-diagonal masks and reports FLOPs, latency, and peak memory together.
  • 2. Related Work: VGTok replaces fixed patchification with per-region granularity chosen from pixels before learned processing, using three patch sizes and no added parameters.Its score, threshold, rank, and retained-token count determine the sequence received by the backbone.

4. Experiments

VGTok is evaluated across matched token budgets on VisDrone and AI-TOD-v2, including transfer, frozen-checkpoint insertion, ablations, and forward-pass costs. It preserves detection accuracy at tight budgets by combining separability-based allocation with structure-aware gating, while reducing encoder computation and memory.

  • VisDrone2019-DET val: 44.22 AP and 35.56 APS at 40% tokens exceed the strongest published VisDrone entries while discarding three fifths of the sequence before the first transformer block.The 40% setting is the lowest budget that exceeds every published AP and APS.
  • VisDrone2019-DET val: 48.04 AP at 80% tokens is achieved by inserting VGTok into the dense checkpoint without retraining, within 0.34 of the 48.38 dense ceiling.The zero-shot result is 5.74 AP above the strongest published validation result.
  • AI-TOD-v2 test: 37.27 AP and 19.51 APvt on AI-TOD-v2 establish new state-of-the-art results when the VisDrone tokenizer and rank are transferred without retuning.At 78.5% tokens, the frozen-checkpoint drop-in reaches 36.29 AP, above the published 35.7 entry.
  • From tokens to FLOPs: At 40% tokens, encoder cost is 2634 GFLOPs versus 8282 dense, while latency falls from 92.3 to 53.0 ms and peak memory from 0.466 to 0.251 GB.Packed grouped attention makes encoder cost fall faster than token count; head cost remains largely fixed by resolution and query count.
  • Method and setup: The top-hat scorer reads multi-scale separability from the surround, and a per-image percentile fixes matched token budgets before the learned encoder.The structure-tensor gate refines only where two-dimensional structure supports subdivision, leaving one-dimensional clutter coarse.

5. Conclusion

VGTok’s training-free criterion and gate deliver strong accuracy across budgets and datasets while supporting an operator-controlled token budget. The supplementary analyses isolate the contributions of the gate, scorer, splitter, and evaluation protocol.

  • 48.38 AP at full retention is 6.08 above the strongest published VisDrone-val entry, while every 40–100% budget exceeds published AP and APS.
  • The same scorer and rank transfer untouched to AI-TOD-v2, reaching state-of-the-art performance on all three reported metrics.
  • The gate contributes more at scarce budgets, reaching +1.28 AP at 20% and only +0.06 AP where tokens are plentiful.
  • Because VGTok is not learned, one checkpoint serves the entire token-budget ladder and the budget remains an operator input after training.
  • Reported differences are evaluated against a 0.16 AP and 0.22 APS seed floor, with budgets calibrated by measured retention tables.

A. Node-level discrimination of λmin

The node-level audit evaluates whether λmin distinguishes clutter from object-bearing structure before detector inference. Across two image pools and budgets, gradient anisotropy outperforms contrast-derived alternatives, while the audit remains separate from full-detector accuracy.

  • λmin identifies one-dimensional clutter such as road markings and field boundaries versus two-dimensional object structure, which contrast alone cannot distinguish.
  • The audit fixes the population under the ungated quadtree descent and labels nodes without overlapping ground-truth boxes as clutter.
  • The audit uses 150 VisDrone validation images and 150 AI-TOD-v2 validation images, with AI-TOD-v2 images disjoint from its scored test split.
  • 0.712–0.750 AUROC is achieved by gradient anisotropy, compared with 0.52–0.65 for four contrast-derived alternatives.
  • On VisDrone, discrimination decreases from 0.731 to 0.614 as the budget tightens because the gate-visible clutter fraction falls from 75% to 55%.
  • The audit isolates the mechanism on 300 images without a detector, while AP measures accuracy separately on the full evaluation split.

B. Pixel-score ablation

The pixel-score ablation holds the detector, budget, and protocol fixed while varying only the training-free patch statistic. The results show meaningful spread among candidate scores, alongside a scorer-specific Single-pass limitation.

  • Eight training-free patch scores span 2.10 AP50, 2.32 AP75, and 2.11 APS at a matched 70% budget.
  • The bake-off varies only the patch score under Single-pass, using the dense checkpoint without training.
  • λmin ranks at the head of the eight candidate scores, while the shipped top-hat remains within the seed floor of it.
  • The eight-score ranking is unchanged on APS; AP50 and AP75 each transpose one adjacent pair within the 0.16 AP seed floor.
  • Upsample MSE is valid under Single-pass because constant scores still select the exact percentile budget, but its quadtree retention flatlines at 0.2493.

C. Zero-shot budget ladders

Zero-shot budget ladders separate the effects of recursive descent and gating at matched retention, without changing weights. Recursive descent supplies the largest measured gain, while the gate’s benefit increases as budgets tighten.

  • The quadtree ladder retains 0.965 and reaches 39.64 APS, then reaches 48.30 at 0.851 and 47.58 AP at 0.700 retention.
  • +1.00 AP and +1.15 APS are the zero-shot gate effects at 40%, flattening to +0.96 and +0.97 at 20%.
  • At 0.400 retention, Single-pass reaches 31.55 AP and 21.99 APS, leaving no depth to reallocate with a single global percentile.
  • +7.90 AP and +9.32 APS at 40% retention measure the recursive descent’s gain over Single-pass under identical scores, weights, and protocol.
  • Comparisons between splitter families are made vertically at fixed retained fractions, not horizontally across their different percentile scales.

D. Scale-resolved results on VisDrone

The structure-tensor gate preferentially improves small-object performance when tokens are scarce, while adding little benefit at loose budgets.

  • +0.56 APS versus +0.49 AP at 40%, and +1.51 APS versus +1.28 AP at 20%.The gate’s gains are larger on the small-object metric at both tight budgets.
  • At 80%, the gate changes performance by −0.03 APS, effectively zero at the reported precision.The stop rule contributes little when tokens are not scarce.
  • The gate declines to refine one-dimensional structure and reallocates those tokens toward the finest scale, where small objects occur.This mechanism explains the stronger APS effect under tight token budgets.

E. AI-TOD-v2 by area band and class

On AI-TOD-v2, the gate improves most area bands at the reported 70% operating point, with gains concentrated in populous and structurally compatible classes while one-dimensional objects can lose performance.

  • Area bands: +0.29 AP at 70%, with gains of +1.24 very-tiny, +0.30 tiny, +0.57 small, and +0.44 medium.At 40%, the very-tiny band instead falls −0.46 while tiny rises +0.37; at 20%, tiny rises +0.36 while very-tiny remains negative.
  • Class results: Six of eight classes improve, and vehicle, person, storage-tank, and ship gain +0.02, +0.10, +0.19, and +0.41 APvt.These are the classes with more than a thousand very-tiny boxes.
  • Class results: Wind-mill loses 1.04 APvt, from 5.59 to 4.55, the largest recorded per-class loss.Its hub and long thin blades match the one-dimensional structure the gate is designed to reject.
  • Class results: Bridge rises +0.09 APvt, while VGTok at 100% tokens reaches 4.84 APvt versus 3.14 ungated.The result shows that the directional prediction is not uniformly borne out across the two one-dimensional classes.

F. Sensitivity to the gate rank r

The shipped gate rank r = 0.20 is the best-performing setting across matched budgets, with a broad local plateau and rank-dependent threshold behavior.

  • r = 0.20 is the AP argmax at every tested budget, while AP falls on both sides of it.The sweep evaluates three matched budgets and identifies an interior optimum rather than an untested default.
  • Moving to r = 0.05 costs 0.38 AP at 40% and 0.37 at 20%, while r = 0.30 costs 0.31 and 0.22 AP.The rank is measurably worse in either direction outside the optimum’s neighborhood.
  • Near the optimum, the margin over r = 0.10 is only 0.02, 0.05, and 0.04 AP across the tested budgets.This local flatness makes the selected setting relatively insensitive to small rank changes.
  • At 70%, r = 0.05 reaches the budget at p 47.12 and r = 0.30 at p 8.66, despite nearly identical AP within 0.05.Different ranks therefore achieve the same token count through increasingly different threshold contributions.

G. Convergence of the dense reference

The dense reference is supported by convergence checks, seed replication, and matched-token audits, establishing that reported comparisons are not explained by training instability or budget mismatch.

  • Convergence: The dense reference reaches 48.38 AP, and epochs 22–28 remain within 0.03 AP while most of the curve is complete by epoch 12.A fresh schedule restart peaks at 48.30 and finishes at 48.15, remaining below the reference.
  • Seed variation: Every trained VisDrone result is a three-seed mean across 24 trained cells and 72 full 8-epoch finetunes.The individual operating-point runs report AP and APS with sample standard deviations.
  • Seed variation: Observed three-finetune ranges span 0.038–0.341 AP and 0.046–0.466 APS, with the widest spread on the ungated control at the loosest budget.The paper quotes a pooled floor rather than treating the widest range as a standard error.
  • Budget audit: All four tokenizers realise 0.4001 of the dense grid at the 40% thresholds, so the gate margin is measured at matched tokens.The audit uses the 548-image VisDrone validation set and a dense grid of 9216 tokens.
  • Budget behavior: At the loosest budget, the gate’s reduction is worth +0.06 AP and is level with the ungated control against a 0.16 floor.With plentiful tokens, the stop rule has little tightening trade-off to exploit.
  • Budget audit: On AI-TOD-v2, retention targets are checked across validation and test splits, and no comparison differs by more than 0.1 percentage point in budget.Both tokenizers overshoot by the same small amount, so the shared bias cancels in their difference.

J. Sensitivity to input resolution

VGTok is evaluated at fixed native resolution because changing input size changes both accuracy and the cost of pruning. Higher resolution narrows the accuracy gap between full and 70% token retention.

  • +1.01 AP at 100% and +1.12 at 70% result from increasing resolution from 2048 to 2560.The same checkpoint and split are used; only input resolution changes.
  • −2.86 AP at 100% and −3.33 at 70% result from reducing resolution from 2048 to 1536.

K. What the λmin gate declines to refine

The λmin gate declines to refine regions whose structure does not support two-dimensional object evidence, but compact two-dimensional clutter remains difficult to distinguish from objects. Its object overlap and accuracy effects therefore depend on the budget and population being ranked.

  • AUROC of 0.712–0.750 measures how well the ranking orders clutter/object pairs, rather than directly measuring the cost of declined refinement.The gate fires on the top r of the ranked population.
  • A declined finest-grid token is still emitted at its own coarse scale, so it represents coarsening rather than dropping the region.This preserves a strict partition while reallocating token budget.
  • 14.6% of declined tokens lie on objects at the 40% operating point, falling to 6.3% at 70% on VisDrone.At 20%, below the reported range, the figure is 29.0%; the statistic and rank remain fixed while the population changes.
  • At tight budgets, the reached nodes become increasingly object-bearing while the clutter fraction falls from 79% to 56%.The gate remains worth +1.28 AP in that setting because freed tokens are re-spent where the descent needs them.
  • Compact two-dimensional clutter can resemble objects, limiting λmin’s AUROC to 0.75 and causing some refinement tokens to be spent on clutter.Examples include roundabouts, tree crowns, rooftop vent clusters, and pallet stacks.
Loading 2608.28706v1…