PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Observation Control for Active Visual Recognition

Dikai Li, Weijia Wang, Yuan-Gen Wang

arXiv:2610.19624v1cs.CV

TL;DR

Fixed-view recognition assigns the same observation budget across inputs, and the public AdaptiveNN-DeiT-S pipeline fixes both crop size and focus-step count. This paper adds adaptive scale, crop refinement, and learned stopping to that recognizer. Across benchmarks it remains competitive while offering threshold-controlled accuracy-compute trade-offs; sequential ImageNet-1K reaches 81.9% top-1 using 2.45 focus steps.

  • Problem

    Fixed-view classifiers use the same observation budget for easy and hard examples, while public AdaptiveNN-DeiT-S fixes crop size and focus-step count.

    Source: §1 · pp. 1–2

  • Method

    The recognizer predicts per-fixation scale, optionally refines crops with foveation and a residual detail-token path, and uses label-grounded stop heads.

    Source: §1 · p. 2, §3.5 · p. 7

  • Results

    The interface remains competitive across benchmarks; sequential ImageNet-1K sustains 81.9% top-1 using 2.45 focus steps at τ = 0.90.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    One checkpoint provides a threshold-controlled accuracy-compute frontier, with net compute benefit depending on realized early termination because the detail path adds per-step cost.

    Source: §7 · p. 14

    Three-seed studies cover Cars, Aircraft, CUB, Dogs, and Oxford-Pet, while the original ImageNet-A, ImageNet-1K, and ImageNet-R table rows use single runs.

    Source: §6 · p. 14

Abstract

from arXiv · show

Visual recognition models usually process a fixed view under a fixed computation budget, although images often benefit from different scales and amounts of spatial evidence. Active vision can adapt by observing a coarse scene and then selecting informative fixations, but the public AdaptiveNN-DeiT-S pipeline still fixes both field of view and observation count. We extend this fixed-step recognizer with a controllable observation interface that adapts both the field of view and the number of fixations to the input. Specifically, the model predicts a per-fixation scale that sets its field of view, refines each selected crop with optional scale-gated foveation and a residual high-resolution detail-token branch, and uses label-grounded stop heads for input-dependent early termination. Experiments across fine-grained benchmarks, full ImageNet-1K, and ImageNet-A/R-derived classification splits show that the interface remains competitive with fixed-step references while exposing a controllable observation budget. The same checkpoint enables threshold-controlled accuracy-compute trade-offs without per-threshold retraining. On sequential ImageNet-1K, the learned stop rule outperforms fixed-step truncation of the same checkpoint, sustaining 81.9% top-1 using 2.45 of the four focus steps and exceeding three-step truncation at lower cost. On each benchmark, a variant combining scale control and detail-token fusion achieves the highest accuracy among the tested component-added non-exit variants. Our code is available at https://github.com/GZHU-DVL/LI-Dikai.

1 Introduction

Fixed-view classifiers use the same observation budget for easy and hard examples, while AdaptiveNN’s public DeiT-S pipeline fixes crop size and focus-step count. The paper adds control over fixation geometry, encoded evidence, and stopping, and evaluates the resulting interface across recognition benchmarks.

1 Introduction

  • The public AdaptiveNN-DeiT-S pipeline retains a fixed crop size and a fixed number of focus steps despite its coarse-to-fine recognition design.Fixed scheduling assigns easy and hard examples the same observation budget, while small discriminative evidence can be diluted by background clutter.Source: §1 · pp. 1–2
  • The interface addresses fixed field of view, uniform crop encoding, and fixed observation count with geometry, evidence, and stopping controls.Scale-adaptive fixations, optional foveation and detail-token fusion, and label-grounded stop heads implement these controls.Source: §1 · p. 2
  • The evaluation spans fine-grained benchmarks, full ImageNet-1K, and ImageNet-A/R-derived classification splits.The analysis examines where scale and detail improve accuracy and where thresholded stopping trades computation for accuracy.Source: §1 · p. 2

2 Related Work

The paper extends the active-recognition glance-and-focus paradigm with adaptive fixation scale and stopping. It distinguishes this sequential observation loop from approaches that adapt computation within a fixed view.

2 Related Work

  • AdaptiveNN extends the glance-and-focus active-recognition structure by expanding the action space and inference policy.The approach builds on sequentially choosing focus regions after an initial glance, alongside related recurrent-attention and spatial-transformer ideas.Source: §2 · pp. 2–3
  • Unlike foveated methods framed as fixed pyramids or standalone sensors, the method uses policy-optimized continuous scale to select one field of view per fixation.Foveation refines the crop according to scale; fine-grained recognition commonly uses part features, attention, or feature pyramids.Source: §2 · p. 3
  • Other non-uniform visual representations include single-fixation foveated sampling for question answering and detection and test-time semantic-feedback sampling.Here, an evolving state selects the next crop’s location and scale, while a stop head decides whether to observe again.Source: §2 · pp. 3–4
  • Early-exit and dynamic-vision methods adapt depth, width, routing, or token use within passive views, so their accuracy-compute frontiers are not directly comparable to active observation.The paper evaluates controllability within one public active-recognition pipeline; its label-grounded stop heads interact with learned fixation scale and provide multiple thresholds from one checkpoint.Source: §2 · p. 4

3 Method

The method extends AdaptiveNN’s glance-and-focus pipeline with adaptive control over crop scale, local detail, and when to stop observing. It combines these mechanisms with classification, distillation, and policy-training objectives.

3 Method; 3.1 Controllable Observation Loop

  • The controller predicts fixation location and field-of-view scale, refines the selected crop, updates the spatial state, and produces focus-step predictions and stop scores.It uses a low-resolution glance to initialize the state and defaults to four focus steps.Source: §3 · p. 4, §3.1 · pp. 4–5

3.2 Baseline AdaptiveNN; 3.3 Scale-Adaptive Fixations

  • The baseline uses two continuous coordinates to select crop position, while the extension adds a scale coordinate to adapt the crop’s field of view.The crop is resized to a fixed focus input size, leaving per-step encoder cost unchanged by scale.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 Foveated Focus Crop

  • Scale-gated foveation preserves the crop center while blending peripheral regions with a degraded version, with its strength increasing for expanded views.The degradation downsamples the crop to half its side length before upsampling; foveation is inactive when its strength is zero.Source: §3.4 · p. 6

3.5 Detail-Token Fusion

  • A shared-center inner crop sampled from the original image supplies a higher-resolution descriptor, preserving detail that resizing the outer crop may discard.The descriptor is pooled from the central 3 × 3 feature-token window and added residually; inner features are detached so the auxiliary path does not overwrite main focus-encoder updates.Source: §3.5 · p. 7

3.6 Confidence-Conditioned Early Exit

  • Per-step stop heads learn task-correct, sufficiently confident stopping targets, and inference exits at the first focus step whose score reaches threshold τ.The training target requires a correct prediction with confidence at least 0.7; τ is a separate inference-time hyperparameter.Source: §3.6 · pp. 7–8
  • Changing τ selects intermediate predictions from the same checkpoint, but returned-prediction accuracy must be measured at the selected exits because later observations can change the predicted class.Simulation applies stopping post hoc; only sequential-step evaluation measures realized step counts and supports compute-saving claims.Source: §3.6 · p. 8

3.7 Training Objective

  • Training combines supervised classification losses, distillation from the final focus prediction to earlier predictions, and a separate PPO update for non-differentiable fixation selection.The fixation reward is the reduction in cross-entropy from the previous prediction to the current focus prediction; scale shares this reward without an auxiliary scale-regression loss or scale-cost penalty.Source: §3.7 · pp. 8–9
  • Components are trained in stages: scale is fixed at the baseline during warmup, then optimized with location, while early-exit fine-tuning starts from the trained +S+F+D checkpoint.The stop loss is enabled only for early-exit fine-tuning, with λstop = 0.05 there and λstop = 0 otherwise.Source: §3.7 · p. 9

4 Experiments

The experiments evaluate recognition across fine-grained, ImageNet-A/R-derived, and large-vocabulary benchmarks, using public evaluation splits. Comparisons include additive component variants and an external fixed-step reference.

4.1 Datasets and Protocol

  • Eight benchmarks cover fine-grained recognition, ImageNet-A/R-derived classification, and large-vocabulary recognition, with accuracy reported on each dataset’s public evaluation split.The benchmarks are FGVC-Aircraft, Stanford Cars, CUB-200-2011, Stanford Dogs, Oxford-IIIT Pet, ImageNet-A, ImageNet-R, and ImageNet-1K validation.Source: §4.1 · p. 9

4.2 Compared Variants

  • The comparison includes an internal AdaptiveNN rerun and additive variants for learned scale, foveation, detail-token fusion, and early exit.Orig disables the new components; +S+F+D is the non-exit full adaptive crop model, and Full+EE adds its confidence-conditioned stop head.Source: §4.2 · pp. 9–10
  • The unmodified official AdaptiveNN codebase is rerun as an external fixed-step reference, while Orig serves as the internal anchor for component effects.The supplementary material reports the difference between Orig and Official separately.Source: §4.2 · p. 10

5 Results

Results compare recognition accuracy against fixed-step references, isolate the dataset-dependent effects of scale, foveation, and detail fusion, and evaluate confidence-conditioned stopping against fixed-step truncation.

5.1 Accuracy and Budget Diagnostics

  • The only negative ∆best values are ImageNet-1K (−0.158) and ImageNet-R (−0.267), where +S+D is stronger.A full four-step Full+EE unroll costs 9.90 versus 5.50 encoder-pass GFLOPs for Orig; realized savings arise only under sequential early exit.Source: §5.1 · p. 10
  • Full+EE exceeds the official rerun on seven of eight datasets and is within 0.002 points on ImageNet-1K.Margins below 0.1 points are treated as parity-level differences.Source: §5.1 · p. 10, Table 1 · p. 11

5.2 Geometry and Evidence Controls

  • Scale alone improves Acc@1 on Cars (+0.560), Dogs (+0.327), Aircraft (+0.390), and ImageNet-1K (+0.118).Among component-added non-exit variants, +S+D or +S+F+D has the highest Acc@1 on each benchmark.Source: §5.2 · p. 11, Table 2 · p. 12
  • Foveation is not uniformly helpful: +S+F is close to +S on Aircraft, CUB, and Dogs, while dataset comparisons favor different component combinations.The single-run ImageNet-A comparison shows a 0.397-point difference; Cars benefits from +S+F+D, while ImageNet-1K, ImageNet-R, and Oxford-Pet favor +S+D.Source: §5.2 · p. 11

5.3 Stopping-Control Trade-Off

  • On ImageNet-1K, sequential τ = 0.70 outperforms fixed two-step truncation by 0.298 percentage points in Acc@1 while using fewer than two focus steps on average.At τ = 0.90, it also outperforms fixed three-step truncation in Acc@1 while using fewer focus steps on average.Source: Figure 3 · p. 13, §5.3 · p. 13, Table 3 · p. 13
  • The ImageNet-1K τ = 0.90 setting uses about 2.45 focus steps but costs 6.48 encoder-pass GFLOPs, compared with 5.50 for four-step Orig.Step reduction, encoder cost, and measured latency are distinct quantities; the runtime timings use a separately fitted max-softmax stopping rule.Source: Table 3 · p. 13, §5.3 · p. 14

6 Limitations

The evaluation has uneven seed coverage, and reported margins can depend on checkpoint selection. Accuracy gains are dataset-dependent, while added detail-path cost and manually chosen thresholds constrain the trade-offs.

6 Limitations

  • Three-seed studies cover Cars, Aircraft, CUB, Dogs, and Oxford-Pet, while original ImageNet-A, ImageNet-1K, and ImageNet-R results are single runs.Checkpoints use best public-split Acc@1, making parity-level margins sensitive to epoch selection.Source: §6 · p. 14
  • Detail-token fusion adds a focus-encoder pass from the second focus step onward, so rarely exiting datasets incur most of its additional cost.Runtime measurements cover ImageNet-1K model-forward execution on one GPU.Source: §6 · p. 14
  • Component interactions vary by dataset: ImageNet-R favors +S+D over active foveation, and the stopping thresholds are set manually.Source: §6 · p. 14

7 Conclusion

The conclusion presents a controllable observation interface that remains competitive across benchmarks and provides a threshold-controlled accuracy-compute frontier from one checkpoint. On sequential ImageNet-1K, its learned stop rule sustains 81.9% top-1 using 2.45 focus steps.

7 Conclusion

  • The method provides a threshold-controlled accuracy-compute frontier from one checkpoint while remaining competitive across benchmark datasets.The interface controls observation geometry, local evidence, and stopping.Source: §7 · p. 14
  • 81.9% top-1 is sustained on sequential ImageNet-1K at τ = 0.90 using 2.45 focus steps.Source: §7 · p. 14
  • Net compute benefit depends on realized early termination because the detail path adds per-step cost.Source: §7 · p. 14

Recognition Supplementary Material

The supplement collects reproducibility material and diagnostics referenced by the main paper.

Recognition Supplementary Material

1 Additional Reproducibility Details

The supplementary material documents training and evaluation choices, paired uncertainty on ImageNet-1K, and training-seed variation on ImageNet-R. The reported ImageNet-R intervals are wide and do not establish equivalent performance.

1 Additional Reproducibility Details

  • Main-table checkpoints use four focus steps, dataset-specific schedules, and public validation/reporting-split Acc@1 selection; scale actions are anchored for eight non-exit warmup epochs.Foveation and detail fusion ramp during training, while early-exit fine-tuning starts from the +S+F+D checkpoint.Source: §1 · p. 17, Table 1 · p. 18
  • Across three seeds, matched-budget +S+F+D−Orig gains are small but positive on Cars, Aircraft, CUB, and Dogs, and negative-to-zero on Oxford-Pet.The authors interpret Oxford-Pet as near-parity, not established equivalence.Source: §1 · p. 17, Table 4 · p. 19

1.1 Paired uncertainty on ImageNet-1K

  • Neither ImageNet-1K McNemar comparison reaches the Holm-adjusted 0.05 threshold; these intervals describe image-level uncertainty, not training-seed variation.Source: §1.1 · p. 18

1.2 Training-seed variation on ImageNet-R

  • The separate ImageNet-R seed study uses training seeds 0–2 on a fixed split, with one GPU and gradient accumulation to an effective batch of 144.Source: §1.2 · p. 19
  • The reported +S+F+D minus +S+D mean difference is +0.100, with interval [−0.622, 0.822]; three-seed intervals remain wide and do not establish equivalent performance.Source: §1.2 · p. 19
  • The budget-matched continuation diagnostic reports Acc@1/Acc@5 and includes a scale-cost-penalty variant run only for Aircraft, CUB, and ImageNet-A.That diagnostic is not used by any reported main-paper run.Source: Table 5 · p. 20

2 Budget-Matched Continuation Diagnostic

A budget-matched continuation compares Orig, +S+F+D, and Full+EE after 30 additional epochs. The proposed components yield dataset-dependent accuracy gains, while the comparison does not assess adaptive operating points.

2 Budget-Matched Continuation Diagnostic

  • With matched effective batch and learning-rate schedule, a 30-epoch continuation of Orig is itself a strong accuracy baseline.The comparison includes Orig, +S+F+D, and Full+EE.Source: §2 · p. 20
  • Under the matched budget, the proposed components yield dataset-dependent gains rather than a uniform improvement.Source: §2 · p. 20
  • The comparison controls the accuracy axis only; it does not give Orig learned per-fixation scale or confidence-conditioned early-exit operating points.Source: §2 · p. 20

3 Early-Exit Diagnostics

The diagnostics examine early-exit behavior, learned scales, and component effects. They report that the +S+D-initialized model does not consistently improve accuracy across stopping rules and budgets, while detail fusion corrects more Stanford Cars errors than it reverses.

3.1 Early exit initialized from +S+D

  • Early-exit models are fine-tuned on ImageNet-R for 20 epochs from different starting configurations and with different effective training batches.The +S+D and +S+F+D batches are 48 and 144, respectively.Source: §3.1 · p. 20
  • The +S+D-initialized model does not consistently improve accuracy across stopping rules and budgets.The comparison retains training-batch differences as a source of confounding.Source: §3.1 · pp. 20–21
  • Accuracy generally increases across focus steps for the full early-exit checkpoint evaluated without stopping.Table 6 reports per-step Acc@1.Source: §3.1 · p. 21, Table 6 · p. 21
  • Mean learned scales stay near the anchor scale on Aircraft, CUB, Dogs, and ImageNet-A, but are larger on Cars, ImageNet-1K, ImageNet-R, and Oxford-Pet.The means are across the four focus steps for the same checkpoints.Source: §3.1 · p. 21, Table 7 · p. 21
  • On Stanford Cars, detail fusion corrected 379 no-detail errors and reversed 290 cases; the examples involve same-make confusions.The comparison holds scale and foveation on and executes four focus steps in both configurations.Source: §3.1 · p. 21, Figure 3 · p. 24
  • The ImageNet-R stop-rule diagnostic uses full-rollout predictions and simulated exits, not realized sequential savings.Thresholds are fit by five-fold cross-fitting on the same 3,000-image evaluation split.Source: §3.1 · pp. 20, 22, Figure 5 · p. 25

4 Computational Cost

The cost section estimates encoder-pass computation, profiles model-forward runtime under a specified inference protocol, and records training time. Adaptive inference reduces mean batch-1 latency versus fixed four-step inference, but has no consistent throughput advantage at larger batch sizes.

4 Computational Cost

  • Full+EE costs an estimated 1.10 + 2.20 K̄ GFLOPs, with K̄ denoting average selected focus steps; the estimate excludes non-encoder operations.The conservative estimate counts a detail pass at every selected step, including the first; four-step costs are 5.50 GFLOPs for Orig and 9.90 GFLOPs for Full+EE.Source: §4 · p. 26, Table 8 · p. 26

4.1 Model-forward runtime and memory

  • Adaptive inference reduces mean batch-1 latency versus fixed four-step Full+EE, while P90 changes little; fixed two-step inference has similar mean latency and lower P90.Profiling uses classifier max-softmax and a single fitted threshold; at the two-step target, the threshold is 0.914015 and timed images average 2.04 focus steps.Source: §4.1 · pp. 26–27
  • At batch sizes 32 and 48, adaptive execution has no consistent throughput advantage over fixed four-step Full+EE inference.The measurements cover 3,200 and 4,800 timed images, respectively.Source: §4.1 · p. 27

4.2 Recorded training cost

  • The recorded three-GPU ImageNet-1K +S+D run takes about 4.2 GPU-hours per epoch and 210 GPU-hours for 50 epochs.Logged early-exit fine-tuning takes approximately 5.1 GPU-hours per epoch, estimated from the available log segment.Source: §4.2 · p. 27
Loading 2610.19624v1…