Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception

1 Introduction

• Uniform full-image processing and resizing can obscure small, important regions, especially in high-resolution scenes where relevant content is small or cluttered.Task-specific information is typically incorporated only after costly feature extraction, whereas early integration could reduce the image area requiring intensive computation.Source: §1 · p. 1
• Coarse glimpses use correlations between target features and a downscaled scene, while fine glimpses iteratively refine locations using a fovea-inspired log-polar sensor.The sensor magnifies details near each glimpse while preserving peripheral information at diminishing resolution.Source: §1 · pp. 2–3
• CF-GAP integrates with existing instance detectors and significantly improves their performance, with small models matching or sometimes surpassing larger versions.The evaluation uses HR-InsDet and Robotools, two challenging benchmarks.Source: §1 · p. 3
• CF-GAP replaces exhaustive full-scene processing with a bio-inspired, task-driven sequence of glimpses that selectively explores high-resolution scenes.The search target guides the sequence toward likely locations; the resulting regions of interest are passed to downstream detectors at high resolution.Source: §1 · pp. 2–3

2 Related Work

• State-of-the-art instance detectors first find proposals across the entire scene, then match them to search-target examples; CF-GAP instead steers detection toward promising regions.The described baselines use SAM or GroundingDINO for proposals and DINOv2 features for matching.Source: §2 · p. 3
• Earlier saccadic-processing approaches span several vision tasks, but none conditions image exploration at inference on task information such as a search target.The original GAP framework lacks a coarse-to-fine, task-driven search strategy.Source: §2 · p. 4
• Prior fovea-inspired imagery mostly encoded a fixed view, whereas CF-GAP uses it to navigate coarse-to-fine glimpsing.Earlier work explored foveal imagery for visual search in simple images and for other vision tasks [3] [9] [18].Source: §2 · p. 4

3 Method

• CF-GAP uses two nested stages of coarse and fine glimpsing, guided by 2D search-map heatmaps of likely target locations, to define a region for downstream verification.The search maps compare scene and search-target features via convolution.Source: §3 · p. 4

3.1 Coarse glimpsing

• The coarse search map correlates averaged target-example features with downscaled scene features, then selects peaks using winner-takes-all and inhibition-of-return.Each coarse location initiates a closer inspection because low-resolution imagery provides only a rough target position.Source: §3.1 · pp. 4–5

3.2 Fine glimpsing

• Fine glimpsing iteratively generates a search map from the current scene glimpse and moves the log-polar sensor to the map centroid for the next glimpse.The sensor preserves high-resolution detail near the glimpse location while retaining distant context at lower resolution, making search-map generation robust against peripheral distractors.Source: §3.2 · pp. 5–6
• Perceiver-based encoders handle log-polar glimpses by compressing scene features into latent embeddings, processing them with self-attention, and decoding a spatial feature map.A distinct set of learnable value embeddings defines the output feature space, and the compression reduces the quadratic cost of subsequent self-attention.Source: Figure 2 · p. 5, §3.2 · p. 6
• The target encoder iteratively attends to multiple target glimpses to produce compact features, which are correlated with the scene feature map to form the fine search map.The architecture shares cross- and self-attention blocks between scene and target encoders.Source: Figure 2 · p. 5, §3.2 · p. 6

3.3 Downstream architecture

• After the prescribed glimpse iterations, the downstream detector receives a fixed-size region, the final fine-glimpse location, and target examples; CF-GAP is detector-agnostic.The location can serve as a spatial prompt or a bounding-box filter, depending on the downstream architecture.Source: §3.3 · p. 7

4 Experiments

• Evaluation covers HR-InsDet and Robotools, reporting AP across IoU thresholds from 0.5 to 0.95 and AP50 at IoU 0.5.HR-InsDet includes clutter, occlusion, and object-size subsets; Robotools tests generalization to novel search targets excluded from training.Source: Instance detection · p. 7
• The baseline set includes OTS-FM with SAM or GroundingDINO proposals, NIDS-Net, and efficient OTS-FM variants using MobileSAM or STT.IDOW is excluded because its fine-tuned weights are unavailable for integration; STT is evaluated only with CF-GAP because it requires a location input.Source: Baselines · pp. 7–8
• CF-GAP supplies small regions and fine-glimpse point prompts to SAM-based baselines, while GroundingDINO detections are filtered by fine-glimpse locations.Standalone baselines instead process the full high-resolution scene with a coarse grid of point prompts.Source: Setup · p. 8
• The experiments use downscaled scenes for coarse glimpsing and full-resolution scenes for fine glimpsing, with 30 coarse and three fine glimpses by default.HR-InsDet scenes are resized to 4096×5460 for experimentation, while Robotools scenes remain at original resolution.Source: Setup · p. 8

5.1 Benchmarking results

• Up to 20% AP improvement on hard HR-InsDet scenes accompanies gains across baselines and Robotools; MobileSAM reaches heavier models’ AP50, but its AP remains lower.The largest HR-InsDet benefits occur on small-object and hard-scene subsets.Source: §5.1 · pp. 8–9, Figure 3 · p. 9, Figure 4 · p. 9

5.2 Analysis and Ablations

• CF-GAP achieves higher AP than exhaustive patching on hard, cluttered scenes while passing over 5× fewer RoIs downstream.The patched baseline splits 6144×8192 scenes into 1024×1024 patches with 50% overlap, yielding 165 RoIs per scene versus 30 from CF-GAP.Source: Naive high-resolution baseline · pp. 9–10, Table 1 · p. 10
• Fine glimpsing most benefits hard scenes and small objects, where it corrects coarse-location errors; its benefit is marginal when objects are easy to find.Coarse search maps can place glimpses away from the object, disrupting downstream models prompted at specific locations.Source: Glimpse locations · pp. 10–11, Table 3 · p. 11
• Log-polar glimpses maintain more stable performance across sizes than crop-based glimpses, avoiding sensitivity to crop size and peripheral distraction.Their peripheral resolution diminishes logarithmically, keeping focus on nearby regions as the captured area changes.Source: Log-polar glimpses · pp. 11–12, Figure 5 · p. 11, Figure 6 · p. 12

5.3 Computational Cost

• With CF-GAP, MobileSAM and STT surpass the original OTS-FMSAM in both efficiency and performance.The comparison uses efficient SAM variants as OTS-FM backbones; the authors note that efficient GroundingDINO variants were unavailable.Source: §5.3 · pp. 13–14, Figure 8 · p. 14
• More coarse glimpses trade efficiency for performance, and downstream-model invocations dominate the reported compute cost.The cost is reported in FLOPS rather than wall-clock runtime, which depends on implementation-specific optimizations.Source: §5.3 · pp. 14–15, Table 5 · p. 14
• In ∼50% of scenes, CF-GAP finds the target within the first 8–10 glimpses, suggesting a robust matching stage could reduce computational cost by stopping early.The proposed early termination depends on recognizing the target during the glimpsing process.Source: Figure 9 · p. 14, §5.3 · p. 15

6 Discussion

• CF-GAP consistently boosts existing instance detectors, with the largest gains for small objects and cluttered scenes; lighter models can be competitive with larger ones.The discussion also describes efficient use of large downstream architectures with fovea-inspired tokenization.Source: §6 · p. 15
• Current CF-GAP relies on texture guidance, may revisit distractors, uses conservatively large fixed-size crops, and incurs cost proportional to coarse glimpses.The paper identifies semantic guidance, adaptive crops, and a stopping mechanism as future directions.Source: §6 · p. 15

A Model Details

• The code can be accessed via a link.Source: A Model Details · p. 19

A.1 Log-polar sensor

• The log-polar sensor samples around a glimpse location, balancing resolution and field of view by magnifying nearby regions.The sampling radius is controlled by hyperparameter ρ.Source: §A.1 · p. 19

A.2 Fine search map

• The target encoder iteratively processes target glimpses into compact target features; three log-polar views are used, with a random fourth view added during training.Source: Input preparation · p. 19, Search target encoder · p. 21
• The scene encoder compresses its feature grid into N latent embeddings before self-attention, reducing quadratic cost while integrating local detail with peripheral context.Source: Scene encoder · p. 20
• A second cross-attention decodes the processed scene latents to spatial resolution, using distinct learnable embeddings to define the output feature map.Source: Scene encoder · p. 20
• Target embeddings are correlated with scene features to form fine search maps, which are averaged across target viewpoints for robustness to viewpoint variation.Source: Feature comparison · p. 21
• Log-polar positional embeddings concatenate angular and radial encodings; cyclic angular frequencies avoid an artificial seam between adjacent angular positions.Source: Positional Embeddings · p. 22

A.3 Location selection

• Coarse glimpses select the highest-valued location in the coarse map, then inhibit that location and its neighborhood to discourage repeat selection.Source: §A.3 · p. 22
• Fine locations use the fine-map centroid: radial position is weighted by normalized map values, while angular averaging accounts for periodicity.Source: §A.3 · p. 23

• The coarse search-map model uses pretrained MobileNetV3 weights without further fine-tuning.Source: §A.4 · p. 23
• The fine search-map model is trained from scratch on synthetic scenes made by pasting target images onto random backgrounds.The synthetic data use a cut-paste-learn framework.Source: §A.4 · p. 23
• Each training sample pairs a scene glimpse with four target glimpses: central, top, and bottom views provide texture, while a random target view specifies the part to localize.Source: §A.4 · pp. 23–24
• Training predicts a spatial map with one positive target location, mapped from the sampled target part into the scene-glimpse coordinates, using cross-entropy over spatial locations.Training uses objects and random backgrounds from HR-InsDet.Source: §A.4 · p. 24

• HR-InsDet scenes were resized to 4096 × 5460 from 6144 × 8192, while Robotools scenes remained at their original 1920 × 1080 resolution.Source: B Experimental details · p. 24
• Coarse glimpsing uses scenes downscaled by a factor of 2, while fine glimpsing uses full-resolution scenes and log-polar images of 245 × 245 pixels.The log-polar sensor radius is 2048 for HR-InsDet and 540 for Robotools.Source: B Experimental details · p. 24
• Each scene receives 30 coarse glimpses and 3 fine glimpses per coarse glimpse, yielding downstream crops of 800 × 800 for HR-InsDet and 360 × 360 for Robotools.Source: B Experimental details · p. 24
• Fine search-map attention uses single-layer, four-head transformers with feature dimension 96, patch size 5 × 5, and 128 learnable embeddings.Source: B Experimental details · p. 24

• Table 6 covers HR-InsDet performance for the paper’s models and prior-work models, including reproduced results for models later evaluated with CF-GAP.OTS-FMSTT is evaluated only with CF-GAP because its STT backbone requires glimpse locations.Source: Table 6 · p. 25
• Table 7 reports performance on Robotools for the paper’s models and models evaluated in prior work.Source: Table 7 · p. 25
• Table 8 analyzes hyperparameter sensitivity for CF-GAP+NIDS-Net on the hard HR-InsDet subset.Source: Table 8 · p. 25

D Justifying the architecture for log-polar processing

• The attention-based architecture outperforms the tested CNN configurations for fine search maps, with the authors attributing the gap to its integration of local detail and broader peripheral context.Source: D Justifying the architecture for… · p. 26
• Table 9 compares CNN-based scene and target encoders with CF-GAP+NIDS-Net on the two most challenging HR-InsDet subsets.Source: Table 9 · p. 26

E Qualitative examples

• Figure 11 shows coarse and fine glimpse locations in six scenes where baseline models miss the search target but the proposed approach succeeds.The top four examples are from HR-InsDet and the bottom two from Robotools.Source: Figure 11 · p. 27

F Failed cases

• CF-GAP failure cases are those in which no glimpse location lands on the search target’s surface.Source: Figure 12 · p. 28, F Failed cases · p. 28
• Small, cluttered or strongly occluded targets can divert glimpses toward distracting regions with similar texture.Source: F Failed cases · pp. 28–29
• Targets that are barely distinguishable from their backgrounds are another source of failure.Source: F Failed cases · p. 29