PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading Decoder Trajectories: Training-Free Counterfactual Query-Trajectory Reliability for Small-Object Detection

Zhaoning Shi, Bo Ma

arXiv:2609.06581v1cs.CVcs.AI

TL;DR

Small-object detection may fail because frozen detectors’ scale knowledge is underactivated and unstable, not because the knowledge is absent. CQTR uses counterfactual scale intervention, decoder-trajectory reliability, cross-scale conflict analysis, and unlabeled routing without parameter updates or target-domain annotations. Across 27 detector–dataset combinations, it consistently improves AP and APs while analyses support latent-response activation and reliability-based correction.

  • Problem

    Small-object detection is challenged by compressed evidence, while existing approaches do not establish whether frozen detectors already contain the needed scale knowledge.

    Source: §I · pp. 1–2

  • Method

    CQTR activates latent scale knowledge through a counterfactual scale branch, interprets spatial-semantic decoder trajectories and cross-scale conflicts, and routes corrections using a small unlabeled subset.

    Source: §I · pp. 1–2, E. Unlabeled Stream-Level Reliability… · p. 4

  • Results

    Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves AP and APs; analyses link scaling to latent responses, trajectories to reliability, and routing to the more effective correction mechanism.

    Source: §I · p. 2, A. Results Across Detectors and Datasets · pp. 6–7, B. Ablation Study · p. 7

  • Takeaways & Limitations

    The paper reframes small-object detection as activating and assessing latent scale knowledge rather than relying solely on external scale augmentation.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

Small-object detection remains challenging because limited pixels cause information loss and suppress the scale knowledge encoded in pretrained detectors. Existing approaches mainly improve representations through multiscale training, architecture redesign, or parameter adaptation, implicitly assuming that frozen models lack the required capability. We challenge this assumption and hypothesize that small-object knowledge already exists in frozen detectors but remains underactivated and unstable during query evolution. To test this hypothesis, we propose Counterfactual Query-Trajectory Reliability (CQTR), a training-free framework that elicits latent responses through counterfactual scale interventions and interprets candidate reliability from decoder-internal spatial convergence, semantic persistence, and cross-scale conflicts. A small unlabeled training subset selects the appropriate correction mechanism for each model-data stream, without parameter updates or target-domain annotations. Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves average precision (AP) and average precision for small objects (APs). Closed-loop analyses further show that scale intervention activates latent responses, trajectory evidence predicts ground-truth support, and unlabeled routing selects the more effective branch. CQTR therefore reframes small-object detection from external scale augmentation to the activation and reliability assessment of latent scale knowledge.

I. INTRODUCTION

Small-object evidence is weakened by compression, raising whether frozen detectors already contain latent scale knowledge that standard inference fails to activate. CQTR addresses this question by activating and assessing such knowledge through counterfactual scaling and decoder-trajectory reliability analysis.

  • Small objects occupy few pixels and feature cells, making their evidence vulnerable to downsampling and background responses.Source: §I · p. 1
  • Existing approaches improve external representations but do not establish whether gains come from newly learned knowledge or reactivated capabilities in frozen detectors.Source: §I · p. 1
  • Counterfactual scaling can reintroduce compressed object evidence but may also amplify background textures and spurious candidates.Source: §I · p. 1
  • CQTR interprets decoder states through spatial convergence and semantic persistence, then detects unreliable candidates using native–counterfactual semantic conflicts.Source: §I · p. 1
  • CQTR uses a small unlabeled training subset to select the correction mechanism for each model-data stream without parameter updates or target-domain annotations.Source: §I · p. 1
  • Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves AP and APs, with analyses linking scaling, trajectory evidence, and routing to reliability.Source: §I · pp. 1–2

II. RELATED WORK

Prior small-object methods enhance representations, spatial resolution, or query optimization, while existing interpretation methods focus mainly on outputs. CQTR instead analyzes frozen-detector query formation through counterfactual scaling, decoder trajectories, and reliability routing.

II. RELATED WORK

  • Representation, spatial-decomposition, and query-optimization methods address small-object detection through feature enhancement, relative-resolution increases, or improved query mechanisms.Source: A. Small-Object Detection · p. 2
  • These approaches primarily learn new scale representations and do not determine whether frozen detectors already contain usable scale knowledge.Source: A. Small-Object Detection · p. 2
  • Post hoc interpretation methods localize image regions influencing predictions, whereas internal-mechanism methods examine intermediate features, activations, or computational paths.Source: B. Internal Model Interpretation · p. 2
  • CQTR analyzes query-formation mechanisms rather than limiting explanation to output visualization.Source: B. Internal Model Interpretation · p. 3
  • CQTR constructs native and counterfactual branches, retains late decoder responses, corrects candidates through two interpretations, and fuses the routed result.Source: A. Problem Formulation and Overall… · p. 3, Figure 2 · p. 3
  • The counterfactual scale is selected adaptively from spatial compression because native resizing can compress objects while uniform enlargement can amplify redundancy and background noise.Source: B. Counterfactual Scale Branch · p. 3

C. Three-Step Spatial-Semantic Query-Trajectory Interpretation

CQTR estimates candidate reliability from how decoder boxes converge spatially and how consistently class support persists across layers. It combines these signals into trajectory evidence for relative confidence recalibration.

  • Final confidence alone ignores how a prediction evolves during decoding, so CQTR measures spatial convergence and semantic persistence from decoder trajectories.Source: C. Three-Step Spatial-Semantic… · p. 4
  • For each candidate, spatial convergence is computed from box transitions over the last three decoder steps using intersection over union.Source: C. Three-Step Spatial-Semantic… · p. 4
  • A larger spatial-convergence value indicates that the query box stabilizes near the end of decoding, whereas a smaller value indicates persistent positional jumps.Source: C. Three-Step Spatial-Semantic… · p. 4
  • Semantic persistence compares the final class response with the strongest response across the last four decoder layers, decreasing when class switching is frequent.Source: C. Three-Step Spatial-Semantic… · p. 4
  • Trajectory evidence is the geometric mean of spatial convergence and semantic persistence, and confidence is recalibrated relative to the current image’s median evidence.Source: C. Three-Step Spatial-Semantic… · p. 4

D. Cross-Scale Semantic-Conflict Interpretation

CQTR identifies cross-scale semantic conflicts when counterfactual candidates overlap native predictions but assign different classes, then softsuppresses only unreliable candidates. Conflict correction and trajectory recalibration remain parallel alternatives selected by an unlabeled router.

Cross-scale conflict detection

  • CQTR softsuppresses counterfactual candidates that strongly overlap high-confidence native predictions but predict different classes.This preserves newly exposed objects while addressing inconsistent semantic explanations across scales.Source: D. Cross-Scale Semantic-Conflict… · p. 4

Conflict measurement

  • Cross-scale spatial overlap is computed after restoring native and counterfactual predictions to the original-image coordinate system.The overlap is confidence-weighted and based on candidate boxes from the two prediction streams.Source: D. Cross-Scale Semantic-Conflict… · p. 4
  • Conflict strength is the maximum overlap with a native candidate of a different class, with larger values indicating stronger contradiction.The conflict-corrected score uses a suppression strength γ and minimum retention factor wmin.Source: D. Cross-Scale Semantic-Conflict… · p. 4

Correction streams

  • Trajectory recalibration and conflict suppression are parallel candidate streams rather than sequential operations, and the unlabeled router selects which stream is used.This prevents indiscriminate application of both corrections.Source: D. Cross-Scale Semantic-Conflict… · p. 4

E. Unlabeled Stream-Level Reliability Routing and Unified Output

CQTR estimates stream-level reliability from unlabeled examples and routes each model-data stream to trajectory recalibration or cross-scale conflict suppression. The selected correction is then retained for the entire stream before producing the unified output.

Routing rationale

  • Query trajectories and cross-scale semantic conflicts capture complementary forms of internal instability, so CQTR selects between their corrections rather than applying both indiscriminately.The two mechanisms address different failure modes.Source: E. Unlabeled Stream-Level Reliability… · p. 4

Unlabeled reliability estimation

  • CQTR samples M unlabeled training images with a fixed random seed to estimate stream-level trajectory dispersion and conflict strength.Higher trajectory dispersion indicates clearer separation of candidate reliability, while larger conflict statistics indicate stronger class competition caused by scale changes.Source: E. Unlabeled Stream-Level Reliability… · p. 4

Routing decision

  • The routing variable applies query-trajectory recalibration when z = 1 and cross-scale semantic-conflict suppression otherwise.The decision uses thresholds for trajectory dispersion and conflict strength.Source: E. Unlabeled Stream-Level Reliability… · p. 4

Unified output

  • After unlabeled calibration, the selected routing variable remains fixed for the entire model-data stream.The corrected counterfactual predictions are then formed under that stream-level choice.Source: E. Unlabeled Stream-Level Reliability… · p. 4
  • The spatial compression ratio ρ(x) determines whether the native branch is retained, while candidate budget Q and NMS threshold η constrain output selection.These choices govern the final unified prediction stage.Source: E. Unlabeled Stream-Level Reliability… · p. 5

IV. EXPERIMENTS

The experiments evaluate CQTR as a frozen, training-free inference framework across three datasets and nine detector families. Cross-domain settings use COCO-pretrained models without target-domain annotations or fine-tuning, under a common evaluation protocol.

Datasets

  • The evaluation uses COCO val2017, TinyPerson test, and VisDrone2019-DET val, containing 5,000, 786, and 548 images, respectively.The latter two datasets test cross-domain small-object detection.Source: §IV · p. 5
  • TinyPerson and VisDrone use COCO-pretrained weights without target-domain training annotations or fine-tuning.This setting tests generalization to cross-domain small-object scenes.Source: §IV · p. 5

Evaluation protocol

  • All datasets use the same COCO-style evaluator reporting AP, AP50, AP75, and APs, with protocol-specific limits and comparisons restricted to paired frozen baselines.TinyPerson does not use its official miss-rate protocol, and VisDrone uses maxDets=100.Source: §IV · p. 5

Detector coverage

  • The study covers nine frozen detectors spanning fixed queries, dynamic anchors, deformable sampling, query selection, and real-time hybrid encoding.All models use COCO-pretrained weights and run under inference mode without gradients.Source: §IV · pp. 5–6

Experimental design

  • The full evaluation contains 27 model-dataset combinations using the same configuration across all detectors and datasets.Experiments cover 5,000, 786, and 548 images for COCO, TinyPerson, and VisDrone, respectively.Source: §IV · p. 6

A. Results Across Detectors and Datasets

CQTR improves both overall and small-object AP across all 27 detector-dataset combinations, with gains spanning diverse detector families and target domains. The reported improvements include substantial APs increases on COCO, TinyPerson, and VisDrone2019.

Cross-dataset results

  • CQTR improves both AP and APs across all 27 detector-dataset combinations.Figures 3 and 4 visualize the overall and small-object AP trends, while complete values are reported in Tables II–IV.Source: A. Results Across Detectors and Datasets · p. 6

COCO

TinyPerson

VisDrone2019

  • On VisDrone2019, improvements range from 0.81 to 2.66 AP and from 0.36 to 2.39 APs.DEIM-D-FINE-S rises from 14.50 to 17.16 AP, while RT-DETRv2-R50 reaches the highest absolute AP of 17.49.Source: A. Results Across Detectors and Datasets · p. 7

Generalization

  • The improvements span diverse detector families and target domains rather than being limited to one architecture or the COCO scale distribution.This conclusion follows the consistent gains reported across the three datasets and nine detectors.Source: A. Results Across Detectors and Datasets · p. 7

B. Ablation Study

The ablation isolates how counterfactual scaling, trajectory evidence, conflict suppression, and unlabeled routing contribute to CQTR. Analyses support scale activation, trajectory-based reliability assessment, and data-driven branch selection.

Ablation Results

  • Counterfactual scaling alone contributes most of the gain, reaching 54.18, 12.99, and 17.13 AP on COCO, TinyPerson, and VisDrone, respectively.The later components are not uniformly beneficial because they address different internal failure modes.Source: B. Ablation Study · p. 7
  • CQTR recovers missed small objects across all three datasets, increasing true positives from 18, 9, and 43 to 23, 28, and 61.False negatives decrease from 18, 68, and 131 to 13, 49, and 113 in the illustrated examples.Source: C. Qualitative Results · p. 7

Latent Scale Activation

Trajectory Reliability

Unlabeled Routing

Trajectory Reliability

  • High-evidence queries converge more stably around small objects than low-evidence queries across the final four decoder layers.Semantic-density maps and representative trajectories provide internal support for trajectory-based reliability estimation.Source: D. Mechanistic Interpretability and… · p. 8

V. CONCLUSION

CQTR is a training-free framework for improving small-object detection with frozen detectors. It combines counterfactual scale intervention, query-trajectory interpretation, and unlabeled reliability routing to activate latent scale knowledge and suppress cross-scale conflicts.

Conclusion

  • CQTR combines a counterfactual scale branch, spatial-semantic query-trajectory interpretation, and unlabeled reliability routing for frozen detectors.The framework is presented as training-free and intended to improve small-object detection.Source: §V · p. 8
  • Experiments across multiple detectors and three datasets demonstrate CQTR's effectiveness, generality, and interpretability.The conclusion frames these properties as the paper's main empirical support.Source: §V · p. 8

APPENDIX

The appendix reports complete quantitative results across 27 detector-dataset combinations. CQTR improves the evaluated metrics across COCO, TinyPerson, and VisDrone2019 under frozen-model protocols without target-domain training annotations.

Quantitative Results

  • All 27 detector-dataset results report AP, AP50, AP75, and APs, with parenthesized values denoting absolute improvements over frozen baselines.All results use frozen parameters and no target-domain training annotations.Source: A. Full Quantitative Results · p. 9

COCO

TinyPerson

  • RT-DETR-R50's TinyPerson AP increases from 9.19 to 13.04, while APs increases from 5.89 to 9.60.CQTR improves AP, AP50, AP75, and APs across all nine detectors on TinyPerson.Source: A. Full Quantitative Results · p. 9

VisDrone2019

  • DEIM-D-FINE-S has the largest VisDrone AP gain, increasing from 14.50 to 17.16, while APs increases from 6.22 to 8.61.RT-DETRv2-R50 achieves the highest absolute VisDrone AP of 17.49.Source: A. Full Quantitative Results · p. 9

Cross-dataset pattern

  • CQTR's small-object detection improvement is more pronounced than its overall AP gain on COCO.The appendix also reports effectiveness in dense aerial scenes with severe scale variation and background interference.Source: A. Full Quantitative Results · p. 9

B. Visualization of Decoder Query Trajectories

The visualization projects decoder query states into semantic-density maps and spatial trajectories for small-object candidates. High-evidence queries remain concentrated around small objects and converge more stably across later decoder layers.

Decoder Visualization

  • The visualization projects query box centers and final-class responses from the final four decoder layers onto the original image.These projections connect spatial locations with semantic responses during iterative decoding.Source: B. Visualization of Decoder Query… · p. 9

Semantic Query Density

  • Semantic query density uses queries whose final predicted area does not exceed the small-object threshold, with a shared normalization range across layers.The shared range allows colors from different decoder layers to be compared directly.Source: B. Visualization of Decoder Query… · p. 9

Semantic Persistence

  • Small-object candidate responses maintain high object-to-background ratios across the four layers, with values of 24.58, 26.44, 26.14, and 26.06.The image contains 80 small-object candidate queries.Source: B. Visualization of Decoder Query… · p. 9

Trajectory Comparison

Visualization Materials

Related papers

Loading 2609.06581v1…