PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading Decoder Trajectories: Training-Free Counterfactual Query-Trajectory Reliability for Small-Object Detection

Zhaoning Shi, Bo Ma

arXiv:2609.06581v1cs.CVcs.AI

TL;DR

Small-object detection may fail because frozen detectors’ scale knowledge is underactivated and unstable, not because the knowledge is absent. CQTR uses counterfactual scale intervention, decoder-trajectory reliability, cross-scale conflict analysis, and unlabeled routing without parameter updates or target-domain annotations. Across 27 detector–dataset combinations, it consistently improves AP and APs while analyses support latent-response activation and reliability-based correction.

  • Problem

    Small-object detection is challenged by compressed evidence, while existing approaches do not establish whether frozen detectors already contain the needed scale knowledge.

  • Method

    CQTR activates latent scale knowledge through a counterfactual scale branch, interprets spatial-semantic decoder trajectories and cross-scale conflicts, and routes corrections using a small unlabeled subset.

  • Results

    Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves AP and APs; analyses link scaling to latent responses, trajectories to reliability, and routing to the more effective correction mechanism.

  • Takeaways & Limitations

    The paper reframes small-object detection as activating and assessing latent scale knowledge rather than relying solely on external scale augmentation.

Abstract

from arXiv · show

Small-object detection remains challenging because limited pixels cause information loss and suppress the scale knowledge encoded in pretrained detectors. Existing approaches mainly improve representations through multiscale training, architecture redesign, or parameter adaptation, implicitly assuming that frozen models lack the required capability. We challenge this assumption and hypothesize that small-object knowledge already exists in frozen detectors but remains underactivated and unstable during query evolution. To test this hypothesis, we propose Counterfactual Query-Trajectory Reliability (CQTR), a training-free framework that elicits latent responses through counterfactual scale interventions and interprets candidate reliability from decoder-internal spatial convergence, semantic persistence, and cross-scale conflicts. A small unlabeled training subset selects the appropriate correction mechanism for each model-data stream, without parameter updates or target-domain annotations. Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves average precision (AP) and average precision for small objects (APs). Closed-loop analyses further show that scale intervention activates latent responses, trajectory evidence predicts ground-truth support, and unlabeled routing selects the more effective branch. CQTR therefore reframes small-object detection from external scale augmentation to the activation and reliability assessment of latent scale knowledge.

I. INTRODUCTION

Small-object evidence is weakened by compression, raising whether frozen detectors already contain latent scale knowledge that standard inference fails to activate. CQTR addresses this question by activating and assessing such knowledge through counterfactual scaling and decoder-trajectory reliability analysis.

  • Small objects occupy few pixels and feature cells, making their evidence vulnerable to downsampling and background responses.
  • Existing approaches improve external representations but do not establish whether gains come from newly learned knowledge or reactivated capabilities in frozen detectors.
  • Counterfactual scaling can reintroduce compressed object evidence but may also amplify background textures and spurious candidates.
  • CQTR interprets decoder states through spatial convergence and semantic persistence, then detects unreliable candidates using native–counterfactual semantic conflicts.
  • CQTR uses a small unlabeled training subset to select the correction mechanism for each model-data stream without parameter updates or target-domain annotations.
  • Across 27 combinations of nine frozen detectors and three datasets, CQTR consistently improves AP and APs, with analyses linking scaling, trajectory evidence, and routing to reliability.

II. RELATED WORK

Prior small-object methods enhance representations, spatial resolution, or query optimization, while existing interpretation methods focus mainly on outputs. CQTR instead analyzes frozen-detector query formation through counterfactual scaling, decoder trajectories, and reliability routing.

  • II. RELATED WORK: Representation, spatial-decomposition, and query-optimization methods address small-object detection through feature enhancement, relative-resolution increases, or improved query mechanisms.
  • II. RELATED WORK: These approaches primarily learn new scale representations and do not determine whether frozen detectors already contain usable scale knowledge.
  • II. RELATED WORK: Post hoc interpretation methods localize image regions influencing predictions, whereas internal-mechanism methods examine intermediate features, activations, or computational paths.
  • II. RELATED WORK: CQTR analyzes query-formation mechanisms rather than limiting explanation to output visualization.
  • II. RELATED WORK: CQTR constructs native and counterfactual branches, retains late decoder responses, corrects candidates through two interpretations, and fuses the routed result.
  • II. RELATED WORK: The counterfactual scale is selected adaptively from spatial compression because native resizing can compress objects while uniform enlargement can amplify redundancy and background noise.

C. Three-Step Spatial-Semantic Query-Trajectory Interpretation

CQTR estimates candidate reliability from how decoder boxes converge spatially and how consistently class support persists across layers. It combines these signals into trajectory evidence for relative confidence recalibration.

  • Final confidence alone ignores how a prediction evolves during decoding, so CQTR measures spatial convergence and semantic persistence from decoder trajectories.
  • For each candidate, spatial convergence is computed from box transitions over the last three decoder steps using intersection over union.
  • A larger spatial-convergence value indicates that the query box stabilizes near the end of decoding, whereas a smaller value indicates persistent positional jumps.
  • Semantic persistence compares the final class response with the strongest response across the last four decoder layers, decreasing when class switching is frequent.
  • Trajectory evidence is the geometric mean of spatial convergence and semantic persistence, and confidence is recalibrated relative to the current image’s median evidence.

D. Cross-Scale Semantic-Conflict Interpretation

CQTR identifies cross-scale semantic conflicts when counterfactual candidates overlap native predictions but assign different classes, then softsuppresses only unreliable candidates. Conflict correction and trajectory recalibration remain parallel alternatives selected by an unlabeled router.

  • Cross-scale conflict detection: CQTR softsuppresses counterfactual candidates that strongly overlap high-confidence native predictions but predict different classes.This preserves newly exposed objects while addressing inconsistent semantic explanations across scales.
  • Conflict measurement: Cross-scale spatial overlap is computed after restoring native and counterfactual predictions to the original-image coordinate system.The overlap is confidence-weighted and based on candidate boxes from the two prediction streams.
  • Conflict measurement: Conflict strength is the maximum overlap with a native candidate of a different class, with larger values indicating stronger contradiction.The conflict-corrected score uses a suppression strength γ and minimum retention factor wmin.
  • Correction streams: Trajectory recalibration and conflict suppression are parallel candidate streams rather than sequential operations, and the unlabeled router selects which stream is used.This prevents indiscriminate application of both corrections.

E. Unlabeled Stream-Level Reliability Routing and Unified Output

CQTR estimates stream-level reliability from unlabeled examples and routes each model-data stream to trajectory recalibration or cross-scale conflict suppression. The selected correction is then retained for the entire stream before producing the unified output.

  • Routing rationale: Query trajectories and cross-scale semantic conflicts capture complementary forms of internal instability, so CQTR selects between their corrections rather than applying both indiscriminately.The two mechanisms address different failure modes.
  • Unlabeled reliability estimation: CQTR samples M unlabeled training images with a fixed random seed to estimate stream-level trajectory dispersion and conflict strength.Higher trajectory dispersion indicates clearer separation of candidate reliability, while larger conflict statistics indicate stronger class competition caused by scale changes.
  • Routing decision: The routing variable applies query-trajectory recalibration when z = 1 and cross-scale semantic-conflict suppression otherwise.The decision uses thresholds for trajectory dispersion and conflict strength.
  • Unified output: After unlabeled calibration, the selected routing variable remains fixed for the entire model-data stream.The corrected counterfactual predictions are then formed under that stream-level choice.
  • Unified output: The spatial compression ratio ρ(x) determines whether the native branch is retained, while candidate budget Q and NMS threshold η constrain output selection.These choices govern the final unified prediction stage.

IV. EXPERIMENTS

The experiments evaluate CQTR as a frozen, training-free inference framework across three datasets and nine detector families. Cross-domain settings use COCO-pretrained models without target-domain annotations or fine-tuning, under a common evaluation protocol.

  • Datasets: The evaluation uses COCO val2017, TinyPerson test, and VisDrone2019-DET val, containing 5,000, 786, and 548 images, respectively.The latter two datasets test cross-domain small-object detection.
  • Datasets: TinyPerson and VisDrone use COCO-pretrained weights without target-domain training annotations or fine-tuning.This setting tests generalization to cross-domain small-object scenes.
  • Evaluation protocol: All datasets use the same COCO-style evaluator reporting AP, AP50, AP75, and APs, with protocol-specific limits and comparisons restricted to paired frozen baselines.TinyPerson does not use its official miss-rate protocol, and VisDrone uses maxDets=100.
  • Detector coverage: The study covers nine frozen detectors spanning fixed queries, dynamic anchors, deformable sampling, query selection, and real-time hybrid encoding.All models use COCO-pretrained weights and run under inference mode without gradients.
  • Experimental design: The full evaluation contains 27 model-dataset combinations using the same configuration across all detectors and datasets.Experiments cover 5,000, 786, and 548 images for COCO, TinyPerson, and VisDrone, respectively.

A. Results Across Detectors and Datasets

CQTR improves both overall and small-object AP across all 27 detector-dataset combinations, with gains spanning diverse detector families and target domains. The reported improvements include substantial APs increases on COCO, TinyPerson, and VisDrone2019.

  • Cross-dataset results: CQTR improves both AP and APs across all 27 detector-dataset combinations.Figures 3 and 4 visualize the overall and small-object AP trends, while complete values are reported in Tables II–IV.
  • COCO: On COCO, gains reach 0.75–1.65 AP and 1.95–3.81 APs.RT-DETR-R50 improves from 53.09 to 54.21 AP and from 34.74 to 38.55 APs.
  • TinyPerson: On TinyPerson, CQTR yields gains of 0.85–3.85 AP and 0.44–3.71 APs.RT-DETR-R50 increases from 9.19 to 13.04 AP.
  • VisDrone2019: On VisDrone2019, improvements range from 0.81 to 2.66 AP and from 0.36 to 2.39 APs.DEIM-D-FINE-S rises from 14.50 to 17.16 AP, while RT-DETRv2-R50 reaches the highest absolute AP of 17.49.
  • Generalization: The improvements span diverse detector families and target domains rather than being limited to one architecture or the COCO scale distribution.This conclusion follows the consistent gains reported across the three datasets and nine detectors.

B. Ablation Study

The ablation isolates how counterfactual scaling, trajectory evidence, conflict suppression, and unlabeled routing contribute to CQTR. Analyses support scale activation, trajectory-based reliability assessment, and data-driven branch selection.

  • Ablation Results: Counterfactual scaling alone contributes most of the gain, reaching 54.18, 12.99, and 17.13 AP on COCO, TinyPerson, and VisDrone, respectively.The later components are not uniformly beneficial because they address different internal failure modes.
  • Ablation Results: CQTR recovers missed small objects across all three datasets, increasing true positives from 18, 9, and 43 to 23, 28, and 61.False negatives decrease from 18, 68, and 131 to 13, 49, and 113 in the illustrated examples.
  • Latent Scale Activation: Scale intervention improves both AP and APs on COCO, TinyPerson, and VisDrone without updating frozen parameters.The especially pronounced COCO APs gain supports eliciting scale-sensitive capability from frozen detectors.
  • Trajectory Reliability: Small-object ground-truth support generally rises with trajectory evidence, with candidates grouped into five equal-frequency evidence bins for post hoc analysis.Ground-truth annotations are not used during CQTR inference or routing.
  • Unlabeled Routing: Unlabeled stream statistics select the more effective correction branch without choosing post hoc from test AP.The routing loop connects internal evidence, branch selection, and performance improvement.
  • Trajectory Reliability: High-evidence queries converge more stably around small objects than low-evidence queries across the final four decoder layers.Semantic-density maps and representative trajectories provide internal support for trajectory-based reliability estimation.

V. CONCLUSION

CQTR is a training-free framework for improving small-object detection with frozen detectors. It combines counterfactual scale intervention, query-trajectory interpretation, and unlabeled reliability routing to activate latent scale knowledge and suppress cross-scale conflicts.

  • Conclusion: CQTR combines a counterfactual scale branch, spatial-semantic query-trajectory interpretation, and unlabeled reliability routing for frozen detectors.The framework is presented as training-free and intended to improve small-object detection.
  • Conclusion: Experiments across multiple detectors and three datasets demonstrate CQTR's effectiveness, generality, and interpretability.The conclusion frames these properties as the paper's main empirical support.

APPENDIX

The appendix reports complete quantitative results across 27 detector-dataset combinations. CQTR improves the evaluated metrics across COCO, TinyPerson, and VisDrone2019 under frozen-model protocols without target-domain training annotations.

  • Quantitative Results: All 27 detector-dataset results report AP, AP50, AP75, and APs, with parenthesized values denoting absolute improvements over frozen baselines.All results use frozen parameters and no target-domain training annotations.
  • COCO: 54.32 is the highest COCO AP, achieved by RT-DETRv2-R50 with CQTR.RT-DETR-R50 has the largest COCO APs increase, from 34.74 to 38.55.
  • TinyPerson: RT-DETR-R50's TinyPerson AP increases from 9.19 to 13.04, while APs increases from 5.89 to 9.60.CQTR improves AP, AP50, AP75, and APs across all nine detectors on TinyPerson.
  • VisDrone2019: DEIM-D-FINE-S has the largest VisDrone AP gain, increasing from 14.50 to 17.16, while APs increases from 6.22 to 8.61.RT-DETRv2-R50 achieves the highest absolute VisDrone AP of 17.49.
  • Cross-dataset pattern: CQTR's small-object detection improvement is more pronounced than its overall AP gain on COCO.The appendix also reports effectiveness in dense aerial scenes with severe scale variation and background interference.

B. Visualization of Decoder Query Trajectories

The visualization projects decoder query states into semantic-density maps and spatial trajectories for small-object candidates. High-evidence queries remain concentrated around small objects and converge more stably across later decoder layers.

  • Decoder Visualization: The visualization projects query box centers and final-class responses from the final four decoder layers onto the original image.These projections connect spatial locations with semantic responses during iterative decoding.
  • Semantic Query Density: Semantic query density uses queries whose final predicted area does not exceed the small-object threshold, with a shared normalization range across layers.The shared range allows colors from different decoder layers to be compared directly.
  • Semantic Persistence: Small-object candidate responses maintain high object-to-background ratios across the four layers, with values of 24.58, 26.44, 26.14, and 26.06.The image contains 80 small-object candidate queries.
  • Trajectory Comparison: High-evidence queries remain spatially convergent around small objects, whereas low-evidence queries are less stable.Figure 9 uses semantic-density maps and representative trajectories from the final four decoder layers.
  • Visualization Materials: The appendix materials include visualization panels for decoder semantic density and detector-versus-CQTR comparisons across detector families.The listed panels cover COCO-small boxes, decoder layers, and multiple frozen detector pairs.
Loading 2609.06581v1…