PaperlayerView exact revision on arXiv

Source-linked AI summary

Losing the name before the box: measuring and repairing what narrow fine-tuning costs a detector outside its deployment vocabulary

Trung Minh Bui, Jongsul Moon, YoungOuk Kim, Jung-Hoon Hwang, Dongin Shin

arXiv:2609.36426v1cs.ROcs.CVcs.LG

TL;DR

In-domain tests do not contain objects omitted from a deployment vocabulary, leaving their coverage unmeasured as pretrained detectors are fine-tuned. The paper follows a pretrained checkpoint and its fine-tuned descendants using held-out top-K proposal coverage. Coverage falls across architectures and domains; a training-free interpolation repair recovers coverage within a stated accuracy cost.

  • Problem

    In-domain evaluation contains no objects whose categories the deployment vocabulary omits, so it cannot report how fine-tuning affects coverage of those objects.

    Source: §1 · p. 2

  • Method

    The paper measures held-out top-K proposal coverage Cτ longitudinally, comparing a pretrained checkpoint with its own fine-tuned descendants.

    Source: §1 · p. 2

  • Results

    87% of detection AP is lost against a fifth of coverage on the architecture scored both ways, and the naming-first ordering holds at all six freeze depths in every run.

    Source: §9 · p. 22

  • Takeaways & Limitations

    Interpolating a quarter of the pretrained state back, including normalization buffers, restores a large share of coverage for under one point of in-domain accuracy.

    Source: §9 · p. 23

    The area filter restricts the measured boxes; removing it increases the IoU 0.5 loss in all four measured seed-0 cells by 16 to 113%.

    Source: §8 · p. 20

Abstract

from arXiv · show

A detector pretrained on a broad corpus is fine-tuned on a narrow domain, its in-domain accuracy improves, and it ships. We ask what happens meanwhile to its coverage of objects the vocabulary never names, which in obstacle detection and inspection carry the risk. No in-domain test set holds an example of one. We give a longitudinal protocol: one pretrained checkpoint against its own fine-tuned descendants. It tracks held-out top-$K$ proposal coverage $C_\tau$: of categories pretraining covered and the vocabulary omits, the share of boxes a detector's top $K$ regions still cover. The quantity is the open-world proposal literature's; the longitudinal reading is not. $C_\tau$ falls while in-domain accuracy rises, on four architectures and three domains, by $5.12$ to $63.35$ points on boxes above $1024$ px$^2$. No in-domain number identifies the fall, and neither does detection average precision, which charges a missed and a misnamed box alike. On the one architecture scoring both, adaptation costs $87\%$ of the AP against a fifth of the coverage, and the naming goes first at all six depths of its freeze ladder, every run. What breaks is structured: three architectures sharing no pretraining run agree on which categories lose coverage, and those a model never learned do not lose any. A repair follows and needs no training: mixing a quarter of the pretrained state back, normalisation statistics included, raises coverage on every cell swept for at most $2.47$ points of in-domain accuracy. Seeing it costs one extra evaluation pass.

1 Introduction

The paper tracks whether fine-tuning a narrow deployment vocabulary damages a pretrained detector’s coverage of objects that vocabulary omits, a loss its in-domain test cannot reveal. It introduces a longitudinal coverage measurement and reports where the loss occurs, how it relates to naming and proposal selection, and a training-free repair.

1 Introduction

  • The longitudinal protocol compares one pretrained checkpoint with its own fine-tuned descendants, extending an established open-world proposal measurement to track retention.The paper also evaluates the interpolation path between checkpoints.Source: §1 · p. 2, Contributions · p. 3
  • Cτ measures a consequence of class-agnostic localization, not the ability itself; reordering proposals or crossing the IoU threshold can lower it.The paper measures coverage at one budget and two thresholds.Source: §1 · p. 2

Contributions.

  • 87% of detection AP is lost against a fifth of coverage on the architecture scored both ways, with naming going first at all six freeze depths in every run.The ordering holds by 3.49 points at the IoU 0.5 knee.Source: Contributions · p. 3

1 Introduction

  • Cτ falls while in-domain accuracy rises across four architectures and three deployment domains, but in-domain metrics cannot identify that fall.Cτ is the share of held-out ground-truth boxes covered by at least one of the detector’s top-K regions at IoU threshold τ.Source: §1 · p. 2, Contributions · p. 3

Contributions.

  • Three architectures with separate pretraining runs agree on which categories lose coverage; categories the model never learned instead gain coverage.Across the learned categories, coverage falls by 2.75 to 9.31 points.Source: Contributions · p. 3
  • At IoU 0.5, the loss is mostly from proposal reordering; at IoU 0.7, about half persists as coordinate change, with the dominant source varying by architecture.The measured account that adaptation pushes unmatched regions to background explains only a minority of the drop.Source: Contributions · p. 3
  • Interpolating 25% of the pretrained state back, including normalization statistics, restores coverage across every swept architecture-domain cell for at most 2.47 points of in-domain accuracy.The repair needs no training run.Source: Contributions · pp. 3–4

2 Related work

This work inherits open-world proposal recall but changes the question from how a training design performs to what fine-tuning does to one pretrained checkpoint. It distinguishes coverage from naming-based detection measures, considers competing accounts of the loss and repair, and positions its longitudinal comparison alongside related protocols.

2.1 The measurement is inherited; the longitudinal use is not

  • Proposal coverage and detection AP answer different questions: AP combines missed boxes with boxes localized but misnamed, while coverage counts proposed regions.Prior error-decomposition methods sort detection errors, whereas this paper evaluates regions proposed by the model.Source: §2.1 · p. 5, §2.4 · p. 7
  • The open-world proposal literature measures recall on categories withheld from training; this paper instead follows a pretrained checkpoint through fine-tuning on narrow-domain images.Its categories are omitted from the deployment vocabulary but were covered in pretraining, making the comparison longitudinal within one checkpoint.Source: §2.1 · pp. 5–6

2.2 Mechanisms proposed for the drop, and what each protocol can see

  • Prior accounts include suppression of unmatched regions and forgetting concentrated in components; neighboring incremental-detection protocols change labels while holding the image distribution fixed.This paper tests which accounts fit its adaptation setting.Source: §2.2 · p. 6, §2.4 · p. 7

2.3 What to move, and what to leave alone

  • Prior repair work studies where to tune and weight interpolation; normalization-statistics methods move statistics toward deployment data and judge accuracy there.This paper compares those repair families against held-out coverage, including the cost of changing normalization statistics.Source: §2.3 · p. 6

2.4 What this paper adds

  • The contribution is a longitudinal reading of class-agnostic pre-NMS coverage on pretrained categories, alongside evidence that no in-domain functional identifies it under an unrestricted hypothesis class.The paper also relates its findings to other protocols and reports that category outcomes depend on prior coverage rather than category name.Source: §2.4 · p. 7

3 Measurement protocol

The protocol reads held-out top-K proposal coverage against the same checkpoint before and after fine-tuning, then tests what in-domain evaluation can identify about that coverage. It specifies the evaluation setup and separates confirmatory tests from exploratory analyses.

3.1 Held-out top-K proposal coverage

  • Coverage is recall without classification: it asks whether a detector’s top-K proposals cover an object, not whether the detector can name it.The table contrasts this with measures over emitted detections and distinguishes categories never learned from those known before fine-tuning but omitted by the deployment vocabulary.Source: §3.1 · p. 8, Table 3 · p. 8

3.2 In-domain evaluation is not sufficient to identify held-out coverage

  • At the same 18.16 mAP on cardd-22, two YOLOX adaptations cover 61.13 and 72.50 points of the held-out group at IoU 0.7, a difference of 11.37 points.Across five seeds of the plain fine-tune, coverage spans 2.25 points, and the highest-accuracy seed has the lowest coverage.Source: §3.2 · p. 8
  • Across 110 YOLOX and RF-DETR runs, in-domain mAP is not a reliable selector of held-out coverage; three of four cells are below chance at both thresholds.YOLOX’s mAP–coverage rank correlations are −0.918 and −0.840, while RF-DETR’s are −0.163 on tabletop-22 and +0.015 on cardd-22.Source: §3.2 · pp. 8–9

3.3 Setup

  • Coverage is measured on COCO val2017 boxes of at least 1024 px^2, with top K = 300, across four architectures and three deployment domains.The setup uses 21,861 boxes outside the deployment vocabularies and 2,225 inside them as a control; RF-DETR contributes its whole query set.Source: §3.3 · p. 9

What was run, and where the standing objections are answered.

  • A rung that trains parameters outside the coordinates read by the ranking score and geometry cannot change coverage, so such cells do not test whether adaptation is free.The harness caught a nominally frozen ladder that was still adapting every BatchNorm buffer in the backbone.Source: What was run, and where the standing… · p. 9
  • The validity control required held-out deployment-split mAP to exceed the pretrained model’s, but the authors also report an all-attempts estimand retaining sub-10-mAP runs.Every full-rung cell passed both; the prespecified criterion failed one ignore-ablation cell and all of voc-20.Source: What was run, and where the standing… · pp. 9–10

3.4 Which tests are confirmatory, and which are not

  • Of seven prespecified hypotheses, three failed: depth ordering inverted on RF-DETR, the displacement constant varied with fitting window, and L2-SP beat interpolation on Faster R-CNN.Other analyses, including α = 0.25, were exploratory rather than confirmatory.Source: §3.4 · p. 10

3.5 Use of large language models

  • The authors used a language model to draft text and write most analysis and auditing code, but the two number-checking scripts share an author with the manuscript.Those scripts check agreement with per-run files, not independent validation; covered tables were also re-derived by hand at least once.Source: §3.5 · p. 10

4 The coverage drop across architectures and domains

Across four architectures, full fine-tuning reduces held-out coverage of novel objects; on OWLv2, detection AP falls much more sharply than coverage. The section also examines how the drop varies across images, object sizes, image distributions, and metrics.

The pretrained rows.

  • Before adaptation, all four architectures score highly on the novel group despite differences in region counts and objectness scores.The protocol was not tuned per architecture, and architecture ordering does not follow whether a model has a purpose-built objectness score.Source: The pretrained rows · p. 10, Table 4 · p. 11

4 The coverage drop across architectures and domains

  • All four architectures lose novel-group coverage after full fine-tuning; the drops grow at IoU 0.7, reaching 52.24 points for OWLv2.Table 4 reports the study’s central measurement on the same 21,861 novel COCO boxes and uses each detector’s top 300 regions.Source: The pretrained rows · p. 10, §4 · p. 10, Table 4 · p. 11

4.1 The name goes before the box

  • On 21,861 held-out boxes, OWLv2’s detection AP falls from 55.64 to 7.17 while its IoU 0.5 coverage falls from 97.96 to 78.00.Across seeds, AP falls by 84.6–89.9% against coverage losses of 9.5–33.9%; the other three architectures cannot be scored for AP on the omitted COCO categories because their classification heads are replaced.Source: §4.1 · p. 11

4.2 The image partition, which the seed bars do not carry

  • Across four further image partitions, partition-level standard deviations are 0.11 at IoU 0.5 and 0.09 at 0.7, below seed standard deviations of 0.95 and 1.27.At this replication, an F-test cannot distinguish the partition effect from zero.Source: §4.2 · p. 11

4.3 What the drop looks like on an image

  • The image examples were selected using a preregistered rule requiring novel-category boxes to cover at least 4% of the image and meet specified pretrained and fine-tuned IoU thresholds.The candidate rate differs by architecture: the same 2,114 eligible boxes yield 19 Faster R-CNN candidates and 586 OWLv2 candidates.Source: §4.3 · p. 12

4.4 Which objects the headline is about

  • Smaller objects lose more coverage in four of seven architecture-domain cells, while image-weighted YOLOX losses are lower than box-weighted losses on both domains.The authors discuss an untested candidate-budget hypothesis: scoring YOLOX over all candidates reduces its measured losses, but neither instance varies K/N with architecture held fixed.Source: §4.4 · p. 12

4.5 Reproduction on a second image distribution

  • The coverage loss also appears on Pascal VOC and OpenImages: all 60 seed-level drops on VOC and all 36 on OpenImages are positive.On OpenImages, Faster R-CNN loses more on cardd-22 than on COCO at IoU 0.7: 40.17 against 30.08.Source: §4.5 · p. 13

4.6 Reproduction under the metric the open-world literature uses

  • Standard AR@K also shows losses in all four tested cells, with each loss between that cell’s Cτ drops at IoU 0.5 and 0.7.These are single-checkpoint measurements requiring a separate forward pass, so they are not comparable row by row with Table 4’s seed means.Source: §4.6 · p. 13

5 What does not explain the drop, and the one split that does

The coverage decline is primarily a selection loss at IoU 0.5, while stricter-threshold losses include substantial coordinate effects; the damage concentrates on categories the checkpoint had learned and is shared across architectures. Several proposed explanations account for only part of the drop, and most of it remains after suppressing unmatched-region background gradients.

5 What does not explain the drop, and the one split that does

  • At IoU 0.5, selection explains most of the coverage drop; at IoU 0.7, placement accounts for about half, with the component split varying by architecture.A budget sweep on Faster R-CNN and scoring every candidate on YOLOX independently support selection as the dominant source at IoU 0.5.Source: §5 · p. 13, §5.1 · p. 14, §5.2 · p. 14
  • The tested alternatives do not replace the category split: displacement is positively associated with overlap loss, but its coefficient varies with the fitting window, and the majority of the strict-threshold drop remains unaccounted for.An indicator for categories named by the deployment vocabulary adds nothing after conditioning on prior coverage.Source: §5 · p. 13, §5.3 · p. 15

5.1 Scoring every candidate: what survives when nothing is ranked away

  • Removing the area floor raises the IoU 0.5 loss by 16 to 113% across four measured cells, while its effect on the IoU 0.7 loss is under a point in the two reported cells.At IoU 0.5, the tabletop-22 loss changes from 10.40 to 13.53 and the cardd-22 loss from 17.13 to 19.88.Source: §5.1 · p. 14

5.2 Holding the candidate set fixed: how much is selection, how much is geometry

  • At IoU 0.5, selecting the fine-tuned model’s top 300 anchors accounts for 60.2% of YOLOX’s drop and 81.5% and 74.8% for Faster R-CNN, averaged over five seeds.The decomposition is factorial, not orthogonal; Faster R-CNN selection includes coordinate-dependent NMS, and its interaction is positive at IoU 0.5 but negative at IoU 0.7.Source: §5.2 · pp. 14–15

5.3 The loss falls on what the checkpoint already had

  • Across 752 LVIS categories, coverage rises by 2.51 to 4.51 points for the 691 categories never learned, while the 61 learned categories lose 2.75 to 9.31 points.The split is by pretraining exposure alone, and all six cells’ intervals clear zero with every seed sharing its cell’s sign.Source: §5.3 · p. 15, Figure 2 · p. 15

5.4 Architectures agree on which categories break

  • Three architectures show positive partial correlations in all twelve category-change comparisons, and most of the sharing persists across deployment domains for two architecture pairs.Partial r ranges from 0.180 to 0.581; most category-level variance remains unexplained.Source: §5.4 · p. 16, Table 5 · p. 16

5.5 Testing the background-suppression account

  • Suppressing unmatched-region background gradients recovers 4.6 to 15.7% of the IoU 0.5 drop without costing in-domain accuracy, while 84 to 95% remains.Extending the intervention to Faster R-CNN’s ROI head raises recovery to 19.5% and 23.0% on cardd-22 and tabletop-22, respectively, but three quarters of the drop still survives.Source: §5.5 · pp. 16–17

6 Freeze depth is a control, not a recipe

Freeze depth does not provide a general rule for preserving coverage: its relationship to damage differs across architectures and domains. The paper recommends measuring each candidate stage and choosing from the resulting trade-off rather than relying on a universal ladder.

6.1 The ordering is architecture-dependent

  • Three of seven freeze ladders have their worst coverage loss before the deepest rung, and the four architectures do not share a depth ordering.RF-DETR’s deepest rung is relatively inexpensive on both domains, while Faster R-CNN’s pattern is monotone on tabletop-22 and inverted on cardd-22.Source: §6.1 · p. 17, §6 · p. 17, Table 6 · p. 18
  • Faster R-CNN’s two shallowest rungs are not measurable because they train only parameters its objectness score does not read; the first measurable rung costs 6.94±0.11 points.The measurement therefore cannot establish whether a genuinely cost-free rung exists.Source: §6.1 · p. 17
  • The all-attempts estimand includes every run and answers the practitioner’s choice-before-outcome question; the admissible analysis excludes five runs below 10 mAP and conditions on successful adaptation.The authors lead with all-attempts because the admissible view conditions on a post-treatment variable and answers a different question.Source: §6.1 · pp. 17–18, §6.2 · p. 18
  • No generic freeze ladder survives the seven measured ladders; adding in-domain mAP to a regression changes the depth relationship, so the reported depth comparisons hold fit quality fixed.A practitioner choosing a rung does not hold fit quality fixed.Source: §6.1 · pp. 17–18

6.2 A measurement procedure in place of a freeze-depth rule

  • The authors recommend evaluating stages for in-domain mAP and held-out coverage, excluding stages that cannot affect the score or failed to fit, and selecting from the survivors’ Pareto front.The proposed procedure leaves the trade-off between coverage and in-domain accuracy to the deployment decision-maker.Source: §6.2 · p. 18

7 A repair that costs no training run

The paper evaluates state interpolation, which mixes pretrained and fine-tuned floating-point tensors without training, as a repair for lost coverage. A quarter-weight mixture transfers across the swept cells, but the repair’s cost depends on the accuracy budget and the interpolation variant matters.

7 A repair that costs no training run

  • Partial freezing is not a reliable substitute: on cardd-22, freezing the backbone and adapting the RPN costs 36.98 ± 2.10 coverage points versus 30.08 ± 0.99 for full fine-tuning.The comparison is based on five runs per approach.Source: §7 · p. 18
  • State interpolation mixes every floating-point tensor with a same-shape pretrained counterpart, including normalization statistics; class-predictor tensors without a matching shape remain from the fine-tuned model.This differs from parameter interpolation, which mixes trainable tensors alone, and buffer restoration, which replaces the running statistics outright.Source: §7 · p. 19
  • Normalization buffers account for 34.9 to 35.7% of recovery on cardd-22 and 53.6 to 55.9% on tabletop-22 when compared with parameter-only interpolation.Re-estimating statistics on deployment data recovers 27.8 to 34.6% of the buffer gap on cardd-22 but is negative on tabletop-22, while held-out coverage falls in both domains.Source: §7 · p. 19
  • The curve’s knee is near α = 0.25, but increasing the mixture beyond it can sharply lower in-domain performance, including RF-DETR cardd-22’s fall from 30.08 to 12.01 to 0.15.At other budgets, leave-one-cell-out selection chooses a different α: the mean ranges from 0.152 at a quarter-point budget to 0.357 at five points.Source: §7 · p. 20
  • With α selected by leave-one-cell-out evaluation, coverage gain is positive in every held-out cell and within 0.25 points of the best α under the same one-mAP-point budget.The cost at α = 0.25 is at most 2.47 points across the six cells; measured-alpha selection returns exactly 0.25 in all six folds.Source: §7 · p. 20

8 Limitations

The reported effects depend on the checkpoint, domain, box selection, and measurement scope; several comparisons and mechanism claims remain conditional. The authors also limit the measurement’s use as a release gate to safety-critical perception stacks.

8 Limitations

  • Removing the area filter raises the IoU 0.5 loss by 16 to 113% in four seed-0 cells, while RF-DETR’s small loss above 1024 px^2 does not hold below 256 px^2.The filtered values are therefore conservative for the measured boxes.Source: §8 · p. 20
  • Because 21,861 boxes come from 4,877 images, image-level resampling widens one coverage interval by about half, comparable to adjacent-rung gaps.The interval changes from [81.17, 82.20] to [80.88, 82.48], small relative to the 5.12 to 63.35 point effects.Source: §8 · p. 20
  • The ordering of naming and coverage losses is established on one architecture and domain, and whether it holds when adaptation is lengthened rather than deepened remains untested.Each architecture has one pretrained checkpoint, so results are conditional on that checkpoint; YOLOX-M variation preserves direction and ordering, not a specific magnitude.Source: §8 · p. 21
  • Mechanism measurements cover three architectures and exactly two, with RF-DETR matching too coarse for individual regions and OWLv2 and Grounding DINO untested.The negative-label ablation is limited to full-rung measurements on two architectures, and Faster R-CNN’s test covers its RPN, not the ROI head’s background class.Source: §8 · p. 21
  • The conditional-mean baseline misses category-to-category variance, while the proposed shared-concentration model captures heterogeneity without explaining it; elongation accounts for under 8% of residual variance.The q_up estimate is extrapolated to X = 0, and the claim concerns category aggregates, not box partitions.Source: §8 · p. 21
  • Three domains cannot isolate distance from difficulty and other differences, and the proxy is a diagnostic outside safety-critical release decisions, with no guarantee of real-world success.The authors state that a model can pass the proxy and still fail on the road; using it costs one held-out evaluation pass before and after adaptation.Source: §8 · pp. 21–22
  • voc-20 fails the validity control because its pretrained baseline exceeds the fine-tuned score; the precondition is a strong baseline plus supervision too thin to beat it on held-out images.The authors report no voc-20 number in their claims.Source: §8 · p. 22
  • Grounding DINO’s scores remain prompt-conditional, OWLv2 is absent from two-domain comparisons, and the novel/control split is vocabulary-specific; mask-proposal behavior is untested.Taking all 900 queries removes ranking but not prompt conditioning.Source: §8 · pp. 21–22

9 Conclusion

Fine-tuning a detector on a narrow deployment domain degrades top-K coverage of objects its vocabulary omits, and in-domain evaluation does not reveal that drop. The conclusion traces distinct thresholded and category-level findings and reports a training-free repair; the paper also releases its measurements and checking scripts.

9 Conclusion

  • Fine-tuning degrades top-K coverage of objects omitted from the deployment vocabulary; on the only architecture scorable both ways, adaptation costs 87% of detection AP against a fifth of coverage.The ordering holds at all six freeze-ladder depths in every run, while checkpoints matched on in-domain mAP differ by 11.37 coverage points.Source: §9 · p. 22
  • At IoU 0.5, the drop nearly disappears when all 8,400 YOLOX anchors are scored instead of its top 300; at IoU 0.7, one-third to one-half remains.These findings are not claimed for the other three architectures.Source: §9 · pp. 22–23
  • Across tested architectures, a vocabulary indicator adds no explanatory value after prior coverage is conditioned on, while tighter thresholds cost more and the geometric coefficient of 2 is not identified.The vocabulary indicator can shift expected category change by at most a third of one standard deviation.Source: §9 · p. 23
  • Mixing a quarter of the pretrained state back, including normalisation buffers, restores a large share of coverage for under one point of in-domain mAP in four of six cells and at most 2.47 in the other two.No shorter schedule or trajectory average beat interpolation with an interval clear of zero; replay followed by interpolation beat interpolation alone when training data and a pretraining-corpus slice were available.Source: §9 · p. 23
  • Ignoring unmatched high-confidence regions during fine-tuning recovers 4.6 to 15.7% of the IoU 0.5 drop, but holding the recognition pathway open for held-out names recovers none; category losses agree across three architectures.The category agreement reaches partial r up to 0.581 after prior coverage is removed.Source: §9 · p. 23

Declarations.

  • The release includes measurements, scripts, and audit programs for checking reported numbers without a GPU, dataset, or checkpoint.The public deployment domains can be rebuilt from their sources with the provided scripts.Source: Declarations · p. 23

Related papers

Loading 2609.36426v1…