PaperlayerView exact revision on arXiv

Source-linked AI summary

Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set

Ruslan Rozumnyi, Matěj Suchánek, Tomáš Vojíř, Klára Janoušková, Jiří Matas

arXiv:2609.01027v1cs.CV

TL;DR

OOD benchmarks often source external images ad hoc, while ID datasets themselves contain label errors that can yield difficult OOD examples. The paper constructs FI-IMAGENET-1K from ReImageNet-rejected ImageNet-1k validation images, verifies candidates through human and multimodal evidence, and retains only confidently labeled external classes. The resulting 655-image benchmark from 522 classes is substantially harder than existing benchmarks, with evaluated classifier-detector combinations remaining above 51% FPR@95.

  • Problem

    OOD benchmarks commonly source images outside the ID data ad hoc, despite label errors within ID datasets offering a natural source of OOD images.

  • Method

    The paper builds FI-IMAGENET-1K from ReImageNet-unassigned validation images and admits only candidates supported by human judgment, MLLM descriptions, reverse image search, and visual comparison with ID classes.

  • Results

    FI-IMAGENET-1K contains 655 OOD images from 522 classes and remains above 51% FPR@95 for all evaluated classifier-detector combinations.

  • Takeaways & Limitations

    FI-IMAGENET-1K provides a challenging benchmark and a general recipe for constructing OOD datasets from verified label errors inside established ID datasets.

  • Takeaways & Limitations

    The oracle uses the correct OOD-label vocabulary and serves as an empirical lower bound rather than a deployable detector.

Abstract

from arXiv · show

Out-of-distribution (OOD) detection predicts whether a test image belongs to none of the predefined classes. To evaluate this task, benchmarks need images from outside the in-distribution (ID) data; typically, these are defined or collected in an ad hoc fashion. Since no ground truth is perfect, ID-labeled datasets themselves contain a natural source of OOD images. We exploit such annotation errors and present Fi-ImageNet-1k, an OOD dataset built from ImageNet-1k validation images that the recent ReImageNet reannotation effort assigned to no ImageNet-1k class. Each image was examined by expert human annotators supported by evidence from MLLMs, VLMs, and reverse image search, comparing it against all visually similar ID classes. We keep only images that could be assigned a specific class outside the ImageNet-1k label space. The resulting Fi-ImageNet-1k, with 655 images from 522 classes, is substantially more challenging than any commonly used OOD dataset. No evaluated combination of classifier and OOD detector achieves a false positive rate below 51% at 95% true positive rate (FPR@95). Compared to the recent NINCO, our dataset is 3.8x more challenging in the FPR@95 metric for state-of-the-art supervised OOD detection methods.

1. Introduction

OOD detection identifies test images outside predefined classes, yet many classification models and VLMs lack mechanisms to recognize such inputs. FI-IMAGENET-1K addresses this gap by converting verified ImageNet-1k label errors into a challenging benchmark.

  • OOD detection identifies images belonging to none of a predefined set of training classes.
  • OOD detection supports safe deployment because real-world inputs cannot be guaranteed to come from a fixed class set.
  • State-of-the-art classifiers and zero-shot VLMs are commonly deployed without mechanisms for recognizing inputs outside their label spaces.
  • Existing ImageNet-based OOD benchmarks typically define labels separately and source OOD images from independent pools.
  • FI-IMAGENET-1K repurposes ImageNet-1k validation label errors, retaining images with verified fine-grained classes outside the 1000-class space.
  • The benchmark contains 655 OOD images from 522 classes, with verified fine-grained labels supporting open-vocabulary research.

2. Related Work

OOD research commonly treats ImageNet-1k labels as a clean reference, while separate analyses show that those labels contain errors. FI-IMAGENET-1K connects these concerns by turning verified internal label errors into near-boundary OOD samples.

  • FI-IMAGENET-1K links OOD detection with ImageNet-1k label-quality research by treating verified label errors as near-boundary OOD samples.
  • Post-hoc OOD detectors score pretrained classifiers without changing training and use logits, activations, weights, features, or hybrid signals.
  • EOE revises the vocabulary by generating plausible outlier categories with an LLM and adding a penalty term to NegLabel.
  • Existing ImageNet-scale benchmarks pair ImageNet-1k validation data with OOD images sourced elsewhere, while NINCO and ImageNet-OOD inspect images for ID contamination.

3. Benchmark Construction

FI-IMAGENET-1K is constructed from ReImageNet-rejected ImageNet-1k validation images, but only candidates with strong evidence for a specific external class are admitted. A three-stage consensus protocol separates OOD, ID, and ambiguous cases.

  • 3. Benchmark Construction: ReImageNet-unassigned images are necessary but insufficient for OOD status, so candidates require high confidence in an external class.
  • 3. Benchmark Construction: Two annotators independently label candidates; agreements are accepted, disagreements proceed to a third annotator, and unresolved cases are marked ambiguous.
  • 3. Benchmark Construction: Annotators assess each candidate using MLLM descriptions, reverse image search, original labels, visually closest ID classes, and empirically confusable classes.
  • 3. Benchmark Construction: The protocol admitted 655 images as OOD, covering 522 concepts, while also returning agreed ID images to the ID data and excluding ambiguous images.
  • 3. Benchmark Construction: Among initially ‘no ImageNet-1k class’ images, 43.2% were certified OOD, whereas only 21.2% of ‘uncertain’ images were certified OOD.

4. Experiments

FI-IMAGENET-1K is evaluated across supervised classifiers, zero-shot VLMs, post-hoc detectors, negative-label vocabularies, and existing OOD benchmarks. The experiments show that the benchmark remains difficult, with negative-vocabulary coverage emerging as a key factor for VLM performance.

  • Evaluation setup: Eight supervised backbones paired with 21 post-hoc detectors are evaluated alongside eight zero-shot VLMs and their OOD detection methods.The evaluation also includes vocabulary-based detectors for VLMs and comparisons with existing ImageNet-scale OOD datasets.
  • Post-hoc supervised OOD detection: 64.4% FPR@95 is the best supervised result, achieved by DINOv3-7B with relative Mahalanobis++; ResNet50 reaches 80.2%.Neither a stronger backbone nor a better post-hoc method sufficiently addresses the benchmark’s challenge.
  • Zero-shot VLMs as OOD detectors: 87.0% to 71.8% is the range of best FPR@95 values for the seven logit-based VLM detectors across models.GL-MCM is best for four of eight VLMs, suggesting patch-level similarities provide OOD information missed by pooled image embeddings.
  • Zero-shot with negative-label vocabularies: 58.6% is the mean FPR@95 for GPT-mined negative labels, improving over WordNet’s 74.6% and outperforming WordNet and EOE for every tested VLM.GPT-mined labels reduce FPR@95 by 6.8 to 22.2 points depending on the VLM.
  • Negative-vocabulary analysis: 58.8% versus 58.6% is the mean FPR@95 for WN+GPT versus GPT alone, and the union helps only three of eight VLMs.WordNet contributes little once fine-grained, per-class GPT negatives are available.
  • Negative-vocabulary analysis: 13.8% is the oracle mean FPR@95, compared with 58.6% for the best deployable vocabulary, leaving a 45-point gap attributed to vocabulary coverage.The oracle uses the correct negative labels, indicating that automatic negative-prompt construction remains unresolved.
  • Comparison to existing OOD benchmarks: 65.0% is FI-IMAGENET-1K’s best FPR@95 with EVA02-L-448, versus 16.9% for NINCO, making it 3.8× harder by this metric.The benchmark also exceeds the hardest full dataset by 16.8 points; sampling analyses show the difference is not explained by dataset size.

5. Conclusions

FI-IMAGENET-1K is an OOD benchmark built from ImageNet-1k validation images rejected by ReImageNet and verified through complementary evidence. Its 655 images from 522 classes are substantially harder than existing benchmarks, while negative-label vocabulary remains a key bottleneck.

  • FI-IMAGENET-1K contains 655 OOD images from 522 classes, each with a verified label.
  • Its construction recipe gathers unassigned or mislabeled ID images, combines complementary content evidence, and compares them against plausible ID classes.
  • 51% is the lowest FPR@95 achieved by the evaluated combinations of eight backbones with 21 post-hoc detectors or eight zero-shot VLMs.
  • NegLabel with SigLIP2-g reaches 6.0% FPR@95 using benchmark labels as negatives, indicating that representation suffices while negative vocabulary is the bottleneck.

Supplementary Material

The supplementary material provides expanded evaluations, analyses of negative-vocabulary design and OOD sample size, additional examples, and implementation documentation.

  • The supplement keeps the main paper’s ID set, OOD set, and ID-anchored FPR@95 threshold while organizing analyses across Sections A–G.
  • Its sections cover complete evaluation results, negative-vocabulary design, OOD sample size, dataset examples, supervised-model extensions, checkpoints, and prompts.

A. Full Evaluation Results

The full evaluations show that FI-IMAGENET-1K remains difficult across supervised model–detector pairs and supervised models, with cleaning substantially improving performance on prior datasets.

  • 64.4% is the lowest observed FPR@95 across all 21 post-hoc detectors and supervised classifier configurations.
  • Mahalanobis-family scores are best or within 2.3 points for every backbone, while activation- and weight-reshaping scores form the weakest group.
  • FI-IMAGENET-1K is the most difficult evaluated dataset for every supervised model, with margins ranging from 7.1 to 18.2 points.
  • EVA02-L-448 achieves mean FPR@95 of 21.2% on other datasets but 65.0% on FI-IMAGENET-1K.
  • Cleaning prior OOD datasets lowers average FPR@95 from 22.6% to 12.2%, with dataset-specific reductions ranging from 0.7 to 25.1 points.
  • Table A1 reports FPR@95 and AUROC for every supervised classifier–detector pairing, grouped into logit-based, activation- and weight-based, feature-based, and hybrid detectors.

B. Ablation Study: Negative Vocabulary

Negative-label vocabulary strongly affects OOD detection, and the best vocabulary depends on how closely the OOD data resemble ImageNet-1k. On FI-IMAGENET-1K, per-class negative siblings perform best, while broader vocabularies perform better across most other datasets.

  • Vocabulary construction: FPR@95 decreases monotonically as GPT-generated negative siblings increase for every VLM.Each generation requests 20 siblings per ImageNet-1k class; merging ten generations produces 26–138 unique candidates per class.
  • Vocabulary construction: The default WordNet pool size M=10,000 exceeds every model’s optimum; Table 2 therefore uses M=5000.The figure reports FPR@95 as a function of pool size, with farthest labels selected first.
  • Vocabulary construction: More merged negatives continue helping popular OOD datasets but not FI-IMAGENET-1K.The merged pool averages 73.6 candidates per ID class when all candidates are retained.
  • Cross-dataset effects: WordNet lowers FPR@95 from 50.4% to 40.7% on NINCO and from 55.5% to 53.6% on ImageNet-OOD, but helps mainly on far-OOD data.The passage contrasts these gains with near-OOD FI-IMAGENET-1K, where WordNet is less effective.
  • Cross-dataset effects: The per-class sibling vocabulary wins only on FI-IMAGENET-1K, whose OOD images lie closest to the ID classes.The union vocabulary is best across the thirteen-dataset mean, at 46.8% versus 47.2% for WordNet and 58.3% for siblings.
  • Cross-dataset effects: Oracle FPR@95 bounds of 4.9–37.4% indicate that vocabulary selection, rather than the scoring function, limits detection.The oracle is an empirical lower bound available for datasets with labels for every image.

C. Ablation Study: Size of the OOD Dataset

Subsampling OOD datasets to 655 images preserves the expected FPR@95 while increasing sampling uncertainty at smaller sample sizes. Detector rankings are usually preserved, but closely spaced top detectors cannot be ranked definitively, and dataset choice matters more than sampling variation.

  • FPR@95 estimation: At n=655, mean FPR@95 differs from the full-dataset rate by at most 0.13 points across 112 backbone–dataset combinations.The experiment uses 1000 random draws without replacement, with each detector selected on the full dataset and then fixed.
  • FPR@95 estimation: The OOD sample size changes FPR@95 precision but not its expected value when the model, detector, ID set, and threshold are fixed.The threshold is the 5th percentile of 48,365 ID scores and is independent of the OOD set.
  • FPR@95 estimation: Sampling uncertainty narrows approximately as 1/√n while the mean remains centered on the full-dataset rate.The observed standard deviation agrees with the finite-population-corrected analytical value within 0.10 points.
  • Uncertainty interpretation: The 1.86-point FI-IMAGENET-1K standard deviation is a plug-in estimate under independent sampling, not fixed-dataset subsampling variability.No larger finite pool of FI-IMAGENET-1K images is available for direct subsampling.
  • Ranking stability: A 655-image draw largely preserves detector rankings, with mean Spearman correlations from 0.95 to 1.00, but may not distinguish the top entries reliably.For EVA02-L-448, correlations range from 0.952 to 0.985.
  • Ranking stability: Dataset choice has a substantially larger effect than 655-image sampling variation: between-dataset spread is 12.7–16.4 points versus 1.39–1.80 points of mean sampling standard deviation.The FI-IMAGENET-1K gap from the hardest external dataset is 16.8 points, approximately nine times its 1.86-point plug-in standard deviation.

D. Additional Dataset Examples

Additional examples show that FI-IMAGENET-1K contains both fine-grained semantic confusions and broader mismatches between depicted concepts and ImageNet-1k labels. Because all images come from the same collection pipeline, the benchmark emphasizes semantic mismatch over systematic capture differences.

  • Example composition: Animals constitute 58.0% of the complete dataset, and half of the displayed additional examples are animals.The dataset proportion is based on the intersection of original ImageNet-1k labels with the respective WordNet subtree.
  • Fine-grained failure modes: The additional examples include fine-grained confusions such as tayra versus weasel and black-footed cat versus lynx.Other examples include concert tuba versus French horn and diesel locomotive versus electric locomotive.
  • Broad semantic mismatches: Other images depict concepts with no close ImageNet-1k class, while their original labels reflect scene, material, shape, or contextual associations.These examples illustrate a broader semantic mismatch rather than a near-neighbor class confusion.
  • Dataset design: Shared ImageNet-1k origin minimizes systematic differences in style, resolution, and capture conditions between ID and OOD data.The authors note that this shared origin does not eliminate all covariate differences.

E. Additional Supervised Models

The expanded supervised evaluation shows that FI-IMAGENET-1K’s high FPR@95 is not confined to the representative architectures used in the main paper. Pre-training regime affects performance, with ImageNet-21k and related regimes outperforming ImageNet-1k-only training within the evaluated families.

  • Expanded classifier evaluation: 64.4% is the lowest FPR@95 among 30 evaluated classifiers and 21 post-hoc detectors, achieved by a DINOv3-7B linear probe with relativ…The classifiers span 5.3M–6.7B parameters, eight architecture families, and five pre-training regimes.
  • Expanded classifier evaluation: None of the 22 additional classifiers improves on the 64.4% minimum.The expanded evaluation therefore tests whether the main-paper result depends on its representative architecture selection.
  • Pre-training effects: Mean best FPR@95 decreases from 79.0% with ImageNet-1k-only training to 73.7% with ImageNet-21k pre-training or the evaluated CLIP regime.JFT noisy-student pre-training gives 75.4% in the same comparison.
  • Pre-training effects: All seven matched model pairs favor ImageNet-21k over ImageNet-1k pre-training, with an average reduction of 4.7 percentage points.This comparison is within the evaluated model families.

F. Implementation Details

The evaluation uses supervised and zero-shot checkpoints, standardized prompt ensembles, and several procedures for generating or validating labels outside ImageNet-1k. Additional details specify detector coverage, hard-negative mining, annotation prompts, and example composition.

  • Zero-shot scoring: Zero-shot scoring averages ℓ2-normalized embeddings from eight label templates before comparing image and text embeddings by cosine similarity.The templates include the bare class name and seven OpenAI ImageNet templates.
  • Checkpoints: Table A8 enumerates 30 supervised classifiers and eight zero-shot VLMs, including frozen-backbone linear probes and their checkpoint or pre-training details.DINOv2-g/14 and DINOv3-7B use single linear layers trained with cross-entropy on ImageNet-1k.
  • Hard-negative mining: NegLabel mines 20 visually confusable non-ImageNet categories for each ID class, repeating the process ten times over the 1000 distinct class names.The rules target a just-outside-the-class-boundary vocabulary using concrete nouns and same-granularity sibling species or visually adjacent objects.
  • Annotation and filtering: Annotation prompts require GPT-5.4 to choose an ImageNet-1k class or provide a fine-grained open-world alternative when the dominant subject is not in the allowed list.Labels matching an ImageNet-1k class name or WordNet synonym are removed, leaving 19,597–19,632 labels per generation.
Loading 2609.01027v1…