PaperlayerView exact revision on arXiv

Source-linked AI summary

NEO-BENCH: A New Multi-Source Benchmark for Generalizable Astronomical Streak Detection

Jiayou He, Jessica Yao

arXiv:2609.06774v1astro-ph.IMcs.CV

TL;DR

Existing streak detectors are commonly evaluated on single-source data, leaving cross-source generalization incompletely assessed. NEO-BENCH brings five heterogeneous sources into a unified benchmark and evaluates within-source and leave-one-source-out performance. Leave-one-source-out F1 decreased in most method–source pairs, and reliable generalization remains an open challenge.

  • Problem

    Strong performance on related but distinct training and test distributions does not necessarily transfer to unseen datasets, while existing evaluations provide limited cross-source evidence.

    Source: §I · p. 2

  • Method

    NEO-BENCH combines five public sources in a common evaluation framework for cross-dataset generalization, using a task-aware representation and within-source and leave-one-source-out evaluation.

    Source: §I · p. 2

  • Results

    Image-level F1 decreased in 14 of 20 method–source pairs, and IoU@0.50 F1 decreased in 13 of 20 pairs under leave-one-source-out evaluation.

    Source: §VI · p. 4

  • Takeaways & Limitations

    Reliable generalization of streak detection across diverse imaging sources remains an open challenge.

    Source: §VII · p. 5

    Because public datasets lack consistent orbit-confirmed NEO-versus-non-NEO labels, NEO-BENCH evaluates generic streak detection rather than confirmed NEO identification.

    Source: §I · p. 2

Abstract

from arXiv · show

Near-Earth Objects (NEOs) can appear as faint streaks in long-exposure astronomical images. Detecting these streaks across diverse observatories requires methods that remain reliable despite differences in image quality, orientation, sky background, and noise. However, existing detectors are commonly evaluated using data from only one source, providing limited evidence of cross-source generalization. We introduce NEO-Bench, a multi-source benchmark containing 8,376 images from five astronomical-image datasets. The sources include the Hubble Space Telescope, a Stellina smart telescope, the United Arab Emirates Meteor Monitoring Network, a TETRA1 telescope using a Celestron C14 with Fastar, and the Roboflow Asteroid dataset. We converted the data to a common YOLO format, audited a sample of labels, and defined within-source and leave-one-source-out evaluation protocols. We evaluated four approaches: Hough, Radon, Gaussian PSF, and YOLO26L. Leave-one-source-out F1 decreased in 14 of 20 image-level method-source pairs and 13 of 20 IoU@0.50 localization pairs. Across the datasets categorized as medium or hard, F1 decreased in 11 of 12 image-level pairs and 9 of 12 localization pairs. These results show that cross-source performance remains inconsistent and that reliable generalization across astronomical imaging sources remains an open challenge. The benchmark, code, and data are publicly available.

I. INTRODUCTION

NEO-BENCH targets the limited evidence for cross-source streak detection by combining five sources in a benchmark designed to measure transfer across instruments and observing conditions. Its labels represent source-accepted streaks, not confirmed NEOs, and the reported results identify domain robustness as a continuing challenge.

I. INTRODUCTION

  • Existing methods are predominantly evaluated on single-source data, providing limited evidence of generalization across telescopes, observing conditions, and labeling protocols.Images can differ in noise, spatial scale, background density, and streak morphology; these variations create domain shift.Source: §I · pp. 1–2
  • NEO-BENCH combines five publicly available sources in a unified framework designed to measure transfer across instruments, imaging conditions, and labeling conventions.Its contributions include 8,376 images, enhanced labels, a task-aware unified representation, and a cross-source protocol.Source: §I · p. 2
  • A streak-positive image contains a source-accepted elongated trace, which may be an asteroid, satellite, debris object, cosmic ray, or meteor rather than a confirmed NEO.The benchmark evaluates generic streak detection because public datasets lack consistent orbit-confirmed NEO-versus-non-NEO labels.Source: §I · p. 2
  • Performance generally decreased on omitted sources despite larger, more diverse training sets, making domain robustness the central challenge identified by NEO-BENCH.The benchmark’s cross-source evaluation is designed to measure transfer and generalization gaps.Source: §I · p. 2

II. RELATED WORK

Prior streak detectors span classical, statistical, and deep-learning approaches, while existing evaluations often remain tied to particular sources. NEO-BENCH integrates multiple sources in a shared format that preserves provenance and makes unseen-source degradation measurable.

II. RELATED WORK

  • Streak detection methods include Hough and Radon line searches, statistical streak models, and deep-learning classification, detection, or segmentation.The statistical models fit streak-like sources such as finite lines blurred by a point-spread function.Source: §II · p. 2
  • NEO-BENCH is presented as the first publicly available benchmark combining multiple astronomical-streak sources in a consistent bounding-box format.The five sources cover smart-telescope streaks, Hubble trails, asteroid streaks, satellite and NEO-like trails, and meteor/non-meteor images.Source: §II · p. 2
  • Unlike a sixth independent source, NEO-BENCH preserves the five sources’ provenance and differences while enabling shared-schema within-source and cross-source evaluation.This design makes unseen-source performance degradation measurable, which single-source benchmarks cannot reveal.Source: §II · p. 2

III. DATASET CONSTRUCTION

The dataset construction standardizes five heterogeneous sources for a common streak-detection benchmark while retaining source-specific labels and provenance. It adds or converts bounding boxes where needed, and the sources differ in their imagery and label coverage.

A. Sources and Direct Inventory

  • Dataset conversion standardized image and label names and represented existing labels as axis-aligned bounding boxes; Table I inventories the five sources after balancing.Images with streaks are streak-positive and have boxes; images without streaks are streak-negative. Fig. 1 shows source examples with ground-truth boxes, but not at a common scale.Source: A. Sources and Direct Inventory · p. 2, Table I · p. 3, Figure 1 · p. 3
  • StreakMind contributes 1,856 images, including 840 with 858 streak instances; its oriented boxes were converted to tight enclosing boxes, and its streaks are faint amid background artifacts.UAEMMN was sampled to 10%, yielding 1,382 images; because its original images lacked boxes, localized boxes were generated and sample-audited.Source: A. Sources and Direct Inventory · pp. 2–3
  • StreaksYoloDataset contains 2,388 images and 1,642 detector-ready boxes from a Stellina telescope; its broad streak class may include satellites, debris, and cosmic rays.Its streaks are described as bright, well-defined, and easy to detect.Source: A. Sources and Direct Inventory · p. 3
  • The Roboflow Asteroid reconstruction contains 1,049 accessible records, but its source provenance is undisclosed and its test split has no negative images.The project credits PEEK and specifies CC BY 4.0.Source: A. Sources and Direct Inventory · p. 3
  • Hubble Asteroid Hunter contributes 1,701 trails across 1,316 observations, with bounding boxes derived from projected sky-coordinate endpoints and padded by five pixels.This is geometry-derived supervision rather than manually drawn boxes, and the dataset has no negative images.Source: A. Sources and Direct Inventory · p. 3

B. Benchmark Tasks

The benchmark defines image-level streak classification and bounding-box object detection as complementary tasks. The former measures presence or absence, while the latter supports human inspection and verification.

B. Benchmark Tasks

  • Image classification asks whether a streak-like target is present or absent across the five sources using their available labels.It is a binary yes/no classification task.Source: B. Benchmark Tasks · p. 3
  • Together, the tasks cover both image-level streak presence and localized streak targets across the five sources.Classification uses available source labels; object detection adds bounding boxes.Source: B. Benchmark Tasks · p. 3
  • Object detection locates streak-like targets in the five sources with bounding boxes to assist human inspection and verification.The task uses bounding-box localization rather than only an image-level presence decision.Source: B. Benchmark Tasks · p. 3

C. Manual Audit and Label Verification

The audit reviewed 500 randomly sampled positive images and found 98.4% observed agreement between the authors. Flagged-image rates ranged from 0% to 6% across the five sources.

C. Manual Audit and Label Verification

  • 98.4% observed agreement was reached on 492 of 500 sampled images, despite disagreement on visually ambiguous cases.The authors report a Matthews correlation coefficient of 0.70 and Gwet’s AC1 of 0.98.Source: C. Manual Audit and Label Verification · p. 3
  • The two authors agreed on all eight images the second author classified as negative, while the first author flagged 16 sampled positives as potentially negative.The cases were visually ambiguous, which the authors say likely contributed to disagreement.Source: C. Manual Audit and Label Verification · p. 3
  • 6% of StreakMind sampled images were flagged, the highest source-specific rate; UAEMMN had 0% flagged.The other rates were 1% for StreaksYoloDataset, 3% for Hubble Asteroid Hunter, and 2% for Roboflow Asteroid.Source: C. Manual Audit and Label Verification · p. 3

IV. EXPERIMENTAL EVALUATION

The evaluation addresses within-source performance and generalization to unseen sources. It compares training or calibration on one source with training or calibration on all but the held-out source.

IV. EXPERIMENTAL EVALUATION

  • The study asks which evaluated method performs best within a source and how well models generalize to unseen sources.These are the two research questions guiding the evaluation.Source: §IV · p. 3

A. Experimental Settings

  • Within-source experiments trained or calibrated methods on one source and evaluated them on that source’s test split.Source: A. Experimental Settings · p. 3
  • Leave-one-out experiments trained or calibrated methods on all but one source and evaluated them on the held-out dataset.Source: A. Experimental Settings · p. 4

B. Metrics

The metrics assess image-level streak classification, changes after withholding a source, and bounding-box localization. IoU@0.50 requires at least 50% overlap for a localization detection to count as a true positive.

B. Metrics

  • Image-level evaluation counts an image as positive when at least one candidate passes the confidence threshold; precision, recall, and F1 are reported for each dataset.Precision is the share of predicted-positive images that are streak-positive, recall is the share of streak-positive images detected, and F1 is their harmonic mean.Source: B. Metrics · p. 4
  • A negative signed F1 change indicates worse performance after withholding the test source.Reporting within-source and leave-one-out results makes the effect of withholding each source visible.Source: B. Metrics · p. 4
  • IoU@0.50 counts a detection as a true positive only when the predicted box overlaps the ground-truth box by at least 50%.For classical and statistical methods, finite line detections or estimated endpoints are widened and converted into rectangular boxes.Source: B. Metrics · p. 4

V. METHODS COMPARED

The comparison spans Hough and Radon line-detection methods, Gaussian PSF fitting, and the learned YOLO26L detector. YOLO26L is tested both within-source and with each source held out in turn.

Hough

  • Hough uses signed, multi-scale directional filters to vote in line-parameter space, then applies endpoint recovery and coherence tests.Candidate scoring uses center-line and sideband responses.Source: §V · p. 4

Radon

  • Radon pads each image and integrates signed responses along trial lines, refining promising angles before endpoint and coherence tests.Source: §V · p. 4

Gaussian PSF

  • Gaussian PSF fits a finite Gaussian-broadened line to Hough and Radon proposals, refining line parameters and requiring likelihood gain and coherence.The refined parameters include position, angle, length, width, amplitude, and background.Source: §V · p. 4

YOLO26L

  • YOLO26L directly predicts streak locations and confidence scores, with models trained within-source or on non-held-out sources for leave-one-source-out testing.For leave-one-source-out runs, the held-out source was used only for testing.Source: §V · p. 4
  • YOLO26L was fine-tuned for 30 epochs from the same pretrained checkpoint, using batch size 16 and 640×640 input images.Testing used best.pt selected by validation mAP50:95; image-level F1-optimal validation thresholds were fixed for testing.Source: §V · p. 4

Classical and statistical methods

  • Hough, Radon, and Gaussian PSF settings were selected using validation data, with test localization precision and recall reported at each method’s F1-optimal operating point.Source: §V · p. 4

VI. EXPERIMENTAL RESULTS

Within-source performance can be strong, but leave-one-source-out results show frequent declines, especially for localization and harder datasets. No evaluated method generalizes robustly across sources.

VI. EXPERIMENTAL RESULTS

  • YOLO26L led within-source evaluation on all five datasets, with mean image-level F1 of 0.987 and localization F1 of 0.763.Its localization F1 ranged from 0.692 on Hubble to 0.851 on Roboflow.Source: §VI · p. 4
  • Non-learned methods had mean localization F1 of 0.231 for Hough, 0.060 for Radon, and 0.085 for Gaussian PSF; only YOLO26L exceeded 0.95 image-level precision and recall on all five within-source tests.The all-positive test splits did not assess the non-learned methods’ ability to reject negative images.Source: §VI · p. 4
  • Image-level F1 fell in 14/20 pairs (mean ∆F1 −0.052), while IoU@0.50 F1 fell in 13/20 (mean ∆F1 −0.105); medium/hard sources worsened in 11/12 and 9/12 pairs, respectively.Localization F1 was unchanged in one medium-or-hard pair.Source: §VI · pp. 4–5, Table II · p. 5
  • YOLO26L localization F1 declined on all five unseen sources by an average of 0.263, with losses ranging from −0.069 on UAEMMN to −0.584 on Roboflow.Its image-level F1 declined on four of five unseen sources; Roboflow increased from 0.987 to 1.000 on an all-positive test split.Source: §VI · p. 5
  • Classical and statistical methods occasionally improved across domains from low within-source performance, but gains were small and inconsistent; none of the methods was consistently robust across sources.YOLO26L’s results meeting the proposed 0.95 threshold held only within-source, with substantial leave-one-out declines.Source: §VI · p. 5

VII. CONCLUSION

NEO-BENCH combines five streak sources to evaluate astronomical streak-detector generalization. Its results show reliable cross-source generalization remains an open challenge.

VII. CONCLUSION

  • NEO-BENCH combines 8,376 images from five streak sources to evaluate the generalizability of streak-detection algorithms.Source: §VII · p. 5
  • The conclusion’s localization comparison is framed as IoU@0.50 precision, recall, and F1 for within-source versus leave-one-source-out evaluation.Source: Table III · p. 5
  • Using NEO-BENCH, the authors conclude that reliable streak-detection generalization across diverse imaging sources remains an open challenge.Source: §VII · p. 5
Loading 2609.06774v1…