PaperlayerView exact revision on arXiv

Source-linked AI summary

RailGen: Improving Railway Intrusion Detection via Agent-Guided Small-Scale Foreign Object Generation

Quan Hao, Ziyang Tao, Chenxi Zhang, Yudong Wang, Rui Shi, Liguo Zhang

arXiv:2608.30727v1cs.CVcs.AI

TL;DR

Railway foreign-object detection must identify scarce, easily confused small intrusions in complex scenes. The paper combines the multimodal RailGen generator with FocalDEIM detection, reporting smaller synthetic objects and higher detection performance than DEIM. The resulting framework targets feature-space incompleteness and feature ambiguity in long-tailed RFOD.

  • Problem

    Railway foreign-object detection faces scarce samples, incomplete rare-object feature spaces, and blurred boundaries between small intrusions and complex backgrounds.

  • Method

    RailGen uses semantically constrained multimodal generation, while FocalDEIM combines Focal Modulation and Focal Loss for generated-data detection training.

  • Results

    5.6% and 7.5% gains over DEIM are reported for mAP50 and mAP50-95, while generated foreign objects are 11.3× and 16.4× smaller than those from FLUX and Nano Banana on average.

  • Takeaways & Limitations

    The framework improves sample diversity and feature separability for rare, small foreign objects in complex railway scenes.

Abstract

from arXiv · show

Small-object detection under long-tailed data distributions is a fundamental yet challenging problem in multimedia. Railway Foreign Object Detection (RFOD) epitomizes this challenge with easily confused small intrusions and scarce samples. To address these issues, we propose a generative-augmented detection paradigm that leverages multimodal image generation to enrich the feature space of rare and small objects. We first construct RailGen, a multimodal image generation agent based on large models. Under semantic constraints, RailGen automatically invokes tools to generate railway scenes, calibrate intrusion positions, extract foreign objects, and fuse them into realistic intrusion effects. This process produces high-quality synthetic samples that effectively densify the feature representations of tail classes and complete the small-object feature space. Within this paradigm, we further propose FocalDEIM, a detection framework designed to enhance training with generated data. FocalDEIM improves dense matching with Focal Modulation for better small-object discrimination and adopts Focal Loss to emphasize hard samples, thereby alleviating blurred inter-class boundaries in complex railway scenes. Experimental results demonstrate that RailGen can generate high-quality small-scale foreign objects, reducing the object pixel area by up to 58x and 13.85x on average. Equipped with these challenging samples, our paradigm surpasses the baseline DEIM by 5.6% and 7.5% in mAP@50 and mAP@(50-95), respectively, and outperforms existing state-of-the-art methods. Ablation studies verify RailGen's feature-space enrichment and FocalDEIM's boundary discrimination. The paradigm provides an effective multimodal generative solution for long-tailed small-object detection in safety-critical applications.

I. INTRODUCTION

Railway foreign-object detection is hindered by scarce, long-tailed samples and visually confusing small intrusions. The paper addresses these problems with RailGen and FocalDEIM, combining realistic synthetic generation with hard-sample-aware detection.

  • Railway foreign-object detection is safety-critical because small objects on overhead lines or tracks can cause facility damage, derailment, and emergency stops.
  • Complex lighting, repetitive track textures, and overhead-line backgrounds make small foreign objects difficult to distinguish from railway scenes.
  • RailGen generates semantically consistent railway intrusion images to enrich rare-object feature spaces under long-tailed data distributions.
  • 5.6% and 7.5% improvements over the DEIM baseline are reported for mAP50 and mAP50-95, respectively.RailGen reduces object pixel area by up to 58× and 13.85× on average.
  • FocalDEIM improves generated-data training with Focal Modulation for small-object discrimination and Focal Loss for hard, easily confused samples.

II. RELATED WORK

Related work establishes RFOD’s scarcity and ambiguity challenges, while existing detectors and generators remain limited by sparse supervision, synthetic-to-real gaps, and weak physical control. The paper positions a multimodal agent and generation-detection collaboration as a response.

  • RFOD research is constrained by severe data scarcity, incomplete feature spaces, and blurred boundaries between rare intrusions and complex railway backgrounds.
  • Transformer-based DETR methods provide global modeling, but one-to-one matching suffers from sparse supervision when positive samples are scarce.
  • Generated training samples are limited by domain gaps between synthetic and real images, restricting performance gains.
  • Existing generative models lack explicit control over intrusion category, placement, scale, and scene compatibility, especially for small objects.
  • RailGen uses multimodal-agent planning and specialized tools for intrusion localization, image generation, segmentation, and foreign-object fusion.

1) Multimodal Semantic Reasoning and Anchor Region Calibration:

RailGen calibrates foreign-object anchor regions by combining multimodal semantic reasoning with railway-specific physical constraints. It then uses deterministic Flow Matching and domain adaptation to generate and prepare railway scenes and object images for fusion.

  • Semantic Reasoning and Anchor Calibration: Anchor regions are candidate foreign-object locations calibrated through multimodal semantic reasoning to satisfy perspective and physical plausibility constraints.Heavy objects may contact the trackbed, while lightweight objects may remain suspended.
  • Semantic Reasoning and Anchor Calibration: The optimal anchor region maximizes both visual naturalness and railway physical consistency.Visual compatibility is combined with track geometry, gravity, and safety-contact constraints.
  • Semantic Reasoning and Anchor Calibration: Railway physical consistency decomposes into contact, gravity, and perspective terms weighted by α1, α2, and α3.These terms constrain support relationships, gravity direction, object scale, and track perspective.
  • Semantic Reasoning and Anchor Calibration: The constrained formulation prevents physically unreasonable anchor positions and avoids contaminating the generated feature space.The paper frames railway safety knowledge and physical priors as computable spatial constraints.
  • Deterministic Generation: Flow Matching uses a probability path from noise to data and a probability-flow ODE for deterministic image synthesis.Inference integrates the ODE from x1 to x0, while Low-Rank Adaptation fine-tunes the railway domain with the pretrained velocity field frozen.
  • Deterministic Generation: After generating high-speed railway scenes and foreign-object images, image segmentation supplies geometric priors for subsequent fusion.The strategy maps a general visual distribution toward the railway-scene manifold.

3) Structure-Aware Physical Conditional Injection Mechanism:

SPCI embeds geometric, structural, physical, and lighting constraints directly into the image-generation process. Its dynamically updated modulation progressively enforces consistency and reduces fusion artifacts.

  • Motivation: Traditional post hoc fusion can produce boundary artifacts, scale inconsistency, and lighting conflicts because it neglects scene geometry and physical constraints.SPCI addresses this by embedding fusion into the generative process.
  • SPCI Representation: The SPCI representation contains five channels for the foreground mask, contour, depth approximation, structural support, and lighting direction.These channels encode complementary conditions for structure-aware physical generation.
  • Physical Constraints: The support constraint models steady-state physical consistency rather than relying only on heuristic gravity constraints.Its semantics distinguish ground-bearing and suspension structures and use object-dependent weights.
  • Physical Constraints: Heavy objects require structural support, whereas lightweight objects such as balloons or kites may remain suspended.Additional channels enforce perspective-depth alignment and lighting coherence.
  • Latent Injection: SPCI injects its condition tensor into latent features through a structure-aware modulation operator guided by railway geometry priors.The modulation uses learnable weights and constraint-specific functions.
  • Progressive Guidance: Dynamic modulation weights jointly apply semantic and structural guidance at each ODE step, progressively enforcing physical consistency along the generation trajectory.The paper reports that this integrated approach reduces artifacts and improves realism.
  • Outcome: SPCI integrates geometric, structural, and physical constraints directly into generation, reducing artifacts and improving realism.

B. FocalDEIM: Focal-Driven Dense Matching for Small Objects

FocalDEIM addresses the remaining discriminability bottleneck in dense matching for small railway foreign objects. It enhances query representations before assignment and uses focal-aware matching to guide learning toward semantically difficult cases.

  • Motivation: RailGen enriches the feature space, but small foreign objects remain difficult to distinguish from track textures and backgrounds during Hungarian matching.Their similar local patterns limit discriminability even when dense matching supplies richer candidates.
  • FocalDEIM: FocalDEIM applies Focal Modulation with context-aware aggregation before Hungarian assignment to enhance small-object query features.The module is designed for complex railway scenes where local object and background patterns overlap.
  • FocalDEIM: A focal-aware matching cost measures semantic consistency between queries and targets and guides more effective learning and faster convergence.

1) Dense Matching in the RailGen-Enriched Feature Space:

DEIM performs dense one-to-one matching in RailGen’s enriched feature space to increase positive supervision. However, randomly initialized queries can still make small objects difficult to assign accurately during early training.

  • DEIM Prediction: DEIM converts Transformer encoder features and N learnable queries into class predictions and four-parameter bounding-box predictions through the decoder.The outputs are represented as Ỹcls ∈ R^N×K and Ỹbox ∈ R^N×4.
  • Hungarian Matching: Hungarian Matching optimally assigns N predictions to M ground-truth targets by minimizing a cost matrix combining classification and regression costs.The setting assumes N ≫ M.
  • Dense Matching: Dense One-to-One Matching applies S image scales and cropping augmentations so each target can match multiple spatially adjacent queries.This expands positive samples by a factor of S and alleviates sparse supervision.
  • Early-Training Bottleneck: Early in training, weak small-object responses and background similarity make accurate Hungarian assignment difficult despite abundant dense-matching candidates.Randomly initialized queries produce decoder features with limited discriminability.

2) Context-Aware Matching Based on Focal Modulation:

FocalDEIM enhances decoder query features with context-aware Focal Modulation before Hungarian matching and uses Focal Loss to emphasize difficult samples. These mechanisms target small-object discrimination, blurred class boundaries, and learning under limited data.

  • Context-aware feature enhancement: FocalBlock refines decoder query representations before Hungarian assignment to improve small-object discrimination against cluttered backgrounds.It aggregates contextual information and enhances query features before matching.
  • Context-aware feature enhancement: Focal Modulation is incorporated into matching costs to enhance small-object features through context aggregation and feature enhancement.The focal matching cost measures semantic consistency between modulated query features and ground-truth target features.
  • Matching strategy: The final matching cost combines weighted classification, regression, and focal modulation costs.This cost matrix is used for Hungarian assignment.
  • Training objective: Focal Loss reweights hard and easily confused samples during training, while the box objective combines GIoU Loss and L1 Loss.The total objective covers matched positives and unmatched negatives.
  • Training objective: FocalDEIM improves small-object learning with limited samples while adding zero inference-time computational overhead.The Focal Modulation module is used only during training-time label assignment and is removed during inference.

IV. RESULTS

The experiments use multiple datasets supporting RailGen generation, detector training, and benchmarking. The RailGen dataset contains generated and manually annotated RFOD images, with 400 generated samples reported as empirically optimal for augmentation.

  • Datasets: Six datasets are used to evaluate image generation, detector training, and benchmarking.The supplied dataset description begins with the source, RailGen, real train, and real validation sets.
  • Datasets: The Source Dataset contains 4,000 real railway scene images and 4,131 foreign-object images for training RailGen.These data support image-generation model training.
  • Datasets: The RailGen Dataset contains 1,318 generated RFOD images manually annotated for label accuracy.Empirically, augmenting with 400 generated samples gives the best detection performance.

2) Evaluation Metrics for Image Generation:

The evaluation measures perceptual quality, physical plausibility, and foreign-object scale across generation methods and datasets. RailGen achieves the strongest reported generation quality and produces substantially smaller objects than competing methods.

  • Evaluation metrics: Generated images are evaluated with Scene Realism, Foreign Object Visual Quality, and Foreign Object Plausibility, each scored from 0 to 10.The average score aggregates SR, FOVQ, and FOP.
  • Evaluation metrics: FO Pixel measures mean foreign-object pixel count, while Avg Ratio and Max Ratio measure average and maximum pixel-count ratios relative to a reference method.These metrics assess whether objects are small but recognizable.
  • Comparison with SOTA methods: RailGen achieves the highest SR, FOVQ, FOP, and Avg scores, followed by Nano Banana, while FLUX performs worst.The comparison includes FLUX, Nano Banana, and RailGen.
  • Comparison with SOTA methods: 198.8 pixels is RailGen’s average object size at 768 × 768 resolution, compared with 2245.2 pixels for FLUX and 3261.8 pixels for Nano Banana.The competing objects are 11.3× and 16.4× larger on average, with maximum ratios of 58.0× and 43.6×.
  • Comparison with existing datasets: RailGen scores 5.21 on average across dataset comparisons, exceeding RailFOD23 at 3.54 and approaching real-scene performance.Real RFOD scenes obtain the highest average score at 6.84.

C. Evaluation of Inter-Class Boundary Clarification via Detection

FocalDEIM improves detection through focal components and RailGen augmentation, with their combination delivering the strongest reported performance and better strict-IoU localization.

  • Detection performance: 3.10 points: FocalDEIM reaches 71.50 mAP50 versus 68.40 for the DEIM baseline under comparable parameter scales.The comparison uses models with 10.0M–12.6M parameters.
  • Detection performance: 74.00 mAP50: RailGen-augmented FocalDEIM surpasses DINO by 4.8 points, combining feature-space expansion with focal-driven discrimination.
  • Ablation study: Combining Focal Modulation and Focal Loss yields 71.50/43.50, while each component alone also improves over the 68.40/38.90 DEIM baseline.The reported single-component results are 70.80/40.70 with FocalBlock and 72.20/41.50 with FocalLoss.
  • RailGen augmentation: The full model achieves 74.00/46.40, improving over the DEIM baseline by +5.6 mAP50 and +7.5 mAP50−95.RailGen improves the baseline from 68.40/38.90 to 72.70/44.70; the gains are reported as mAP50/mAP50−95.
  • RailGen augmentation: RailGen can slightly reduce mAP50 in one FocalLoss configuration while increasing mAP50−95, indicating stronger localization under stricter IoU thresholds.For DEIM + FocalLoss, mAP50 changes from 72.20 to 71.20, while mAP50−95 changes from 41.50 to 45.90.

E. Sensitivity Analysis of Focal Cost for Boundary Clarification in Hungarian Matching

The focal matching cost is tuned through a grid search, with λfocal = 0.5 providing the best balance between focal-aware discrimination and spatial localization.

  • Sensitivity analysis: λfocal = 0.5 achieves the optimal balance in the Hungarian matching sensitivity analysis.The tested values are {0.1, 0.5, 1.0, 2.0}.
  • Sensitivity analysis: Weights <= 0.1 fail to distinguish small objects from background clutter, whereas weights >= 1.0 overemphasize feature similarity at the expense of spatial localization.
  • Matching objective: λfocal balances the focal-aware cost with the original classification and localization terms in the Hungarian matching objective.The selected value is adopted in all experiments.
  • Feature-space expansion: RailGen improves mAP50 and mAP50−95 by 2.50 and 2.90 points over the real-only baseline on the Real Val Set.The comparison evaluates detectors trained with mixtures of augmented and real training data.
  • Feature-space expansion: Semantically aligned synthetic data is more effective for feature-space expansion than existing datasets or data from related infrastructure scenarios.RailFOD23 improves mAP50−95 by 2.51 points but decreases mAP50 by 0.24 points; CES provides more modest gains.
Loading 2608.30727v1…