PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Object Detection with Privileged Information: A Model-Agnostic Teacher-Student Approach

Matthias Bartolo, Dylan Seychell, Gabriel Hili, Matthew Montebello, Carl James Debono, Saviour Formosa, Konstantinos Makantasis

arXiv:2601.02016v1cs.CVcs.AIcs.ETcs.LG

TL;DR

Object detectors do not fully exploit fine-grained information in annotated images, while complex models, training demands, and costly annotation constrain accuracy. This paper introduces model-agnostic teacher–student LUPI training that uses privileged information unavailable at inference. Across the evaluated architectures and datasets, LUPI students improve detection accuracy without increasing inference efficiency costs or model size, although training takes longer.

  • Problem

    Object detectors do not fully exploit fine-grained information in annotated images, while complex architectures, extensive training, and costly annotation constrain efforts to improve accuracy.

    Source: §I · p. 1

  • Method

    A model-agnostic teacher–student framework trains teachers with privileged information and distills their guidance to RGB-only students.

    Source: §I · p. 2, A. Teacher–Student Framework for LUPI · p. 5

  • Results

    Across models and datasets, LUPI improves student detection accuracy without increasing model depth, parameter count, or inference time, while training both teacher and student takes longer.

    Source: E. Performance and Efficiency Analysis · p. 8, F. Discussion · p. 9, §VIII · p. 10

  • Takeaways & Limitations

    The findings support LUPI as a practical strategy for enhancing object detection while maintaining identical inference efficiency.

    Source: §VIII · p. 10

    Privileged-information generation can be affected by same-category overlap, occlusion from larger boxes, and limited color differentiation in masks; training both models lengthens training.

    Source: §VIII · p. 10

Abstract

from arXiv · show

This paper investigates the integration of the Learning Using Privileged Information (LUPI) paradigm in object detection to exploit fine-grained, descriptive information available during training but not at inference. We introduce a general, model-agnostic methodology for injecting privileged information-such as bounding box masks, saliency maps, and depth cues-into deep learning-based object detectors through a teacher-student architecture. Experiments are conducted across five state-of-the-art object detection models and multiple public benchmarks, including UAV-based litter detection datasets and Pascal VOC 2012, to assess the impact on accuracy, generalization, and computational efficiency. Our results demonstrate that LUPI-trained students consistently outperform their baseline counterparts, achieving significant boosts in detection accuracy with no increase in inference complexity or model size. Performance improvements are especially marked for medium and large objects, while ablation studies reveal that intermediate weighting of teacher guidance optimally balances learning from privileged and standard inputs. The findings affirm that the LUPI framework provides an effective and practical strategy for advancing object detection systems in both resource-constrained and real-world settings.

I. INTRODUCTION

Object detection remains constrained by accuracy demands, complex architectures, costly annotations, and substantial training resources. The paper applies LUPI to use fine-grained training-only information, while situating the approach among varied detection paradigms.

I. INTRODUCTION

  • Detection accuracy remains difficult to achieve because state-of-the-art models may require complex architectures, domain-specific fine-tuning, extensive training resources, and costly annotation.Large annotated datasets demand costly, labour-intensive annotation to improve detection accuracy.Source: §I · p. 1
  • Fine-grained information that annotated images contain but detectors underuse can be constructed and leveraged during training through the LUPI paradigm.LUPI supplies supplementary information during training that is unavailable at inference; examples include depth cues and saliency maps.Source: §I · pp. 1–2
  • The proposed model-agnostic LUPI methodology injects privileged information into object detectors and is evaluated across five pretrained models, UAV litter datasets, and Pascal VOC.The study also examines object scales, COCO metrics, information types, inference time, and model size.Source: §I · p. 2

II. RELATED WORK

  • The related-work section reviews deep-learning object detection and LUPI applications in computer vision.It frames detection as involving classification and localization.Source: §II · p. 2

A. Deep Learning for Object Detection

  • Object detection approaches include one-stage, two-stage, transformer-based, and other deep-learning methods, with different paradigms addressing different aspects of the task.One-stage detectors prioritize speed and efficiency, while two-stage detectors excel in complex scenes; transformer-based models flexibly model object relationships.Source: A. Deep Learning for Object Detection · p. 2

B. Learning Using Privileged Information in Computer Vision

Prior LUPI work spans object localization and image classification, while knowledge distillation offers a related teacher–student mechanism. The paper distinguishes LUPI by its training-to-inference information asymmetry.

B. Learning Using Privileged Information in Computer Vision

  • Earlier LUPI object-localization studies reported marginal or limited improvements, and the cited early work predated widespread deep-learning adoption.Feyereisl et al. (2014) used segmentation masks and SURF features on Caltech-UCSD Birds; Sun et al. (2018) studied the same dataset.Source: B. Learning Using Privileged… · p. 2
  • LUPI has also been explored in image classification using semantic attributes, bounding boxes, textual descriptions, annotator rationale, high-resolution images, tags, audio, and physiological features.The cited studies report measurable improvements in SVM+ and enhanced multi-label classification performance.Source: B. Learning Using Privileged… · p. 2
  • Knowledge distillation transfers information from a teacher to a student through feature or logit matching, while LUPI transfers information unavailable to the student at inference.Distillation commonly seeks a compact student performing comparably to a larger teacher with identical input information; LUPI instead addresses information asymmetry.Source: B. Learning Using Privileged… · pp. 2–3

III. METHODOLOGY

The formulation separates standard image inputs used for prediction from privileged information available during training. Each training example pairs those inputs with object-level bounding-box and class labels.

A. Problem Formulation

  • Each training sample contains a standard image, privileged information available only during training, and ground-truth labels.For each depicted object, the label includes a bounding box and class label.Source: A. Problem Formulation · p. 3
  • The ground-truth label for an image is a set of bounding-box and class-label pairs, with M denoting the number of depicted objects.The objective is to estimate a function parameterized by network weights.Source: A. Problem Formulation · p. 3
  • The network predicts from standard inputs X, but its parameters are estimated using X, labels Y, and privileged information X*.The function is implemented by a neural network whose parameters correspond to its weights.Source: A. Problem Formulation · p. 3

B. Proposed Approach

The teacher uses standard and privileged inputs to form richer representations, while the student learns from standard inputs alone by aligning its features with the teacher. Training balances detection supervision and feature transfer.

B. Proposed Approach

  • The teacher receives standard and privileged inputs, while the student sees only standard inputs and is trained to align its intermediate latent representations with the teacher’s.The aligned layer has the same number of hidden neurons in both networks, enabling direct representation comparison.Source: B. Proposed Approach · p. 3
  • Among the investigated privileged-information forms, bounding-box masks achieved the highest improvement, while saliency- and depth-based contributions remained modest.The comparison used the RetinaNet model on the SODA 1-metre dataset and also considered fused representations.Source: Figure 3 · p. 4
  • A combined objective balances standard detection supervision with teacher-to-student knowledge transfer, with α controlling the relative weight of ground-truth supervision and teacher guidance.The transfer term measures cosine distance between teacher and student feature vectors.Source: B. Proposed Approach · p. 4

C. Privileged Information for Object Detection

The section compares privileged cues in teacher–student training and finds bounding-box masks outperform saliency and depth alternatives. The mask representation combines localization and class cues in a compact form.

C. Privileged Information for Object Detection

  • Among the investigated alternatives, bounding-box masks achieved the highest improvement, while saliency- and depth-based contributions remained modest.The alternatives were evaluated within the teacher–student training framework.Source: C. Privileged Information for Object… · p. 4
  • Bounding-box masks guide attention toward object regions by encoding localization and class cues as grayscale-filled boxes on a black background.This structured spatial-signal choice aligns with motivation from physical reasoning about objects and prior work on saliency and depth cues.Source: C. Privileged Information for Object… · p. 4
  • Masks use ground-truth annotations available only during training; bounding boxes were preferred over polygonal or segmentation masks for simplicity, dataset consistency, and stable performance.Source: C. Privileged Information for Object… · p. 4

IV. IMPLEMENTATION

The next section describes the practical implementation and training setup of the LUPI-based detection framework.

  • The section that follows outlines the framework’s practical implementation and training setup.Source: §IV · p. 4

A. Teacher–Student Framework for LUPI

The framework trains a teacher with privileged inputs and transfers its feature representations to an RGB-only student. The implementation standardizes models and training procedures before evaluating the approach across models and conditions.

A. Teacher–Student Framework for LUPI

  • The teacher receives RGB plus privileged input, while an RGB-only student mirrors its architecture and learns through detection losses and a weighted feature-transfer term.The teacher’s input layer is extended to four channels; the student uses teacher features from the final backbone layer, and α balances direct supervision with teacher guidance.Source: A. Teacher–Student Framework for LUPI · pp. 4–5

B. Object Detection Models and Training Protocol

  • Five one- and two-stage detectors used standardized training and preprocessing; transfer tapped the final backbone layer to isolate privileged-information effects.Models used open-source architectures, and the complete training pipeline is publicly available.Source: B. Object Detection Models and… · p. 5

V. EVALUATION STRATEGY

  • The evaluation strategy assesses robustness through datasets, metrics, and performance analyses across models and conditions.Source: §V · p. 5

A. Datasets and Metrics

The evaluation spans controlled UAV-litter experiments, transfer to other litter datasets, and Pascal VOC 2012, with ablations examining teacher-supervision balance.

A. Datasets and Metrics

  • The study uses SODA subsets for controlled litter-detection scenarios, cross-dataset tests on BDW and UAVVaste, and Pascal VOC 2012 to assess broader applicability.Metrics include mAP, precision, recall, F1, mAR, and COCO-style measures across object scales; runtime, Grad-CAM, and teacher-versus-student comparisons are also examined.Source: A. Datasets and Metrics · p. 5

B. Within- and Cross-Dataset Experiments

  • Within SODA, five architectures are tested as teachers and students across binary and multi-label scenarios, with α varied from 0 to 1 in 0.25 steps.BDW tests SODA-trained models at 1-metre altitude without retraining; UAVVaste uses models trained on tiled SODA images across multiple altitudes, with binary detection and runtime considered.Source: B. Within- and Cross-Dataset Experiments · pp. 5–6
  • Across teacher evaluations, Faster R-CNN had the highest average performance, RetinaNet and FCOS performed similarly, and SSD and SSDLite had lower results.Source: Table I · p. 6

C. Pascal VOC 2012 Experiment

  • Pascal VOC 2012 evaluates multi-label detection across 20 object categories by comparing RGB-only baselines, LUPI students, and teachers under identical protocols.Source: C. Pascal VOC 2012 Experiment · p. 6

D. Ablation Study on Teacher–Student Balance

  • Ablations vary α across SODA binary and multi-label tasks and Pascal VOC 2012 to identify task-specific weighting that balances teacher supervision with ground-truth labels.The tested values range from 0 to 1 in increments of 0.25.Source: D. Ablation Study on Teacher–Student… · p. 6

VI. RESULTS AND DISCUSSION

The results show strong teacher performance and consistent student gains over RGB-only baselines, with improvements varying by evaluation setting and teacher-guidance weight.

B. Baseline vs. Student Model Comparison

  • LUPI-trained students consistently improve strict mAP and F1 over RGB-only baselines, with the largest gains in within-dataset UAV litter detection.Gains are smaller but meaningful in cross-dataset evaluations; Faster R-CNN, FCOS, and RetinaNet benefit most in UAV scenarios, while SSD and SSDLite improve more clearly on Pascal VOC.Source: Figure 4 · p. 7, B. Baseline vs. Student Model Comparison · p. 7

A. Teacher Model Performance

  • Teacher models achieve strict mAP and mAR values approaching 1, with consistently strong performance for medium and large objects.Performance decreases slightly in cross-dataset and multi-label settings; Faster R-CNN has the highest average mAP, followed by RetinaNet and FCOS.Source: A. Teacher Model Performance · p. 7

C. The Effect of Balancing the α Parameter

  • Intermediate teacher weights α=0.25 and α=0.5 generally perform best, balancing ground-truth labels and teacher knowledge across datasets and models.α=0 denotes the baseline, α=1 full teacher supervision; α=0.75 occasionally performs well, while α=1 tends to underperform.Source: C. The Effect of Balancing the α… · p. 7
  • Small objects show limited improvement, while medium and large objects benefit more; Faster R-CNN's Pascal VOC baseline marginally exceeds its student counterpart.The authors suggest additional teacher region proposals may introduce ambiguous supervision in that Faster R-CNN case.Source: C. The Effect of Balancing the α… · p. 7

D. Interpretability Analysis

LUPI-trained students show more focused attention on litter than baselines, while the broader discussion reports gains that vary by architecture and dataset. Training requires additional workload, but inference efficiency remains consistent.

D. Interpretability Analysis

  • Across datasets and experiments, α = 0.25 and 0.5 generally perform best, α = 0.75 also performs well, and α = 1 has lower average performance.The comparison uses COCO metrics; red arrows mark top strict map@50–95 performance.Source: Figure 5 · p. 8
  • LUPI-trained students focus more sharply on litter than baselines, with higher-confidence detections and fewer misclassifications.The SODA 1-metre Grad-CAM comparison shows student attention concentrated on litter and more diffuse baseline activation across the background.Source: D. Interpretability Analysis · p. 8, Figure 6 · p. 9

E. Performance and Efficiency Analysis

  • Teacher–student training effectively doubles workload, while baseline and LUPI students remain nearly identical in size, parameters, GFLOPS, and FPS.Similar efficiency trends were observed across datasets, though only Pascal VOC 2012 results are shown.Source: E. Performance and Efficiency Analysis · pp. 8–9

F. Discussion

  • LUPI consistently improved student learning, but gains varied by architecture and dataset; moderate teacher weighting helped balance supervision, while excess could confuse complex multi-label cases.Source: F. Discussion · p. 9

VII. PRACTICAL APPLICATIONS

The paper presents LUPI-based object detection as suitable for lightweight, resource-constrained applications. Its students improve accuracy without changing baseline architecture or inference efficiency, although mask generation and longer training remain limitations.

VII. PRACTICAL APPLICATIONS

  • The approach is presented as suited to UAV monitoring, traffic analysis, and surveillance where fast, reliable detection is needed on resource-constrained platforms.Source: §VII · p. 10

VIII. CONCLUSION

  • Across multiple architectures and datasets, privileged information during training enhances detection accuracy without increasing model depth, parameter count, or inference time.LUPI students remain identical to their baselines in architecture and efficiency while using additional teacher guidance during training.Source: §VIII · p. 10
  • Privileged-information masks can be affected by same-category overlap, occlusion from larger boxes, and limited color differentiation; training both teacher and student also lengthens training.Source: §VIII · p. 10
  • Future directions include newer detection architectures, richer privileged signals such as semantic maps or attention cues, and adaptation to object segmentation.Source: §VIII · p. 10
Loading 2601.02016v1…