PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

Kaiser Hamid, Can Cui, Nade Liang

arXiv:2608.25344v1cs.CVcs.AI

TL;DR

Coarse video labels often omit when risk evidence emerges and which entities support a prediction, although recovering those details normally requires costly fine-grained annotation. CoRE freezes a coarse predictor, uses structured interventions to create graded effect targets, and distills them into a student that predicts temporal and entity support directly. Across RISEE, DoTA, and UCF-Crime, the framework recovers informative fine-grained support from coarse supervision, with strong DoTA temporal localization and competitive UCF-Crime performance.

  • Problem

    Video-level risk judgments do not specify the temporal regions or scene entities supporting them, while fine-grained annotations are substantially more expensive.

  • Method

    CoRE freezes a video-level predictor, measures prediction changes under structured temporal or entity interventions, and distills graded effects into direct support predictors.

  • Results

    Across RISEE, DoTA, and UCF-Crime, CoRE learns informative fine-grained support from coarse supervision, with strong temporal localization on DoTA and competitive performance on UCF-Crime.

  • Takeaways & Limitations

    Coarse video supervision can recover both a prediction and evidence supporting it without corresponding fine-grained labels.

  • Takeaways & Limitations

    The learned support describes dependence of the coarse prediction on candidate evidence and does not establish physical causality.

Abstract

from arXiv · show

Perceived risk in driving evolves over time and may be supported by specific scene entities, yet supervision is typically limited to coarse video-level judgments. Learning \emph{when} supporting evidence emerges and \emph{which entities} support a risk predictor would ordinarily require costly temporal- and entity-level annotations. We introduce \textbf{CoRE}, a weakly supervised coarse-to-fine framework that learns fine-grained prediction support from coarse video supervision. CoRE first trains a video-level predictor and then freezes it. Structured interventions over candidate temporal regions or entity tracks measure how each candidate changes the coarse prediction, producing graded prediction-effect targets. These targets are distilled into a student that directly predicts temporal and entity support from the original video, without requiring interventions at inference. We evaluate this learning principle across three complementary settings: RISEE tests perceived-risk support from subjective clip-level judgments without temporal or entity-level risk annotations; DoTA provides independent temporal event annotations for evaluating weakly supervised traffic-anomaly localization; and UCF-Crime tests whether the same coarse-to-fine mechanism extends to a standard non-driving anomaly-detection benchmark. Across these settings, CoRE learns informative fine-grained support from coarse supervision, with strong temporal localization on DoTA and competitive performance on UCF-Crime. These results show that coarse video predictions can provide useful supervision for recovering the fine-grained evidence supporting them, without requiring corresponding fine-grained labels.

1. Introduction

CoRE addresses the mismatch between coarse video-level supervision and the temporally and entity-specific evidence underlying perceived-risk predictions. It uses prediction changes from structured interventions as weak supervision for direct fine-grained support prediction.

  • Motivation: Driving videos often provide only overall risk scores or event labels, leaving the timing and contributing scene entities unspecified.Temporal- and entity-level annotations are more expensive than video-level judgments.
  • Motivation: Existing weakly supervised localization methods can recover fine-grained scores from coarse labels but may emphasize only the most discriminative snippets.Accurate coarse prediction and reliable fine-grained support can diverge.
  • Motivation: Attention or multiple-instance scores should not automatically be interpreted as faithful prediction support.A coarse predictor may rely on a few salient observations while other regions or entities also contribute.
  • CoRE: CoRE freezes a video-level predictor, measures prediction changes under temporal or entity interventions, and distills graded effects into a direct support predictor.The student predicts temporal and entity support in a single forward pass, while learned support reflects prediction dependence rather than physical causality.
  • Evaluation: CoRE is evaluated on RISEE, DoTA, and UCF-Crime to test subjective risk support, temporal localization, and extension beyond driving.The evaluation spans clip-level perceived-risk judgments, independent temporal annotations, and a non-driving anomaly benchmark.

2. Related Work

Related work shows that coarse video labels can support temporal instance scoring, while attribution and perturbation methods offer complementary ways to assess prediction evidence. CoRE distinguishes itself by converting intervention responses into direct support supervision for temporal and entity candidates.

  • Driving risk and accident understanding: Driving-risk research studies accident anticipation, traffic-anomaly understanding, and influential scene entities in driving interactions.DoTA supplies temporal, spatial, and categorical traffic-anomaly annotations, while ROAD represents road events with structured labels.
  • Weakly supervised temporal localization: Weakly supervised temporal localization recovers action intervals from video-level category labels using temporal selection, refinement, and fine-grained representations.Representative methods address incomplete localization and foreground–background ambiguity.
  • Weakly supervised video anomaly detection: Weakly supervised anomaly detection learns fine-grained anomaly scores from video-level normal/abnormal labels through multiple-instance and instance-discovery methods.The cited line includes self-training, feature-magnitude learning, normality-guided MIL, and debiased instance learning.
  • CoRE overview: CoRE's pipeline trains and freezes a coarse predictor, intervenes on temporal and entity candidates, and converts the resulting effects into graded support targets.The figure organizes these stages as coarse prediction, prediction-effect target construction, and candidate scoring.
  • Attribution, perturbation, and distillation: Unlike methods that derive temporal supervision from scores, statistics, prompts, or representation transfer, CoRE distills candidate-level prediction effects into a direct support model.Interventions are confined to target construction, while support is produced directly at inference.

3. Method

CoRE learns fine-grained temporal and entity support from coarse video supervision by distilling prediction effects from a frozen coarse predictor into a direct student model. Structured interventions construct graded support targets, while inference uses a single pass over the original video.

  • 3.1. Problem Formulation: CoRE assumes videos have coarse targets while temporal boundaries and entity relevance annotations are unavailable during training.
  • 3.2. Coarse Video Predictor: A teacher is trained with video-level supervision, produces a coarse response, and is frozen before support-target construction.The coarse response may be continuous or categorical, using a normalized prediction or target-class score/probability.
  • 3.3. Prediction-Effect Target Construction: Structured interventions perturb one temporal region or entity track while preserving the remaining input, and prediction changes quantify candidate support.Larger positive effects indicate stronger prediction support under the specified intervention.
  • 3.3. Prediction-Effect Target Construction: CoRE retains the full graded effect pattern, converts it into task-specific support targets, and caches the targets as fixed student supervision.This preserves relative effect strength instead of selecting only the highest-effect candidate.
  • 3.4. Temporal Support: Temporal and entity branches apply the same rule: score candidate interventions, construct graded targets, and distill them into direct support predictors.The branches differ only in whether candidates are temporal regions or entity tracks.
  • 3.6. Support Distillation and Inference: At inference, interventions and the teacher are removed; the student processes the original video once and predicts coarse response together with available temporal and entity support.Fine-grained scores indicate where the prediction is supported, while the coarse response determines overall prediction strength.

4. Experiments

CoRE is evaluated across subjective perceived-risk support, weakly supervised driving anomaly localization, and non-driving anomaly detection, using coarse labels for training and fine-grained annotations only for evaluation where available.

  • Experimental Setup: RISEE uses clip-level human ratings, while fine-grained support is evaluated through held-out candidate interventions rather than temporal or entity-level ground truth.The protocol preserves identical scenario-level five-fold splits across methods and activates CoRE’s entity-support branch.
  • Experimental Setup: DoTA compares protocol-matched methods on shared feature banks and reports frame, macro, event-level F1, and best temporal-IoU metrics.The evaluation uses a weakly supervised split with temporal annotations reserved for assessment.
  • Main Results: 0.320 selected drop, 0.245 gain over random, and 0.542 effect correlation show that CoRE identifies RISEE temporal evidence more faithfully than Attention MIL and Soft top-k MIL.The baselines’ selected regions produce small prediction drops and negative correlation with measured window effects.
  • Main Results: 0.735 frame AUC, 0.374 F1@0.5, and 0.440 best tIoU are achieved by CoRE with ResNet-50 on DoTA, improving from 0.642, 0.237, and 0.370 respectively.With CLIP ViT-B/32, CoRE reaches 0.744 AUC, 0.514 AP, 0.364 F1@0.5, and 0.429 tIoU.
  • Ablation and Analysis: Removing prediction-effect supervision reduces DoTA AUC from 0.744 to 0.419 and F1@0.5 from 0.364 to 0.139.Removing multi-scale candidates nearly eliminates event localization, reaching 0.040 F1@0.5 and 0.098 tIoU.

5. Conclusion

CoRE converts structured intervention effects into fine-grained support supervision and predicts temporal and entity support without interventions at inference. Results across RISEE, DoTA, and UCF-Crime support recovering evidence from coarse video supervision.

  • CoRE turns prediction changes under structured interventions into fine-grained support supervision.
  • A student distills intervention effects to predict temporal and entity support in a single forward pass.
  • RISEE demonstrates support learning from clip-level perceived-risk judgments without temporal or object-level labels.
  • DoTA validates temporal support against independent event annotations, while UCF-Crime tests extension beyond driving.
  • CoRE recovers both video predictions and their supporting evidence from coarse supervision.

A. Supplementary Details

The supplement specifies dataset protocols and implementation details for applying CoRE under weak video-level supervision. It also defines intervention-based target construction for temporal and entity support.

  • CoRE uses only video-level labels for training while inferring fine-grained temporal or object support without dense supervision.
  • RISEE: RISEE contains 179 egocentric driving scenarios with aggregated human risk ratings but no temporal or object-level risk labels.Support is evaluated through prediction-effect tests rather than supervised localization accuracy.
  • RISEE: RISEE temporal support uses 16 positions and 13 overlapping four-position candidate windows.
  • DoTA: DoTA training uses 2,420 anomalous and 3,234 normal videos, with 269 anomalous and 358 normal videos for validation.Evaluation uses the released 1,140-video DoTA test split.
  • UCF-Crime: UCF-Crime follows a weakly supervised protocol with 1,610 training and 290 test videos.The protocol holds out 161 training videos for validation and uses 1,449 for optimization.
  • Interventions and targets: Local-mean replacement is the default intervention, while blur provides an independent RISEE robustness operator.Selected candidates are perturbed while the rest of the input is preserved; effects become task-specific support targets.

B.1. Cross-Operator Robustness

Cross-operator experiments test whether CoRE’s learned support depends on the intervention operator used to construct targets. Temporal and object support remain stable across operator mismatches.

  • The cross-operator experiment evaluates students under matched and mismatched training and evaluation operators.
  • Students trained with one intervention operator retain stable temporal and object support when evaluated with the other.
  • High agreement between candidate effects supports local-mean replacement as the default and blur as an independent robustness check.

B.3. Teacher Robustness

Teacher-robustness experiments vary only the coarse risk teacher while keeping candidate construction, student design, training, and evaluation fixed. All teachers produce positive support-selection gains.

  • Teacher robustness is tested while candidate construction, student design, training, and evaluation remain fixed.
  • The canonical teacher gives the strongest coarse prediction and selected drop.
  • The Attention-MIL teacher gives the largest gain over random.
  • All teachers produce positive support-selection gains, indicating the approach is not tied to one coarse-head architecture.

C. Attention Versus CoRE Support

CoRE distinguishes attention visualization from support learned using measured prediction effects. Under the same Attention-MIL teacher, intervention-distilled support improves every reported metric over direct attention.

  • Attention weights are not optimized to match prediction effects, so they need not faithfully represent prediction support.
  • CoRE is evaluated without dense temporal or object labels during training.
  • Table 7 reports selection metrics shared by CoRE's temporal and object support across intervention operators.
  • Table 8 evaluates agreement between local-mean and blur intervention effects, while Table 9 varies the coarse risk teacher on RISEE.
  • Table 10 compares direct Attention-MIL attention with CoRE support distilled from intervention effects of the same teacher.
  • Intervention-distilled support improves every metric over direct attention under the same teacher.

D. Runtime

CoRE performs intervention-based target construction offline and uses direct student inference at test time. Its student heads are lightweight relative to feature extraction, avoiding candidate-wise intervention evaluation during inference.

  • Target construction is performed offline, while test-time CoRE uses direct student inference without candidate-wise interventions.
  • CoRE avoids the candidate-wise intervention cost at inference, although offline target construction costs similarly to naive intervention explanation.
  • The student heads are lightweight relative to feature extraction.
Loading 2608.25344v1…