Source-linked AI summary
FrameScope: Temporal Data Valuation for Stream Active Learning in Autonomous Vehicle Systems
Yuheng Zhu, Man-Ki Yoon
TL;DR
Autonomous vehicles need continuous learning to adapt perception models to dynamic environments, but streaming visual data makes heuristic selection inefficient and temporally incomplete. FrameScope extends neural tangent kernel theory with temporal valuation, selects frames locally, and queries cloud labels only for selected frames; experiments across multiple domain shifts report consistent improvements, higher sampling efficiency, and reduced catastrophic forgetting.
Problem
Dynamic driving environments produce novel, underrepresented scenarios and massive visual streams in which heuristic selection can miss valuable frames or choose redundant data.
Method
FrameScope extends neural tangent kernel theory to temporal data valuation by weighting temporal position and scene dynamics for local frame selection, with cloud labeling and on-vehicle adaptation.
Results
FrameScope consistently outperforms baseline methods across geographic, weather, and time domain shifts, achieving highest AP@50 scores of 0.529, 0.494, and 0.519, respectively.
Takeaways & Limitations
Local valuation and selective cloud annotation support continuous autonomous-vehicle adaptation under bandwidth and computational constraints while reducing catastrophic forgetting.
Takeaways & Limitations
The approach assumes inter-frame scene dynamics are positively correlated with learning value and uses temporal-window position weighting to estimate frame importance.
Abstract
from arXiv · showhide
Autonomous vehicles operate in dynamic, ever-changing environments where new scenarios and edge cases constantly emerge. As a result, static learning models are inadequate for ensuring safe and reliable operation. Continuous learning is essential for adapting to these evolving conditions and maintaining robust performance across diverse real-world settings. However, autonomous vehicles generate massive streams of visual data during operation, and existing continuous learning approaches typically rely on heuristic sampling methods that fail to capture temporal dynamics, often overlooking critical learning opportunities or selecting redundant frames. In this paper, we introduce FrameScope, a temporal data valuation framework for continuous learning in autonomous vehicles. FrameScope extends neural tangent kernel theory to temporal domains, enabling principled valuation of streaming visual data. Unlike cloud-centric methods that transmit all video data for processing, our approach performs principled, local frame selection on the vehicle and queries a cloud-based oracle model only for labels of those high-value frames. Extensive experiments across multiple domain shifts show that FrameScope consistently outperforms existing methods, achieving higher sample efficiency and significantly reducing catastrophic forgetting in autonomous vehicle perception. By valuing data on the vehicle and querying only labels for selected frames, FrameScope reduces bandwidth requirements, enabling scalable operation with a lightweight cloud labeling service.
1 Introduction
Autonomous vehicles need continuous learning because dynamic driving environments expose perception models to novel scenarios and possible catastrophic forgetting. FrameScope addresses streaming-data selection by valuing frames temporally, selecting locally, and querying cloud labels only for high-value frames.
- Dynamic driving environments create novel scenarios that may be underrepresented in initial training data, requiring continuous learning without catastrophic forgetting.
- Thousands of streamed frames per minute include redundant information alongside critical edge cases, making principled and computationally efficient frame selection a central bottleneck.
- Heuristic sampling can miss learning opportunities or select redundant data, while Shapley values and influence functions are computationally prohibitive for edge deployment.
- Pool-based valuation assumes prior access to the full dataset and treats samples independently, whereas stream-based valuation must assess sequentially arriving frames without future information.
- FrameScope extends data valuation to the temporal domain through Temporal Neural Tangent Kernel, temporal position weighting, and scene dynamics weighting.
- Experiments across geographic, weather, and time domain shifts report improved performance, sampling efficiency, and mitigation of catastrophic forgetting over baseline methods.
- The complete system integrates on-vehicle frame selection and retraining with a cloud oracle that labels only selected high-value frames.
2 Related Work
Related work spans data valuation, active learning, deep-learning selection, and streaming continuous learning. Existing approaches provide useful selection or bandwidth strategies but leave principled temporal valuation insufficiently addressed.
- Traditional data valuation methods: Data Shapley frames data points as players whose utility is model performance, but exact valuation has exponential complexity across data subsets.
- Traditional active learning approaches: Traditional active learning selects informative unlabeled examples using uncertainty, committee disagreement, or expected model change.
- Deep learning-based active learning: Deep-learning methods combine uncertainty estimation with representativeness to improve computational efficiency and reduce redundant selection.
- Streaming and continuous active learning: Stream-based and continuous active learning must handle concept drift, limited storage, evolving distributions, and real-time constraints.
- Streaming and continuous active learning: AdaptAV uses heuristic periodic, density-based, and event-based sampling rather than principled data valuation, potentially selecting suboptimal samples.
- Streaming and continuous active learning: Hawk performs bandwidth-aware selective transmission for rare-positive acquisition and serves as a baseline for edge-based continuous improvement.
3 FrameScope System Architecture
FrameScope places valuation, query selection, and model adaptation on the vehicle while using the cloud as a labeling oracle. This architecture targets bandwidth, computational, and fleet-scaling constraints by transmitting only selected frames.
- Cloud-only retraining repeatedly transfers large model weights and creates per-vehicle state and centralized scaling pressures.
- FrameScope performs on-vehicle adaptation through selective data valuation and query-based learning, using the cloud only to label queried frames.
- Figure 2’s closed loop connects a camera stream, on-vehicle model adaptation, and a cloud-based oracle model.
- The on-vehicle adaptation loop values incoming frames, filters queries under labeling budgets, and retrains the perception model on GPU resources.
- The update policy can retrain opportunistically from the current model and use a replay buffer to preserve prior competencies while keeping the cloud stateless.
4 Stream Data Valuation using Temporal Neural Tangent Kernel
FrameScope adapts data valuation to streaming video by combining temporal neural tangent kernel scoring with frame-level domain discrepancy. Its temporal design values frames using overlapping windows, temporal position, scene dynamics, and local stream information.
- DAVINZ’s static-data assumptions create temporal blindness, granularity mismatch, and difficulty handling dynamic scene complexity in video streams.
- FrameScope introduces TNTK, frame-level MMD, and multi-scale timing modeling for stream-based active learning.
- FrameScope selects valuable frames online under a labeling budget without observing the entire future dataset.
- FrameScope combines temporal valuation with frame-level domain discrepancy to assess each frame’s learning value in context.
- TNTK computes NTK values over overlapping sliding windows, preserving frame-level valuation while capturing temporal dependencies.
- Temporal position weighting emphasizes central frames, while scene dynamics weighting prioritizes frames associated with significant visual changes.
- A rare dynamic event receives a high Scene Dynamics Weight, demonstrating that the mechanism distinguishes valuable dynamic events from static scenes.
5 Evaluation
Across geographic, weather, and time domain shifts, FrameScope improves continual-learning performance, sample efficiency, and communication efficiency while mitigating catastrophic forgetting. Its on-vehicle selection remains effective under constrained sampling budgets and avoids the instability seen with some baselines.
- Domain-shift performance: FrameScope achieves the lowest final validation loss with smooth convergence and no significant degradation during geographic domain transitions.The evaluation covers sequential shifts through increasingly complex environments, including highways, residential areas, city streets, gas stations, and parking lots.
- Baseline comparison: Event-based and density-based sampling fluctuate more during transitions because they can miss gradual shifts or rely on object counts that do not capture visual complexity.FrameScope instead combines gradient-based learning-potential estimates with temporal context for frame selection.
- Domain-shift performance: FrameScope consistently outperforms baseline methods across geographic, weather, and time domain shifts, achieving the highest AP@50 scores in each scenario.Its AP@50 scores are 0.529 for geographic, 0.494 for time, and 0.519 for weather, with statistically significant paired differences.
- Resource efficiency: FrameScope filtering on the vehicle reduces uploaded bytes by 90.3% under a 6,000-frame per-phase budget and transmits only selected frames for oracle annotation.Cloud-training baselines upload all frames, while Filter-on-Cloud methods also upload all frames for scoring and filtering.
- Sampling efficiency: At a 20% sampling rate, FrameScope reaches average precision 0.522 versus 0.552 for the 100% baseline, a 4.7% drop while using 80% fewer samples.At 10% sampling, it achieves average precision 0.494, described as roughly 90% of full-dataset performance.
- Resource efficiency: Bandwidth scales linearly with sampling rate: moving from 5% to 20% multiplies transmission costs by four, while using 100% causes a 20-fold increase.The 10% configuration is presented as a practical balance between performance and bandwidth consumption.
6 Conclusion
FrameScope extends neural tangent kernel theory to temporal data valuation for autonomous vehicle systems, addressing heuristic methods that overlook temporal dynamics. The TNTK framework incorporates frame-to-frame dependencies and scene dynamics to support principled frame selection under practical deployment constraints.
- FrameScope extends neural tangent kernel theory to the temporal domain for autonomous vehicle data valuation.
- TNTK incorporates temporal dependencies among frames and scene dynamics to enable principled frame selection.
- FrameScope addresses heuristic sampling methods that fail to capture temporal dynamics in video streams.
- Experiments demonstrate practical utility in real-world settings, particularly under bandwidth and computational constraints.
- The framework connects theoretical data valuation with practical streaming applications for continuous learning.