PaperlayerView exact revision on arXiv

Source-linked AI summary

Bend the Clock: Predicting Ahead to Beat Latency in Event-Based Object Detection

Biswadeep Sen, Benoit R. Cottereau, Nicolas Cuperlier, Terence Sim

arXiv:2609.26919v1cs.ROcs.CV

TL;DR

Event detectors can become stale while their predictions are computed, a mismatch that observation-time evaluation overlooks. ChronoFuse uses causal cross-time fusion to predict object states for output availability, recovering latency-lost accuracy across driving and drone benchmarks and improving performance under extreme motion.

  • Problem

    Observation-time evaluation compares detections with the scene at input time, although robotic systems receive them after computation latency, when the scene may have changed.

    Source: §I · p. 1

  • Method

    ChronoFuse uses future-target supervision and causal fusion of current and cached multi-scale features to predict object states at availability time without future observations.

    Source: §I · pp. 1–2, A. Latency-Compensated Detection · p. 3

  • Results

    ChronoFuse recovers 71% of latency-lost accuracy on 1Mpx and 90.8% on FRED, and reaches 20.95 versus 2.25 sAP on EV-Flying.

    Source: §I · p. 2

  • Takeaways & Limitations

    Availability-time prediction provides more timely object-state estimates for downstream robotic decisions, including navigation, interception, and multi-robot coordination.

    Source: §V · pp. 7–8

Abstract

from arXiv · show

Event cameras promise low-latency perception for high-speed robotic systems, where even short delays can render detections stale by the time they inform downstream robotic decisions. Yet modern event detectors still require tens of milliseconds of computation before their predictions become available. Conventional evaluation ignores this delay by comparing predictions with annotations at the observation timestamp, even though the scene may have changed by the time those predictions are produced. We study this observation-availability mismatch in event-based multi-object detection and show that state-of-the-art event detectors degrade substantially when evaluated at prediction availability rather than observation time. To address this, we introduce ChronoFuse, a causal availability-time detector that predicts object states for when its output becomes available rather than for when its input was observed. ChronoFuse performs causal cross-time fusion over a multi-scale feature hierarchy, combining current representations with cached temporal features to expose short-term temporal cues without using future observations. The fusion pathway is lightweight, adding only 0.17 million parameters and 0.84 ms of mean end-to-end latency overhead. ChronoFuse recovers 71% of the accuracy lost to latency on 1Mpx driving data and 90.8% under rapid drone motion on FRED, nearly restoring zero-delay performance. Under the extreme motion of EV-Flying, ChronoFuse reaches 20.95 sAP, compared with 2.25 for the strongest standard event detector (9.3x gain). These results show that predicting ahead can be critical for robots operating in fast-changing scenes, including autonomous driving, agile flight, and robotic interception.

I. INTRODUCTION

Event detectors can produce accurate boxes that are stale by the time robotic systems receive them. ChronoFuse predicts object states at output availability using causal temporal fusion, recovering latency-lost accuracy across driving, drone, and bird/insect benchmarks.

I. INTRODUCTION

  • Event cameras offer microsecond-scale temporal resolution and little motion blur, but sensing speed alone does not make predictions timely because processing delays their availability.The paper frames the needed prediction around the state available to downstream planning and control.Source: §I · p. 1
  • Offline evaluation compares detections with annotations at observation time, while robotic systems receive them after latency, when objects may have moved, appeared, or left the scene.Even perfect boxes from time t retain just 72.96 mAP on 1Mpx after 33.3 ms.Source: §I · p. 1
  • Prior streaming methods compensate for delay in frame-based inputs, while event detectors use temporal memory but still describe the latest observation rather than availability-time scenes.The paper identifies event-specific temporal representations as a reason frame-centric designs do not directly fit.Source: §I · p. 1
  • ChronoFuse predicts classes and boxes for output availability by fusing current and cached multi-scale features causally, adding 0.17M parameters and 0.84 ms mean end-to-end latency.It decodes availability-time detections directly, without using future observations or extrapolating completed boxes.Source: §I · pp. 1–2
  • 71% of latency-lost accuracy is recovered on 1Mpx, 90.8% on FRED, and EV-Flying reaches 20.95 versus 2.25 sAP (9.3×).The FRED result nearly restores same-time performance; EV-Flying tests extreme bird and insect motion.Source: §I · p. 2

II. RELATED WORK

Event detection has advanced in accuracy and efficiency, but existing detectors remain observation-time predictors. Streaming approaches compensate for delay in RGB video, while event-based work has focused on faster processing, tracking, or post-detection motion compensation.

A. Event-Based Object Detection

  • Event detectors progressed from recurrent and RVT models to sparse-attention and state-space designs, but remain observation-time predictors; RED’s one-step-ahead regression is only auxiliary.SSLA-Det’s evaluation is limited to the smaller Gen1 and N-Caltech101 benchmarks.Source: A. Event-Based Object Detection · pp. 2–3

B. Streaming and Latency-Aware Perception

  • RGB streaming methods predict future detections or fuse temporal features to offset inference delay, but rely on regularly sampled frames and frame-centric designs –, [11],.Transtreaming extends prediction across multiple future horizons.Source: B. Streaming and Latency-Aware… · p. 3
  • Event-based alternatives reduce processing delay, track an initialized target, extrapolate completed detections, or forecast events; ChronoFuse instead predicts detections directly at availability time –.Motion extrapolation depends on association and motion assumptions and cannot recover objects missed before extrapolation; event forecasts require another inference stage.Source: B. Streaming and Latency-Aware… · p. 3

III. METHOD

ChronoFuse treats detector latency as a temporal-alignment problem: it predicts the scene at output availability and uses causal cross-time fusion to capture recent evolution. Its inputs contain no events after the observation cutoff.

III. METHOD

  • The Cross-Time Fusion Module combines the current feature pyramid with the preceding cached pyramid before passing the fused representation to the detection head.The recurrent encoder produces the current pyramid and updates its state; the RVT encoder and feature pyramid instantiate this network.Source: §III · p. 3, A. Latency-Compensated Detection · p. 3, Figure 2 · p. 4

A. Latency-Compensated Detection

  • At each cutoff, the causal input contains events from the preceding integration window and excludes every event occurring after that cutoff.The event representation encodes each event's pixel location, timestamp, and polarity.Source: A. Latency-Compensated Detection · p. 3
  • ChronoFuse shifts its target from the scene at observation time to the scene at the prediction horizon, while keeping the same causal observation support.The future target is the valid annotation closest to t_k + ∆; events in (t_k, t_k + ∆] remain unavailable.Source: A. Latency-Compensated Detection · pp. 3–4
  • The detection head produces raw box, objectness, and class predictions, which are decoded and post-processed to obtain the final prediction for t_k + ∆.The horizon, integration window, processing interval, and measured runtime are distinct temporal quantities.Source: A. Latency-Compensated Detection · pp. 3–4

B. Future-Target Supervision

Future-target supervision trains ChronoFuse to predict object configurations one prediction horizon after the causal observation cutoff.

B. Future-Target Supervision

  • ChronoFuse is trained against the object configuration one prediction horizon after the observation cutoff, rather than the observed configuration.The target is the valid annotation temporally closest to t_k + ∆ within the same sequence.Source: B. Future-Target Supervision · p. 4
  • The target annotation is selected by temporal proximity to t_k + ∆ and must be valid and from the same sequence.Source: B. Future-Target Supervision · p. 4

C. Cross-Time Fusion Module

The Cross-Time Fusion Module combines cached and current multi-scale features through a learned residual correction, while preserving the current representation.

C. Cross-Time Fusion Module

  • Cached temporal information is limited to previously computed features, and boundary initialization prevents information from leaking across sequences.Source: C. Cross-Time Fusion Module · p. 4
  • The module projects cached and current features at each pyramid level, concatenates them in past–present order, and adds the result to the current feature.Projection parameters are shared between the cached and current branches but are independent across pyramid levels; each branch produces Cℓ/2 channels.Source: C. Cross-Time Fusion Module · p. 4
  • The residual path retains the complete current representation, and the module learns its correction directly from future-target supervision without explicit flow, velocity, or correspondence estimation.Source: C. Cross-Time Fusion Module · p. 4
  • At the first step, the initial model preserves the pretrained detection function, allowing optimization to learn the temporal residual progressively.Source: C. Cross-Time Fusion Module · p. 4

D. Optimization and Causal Execution

Optimization uses future boxes for assignment while retaining the original detection losses; causal execution maintains and resets temporal state and feature caches.

D. Optimization and Causal Execution

  • For the YOLOX-style head, SimOTA assignment is recomputed using future boxes while the regression, objectness, and classification losses remain unchanged.Source: D. Optimization and Causal Execution · p. 4
  • Training steps without a valid future target contribute no detection loss but still update the temporal state and feature cache.Source: D. Optimization and Causal Execution · p. 4
  • The FT-only ablation retains future targets, detector, and training objective while removing fusion, isolating the contribution of explicit cross-time feature fusion.Source: D. Optimization and Causal Execution · p. 5
  • ChronoFuse caches only the unfused current pyramid, does not recursively accumulate temporal residuals, and resets its cache and temporal state at sequence boundaries.It requires no additional encoder evaluation and stores one feature pyramid per stream.Source: D. Optimization and Causal Execution · p. 5

IV. EXPERIMENTS

The experiments test whether predicting object locations ahead of the current event timestamp can offset inference-latency accuracy loss, outperform motion extrapolation, and remain effective as motion increases.

IV. EXPERIMENTS

  • The experiments assess latency degradation, learned future prediction versus motion extrapolation, robustness to faster motion, and the contributions of future-fusion components.These four questions define the evaluation scope.Source: §IV · p. 5

A. Experimental Setup

The evaluation spans driving, aerial, and fast biological targets, using streaming metrics that score detections when predictions become available. It compares event detectors and motion baselines at a common latency horizon and measures ChronoFuse’s computational overhead.

A. Experimental Setup

  • The datasets span driving scenes (1Mpx), small fast-moving drones (FRED), and rapidly moving birds and insects (EV-Flying).The progression tests latency compensation as motion becomes less constrained and object scale decreases.Source: A. Experimental Setup · p. 5
  • Streaming AP evaluates detection accuracy at prediction availability, with COCO-style sAP50:95, sAP50, and sAP75 reported as percentages.The evaluation compares predictions with the scene at t + ∆ rather than the input timestamp.Source: A. Experimental Setup · p. 5
  • The common streaming horizon is ∆ = 33.33 ms, and all timings include representation construction, inference, transfer, decoding, and NMS.The horizon lies within contemporary event-detector end-to-end latencies; 1Mpx uses 50 ms event windows and FRED and EV-Flying use 33.33 ms windows.Source: A. Experimental Setup · p. 5
  • ChronoFuse adds 0.84 ms (2.84%) mean end-to-end latency and 0.17M parameters (0.93%) relative to RVT-B.It adds a 1.64 MiB feature cache per stream, requires no additional encoder pass, and keeps mean runtime below the 33.33 ms horizon.Source: A. Experimental Setup · p. 5
  • The baselines include RVT, SAST, S5-ViT, SMamba, and causal Constant Velocity and Kalman extrapolation on RVT detections.All extrapolation variants use only information available up to t and predict at t + ∆.Source: A. Experimental Setup · p. 5

B. Main Results

ChronoFuse recovers latency-related accuracy across driving, drone, and fast biological-target benchmarks, with larger benefits under more severe motion. The results also show improved localization and tighter qualitative alignment than RVT and RVT with Kalman extrapolation.

B. Main Results

  • 46.20 sAP on 1Mpx recovers 71.8% of RVT’s latency-induced loss and exceeds RVT at prediction availability by 3.03 sAP.The 33.33 ms delay reduces RVT from 47.39 to 43.17 sAP; ChronoFuse comes within 1.19 points of the zero-delay reference.Source: B. Main Results · p. 5, Table II · p. 6
  • ChronoFuse improves precise localization: sAP75 increases from 42.84 to 47.60, a 4.76-point gain at the stricter threshold, indicating improved spatial alignment.Class AP improvements are also consistent across pedestrians, two-wheelers, and cars: 3.46, 3.18, and 2.45 points, respectively.Source: B. Main Results · pp. 5–6
  • ChronoFuse maintains tighter spatial alignment than RVT and RVT + Kalman across near-static and fast-moving-drone examples at the same availability horizon.The figure distinguishes ground truth with green dashed boxes and predictions with solid magenta boxes.Source: Figure 3 · p. 6
  • Latency-related RVT loss on 1Mpx rises from 0.53 sAP for low motion to 2.23 for medium motion and 10.06 for high motion; ChronoFuse recovers 7.76 sAP in the high-motion group.It restores 77% of the high-motion loss; the corresponding high-motion sAP75 loss reaches 15.44 points.Source: B. Main Results · p. 6
  • 44.99 sAP on FRED recovers 90.8% of latency-induced loss, compared with 26.88 sAP for RVT at the 33.33 ms availability horizon.ChronoFuse remains within 1.83 points of the 46.82 sAP observation-time reference.Source: B. Main Results · p. 7
  • 20.95 sAP—versus at most 2.25 sAP for existing event detectors—on EV-Flying under prediction-availability evaluation, a 9.3× gain over the strongest existing-detector result.EV-Flying is the most demanding regime, with tiny birds and insects moving a large fraction of their box size within one detection interval.Source: B. Main Results · p. 7
  • Across benchmarks, stale-detection degradation grows with target motion severity, while predictive latency compensation becomes increasingly valuable.The reported trend is summarized in Fig. 4(a,b).Source: B. Main Results · p. 7

C. Comparison with RGB Streaming Detection

On synchronized FRED sequences, ChronoFuse outperforms the RGB streaming reference under matched timing, though the modalities and architectures differ. Mixed-speed FRED training improves average sAP across speeds, while performance degrades under the largest speed shift.

C. Comparison with RGB Streaming Detection

  • 44.88 sAP for ChronoFuse versus 22.76 sAP for StreamYOLO on synchronized FRED sequences, a 22.12 sAP gain.The comparison is a matched-timing cross-modal reference, not a controlled modality ablation; ChronoFuse also gains 29.12 sAP75.Source: C. Comparison with RGB Streaming… · p. 7

D. Motion-speed robustness

  • Mixed-speed training raises ChronoFuse’s Avg. sAP from 21.01 to 33.39, versus 12.10 for RVT and 11.82 for SMamba.ChronoFuse retains 30.54 and 23.21 sAP at 3× and 4× speed, respectively, with only a modest reduction in nominal-speed accuracy; Avg. sAP averages all four speeds.Source: D. Motion-speed robustness · p. 7, Table III · p. 7
  • The ablation table reports sAP on 1Mpx and EV-Flying at ∆ = 33.33 ms.The supplied caption gives the datasets and horizon but no ablation values.Source: Table IV · p. 7

E. Ablation Study

ChronoFuse recovers much of the latency-induced accuracy loss across driving and drone data, and both temporal fusion and the residual path contribute to its ablation performance. The conclusion also reports improved streaming accuracy on synchronized FRED data and identifies fixed-horizon prediction as a scope for future work.

E. Ablation Study

  • ChronoFuse scores 46.20 sAP on 1Mpx and 20.95 sAP on EV-Flying, versus 45.75 and 16.54 without fusion and 44.97 and 18.82 without the residual path.The ablation uses the same future-target protocol; FT-only bypasses fusion, while the no-residual variant removes the current-feature identity path.Source: E. Ablation Study · p. 7

V. CONCLUSION

  • ChronoFuse recovers approximately 71% and 91% of RVT’s latency-induced accuracy loss on 1Mpx and FRED, respectively.Source: §V · p. 7
  • On synchronized FRED data, ChronoFuse achieves nearly twice the streaming accuracy of RGB StreamYOLO under the same evaluation protocol.Source: §V · p. 8
  • The conclusion identifies latency-conditioned models that adapt predictions to changing computational loads and system delays as future work.Source: §V · p. 8
Loading 2609.26919v1…