PaperlayerView exact revision on arXiv

Source-linked AI summary

DeDelayed: Deleting Remote Inference Delay via On-Device Correction

Dan Jacobellis, Mateen Ulhaq, Fabien Racapé, Hyomin Choi, Neeraja J. Yadwadkar

arXiv:2510.13714v3eess.IVcs.AIcs.CVcs.LG

TL;DR

Real-time video inference must balance costly local processing with cloud-model delay, which can undermine deadline-bound predictions. DeDelayed combines future-predictive remote features with fresh local inference; across realistic network conditions, it surpasses strong local-only and remote-only baselines, particularly at longer latencies and with high-motion content.

  • Problem

    Existing real-time video systems may lack a local fallback, fail to account for latency’s effect on accuracy, or reduce video resolution to control computation.

    Source: §1 · p. 1

  • Method

    DeDelayed conditions a remote video model on delay and fuses its predictive features with features from a local model processing the current frame.

    Source: §1 · p. 2

  • Results

    Across realistic network conditions, DeDelayed surpasses strong local-only and remote-only baselines, with particular advantage at longer latencies and on high-motion content.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The framework combines remote semantic features with fresh local alignment in a driving-scene segmentation system that avoids pitfalls limiting prior approaches’ practicality.

    Source: §1 · p. 2

    The stated guarantee of performance no worse than either subsystem depends on training them to their conventional maximum accuracies, for example by freezing image backbones during final joint training.

    Source: §4 · p. 4

Abstract

from arXiv · show

Video comprises the vast majority of bits that are generated daily, and is the primary signal driving current innovations in robotics, remote sensing, and wearable technology. Yet, the most powerful video understanding models are too expensive for the resource-constrained platforms used in these applications. One approach is to offload inference to the cloud; this gives access to GPUs capable of processing high-resolution videos in real time. But even with reliable, high-bandwidth communication channels, the combined latency of video encoding, model inference, and round-trip communication prohibits use for certain real-time applications. The alternative is to use fully local inference; but this places extreme constraints on computational and power costs, requiring smaller models and lower resolution, leading to degraded accuracy. To address these challenges, we propose Dedelayed, a real-time inference system that divides computation between a remote model operating on delayed video frames and a local model with access to the current frame. The remote model is trained to make predictions on anticipated future frames, which the local model incorporates into its prediction for the current frame. The local and remote models are jointly optimized with an autoencoder that limits the transmission bitrate required by the available downlink communication channel. We evaluate Dedelayed on the task of real-time streaming video segmentation using the BDD100k driving dataset. For a round trip delay of 100 ms, Dedelayed improves performance by 6.4 mIoU compared to fully local inference and 9.8 mIoU compared to remote inference -- an equivalent improvement to using a model ten times larger. We release our training code, pretrained models, and python library at https://github.com/InterDigitalInc/dedelayed .

1. Introduction

Real-time video inference must balance the accuracy of cloud models against the delay and fallback constraints of hard deadlines. DeDelayed combines future-predictive remote features with current-frame local inference to address that trade-off.

1. Introduction

  • Hard real-time applications need a guaranteed deadline because late outputs can be catastrophic, while soft real-time applications can still use late outputs.In soft real-time settings, low typical latency can make cloud offloading worthwhile for on-device power savings.Source: §1 · p. 1
  • Existing real-time video systems may lack a local-only fallback, ignore latency’s effect on accuracy, or reduce spatiotemporal resolution and omit visual detail.The paper frames these as three common pitfalls of existing approaches.Source: §1 · p. 1
  • DeDelayed fuses delayed, high-level remote features with fresh local features that align them to the current scene.The local model can operate without remote outputs; the remote model processes high-resolution frames with a 3D transformer that understands motion.Source: §1 · p. 2
  • The authors report a driving-scene video segmentation system that outperforms local or remote inference while avoiding pitfalls that limit prior systems’ practicality.The framework combines a future-predicting remote model with the current input to a local model.Source: §1 · p. 2

2. Background

Video sensors can capture rich, high-resolution streams efficiently, but processing those streams locally is costly. Remote processing offers greater compute, while delay and network unreliability motivate local fallback and temporally predictive inference.

2. Background

  • Video sensors ingest high-resolution streams efficiently, but advanced video models can consume far more energy per pixel, motivating transmission to cloud GPUs.The passage contrasts sensor ingestion at tens or hundreds of megapixels per watt-second with advanced video AI at as few as 500 pixels per watt-second.Source: §2 · pp. 2–3
  • A pixel-predicting 3D transformer captures motion in objects, signs, and road markings, but not high-frequency details.Figure 2 compares an original frame, a future-frame difference, the transformer’s pixel prediction, and its difference from the true future frame.Source: Figure 2 · p. 3
  • Combining remote and local activation maps can classify and localize distant pedestrians that are difficult to discern at low resolution.The remote server uses higher video detail for classification, while the local model adjusts positions using the current frame.Source: Figure 3 · p. 3
  • Remote processing can be unreliable for real-time tasks, so deadline-bound systems need a fallback independent of the server.The passage notes that limited local-model accuracy and reliability can leave human operators as fallback in some systems.Source: §2 · p. 3

3. Related work

Related work addresses real-time video perception through efficiency improvements, offloading, temporal prediction, and remote collaboration. DeDelayed is positioned as combining dynamic delay conditioning with on-device feature fusion.

Improving the accuracy-efficiency trade-off

  • For low power budgets below 10 Watts, prior local systems may reduce spatial or temporal resolution by orders of magnitude, significantly reducing accuracy.At moderately powered hardware levels of 10–100 Watts, recent 2D models can segment high-resolution images at greater than 30 fps.Source: §3 · p. 3

Offloading computation

  • Many offloading methods allocate local computation to a pipeline that ends in the cloud, risking stale on-device predictions and leaving little room for local fallback.This trade-off may be acceptable when predictions are only used in the cloud, such as remote sensing.Source: §3 · pp. 3–4

Temporally predictive video modeling

  • Prior temporal-prediction methods repurpose mechanisms from other applications, rather than optimizing predictive behavior jointly with the target perception task.Examples include optical flow, codec motion vectors, and generative video world models.Source: §3 · p. 4

Dynamically augmenting the on-device model via remote collaboration during streaming inference

  • Remote collaboration methods augment on-device models, but Knowledge Boosting requires models trained for fixed delays and was evaluated on audio in a controlled local network.DeDelayed instead conditions its remote model on measured runtime delay to adapt across varying latency.Source: §3 · p. 4

4. Method

DeDelayed combines fresh local inputs with delayed, delay-aware remote predictions to produce timely outputs. The two models are trained end to end against the current frame’s task loss.

4. Method

  • The local and remote subsystems provide complementary capabilities: the local model has current data but limited capacity, while remote inference is accurate but delayed.When trained to preserve each subsystem’s conventional accuracy, DeDelayed is described as never worse than either one alone.Source: §4 · p. 4
  • DeDelayed fuses a fresh frame with delayed remote features in a lightweight local model to produce a timely current-frame result.The remote model processes earlier frames and measured delay; the client pairs its response with the current frame.Source: §4 · p. 4, Figure 4 · p. 4
  • The remote model predicts from a short clip ending at t−τ, conditioning its output on the measured delay τ.The resulting remote features are combined with the current frame by the local model.Source: §4 · pp. 4–5
  • The full system is trained end to end with a task-specific loss comparing its current-time prediction against current-frame ground truth.For semantic segmentation, the loss is typically cross-entropy.Source: §4 · p. 5

5. Design and implementation

DeDelayed combines a high-resolution remote video model with a lightweight local model that processes the latest frame. Delay-conditioned remote features are adapted for downlink transmission and fused into the local model.

System overview

  • The system sends frames to a remote model, then fuses its returned features with a fresh local frame before producing labels.The remote model maintains a context window and conditions features on measured delay; the local model completes the decode.Source: §5 · p. 5

Latency-aware remote video model

  • The remote model uses compressed high-resolution video, a 2D ViT backbone, temporal 3D ViT layers, and a delay embedding to predict a future-frame target.Training samples delays from 0 to 5 frames; the implementation uses four context frames and targets 33 ms latency on a GPU testbed.Source: §5 · p. 5

Local image model incorporating remote side information

  • The local model processes the latest frame at low resolution, with remote features added to its intermediate activation map.It is designed to operate up to 704 × 480 resolution while meeting a 33 ms target on a CPU testbed.Source: §5 · p. 6

Joint prediction with autoencoded resolution adapter

  • A dimensionality-reducing autoencoder adapts remote outputs to the local model’s resolution and channels for downlink transmission, after which both models are jointly trained.Adaptive spatial pooling and a channel bottleneck produce a low-resolution, low-channel activation map.Source: §5 · p. 6

6. Evaluation

The evaluation tests real-time driving-scene segmentation on BDD100K under delays up to five frames. Training first adapts pretrained models to segmentation and then jointly trains local and remote components for the target setup.

Experimental setup

  • Evaluation uses BDD100K driving video at 30 fps with the standard 19-label Cityscapes taxonomy and generated training and validation pseudo-labels.DepthAnything supplies validation pseudo-labels, while EoMT labels 70,000 training images.Source: Experimental setup · p. 6
  • The system is evaluated at known delays from 0 to 5 frames, or 0 to 167 ms at 30 fps.Training samples the delay uniformly over that range.Source: Experimental setup · p. 6

Training details

  • Training proceeds from ImageNet classification and Cityscapes segmentation pretraining to BDD100K fine-tuning, followed by joint local-remote training.The remote model uses a delay-aware future-label objective; the final stage replaces task-specific MLP layers with an entropy bottleneck and resolution adapter.Source: Training details · pp. 6–7
  • Remote-model pretraining transforms a fixed context of K frames with per-frame 2D vision features, spatial merging, a delay embedding, and 3D video transformer layers.The output predicts the frame D frames ahead of the latest remote input.Source: Figure 6 · p. 7

7. Results

DeDelayed largely preserves accuracy as network delay increases, and at the maximum tested delay it outperforms local inference. Conventional remote inference degrades with delay, while temporal prediction extends its usefulness.

Results

  • Beyond 2 frames (67 ms), conventional remote inference falls below local inference, reaching 8%pt. mIoU lower at 167 ms; temporal prediction extends remote utility to 5 frames.At low communication delay, conventional remote inference is roughly 5%pt. mIoU above local inference.Source: §7 · p. 8
  • At 5 frames (167 ms), DeDelayed improves mIoU by 6.7%pt. over the local baseline, a difference typically requiring a model roughly 10 times larger.The figure plots segmentation accuracy (mIoU) against round-trip latency.Source: Figure 7 · p. 8, §7 · p. 8
  • With local inference delays of ≤8 ms, local plus remote predictive inference is more accurate across all tested network round-trip delays.The comparison covers varying local and remote subsystem delays.Source: §7 · p. 8

Additional experiments

  • The most recently received remote features generally provide the largest accuracy boost; if features are extremely stale or missing, the local model may ignore them.Source: §7 · p. 8

8. Conclusion

DeDelayed targets prediction staleness caused by network delay in real-time systems using remote computation. The conclusion highlights its performance across realistic conditions and outlines directions for future work.

8. Conclusion

  • DeDelayed addresses prediction staleness induced by network delay in real-time systems that rely on remote computation.It conditions the remote model on delay and fuses remote features with fresh local features.Source: §8 · p. 8
  • Across realistic network conditions, DeDelayed surpasses strong local-only and remote-only baselines, especially at longer latencies and with high-motion content.Source: §8 · p. 8
  • Future work includes variable and stochastic delay distributions, high-motion data, lighter local models, and local future prediction.Source: §8 · p. 8

Appendix

The code combines remote features conditioned on delay with a local model that processes the current input. Remote features are added to local activations before the remaining local prediction layers.

Appendix

  • The fused model sends current local input and remote input with delay to separate local and remote models, then returns the local prediction.Source: §8.1 · p. 11
  • The remote model adds a delay embedding to image features, processes them with a 3D video backbone, and produces features for fusion or segmentation logits.Source: §8.1 · p. 11
  • The local model adds remote features to its extracted features before applying its remaining layers and segmentation head.Source: §8.1 · p. 11
  • Training and evaluation offset frames and target labels by the appropriate delay; training minimizes per-pixel cross-entropy and evaluation uses mIoU.Source: §8.1 · p. 11

8.2. Additional architecture details

The appendix describes the delay embedding and 3D video backbone, then examines delay mismatch, local input resolution, delay conditioning, and uplink compression. It reports that delay mismatch is more damaging when the input delay underestimates observed delay, and that explicit delay conditioning improves mIoU.

8.2. Additional architecture details

  • A two-layer MLP maps the delay input from R to RC, and the 3D ViT uses 12 blocks with 256 input and output channels.Source: §8.2 · p. 11

8.3. Effect of delay jitter

  • Accuracy peaks when input and observed delays match; when they differ, overestimating delay causes a smaller drop than underestimating it.The model was not explicitly trained for mismatched delays or delays beyond 5 frames, yet performs well at those out-of-distribution delays; a precomputed matrix can guide selection among received remote features or local-only fallback.Source: §8.3 · p. 11, Figure 8 · p. 11

8.4. Local input resolution

  • After fine-tuning at 224, 320, and 480 px, the remote-assisted local model achieves better accuracy under round-trip latency than other solutions, including at 224 px.The fused model was trained at 480 px and fine-tuned for 10 additional epochs at each resolution.Source: §8.4 · p. 12, Figure 9 · p. 12

8.5. Delay-input ablation

  • Removing delay conditioning causes an average drop of 0.015 mIoU for local 224 px, indicating that explicit delay inputs are beneficial.Source: §8.5 · p. 12, Table 4 · p. 12

8.6. Robustness to uplink compression

  • The evaluation tests robustness to x264/x265 uplink compression using veryfast, zerolatency, and bframes=0 settings.Source: §8.6 · p. 12, Table 5 · p. 12

8.7. Compute and runtime breakdown

The appendix covers component compute and runtime, evaluation on Nymeria, occlusion examples, and validation pseudolabels. It reports that DeDelayed alone detects a newly unoccluded object in the illustrated example and describes the sampled pseudolabels as close to human-annotated quality, with exceptions.

8.7. Compute and runtime breakdown

  • Table 6 presents a compute and runtime breakdown by component.Source: Table 6 · p. 12

8.8. Additional dataset evaluation

  • Models are trained on 35k Nymeria clips and evaluated on 2k clips from 25 disjoint sequences with unseen environments and actions, using semantic segmentation and the ADE20K taxonomy.Source: §8.8 · p. 12, Table 7 · p. 12

8.9. Occlusion

  • In the 167 ms RTT example, DeDelayed detects a newly unoccluded cyclist that methods without local final inference miss.Source: Figure 10 · p. 12, §8.9 · p. 12

8.10. Sample validation pseudolabels

  • The sampled validation pseudolabels are described as close to human-annotated quality, with a few exceptions and occasional speckles or jagged boundaries near partial occlusions.The figure shows 20 randomly selected samples colored with the Cityscapes palette.Source: §8.10 · pp. 12–13, Figure 11 · p. 14, Figure 12 · p. 14
Loading 2510.13714v3…