PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

Zetian Song, Qize Yu, Yue Chen, Bohan Zhou, Bowen Ping, Yuqi Ye, Haocheng Tang, Haotian Zhang, Qizhe Wang, Shiyu Huang, Jiaqi Zhang, Siwei Ma, Wen Gao

arXiv:2610.33079v1cs.CVeess.IV

TL;DR

Random temporal sampling in AI training exposes a mismatch with video storage designed for sequential playback, where decoding can require unrequested frames and reference chains. The paper co-designs dependency-aware sparse decoding with optional DDRA encoding to reduce that work while retaining inter-frame prediction. At comparable reconstruction quality, its H.265 variant achieves 1.45× LeRobot v3 throughput with 58.7% less media payload.

  • Problem

    Randomly sampled video-training workloads face a mismatch with playback-oriented codecs, while independent frames and short GOPs reduce decoding work at the cost of larger storage and transfer volumes.

    Source: §1 · pp. 1–2

  • Method

    The method co-designs online dependency-aware sparse decoding, which skips unrelated frames, with optional offline DDRA encoding that limits each interior frame to two intra-coded anchors.

    Source: §1 · p. 2

  • Results

    1.45× the throughput of LeRobot v3 with 58.7% less media payload is achieved by the H.265 variant at comparable reconstruction quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On the reported benchmark, bounded reference dependencies provide a practical operating point between playback-oriented compression and image-oriented random access.

    Source: §5 · p. 10

    Longer intra intervals reduce anchor frequency but can increase interior-frame bits, and removing B-to-B references limits prediction flexibility.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Large-scale AI training increasingly relies on massive video datasets with random temporal sampling patterns. However, existing video storage paradigms are optimized for sequential playback and employ sophisticated inter-frame reference structures to maximize compression efficiency. When applied to the random-sampling workloads of AI training, these paradigms face substantial decoding-efficiency bottlenecks. Workarounds such as independent JPEG frames and short groups of pictures (GOPs) reduce decoding work at the cost of larger storage and transfer volumes. In this paper, we identify two sources of decode amplification: sequential reconstruction of unrelated frames and reconstruction of the reference chains required by sampled targets. We address both sources by co-designing the decoding schedule and video reference structure. Dependency-aware sparse decoding reconstructs only requested frames and their required references, skipping unrelated reconstruction without transcoding the source videos. Optional offline Dependency-Decoupled Random Access (DDRA) encoding further reduces reference reconstruction by restricting each interior frame to two bounding intra-coded anchors. Together, sparse decoding and DDRA retain inter-frame prediction while bounding reconstruction to at most three frames per requested target, independent of the intra period. Experiments on a large-scale robotics corpus show that, at comparable reconstruction quality, our method achieves $1.45\times$ the loading throughput of LeRobot v3 with 58.7% less media payload.

1 Introduction

AI training samples videos randomly, but conventional video storage is designed for sequential playback, creating a tension between compact representation and efficient temporal access. The paper co-designs sparse decoding and video reference structures to reduce decoding work while retaining inter-frame prediction.

1 Introduction

  • Training repeatedly samples short windows or sparse frames, shuffles requests across videos, and revisits long recordings, so sequential throughput alone is insufficient.Storage must combine compact representation, fine-grained temporal access, and sustained delivery to accelerators.Source: §1 · p. 1
  • Playback-oriented codecs amplify decoding through sequential reconstruction of unrequested frames and reconstruction of transitive B-frame reference chains.Both consume CPU time without directly producing requested training samples.Source: §1 · p. 1
  • Independent images and mini-GOPs better match sparse access but sacrifice inter-frame redundancy, increasing persistent-storage use and training bandwidth demand.The trade-off is either eliminating temporal dependencies or using frequent intra-coded anchors.Source: §1 · pp. 1–2
  • Dependency-aware sparse decoding skips unrelated frames, while optional DDRA restricts each interior frame to two bounding intra-coded anchors, bounding reconstruction to the target and its anchors.The approach retains inter-frame prediction and the decoded-frame interface expected by existing training pipelines.Source: §1 · p. 2
  • 1.45× the throughput of LeRobot v3 with 58.7% less media payload at comparable reconstruction quality is achieved by the H.265 variant.The comparison uses average loading times across four access patterns; Figure 1 summarizes the storage–throughput trade-off.Source: §1 · p. 2, Figure 1 · p. 2

2 Background and Motivation

Video coding exploits temporal redundancy through reference pictures, but those dependencies can require reconstructing unrequested frames. The paper frames random-access training as a co-design problem spanning data organization, encoded references, and decoding schedules.

2.1 Video Coding Framework

  • The proposed co-design combines dependency-aware sparse decoding with DDRA encoding to eliminate unrelated reconstruction and bound each interior target’s dependency closure while retaining inter-frame prediction.The figure contrasts hierarchical coding with intermediate B-frame references against DDRA’s two-anchor structure.Source: §2.1 · p. 2, Figure 2 · p. 4
  • Temporal prediction compresses video using reconstructed reference pictures, but hierarchical B-frame references can create chains that sparse requests must reconstruct.References must be decoded before dependents, so coding order can differ from presentation order.Source: §2.1 · pp. 2–3
  • Selective decoding can remove unrelated pictures in a sequentially decoded prefix, but it cannot shorten the encoded dependency chains.Existing loaders improve seeking and parallelism while leaving the reference graph unchanged.Source: §2.1 · p. 3

2.2 Existing Data Organizations and Pipelines

  • Training data is transferred from storage and decoded into RGB tensors, making both I/O and decoding potential bottlenecks.Data organization affects sample access, while visual encoding affects transfer and decoding cost.Source: §2.2 · p. 3
  • Inter-frame coding reduces transfer volume but may require extra reference reconstruction; independent frames remove dependencies but require larger payloads, while short GOPs compromise between them.The paper’s LeRobot v3 baseline uses AV1 with a two-frame GOP.Source: §2.2 · p. 3

3 Method

The method co-designs dependency-aware decoding and an optional encoding configuration to reduce unrelated and reference-frame reconstruction. DDRA limits each interior frame to two bounding intra-coded anchors, while sparse decoding reconstructs only requested frames and their dependencies.

3.1 Dependency-Aware Sparse Decoding

  • Sparse decoding parses the reference graph, traverses the dependency closure in decoding order, and reconstructs each required frame once while skipping unrelated pixel reconstruction.A state-only path for skipped frames preserves decoder state; on a validated reference graph, the schedule removes unrelated reconstruction.Source: §3.1 · pp. 4–5

3.2 Dependency-Decoupled Encoding

  • DDRA restricts each interior frame to two bounding intra-coded anchors, making reference depth independent of temporal distance and bounding reconstruction to at most three frames per target.Longer intra periods reduce anchor frequency, but can require more bits for distant predictors; removing B-to-B references also limits prediction flexibility.Source: §3.2 · p. 5

3.3 Decode-Amplification Analysis

  • For K interior targets in one interval, DDRA reconstructs K + 2 pictures, giving amplification 1 + 2/K; across intervals, cost depends on distinct anchors.The request pattern and intra period jointly determine how far amplification falls below the per-frame upper bound.Source: §3.3 · p. 6
  • At stride 30, DDRA + Sparse lowers expected decode amplification to 2.01, versus 5.04 for Sparse and 16.40 for Sequential.At d = 4, increasing K from 1 to 30 lowers DDRA’s expectation from 2.94 to 1.16; at K = 10 and d = 4, increasing TI from 4 to 32 lowers it from 1.83 to 1.28, while Sparse changes from 2.33 to 2.63.Source: §3.3 · p. 6

4 Evaluation

The evaluation compares video-loading efficiency, storage cost, and downstream training performance, then tests how access patterns, bandwidth, and intra-period settings affect results. DDRA improves loading efficiency while retaining comparable task performance at substantially lower media payload than LeRobot v3.

4.2 Loading Efficiency and Storage Cost

  • 2.22× and 2.12× lower random-access loading time: DDRA H.264 and H.265 reduce batch time from 297.8 to 134.1 and 140.6 ms, respectively.Both also have lower mean and p95 loading times than LeRobot v3 and JPEG in every main-table pattern.Source: §4.2 · p. 7
  • 1.42–1.45× LeRobot v3’s throughput with 29.8–58.7% less payload: DDRA H.264/H.265 improve loading throughput at aligned quality.Against JPEG, DDRA provides 1.99–2.02× throughput with 83.0–90.0% less payload; H.265 uses less storage than H.264 at comparable loading times.Source: §4.2 · p. 7

4.3 Training Performance under Compression

  • At approximately 100× compression relative to Original, DDRA H.265 shows no statistically significant success-rate change for VLA or WAM, while using approximately 50% less storage and media-transfer bandwidth than LeRobot v3.For VLM, transcoding does not necessarily reduce downstream accuracy when video quality remains high.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Ablation Study

  • At a 100 MiB/s node-wide cap, DDRA H.264 and H.265 load 6.38 and 10.79 ep/s, versus 4.36 ep/s for LeRobot v3 and 1.06 ep/s for JPEG.At 800 MiB/s, DDRA H.264 exceeds Ours native in loading throughput; Figure 5 reports average and p95 loading times and aggregate object-storage response throughput.Source: §4.4 · p. 8, Figure 5 · p. 9
  • 1.44–4.41× for H.264 and 1.59–3.27× for H.265: DDRA’s loading gains persist across all 27 tested access/resource configurations.The sweeps vary requested frame count, stride, random span, batch size, workers, and CPU allocation.Source: §4.4 · p. 8
  • At intra period 32, average loading time is 14.8% lower than at period 8, while payload is 4.7% higher; period 8 has the smallest payload and period 32 the lowest average loading time.A larger period lets sampled frames share keyframes, reducing distinct references decoded, but can increase bitrate as anchor distance grows.Source: §4.4 · pp. 8–9, Figure 6 · p. 9
  • 2.24× random-access speedup: sparse decoding reduces reconstruction from 233.9 to 92.3 frames per episode and loading time from 668.3 to 297.8 ms; adding DDRA reaches 43.1 frames and 134.1 ms.DDRA adds a 2.22× speedup over sparse decoding alone; across patterns, the combined design achieves 2.73–4.98× speedup over Sequential.Source: §4.4 · p. 9, Table 2 · p. 9

5 Conclusion

The conclusion presents workload-aware video storage as a way to balance compression with random access by modifying codec reference structures. On the controlled three-dataset benchmark, DDRA achieves high sampled-episode throughput with substantially less payload than JPEG, while the preferred representation depends on request span and bandwidth.

5 Conclusion

  • DDRA reduces random-access average loading time by 2.22× with H.264 and 2.12× with H.265 relative to native sparse H.264 loading.Across four access patterns, the representations achieve 64.66 and 65.75 ep/s while using 83.0–90.0% less media payload than JPEG.Source: §5 · p. 10
  • The controlled three-dataset benchmark identifies bounded reference dependencies as a practical operating point between playback-oriented compression and image-oriented random access.The preferred representation depends on request span and bandwidth.Source: §5 · p. 10
  • Reported loading rates do not establish full-tensor delivery rates or training-time speedups.Source: §5 · p. 10

AI use statement

  • Generative AI tools were used primarily to polish the manuscript and assist with code development and debugging.Source: AI use statement · p. 10

A.1 Corpus, Sampling, and Implementations

The appendix specifies the loading corpus, sampling patterns, representation implementations, quality calibration, timing methodology, and decoding validation. It reports that the combined sparse-decoding and DDRA design reduces reconstruction work and loading time, with pixel-identical outputs in its correctness tests.

A.1 Corpus, Sampling, and Implementations

  • The corpus contains 3,000 episodes and 8,908 camera streams from RealSourceWorld, RoboCasa, and Table30v2, with every camera and the full episode duration retained.Each request selects ten frames per camera; Random samples ten frames without replacement from a deterministic 100-frame window, while other patterns use consecutive or fixed-stride timesteps.Source: §A.1 · p. 14
  • The comparison uses a common episode-level Lance-on-OSS path; native decoders read original bitstreams, while JPEG, LeRobot v3, and DDRA use their specified transcoded representations.Ours native uses sparse decoding; Ours-DDRA combines sparse decoding with open-GOP H.264/H.265 encoding at intra period 32.Source: §A.1 · p. 14

A.2 Reconstruction-Quality Calibration

  • All selected transcoded settings are within 0.18 dB of LeRobot v3 in dataset-macro-averaged RGB PSNR, a calibration of reconstruction quality rather than downstream learning performance.The caption identifies Table 3 as reporting quality settings and dataset-macro-averaged RGB PSNR.Source: §A.2 · p. 14, Table 3 · p. 14

A.3 Timing, Aggregation, and Throughput

  • The main loading measurements include remote I/O, reference-graph construction, RGB decoding, and metadata transfer, but exclude full RGB-tensor transfer, GPU/model computation, and offline conversion.Mean and p95 are computed within each dataset and averaged equally; reported p95 is not the percentile across all pooled batches.Source: §A.3 · pp. 14–15
  • 2.73×, 4.98×, and 4.33×: the combined design’s loading speedups over Sequential for Contiguous, Random, and Stride-4, respectively.Across patterns, sparse decoding cuts loading time by 32.9–55.4% versus Sequential, and DDRA cuts it by a further 45.5–55.1% versus sparse decoding alone.Source: §A.3 · p. 15

A.4 Sparse-Decoding and Encoding Details

  • Sparse decoding’s validated path reconstructs only the dependency closure, using a header pass to track references and buffer state before pixel reconstruction.Overlapping execution ranges are merged so shared anchors are reconstructed once per call; DDRA uses open-GOP encoding with intra period 32.Source: §A.4 · p. 15

A.5 Decoding Correctness

  • All 100,000 returned frames in the correctness tests are pixel-identical to full sequential FFmpeg RGB24 decoding, with maximum absolute RGB error zero.The 10,000 requests cover first and last frames, open-GOP boundaries, and duplicate or out-of-order requests.Source: §A.5 · p. 15

B Training-Performance Protocol and Additional Results

The additional training-performance protocol sweeps compression settings across VLA, WAM, and VLM configurations. These task-performance sweeps use different original visual representations by family and are separate from the quality-aligned loading benchmark.

B Training-Performance Protocol and Additional Results

B.1 Data, Training, and Evaluation

The section specifies the corpora, training configurations, and evaluation workloads used for VLA, WAM, VLM, and LeRobot v3 comparisons.

B.1 Data, Training, and Evaluation

  • VLA uses 1,693 LIBERO episodes and 273,465 frames; Original stores lossless PNG observations totaling 32.5015 GiB.Original and DDRA use eight H800 GPUs, BF16, global batch size 256, and 30,000 optimizer updates; evaluation comprises 2,000 rollouts.Source: §B.1 · p. 16
  • WAM is fine-tuned from OpenWAM’s pretrained checkpoint on the same corpus, using its default LIBERO recipe and 2,000 evaluation rollouts.Its video backbone and ActionDiT use joint self-attention and mutual video/action visibility; evaluation covers four suites with a 600-step limit, extended to 700 for LIBERO-Long.Source: §B.1 · p. 16
  • LeRobot v3 uses two-frame-GOP AV1 at CRF20 and CRF50, plus VBR targets of 100× and 200×, in VLA runs with 2,000 evaluation rollouts.The VBR settings achieve compression factors of 93.375× and 188.224×, respectively.Source: §B.1 · p. 16
  • VLM fine-tuning uses Qwen3.5-2B on 21,437 videos totaling 9,706,740 frames, with 1,005 optimizer updates and 3,800 MVBench questions per configuration.Training uses eight H800 GPUs, BF16, ZeRO-2, batch size 64, and three epochs; no inference failures are reported.Source: §B.1 · p. 16

B.2 Storage Metric and Task Performance

The section compares storage and task performance across compression settings, then interprets DDRA’s LIBERO results against Original using confidence intervals and plotted bands.

B.2 Storage Metric and Task Performance

  • Approximately 2.0× and 2.1× compression gains correspond to about 49% and 53% less storage for VLA and WAM, respectively.At a fixed video-consumption rate, media-transfer bandwidth decreases in the same proportion.Source: §B.2 · p. 16
  • S = 1/C; S below 1 indicates storage reduction and S above 1 indicates expansion, with compression anchors of PNG for LIBERO and H.264 for VLM.Figure 4 uses a logarithmic compression axis, and its insets match success rates by interpolation in log C.Source: §B.2 · p. 16
  • VLM accuracy at CRF18 and CRF28 is close to or above Original’s 65.87%, while CRF48 reaches 63.82% for H.264 and 64.03% for H.265.The results support preserving VLM performance in the high-video-quality regime.Source: §B.2 · p. 16

B.3 Statistical Interpretation

  • At 101.383× compression, DDRA (H.265) at CRF28 has VLA and WAM SR-difference intervals of [−0.84, 1.24] and [−0.73, 0.43] percentage points; both include zero.The reported p-values are 0.706 and 0.611, so the changes are not statistically significant under the two-proportion normal approximation.Source: §B.3 · p. 17

C Access and Resource Sensitivity Details

The access/resource sweeps examine how loading time changes with requested frames, temporal access patterns, batch size, workers, and CPU allocation. At fixed 16-core allocation, increasing workers from eight to 64 reduces DDRA H.265 mean loading time from 358.9 to 46.7 ms/batch.

C Access and Resource Sensitivity Details

  • The sweeps vary requested frame count, fixed stride, random candidate span, batch size, worker count, and CPU allocation from a ten-frame baseline configuration.There are 27 distinct configurations, each evaluated with all eight methods; Figure 7 displays six methods and omits batch-size-two and eight-DataLoader-worker configurations.Source: C Access and Resource Sensitivity… · p. 17
  • 358.9 to 46.7 ms/batch: increasing workers from eight to 64 at 16 cores reduces DDRA H.265 mean loading time.Increasing CPU allocation from eight to 32 cores with 32 workers reduces mean loading time from 132.6 to 71.1 ms/batch; 64 workers do not require 64 CPU cores.Source: C Access and Resource Sensitivity… · p. 17
  • The measurements characterize combined I/O, decoding, and scheduling trends, and small nonmonotonic changes between adjacent configurations are not assigned a causal interpretation.They do not isolate decoder-kernel scaling or provide confidence intervals.Source: C Access and Resource Sensitivity… · p. 17
  • Figure 7 reports average loading time over 100 measured batches, macro-averaged across datasets, with lower values indicating faster loading.Panels share a linear time scale; each point uses ten warm-up batches and measured batches split 34/33/33 across the three datasets.Source: C Access and Resource Sensitivity… · p. 17, Figure 7 · p. 18

D Bandwidth Experiment Details

The bandwidth experiment evaluates all methods’ Lance reads from object storage under node-wide response caps. Loading times are measured separately from proxy throughput, whose accounting window includes startup, prefetch, warm-up, and shutdown.

D Bandwidth Experiment Details

  • The experiment applies node-wide object-storage response caps of 20, 50, 100, 200, 400, and 800 MiB/s to all methods’ Lance reads.Reads pass through a caching-disabled proxy, use ten contiguous frames per camera, and follow the same deterministic episode sequence across methods and caps.Source: D Bandwidth Experiment Details · p. 18
  • Average and p95 loading times use only the 100 measured batches, following ten warm-up batches.Measured batches are split 34/33/33 across the three datasets.Source: D Bandwidth Experiment Details · p. 18
  • Proxy throughput divides total response bytes by elapsed time from iterator startup through worker shutdown, so it differs from Figure 1 episode throughput in unit and accounting window.The configured cap bounds available bandwidth.Source: D Bandwidth Experiment Details · p. 18

Related papers

Loading 2610.33079v1…