PaperlayerView exact revision on arXiv

Source-linked AI summary

Octree-based Video Representation

Rungui Zhou, Chuanzhi Zhou, Yuk-Kit Hou, Peng-Shuai Wang

arXiv:2609.33100v2cs.CV

TL;DR

Video models commonly use uniform grids despite variation in visual complexity across space and time. OctVideo represents clips with an error-guided octree and a VAE that maps its cells to a latent grid for selective decoding. It reports reconstruction, compute, and recognition results across K400 and zero-shot DAVIS evaluation.

  • Problem

    Video models commonly process clips on uniform grids despite variation in visual complexity, while existing approaches do not combine adaptive cell sizes with a locally reconstructible RGB signal for neural processing.

    Source: §1 · pp. 1–2

  • Method

    OctVideo uses an error-guided octree whose leaves store local RGB values and spatio-temporal gradients, with learned residual features, and a VAE that maps serialized cells to a latent grid for selective decoding.

    Source: §1 · p. 2

  • Results

    OctVideo reports competitive reconstruction and recognition with reduced computation and tokens: 36.12 dB PSNR and 189.4 GFLOPs on K400, plus zero-shot DAVIS transfer.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    OctVideo provides an adaptive representation that preserves a directly reconstructible signal while supporting neural reconstruction and recognition.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Video models commonly use uniform grids even though visual complexity varies substantially across space and time. We introduce OctVideo, which approximates a video clip with an octree. This hierarchy recursively partitions a spatio-temporal volume into eight subvolumes, so that smooth regions remain coarse while detailed regions receive finer cells. Each leaf stores local RGB values and spatio-temporal gradients, supplemented by a lightweight learned residual. For reconstruction, a Conv1D VAE maps the serialized cells to a regular latent grid and selectively refines details during decoding. Our VAE achieves 36.12 dB PSNR with 38.2M parameters and 189.4 GFLOPs per clip on Kinetics-400 (K400). It also generalizes zero-shot to the high-resolution Densely Annotated VIdeo Segmentation (DAVIS) 2016 dataset with reconstruction quality comparable to the best evaluated models. On both datasets, it requires the fewest model FLOPs and achieves the fastest encoding and decoding among the evaluated models. OctVideo also supports video understanding, achieving competitive recognition performance with few input tokens when trained from scratch. By exploiting the redundancy already present in video signals and efficiently processing sparse structures, OctVideo provides an efficient representation for video.

1 INTRODUCTION

OctVideo adapts video representation to local complexity, preserving coarse descriptions in smooth regions and finer cells where detail requires them. Its octree representation supports reconstruction and recognition with reduced computation and token counts.

1 INTRODUCTION

  • Existing codecs are not designed for direct neural processing, while regular-grid VAEs and recognition token reduction do not combine adaptive cell sizes with a locally reconstructible RGB signal.OctVideo’s stated goal is a compact representation combining content-adaptive allocation, direct reconstruction, and efficient neural processing.Source: §1 · p. 2
  • Applying hierarchical adaptive structures to video requires retaining local RGB variation across space and time and translating irregular partitions into neural-computation savings.The introduction identifies these as linked challenges for bringing adaptive representations to video.Source: §1 · p. 2
  • OctVideo stores local RGB values and spatio-temporal gradients in each leaf, using approximation error to keep smooth regions coarse and refine more detailed regions.A lightweight learned residual enriches each leaf without changing the error-guided partition.Source: §1 · p. 2
  • Its VAE serializes mixed-depth leaves in Morton order, maps encoded features to a regular latent grid, and selectively refines regions during decoding.The representation also supports recognition through patch-wise attention and sibling pooling along the octree hierarchy.Source: §1 · p. 2
  • 36.12 dB PSNR with 189.4 GFLOPs on K400; OctVideo also transfers zero-shot to DAVIS and achieves competitive recognition with fewer input tokens.On K400, it uses approximately 3,700 adaptive cells, 17.9× fewer than the 65,536 patches in the regular finest-level partition.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses video compression, adaptive representations, latent-grid processing, and token reduction separately. OctVideo applies adaptive local approximation to video while retaining a directly reconstructible signal for neural processing.

2 RELATED WORK

  • Video codecs remove redundancy for storage and transmission, but their compressed streams are not designed for direct neural processing.OctVideo instead constructs an explicit spatio-temporal representation whose cells can be processed directly and decoded locally.Source: §2 · p. 3
  • Adaptive geometry and learned octree methods represent smooth regions coarsely and complex regions finely; OctVideo extends local approximation to dense RGB video across space and time.Its value-gradient descriptors support analytic reconstruction and provide the error criterion for refinement.Source: §2 · p. 3
  • Video VAEs use spatio-temporal latent grids, while newer methods improve semantic utility or compression without directly decoding the local signal.OctVideo adapts cell size while preserving full spatio-temporal coverage and refining regions with high approximation error.Source: §2 · p. 3
  • Token pruning and merging reduce recognition cost but discard regions or replace distinct local signals, limiting faithful reconstruction; OctVideo coarsens the signal while representing every location.Its larger cells store RGB values and gradients in smooth regions, while higher-error regions receive smaller cells.Source: §2 · p. 3

3 METHOD

OctVideo allocates cells using local approximation error, then connects the resulting mixed-depth representation to reconstruction and classification networks. Its VAE maps octree features to a latent grid and selectively refines regions, while its classifier processes tokens hierarchically.

3.1 OCTVIDEO REPRESENTATION

  • For each leaf, RGB values and coordinate gradients form a 12-dimensional descriptor fitted by local least squares; flattening 32 such descriptors yields a 384-dimensional token regardless of coverage.A lightweight 3D convolutional branch extracts features over finest-level patches, aggregates covered patches by average and max pooling, and projects them to a learned residual without changing the partition.Source: §3.1 · p. 4

3.2 SEQUENCE-BASED VIDEO VAE

  • The VAE serializes leaves in Morton order, uses regular sequence convolutions, and maps encoded features to a compact regular latent grid according to cell bounds.Coarse-leaf features are copied across covered latent cells, while eight finest-level siblings are averaged within a latent cell.Source: §3.2 · p. 5
  • A decoder split head selects cells for refinement, projects selected cells into eight child features, and combines coarse and refined features before predicting RGB video.Training uses the input octree to supervise split targets; inference uses predicted split masks.Source: §3.2 · p. 5

3.3 HIERARCHICAL ATTENTION FOR VIDEO CLASSIFICATION

  • The classifier applies self-attention within token patches, then mean-pools complete groups of eight finest-level siblings between stages while shallower leaves pass through unchanged.A learned query aggregates the final sparse memory for classification readout.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments assess OctVideo’s representation, VAE reconstruction and design choices, and video understanding. They examine reconstruction quality and efficiency alongside semantic retention and recognition.

4.1 REPRESENTATION ANALYSIS

  • 3,665 adaptive leaves per K400 view—17.9× fewer than the 65,536-patch finest-level partition—achieve 34.72 dB PSNR under direct reconstruction.PSNR is computed per clip and then averaged.Source: §4.1 · p. 6

4.2 VAE RECONSTRUCTION

  • OctVideo uses the fewest parameters and FLOPs and the shortest encoding and decoding times among compared VAEs, with 36.12 dB PSNR, 0.960 SSIM, and 0.0220 LPIPS.The comparison uses K400 clips at 256 × 256 with 4 × 8 × 8 compression and 16 latent channels.Source: §4.2 · p. 6, Table 1 · p. 7

4.2.1 ZERO-SHOT DAVIS RECONSTRUCTION

  • On all 50 original-resolution DAVIS 2016 sequences, OctVideo yields visually competitive reconstructions with substantially lower VAE runtime.The comparison uses frozen K400 checkpoints without DAVIS fine-tuning or model selection.Source: §4.2.1 · p. 7, Table 2 · p. 7

4.3 VAE ABLATION STUDIES

  • At matched computational cost, the small Conv3D baseline trails OctVideo by 1.88 dB and has substantially worse LPIPS; a larger dense baseline reaches 34.12 versus 34.18 dB at 1,780.5 versus 239.7 GFLOPs.Among the evaluated dense baselines, increased capacity approaches OctVideo’s reconstruction quality at substantially higher computational cost.Source: Table 3 · p. 7, §4.3 · p. 8
  • Removing spatio-temporal gradients lowers PSNR by 1.48 dB, while removing the dense latent residual lowers it by 1.98 dB; the residual branch alone reaches 31.87 dB and has LPIPS 0.0685 versus 0.0382 for the full model.The gradient-removal variant also degrades SSIM and LPIPS; the results characterize gradients as useful signal priors and the two pathways as complementary.Source: §4.3 · p. 8
  • The random-tree control loses 0.68 dB PSNR, while fully expanding the decoder raises compute from 189.4 to 835.9 GFLOPs and produces PSNR 0.28 dB below selective decoding.The controls support error-guided input-cell allocation and decoder refinement targets.Source: §4.3 · p. 8

4.4 VIDEO UNDERSTANDING

  • A frozen Video Swin model’s Top-1 accuracy falls from 82.27% on original videos to 79.51% on analytic OctVideo reconstructions.This comparison uses neither a VAE nor adaptation to reconstructed inputs.Source: §4.4 · p. 9, Table 4 · p. 9
  • With 31M parameters and approximately 4,200 input tokens, OctVideo reaches 80.1% Top-1 and 94.4% Top-5 accuracy on K400.Transferring the K400 checkpoint to UCF101 split 1 yields 96.43% Top-1 and 99.52% Top-5 accuracy.Source: §4.4 · p. 9, Table 5 · p. 9

5 CONCLUSION

OctVideo combines local signal approximation and adaptive refinement with neural processing for reconstruction and recognition. The conclusion reports competitive fidelity at low computational cost and identifies finer detail and fast-decoding applications as future directions.

Conclusion

  • OctVideo stores RGB values and gradients per cell, refining the octree according to local approximation error while retaining learned residual features for neural processing.Smooth regions receive coarse descriptions, while detailed regions receive finer cells.Source: §5 · p. 9
  • Experiments on Kinetics-400 and zero-shot DAVIS transfer demonstrate competitive pixel-level fidelity at low computational cost.The conclusion also reports controlled UCF101 ablations supporting explicit gradients and error-guided input and decoder topologies.Source: §5 · p. 9
  • Recognition experiments show that adaptive cells retain semantic information and support learned classification through patch-wise attention and sibling pooling.The conclusion describes this as complementary evidence for the representation's use beyond reconstruction.Source: §5 · p. 9
  • Improving fine spatio-temporal detail and studying fast decoding in repeated video sampling and evaluation are identified as future directions.The conclusion states that a compact signal description can also provide useful structure for neural computation.Source: §5 · p. 9

A.1 VIDEO UNDERSTANDING DETAILS

The appendix describes how OctVideo's adaptive tokens are pooled for recognition and specifies the protocols for UCF101 transfer and comparison with RGB-based methods.

Recognition hierarchy

  • Sibling pooling replaces complete groups of eight children with their parent, while shallower leaves pass through unchanged.The operation is parameter-free.Source: §A.1 · p. 13
  • Table 6 describes the K400 recognition token hierarchy for one 32 × 256 × 256 view sampled at temporal interval 2, including classifier states after sibling pooling.The caption identifies the states represented in the table but does not report their values.Source: Table 6 · p. 13

UCF101 transfer

  • UCF101 transfer fine-tunes a K400-supervised OctVideo checkpoint on official split 1 without using UCF101 videos in pre-training.Fine-tuning uses AdamW for 50 epochs, with a base learning rate of 3 × 10−5 and weight decay of 0.05.Source: §A.1 · p. 13
  • Table 7 covers UCF101 transfer and representative RGB-based methods; OctVideo uses four temporal clips and three spatial crops on official split 1.Published methods retain their original protocols, so the caption does not establish a protocol-matched comparison.Source: Table 7 · p. 13

A.2 TRAINING AND EVALUATION PROTOCOLS

The appendix specifies optimization and augmentation for classification, alongside the high-resolution DAVIS reconstruction setup and metric computation.

Optimization

  • K400 classification trains for 200 epochs with AdamW, while the VAE trains for 50 epochs with MuonAdamW and no adversarial loss.Both use BF16 automatic mixed precision and an exponential moving average rate of 0.999.Source: §A.2 · p. 13

Classification augmentation

  • Classification augmentation uses two augmented views per training video, RandAugment, Mixup or CutMix, random erasing, crops, and horizontal flips.Mixup or CutMix is applied with probability 1.0, with equal probability for either operation.Source: §A.2 · p. 13

High-resolution reconstruction

  • The DAVIS comparison uses frozen checkpoints and one center-aligned clip from each of 50 sequences, with models evaluated at their native lengths.HunyuanVideo, LeanVAE, and OctVideo use non-overlapping 256 × 256 tiles; Wan2.1 uses native full-frame inference.Source: §A.2 · pp. 13–14

Evaluation metrics

  • DAVIS metrics are averaged equally across sequences over valid aligned frames after restoring original resolution.PSNR and SSIM use RGB values in [0, 1], LPIPS uses AlexNet for every method, and rFVD uses one Kinetics-400 I3D feature per aligned clip.Source: §A.2 · p. 14

A.3 NETWORK ARCHITECTURES

The appendix details the VAE's octree-derived token pathway and residual branches, then describes the classifier's hierarchical token pooling and attention architecture.

VAE architecture

  • Each VAE leaf stores a 2 × 4 × 4 grid of 12-dimensional RGB-and-gradient descriptors, flattened into a 384-dimensional token.A depth-d_c leaf covers (2s, 4s, 4s) video samples, with s = 2^(D−d_c).Source: §A.3 · p. 14

VAE encoder

  • The encoder adds per-leaf residual features and depth embeddings to analytic tokens, processes them with eight Morton Conv1D blocks, then copy/pool maps features to an 8 × 32 × 32 × 128 latent grid.Coarse leaves are copied to covered locations, while complete groups of eight finest-depth siblings are averaged.Source: §A.3 · p. 14

VAE residual branches

  • Per-leaf and dense latent residual branches supplement the analytic pathway, with the branches activated after 10% and 20% of training epochs, respectively.The dense residual is added to the copy/pool grid before the latent-grid block; Figure 6 depicts the encoder branches and their placement.Source: §A.3 · p. 14, Figure 6 · p. 14

VAE objective and optimization

  • The VAE uses RGB ℓ1 reconstruction, weighted AlexNet LPIPS, a ramped KL term, and split-mask supervision for selective decoder expansion.Ground-truth split masks determine expansion during training; inference uses a sigmoid threshold of 0.5.Source: §A.3 · p. 15

Classification network

  • The classifier uses analytic 384-dimensional tokens, staged transformer blocks with attention over contiguous groups of 256 hierarchy-ordered tokens, and learned single-query attention readout.After stages one and two, complete groups of eight depth-6 and depth-5 siblings are averaged into parents, while unmatched shallower leaves pass through.Source: §A.3 · p. 15

Sampling protocols

  • Recognition samples 32 frames at interval 2 and averages approximately 4,200 leaves, whereas the VAE uses 32 consecutive frames and averages 3,665 leaves on K400 validation.The larger recognition count is attributed to its greater temporal span and variation, not augmentation or a different tree threshold.Source: §A.3 · pp. 15–16

A.4 TREE CONSTRUCTION AND DIRECT RECONSTRUCTION

OctVideo recursively splits cells whose local approximation error exceeds depth-specific thresholds, retaining smooth regions as coarse leaves. Its analytic reconstruction reaches 34.72 dB PSNR on K400, while controlled comparisons show the importance of gradients and error-guided placement.

A.4 TREE CONSTRUCTION AND DIRECT RECONSTRUCTION

  • Starting from depth 3, cells exceeding the approximation-error threshold split recursively into eight children; depth-6 cells are retained, and leaves cover the video without overlap.The depth-3, 4, and 5 thresholds are 0.7, 0.8, and 1.0; leaf centers determine Morton ordering.Source: §A.4 · p. 16
  • The default threshold schedule (0.7, 0.8, 1.0) averages 3,665 leaves and 34.72 dB, while smaller threshold multipliers increase leaves and improve analytic reconstruction.The thresholds are scaled together on K400 validation.Source: Figure 7 · p. 16
  • GPU construction tests active cells in parallel level by level and stores accepted leaves in flat tensors, avoiding per-cell Python recursion and pointer traversal.Source: §A.4 · p. 16
  • 34.72 dB PSNR is achieved by OctVideo’s analytic reconstruction on K400 without a VAE or learned residual features.PSNR is computed per clip and then averaged.Source: §A.4 · p. 16
  • Removing gradients from the adaptive tree introduces block artifacts and weakens thin contours, while a random tree with matched per-depth cell counts concentrates severe errors on foreground structure.The full representation allocates finer cells to the dancer while keeping the smooth ground coarse.Source: §A.4 · p. 16, Figure 8 · p. 17

A.5 KINETICS-400 VAE RECONSTRUCTIONS

The figure presents six VAE reconstructions alongside the original clip, with each row showing the same source frame across methods. The compared models use between 29 and 33 input frames.

A.5 KINETICS-400 VAE RECONSTRUCTIONS

  • Each row shows the same source frame across the original clip and six reconstructions.Source: Figure 9 · p. 17
  • OctVideo, CogVideoX, and Wan2.1 use 32 input frames.Source: Figure 9 · p. 17
  • WF-VAE uses 33 input frames, while IV-VAE and LeanVAE use 29.Source: Figure 9 · p. 17

Related papers

Loading 2609.33100v2…