Source-linked AI summary
ByteAction: Byte-space Action Recognition Foundation Model
Fangcheng Li, Zhen Yu, Kejun Wu, Qiong Liu, You Yang
TL;DR
Action recognition from corrupted image bitstreams must avoid dependence on successful pixel decoding. ByteAction uses dual corrupted views with BPA and corruption consistency training, achieving state-of-the-art corruption robustness across three datasets while retaining competitive intact performance.
Problem
Pixel-domain recognition depends on decoded images, while direct byte-space action recognition must learn semantics from long, discrete sequences affected by corruption.
Method
ByteAction uses two corruption-severity views, two-dimensional byte-matrix erasure through BPA, and bidirectional-KL consistency training with shared byte-level encoding.
Results
ByteAction achieves state-of-the-art corruption robustness across Stanford40, PPMI, and PASCAL VOC 2012 Action while maintaining competitive intact-bitstream performance.
Takeaways & Limitations
ByteAction provides robust action recognition directly from corrupted image bitstreams without pixel decoding or intact references during training.
Takeaways & Limitations
Pixel-based comparison methods remain ineffective when bitstream corruption prevents reliable image reconstruction because they require fully decoded visual inputs.
Abstract
from arXiv · showhide
Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.
1. Introduction
Pixel-domain action recognition depends on successful image decoding, which corruption can degrade or prevent. ByteAction addresses this challenge with teacher-free dual-view learning, BPA, and consistency training for robust recognition directly from corrupted bitstreams.
- Corrupted compressed files can degrade image quality or prevent decoding, making pixel-domain action recognition unreliable.
- Byte-space Action Recognition recognizes human actions directly from compressed bitstreams without relying on pixel decoding.
- Existing BRACE-based robustness training aligns corrupted representations with intact anchors through an additional frozen branch and requires clean-corrupted pairing.
- ByteAction learns from two corrupted views with different severity levels instead of using intact references or a teacher model.
- BPA reshapes one-dimensional byte sequences into two-dimensional matrices and applies region-level erasure to diversify bitstream patterns.
- Corruption Consistency Training uses bidirectional KL divergence to stabilize predictions across corruption severities without additional models or intact references.
- ByteAction achieves state-of-the-art corruption robustness across Stanford40, PPMI, and PASCAL VOC 2012 Action while maintaining competitive intact-bitstream performance.
2. Related Work
Related work spans pixel-based action and human-object understanding, compressed-data analysis, byte-level visual recognition, and multimodal foundation models. Pixel-based methods remain constrained by their reliance on fully decoded visual inputs, motivating direct bitstream analysis.
- Action Recognition: Still-image action recognition methods model spatial cues using hand-crafted features, pose estimation, part detectors, and attention-based architectures.
- Action Recognition: Human-object interaction detection jointly predicts humans, objects, and their relationships using multi-branch and vision-language architectures.
- Action Recognition: Pixel-based action recognition methods fail when bitstream corruption prevents reliable image reconstruction because they require fully decoded visual inputs.
- Compressed and Byte-level Understanding: Compressed-data methods reduce reconstruction cost but still depend on correctly parsing headers, quantization tables, and entropy-coding structures.
- Compressed and Byte-level Understanding: Byte-level models process raw bytes end to end, with ByteFormer applying learnable byte embeddings to compressed image streams for visual recognition.
- Multimodal Foundation Models: Multimodal foundation models integrate modalities such as text, images, video, audio, and sensor signals, while vision-language models support transfer and open-domain understanding.
- Multimodal Foundation Models: ByteAction’s framework combines dual corruption views, BPA, shared ByteFormer encoding, classification, and consistency objectives.
3. The Proposed ByteAction
ByteAction uses two differently corrupted byte views, BPA, and a shared ByteFormer backbone to recognize actions without pixel decoding. Its objective combines branch-wise classification with consistency regularization to stabilize predictions across corruption severities.
- Overview: ByteAction generates weakly and strongly corrupted byte views, augments both with BPA, and processes them through a shared-weight ByteFormer backbone.Each branch produces an action prediction, while training uses both views jointly.
- Bitstream Pattern Augmentation: Spatially coherent gradient-sensitivity patterns suggest that JPEG encoding preserves implicit two-dimensional structure in the byte sequence.The sequence is reshaped into a matrix to expose dependencies inherited from JPEG’s block-wise scanning order.
- Bitstream Pattern Augmentation: BPA reshapes one-dimensional byte sequences into two-dimensional matrices, applies region-level erasure, and flattens the results back into sequences.The augmentation diversifies regional missing or corrupted patterns in byte space.
- Corruption Consistency Training: Corruption Consistency Training aligns predictions for weakly and strongly corrupted views rather than requiring reconstruction of lost information or intact reference representations.The two severity levels are processed by the same network and constrained toward stable predictions.
- Training Objective: The consistency loss uses symmetrized KL divergence between softened prediction distributions, with temperature smoothing and mutual constraint between branches.The total loss adds classification losses for both branches to the weighted consistency term.
4.1. Experimental Setup
The experiments evaluate action recognition on three image-bitstream datasets under intact and corrupted conditions using standardized corruption settings and accuracy-based metrics. ByteAction is compared with pixel-, compressed-, and bitstream-domain methods under a unified preprocessing and training setup.
- Evaluation Datasets and Corruption Protocol: The evaluation uses image bitstreams from Stanford40, PPMI, and PASCAL VOC 2012 Action under the RBCS corruption protocol.Corruption is parameterized by segment size, segment-selection probability, flip ratio, and byte-loss probability.
- Evaluation Datasets and Corruption Protocol: The BAR-D evaluation contains one intact setting and twelve corrupted settings combining four severity levels with three corruption types.The corruption types are Flip, Loss, and Mixed; severities range from Light to Extreme.
- Evaluation Metrics: The study reports Top-1 Accuracy (%) and mean Average Precision (mAP, %) as evaluation metrics.Undecodable samples for decoding-dependent methods count as misclassifications with zero prediction confidence.
- Compared Methods: ByteAction is compared with pixel-, compressed-, and bitstream-domain state-of-the-art methods.The comparison includes five pixel-domain architectures and compressed-domain methods operating on partially decoded DCT coefficients.
- Implementation Details: All methods use a unified preprocessing pipeline that resizes images to 256×256, center-crops to 224×224, re-encodes them as JPEG, and applies fixed-seed corruption.Bitstream-domain models consume raw bytes truncated or zero-padded to N=50,000.
- Implementation Details: ByteAction is initialized from ImageNet-pretrained ByteFormer weights and fine-tuned end-to-end with AdamW at learning rate 3 × 10^-5 and weight decay 0.05.Its weak and strong training branches sample corruption parameters from separate severity ranges.
4.2. Comparison with SOTA Methods in Different Domains
ByteAction is evaluated against pixel-, compressed-, and bitstream-domain methods, with robustness comparisons focused on corrupted bitstreams. It achieves the strongest corrupted-bitstream performance across datasets and corruption types while preserving discriminative representations.
- Pixel- and compressed-domain methods degrade to random-chance levels under corruption because JPEG decoding fails on damaged bitstreams.
- ByteFormer’s Stanford40 Accuracy falls from 61.33% to 5.91% across the four corruption levels.
- 44.60% Corrupt Average Accuracy on Stanford40 improves over ByteFormer’s 24.36%, while PPMI and VOC2012 improve to 31.27% and 47.20%, respectively.The corresponding baselines are 16.42% on PPMI and 28.68% on VOC2012.
- ByteAction achieves the highest Accuracy and mAP across Flip, Loss, and Mixed corruption types on all three datasets.Figure 4 averages results across four severity levels for each corruption type and overall.
- Under Medium-Mixed corruption on VOC2012 Action, ByteAction maintains separated t-SNE feature clusters, unlike ByteFormer’s interleaved clusters.
- Across 12 corruption scenarios, ByteAction shows the strongest confusion-matrix diagonal and reduced inter-class confusion.
4.3. Ablation Studies
The ablation study evaluates BPA and Corruption Consistency Training as additions to ByteFormer. Each component improves the baseline, while their combination produces the best overall performance across the three datasets.
- Adding either BPA or Corruption Consistency Training consistently improves performance over the ByteFormer baseline.
- Combining BPA and Corruption Consistency Training yields the best overall performance on all three datasets.The reported improvement over single-component variants attributes different benefits to pattern enrichment and prediction stabilization.
5. Conclusion
ByteAction recognizes actions directly from corrupted image bitstreams without pixel decoding through BPA and Corruption Consistency Training. Experiments across three datasets show robust corrupted-bitstream recognition with competitive intact performance.
- ByteAction performs action recognition directly from corrupted image bitstreams without pixel decoding.
- BPA reshapes one-dimensional byte sequences into two-dimensional matrices and applies region-level erasure to diversify training patterns.
- Corruption Consistency Training aligns predictions from weakly and strongly corrupted views through bidirectional KL divergence.
- Experiments on Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate state-of-the-art robustness across diverse corruption settings with competitive intact-bitstream performance.
- The results support learning directly from image bitstreams as a direction for decoding-free visual recognition in privacy-sensitive scenarios.
CRediT authorship contribution statement
The contribution statement assigns writing, methodology, analysis, data, validation, supervision, resources, and project-management roles across the five authors.
- Fangcheng Li contributed writing, methodology, formal analysis, data curation, and visualization.
- Zhen Yu contributed writing, validation, and investigation.
- Kejun Wu contributed writing, methodology, and funding acquisition.
- Qiong Liu contributed supervision and project administration, while You Yang contributed supervision and resources.
Data availability
The data will be made available on request.
- Data will be made available on request.