Source-linked AI summary
Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors
Chaoran Huang, Fangcheng Li, Tianyi Liu, Wenyang Liu, Kejun Wu
TL;DR
Bitstream corruption can severely degrade or prevent reliable pixel reconstruction, limiting conventional visual understanding of harshly corrupted videos. BLMSP learns semantic priors directly from corrupted bytes and injects them into downstream models, improving restoration, captioning, and pose estimation across evaluated corruption settings.
Problem
Severe bitstream corruption can degrade or prevent reliable decoded pixels, challenging visual understanding when conventional pipelines assume correct decoding and sufficient reconstructed information.
Method
BLMSP models corrupted byte sequences with VBBM and injects learned bitstream-native semantic priors into captioning, pose estimation, and restoration models.
Results
Improvements were consistent across evaluated model–codec pairs for captioning, pose estimation, and restoration after semantic-prior injection.
Takeaways & Limitations
Corrupted bitstreams can provide reusable action-level semantic priors that complement degraded visual evidence in BcHVU.
Takeaways & Limitations
The approach assumes correctly decoded compressed bitstreams in conventional pipelines, while its scope under unseen codecs and natural transmission errors remains unexplored.
Abstract
from arXiv · showhide
Bitstream-corrupted Harsh Visual Understanding (BcHVU) aims to understand harshly degraded videos originally decoded from a severely corrupted bitstream in real-world multimedia communication. The ill-posed nature of BcHVU poses a major challenge for existing vision models, as even subtle bitstream corruption can lead to irreversible pixel distortion and significant semantic loss. To address these challenges in BcHVU, we propose Bitstream Language Modeling as Robust Semantic Priors (BLMSP), a framework for learning and injecting bitstream-native semantic cues. Our proposed BLMSP framework learns to extract bitstream-native semantic cues by bitstream language modeling, and leverages them as priors by injecting into off-the-shelf vision models of BcHVU tasks. Specifically, we present a Video Bitstream Byte Model (VBBM) that integrates byte-level modeling and cross-codec semantic distillation, enabling it to interpret robust semantics from byte sequences in multiple corrupted bitstream formats. The learned bitstream semantics are leveraged as robust priors and fused into BcHVU model backbones for improving the quality of video restoration, captioning, and human pose estimation. To train BLMSP, we construct a large-scale multi-source Corrupted-bitstream Harsh-video Paired (CHP) dataset containing 607k corrupted bitstream segments and 287k paired harsh video clips. Extensive experimental results show that the learned bitstream priors improve video restoration, captioning, and human pose estimation by 2.51 dB in PSNR, 0.20 in CIDEr, and 0.18 in PCK@0.2 on average, respectively. These results demonstrate that corrupted bitstream can serve as robust semantic priors in solving pixel distortion and semantic loss in BcHVU.
Introduction
BcHVU targets video understanding when severe bitstream corruption causes degraded or partially decodable inputs and semantic loss. BLMSP addresses this challenge by learning semantics directly from compressed bytes and injecting them as reusable priors into downstream visual models.
- Motivation: Most downstream video systems decode compressed content into pixels before applying vision models, assuming correct decoding and sufficient reconstructed visual and semantic information.This assumption becomes problematic when compressed bitstreams are corrupted.
- Challenge: Severe bitstream corruption can cause blocking artifacts, color distortion, missing regions, and errors spanning multiple spatial regions or consecutive frames.Corruption may arise from bit errors, packet loss, unstable communication links, or file truncation.
- Related work: Raw corrupted bitstreams can retain semantic information, as byte-level attention, language models, and direct byte modeling support file, image, and action understanding.Prior work also explores semantic understanding and captioning directly from corrupted bitstreams, while compressed-domain methods use codec-derived representations but require partial decoding or codec-specific syntax extraction.
- Problem formulation: BcHVU studies understanding videos with severely corrupted compressed bitstreams and formulates byte-level semantics as reusable priors for heterogeneous downstream models.The framework learns semantic representations directly from compressed byte sequences for downstream visual understanding.
- Proposed approach: BLMSP learns bitstream-native action semantics from damaged bytes through VBBM, cross-codec semantic distillation, and task-specific prior injection.These priors provide high-level action cues when decoded pixels are unreliable.
Methodology
BLMSP combines partially recoverable visual inputs with direct semantic modeling of corrupted byte sequences, then injects confidence-aware bitstream priors into downstream video tasks. Its VBBM uses byte-level modeling and cross-codec distillation to preserve semantics across corrupted codecs, supported by the CHP dataset for systematic evaluation.
- Framework: BLMSP uses a dual-branch design combining permissively decoded frames with VBBM semantic predictions extracted directly from corrupted byte sequences.The visual branch feeds recoverable frames to task-specific models, while VBBM operates without reliable pixel reconstruction.
- Prior injection: The priors guide captioning and restoration through confidence-aware text prompts, while pose estimation fuses semantic embeddings with visual features.These task-specific pathways inject bitstream semantics into downstream models.
- Bitstream semantic modeling: ByteFormer models coding patterns, byte dependencies, and codec syntax, while an MJPEG teacher transfers semantic knowledge to an H.264 student through cross-codec distillation.The distillation addresses semantics distributed across H.264 inter-frame prediction dependencies.
- Bitstream semantic modeling: The H.264 student combines supervised classification, output-level distillation, and intermediate feature alignment to preserve task-discriminative semantics under inter-frame compression.The objective uses cross-entropy, temperature-scaled KL divergence, and feature-alignment losses.
- Prior injection: Confidence-aware Top-k priors preserve semantic uncertainty by selecting fewer candidate classes at higher Top-1 confidence and more candidates at lower confidence.Class logits are converted into temperature-scaled probabilities before adaptive gating.
- CHP dataset: CHP addresses scarce paired corrupted-codec benchmarks by enabling systematic evaluation across coding formats and corruption conditions.It is constructed from UCF101, Penn Action, and Sub-JHMDB, with corrupted MJPEG and H.264 streams generated using an adapted corruption simulator.
Experiments
Experiments show that VBBM extracts robust action semantics from clean and corrupted bitstreams, and that injecting these semantic priors consistently improves captioning, pose estimation, and video restoration. Qualitative examples further illustrate improved recovery of events and visual content under corruption.
- Training strategy: VBBM training uses MJPEG classification, cross-codec distillation into H.264, and separate fine-tuning on the CHP dataset.The MJPEG teacher trains for 50 epochs, the H.264 student for 200 epochs, and distillation uses 3-second clips at CRF 25–30.
- Video captioning: Semantic-prior injection improves every reported captioning model–codec pair across lexical-overlap metrics and CIDEr on corrupted UCF101 videos.The evaluation compares decoded corrupted frames alone with captions conditioned on VBBM-predicted action classes and confidence scores.
- Human pose estimation: Injecting semantic priors consistently improves PCK@0.2 and PCK@0.1 across all evaluated pose models and codec settings.VBBM action priors are injected into five estimators on Penn Action and Sub-JHMDB under identical corrupted inputs.
- Video restoration: Prior injection improves PSNR and SSIM while decreasing LPIPS and DISTS for every reported restoration model–codec pair.The evaluation uses corrupted MJPEG and H.264 bitstreams decoded into degraded frames and compares restoration with versus without semantic priors.
Conclusion
BLMSP addresses Bitstream-corrupted Harsh Visual Understanding by extracting semantic priors directly from corrupted bytes through VBBM and injecting them into downstream vision models. Its complementary action-level cues improve captioning, pose estimation, and video restoration under MJPEG and H.264 corruption.
- Method: BLMSP extracts semantic priors directly from corrupted bytes through the Video Bitstream Byte Model (VBBM).This contrasts with decode-then-understand pipelines that rely solely on unreliable reconstructed pixels.
- Downstream integration: BLMSP injects bitstream-derived semantic priors into captioning, pose estimation, and video restoration models.The framework targets these three downstream tasks within Bitstream-corrupted Harsh Visual Understanding.
- Results: BLMSP improves downstream performance under MJPEG and H.264 corruption by providing complementary action-level cues.The conclusion characterizes these cues as complementary to information available from corrupted reconstructed pixels.