PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Bitstream Fine-grained Understanding for Privacy-Friendly AIoT

Zhen Yu, Wenyang Liu, Kejun Wu, Chengwang Xiao, Renjie Qiao, Chengtao Cai

arXiv:2610.08414v1cs.CV

TL;DR

Fine-grained understanding is often performed on decoded images, while compressed bitstreams can be corrupted and decoded-pixel processing exposes visual content in the pipeline. BFG encodes bitstream semantics and generates descriptions, with CFU-D supporting evaluation across corruption types and severity levels. Experiments report stable captioning under corruption while pixel-domain vision-language models degrade or fail to decode.

  • Problem

    Many fine-grained methods rely on decoded images, while byte-level foundation models for unified classification and description under a privacy-preserving design remain largely unexplored.

    Source: §I · p. 1, A. Fine-grained Classification · p. 2

  • Method

    BFG uses BSeE to extract semantics directly from image bytes and FSeG to generate detailed descriptions autoregressively, and CFU-D supplies intact and corrupted variants for evaluation.

    Source: §I · p. 2, 2) Fine-grained Semantic Generator… · p. 5

  • Results

    BFG's CIDEr decreases by approximately 1% to 2% under replacement and dropping, while pixel-domain vision-language models suffer severe degradation or decoding failure.

    Source: C. Evaluations on Robustness of… · p. 8, §V · p. 12

  • Takeaways & Limitations

    BFG performs fine-grained inference without explicitly materializing an RGB image during inference, reducing pixel-domain visual exposure in the processing pipeline.

    Source: §I · p. 1, §V · p. 12

Abstract

from arXiv · show

Image Bitstream Fine-grained Understanding (IBFU) aims to directly perform fine-grained classification and semantic description generation from encoded image byte sequences. In contrast to conventional pixel-domain visual understanding, IBFU conducts semantic analysis without fully decoding images into the pixel domain. Since pixel-level visual content is not explicitly reconstructed during inference, this paradigm reduces visual exposure within the processing pipeline and suits privacy-friendly Artificial Intelligence of Things (AIoT) applications. In this paper, we propose Bitstream Fine-grained Generator (BFG), a novel foundation model tailored for IBFU. BFG consists of two main components: a Bitstream Semantic Encoder (BSeE) and a Fine-grained Semantic Generator (FSeG). BSeE directly models semantic representations from encoded image bitstreams without explicit pixel reconstruction, while FSeG transforms the extracted bitstream semantics into detailed natural-language descriptions through autoregressive generation. To train BFG and comprehensively evaluate IBFU in practical AIoT scenarios, where image bitstreams may suffer corruption during transmission and storage, we construct a large-scale Corrupted-bitstream Fine-grained Understanding dataset (CFU-D), containing both intact bitstreams and corrupted variants across multiple corruption types and severity levels. Experiments show that BFG maintains stable fine-grained caption generation under bitstream corruption. For example, the performance only has slight change from 0.6339 to 0.6077 in terms of average CIDEr score on Stanford Dogs Caption dataset, while vision-language models, such as Qwen-VL-Chat, BLIP-2, GLM, Gemini, and GPT suffer severe performance decrease. This paper provides a practical paradigm for privacy-friendly fine-grained understanding in AIoT.

I. INTRODUCTION

Fine-grained visual understanding commonly operates on decoded images, although real-world bitstreams can be corrupted and methods relying on valid JPEG structure may be unreliable. The paper introduces BFG and CFU-D for bitstream-native fine-grained classification and description, reporting stable performance under corruption.

I. INTRODUCTION

  • Most fine-grained methods use fully decoded images, while transmission or storage faults can corrupt bitstreams and structurally damaged JPEGs can undermine methods based on partially decoded DCT features.The introduction situates this challenge alongside prior bitstream-understanding studies –.Source: §I · p. 1
  • CFU-D adds descriptions to Stanford Dogs and corruption annotations to Stanford Dogs and Flickr30k for evaluating classification, description quality, and robustness.The dataset includes intact bitstreams and corrupted variants across multiple corruption types and severity levels.Source: §I · p. 2
  • BFG uses BSeE to extract semantics from image bytes without pixel reconstruction and FSeG to generate descriptions autoregressively, supporting classification and description on one bitstream representation.FSeG combines local byte evidence with global context.Source: §I · p. 2
  • BFG maintains stable fine-grained classification and captioning under bitstream corruption, while pixel-domain models suffer significant performance decreases.The supplied introduction states this comparison without reporting specific scores.Source: §I · p. 2

II. RELATED WORK

The related work surveys fine-grained visual modeling, byte-level bitstream understanding, and privacy concerns in AIoT. It identifies gaps in unified byte-input classification and description, robustness to corrupted bitstreams, and reducing visual exposure during analysis.

A. Fine-grained Classification

  • Fine-grained research has progressed from local-region methods to weakly supervised, self-supervised, and vision-language approaches, but few foundation models operate directly on bytes for joint classification and description.The stated gap also includes a privacy-preserving design.Source: A. Fine-grained Classification · p. 2

B. Bitstream Understanding

  • Byte-level and compressed-domain methods reduce reliance on full decoding, but most bitstream approaches assume complete, stable inputs, leaving corruption robustness open.The surveyed work includes byte representations, hierarchical chunking, and dynamic token merging.Source: B. Bitstream Understanding · p. 2

C. Privacy Preservation in AIoT Systems

  • Encryption, authentication, and edge processing protect data in transit or storage, but decoded-pixel inference can still expose visual details during analysis.Anonymization may remove fine-grained cues, while encrypted images generally must be decrypted before pixel-based inference.Source: C. Privacy Preservation in AIoT Systems · pp. 2–3

III. METHOD

IBFU performs fine-grained semantic inference from encoded bytes rather than decoded pixels, including under byte corruption. BFG processes those bytes through BSeE and uses FSeG to produce detailed descriptions token by token.

A. Problem Definition

  • IBFU takes an encoded byte sequence as its visual input and does not explicitly reconstruct a pixel tensor during inference.Conventional visual understanding instead decodes the bytes before semantic inference.Source: A. Problem Definition · p. 3
  • IBFU covers both category-distribution prediction over visually similar subcategories and generation of a fine-grained attribute description.The description may express breed, appearance, pose, background, and atmosphere; the task also considers corrupted bitstreams.Source: A. Problem Definition · p. 3

B. BFG Architecture

  • BFG sends the ordered byte sequence through BSeE, projects its features to the language decoder's dimension, and has FSeG predict descriptions one token at a time.The generated descriptions cover breed, appearance, pose, background, and atmosphere.Source: B. BFG Architecture · p. 3

1) Bitstream Semantic Encoder (BSeE):

BSeE converts JPEG bytes into a reduced, position-aware feature sequence, then encodes local windows and successively broader context. Its features feed a language projector, while the direct classification pathway pools them for a classifier.

1) Bitstream Semantic Encoder (BSeE):

  • The byte-embedding layer maps each byte to a continuous vector, and Conv1D token reduction summarizes neighboring embeddings before attention.Positional encoding is added after reduction, and the validity mask is reduced by the same stride.Source: 1) Bitstream Semantic Encoder (BSeE) · p. 3
  • BSeE uses windowed self-attention with shifted windows and token merging to model local byte patterns and progressively larger bitstream contexts.Successive encoder blocks apply these operations before a final linear layer produces the 192-dimensional feature sequence.Source: 1) Bitstream Semantic Encoder (BSeE) · p. 3
  • The bitstream–language projector transforms BSeE features before sending them to the language decoder, producing features with the decoder's 512-dimensional word-representation size.The described projection applies a linear transformation, CELU, LayerNorm, and Dropout.Source: 1) Bitstream Semantic Encoder (BSeE) · p. 3
  • BFG-Direct pools valid BSeE features and passes the pooled vector to a 120-class linear head.The overall framework also uses BSeE features as input to a language decoder for description generation.Source: 1) Bitstream Semantic Encoder (BSeE) · p. 4, Figure 2 · p. 4

2) Fine-grained Semantic Generator (FSeG):

FSeG combines bitstream features with prior text tokens, using local and global bitstream information to generate descriptions autoregressively.

2) Fine-grained Semantic Generator (FSeG):

  • During training, FSeG receives the ground-truth description shifted right; during inference, it receives ⟨BOS⟩ followed by previously generated words.Its other input is the projected bitstream feature sequence PB.Source: 2) Fine-grained Semantic Generator… · p. 4
  • Masked mean pooling produces a 512-dimensional global feature from valid projected bitstream tokens, while the local sequence PB preserves position-specific evidence.Both representations are reused in every decoder layer.Source: 2) Fine-grained Semantic Generator… · p. 4
  • Causal self-attention processes prior text tokens, global-context fusion combines their features with gB, and cross-attention retrieves weighted information from projected bitstream memory.The decoder repeats this process across three stacked layers.Source: 2) Fine-grained Semantic Generator… · pp. 4–5
  • FSeG autoregressively generates a natural-language description, appending each selected next token until ⟨EOS⟩ or the maximum length.The resulting prose describes the breed, appearance, pose, background, and atmosphere rather than five independent classification scores.Source: 2) Fine-grained Semantic Generator… · p. 5

C. Construction of CFU-D

CFU-D pairs intact image bitstreams with corrupted variants and semantic supervision for evaluating classification and description under bitstream faults.

C. Construction of CFU-D

  • CFU-D supports both classification and description on intact and corrupted bitstreams, linking each derived sample to its source and original split to prevent variants crossing train-test partitions.Its construction comprises dataset composition and description generation, bitstream serialization, and corruption generation.Source: C. Construction of CFU-D · p. 5
  • Stanford Dogs supplies breed labels and descriptions covering breed, appearance, pose or activity, background, and atmosphere, while Flickr30k adds diverse scenes and human-written references.Dog descriptions are generated using the ground-truth breed name and checked for required descriptive fields; a random subset was manually inspected.Source: C. Construction of CFU-D · p. 5
  • Serialization pairs each source image's byte sequence with its semantic targets and retains the intact stream, valid length, available category label, text references, split, and encoding information.This separates the visual carrier from task supervision for classification and generation.Source: C. Construction of CFU-D · p. 5
  • Replacement, deletion, and repetition create corrupted variants at multiple severity levels, respectively modeling byte-value errors, information loss, and redundant insertion.Each corruption operator is applied independently.Source: C. Construction of CFU-D · p. 5

IV. EXPERIMENTS

The experiments use Stanford Dogs for fine-grained recognition and description and Flickr30k for general description, with implementation details and runtime reported.

1) Datasets:

  • Stanford Dogs uses 12,000 training and 8,580 test images, while Flickr30k contains 31,783 images with five human-written captions each.The datasets support fine-grained recognition and description, and general semantic description, respectively.Source: 1) Datasets · pp. 5–6

2) Implementation Details:

  • BFG has 35.257M parameters, including a frozen 15.580M-parameter ByteFormer and 19.677M trainable parameters, and is trained for 20 epochs.The implementation uses JPEG quality 90, sequences of 16,384 positions, and greedy autoregressive caption decoding.Source: 2) Implementation Details · p. 6
  • 236.6 ms per sample is the average caption-generation time for batch size 1 and 16,384-byte inputs on an NVIDIA GeForce RTX 4090.The pipeline achieves 4.2 samples/s and uses 223.1 MiB of peak CUDA memory.Source: 2) Implementation Details · p. 6

3) Comparison Methods and Evaluation Metrics:

The evaluation compares bitstream and pixel-domain methods for fine-grained classification and captioning under controlled corruption conditions. BFG-Direct achieves stronger breed recognition than the other bitstream-domain models in the reported classification comparison.

3) Comparison Methods and Evaluation Metrics:

  • The comparison includes BFG-Direct, DCT-CNN, BLOCK-CNN, bGPT, MBLM, MegaByte, and MambaByte on the official Stanford Dogs split.BFG-Direct uses a ByteFormer encoder with a 120-class linear head and is trained on JPEG bytes and breed labels.Source: 3) Comparison Methods and Evaluation… · p. 7
  • Evaluation reports accuracy, macro precision, macro recall, macro F1-score, and balanced accuracy for fine-grained classification.BFG-Direct is initialized with ImageNet-pretrained weights.Source: 3) Comparison Methods and Evaluation… · p. 7
  • Captioning comparisons include BFG and five vision-language models; Gemini-2.5-Flash and GPT-5.1 are tested on fixed subsets because of API cost.Pixel-domain models require JPEG decoding, and caption quality and decoding availability are evaluated under the same corruption conditions.Source: 3) Comparison Methods and Evaluation… · p. 7
  • Replace, Drop, and Repeat model byte-value corruption, byte-position loss, and redundant insertion at specified severity levels.Replace and Drop use 2%, 5%, or 10%; Repeat uses 10% or 20% of positions, repeating selected bytes two to four times.Source: 3) Comparison Methods and Evaluation… · p. 7

B. Evaluations on Bitstream-Domain Fine-Grained Classification Task

  • 58.51% breed accuracy on 8,580 official-test images is reported for BFG-Direct, which outperforms the other bitstream-domain models in the controlled comparison.BFG-Direct uses a standard 120-way classification head.Source: B. Evaluations on Bitstream-Domain… · p. 7
  • Confusion matrices show BFG’s strong diagonal, while bGPT predicts every sample as afghan hound and several other models have sparse predictions.BLOCK-CNN and DCT-CNN show diagonal values of only 12–25% in the displayed 10-breed matrices.Source: B. Evaluations on Bitstream-Domain… · pp. 7–8

C. Evaluations on Robustness of Bitstream-Domain Caption Task

BFG’s captioning remains stable under several bitstream corruptions, while pixel-domain vision-language models experience decoding failures and substantial quality losses. The results also show that repetition is a more difficult corruption for BFG than replacement or dropping.

C. Evaluations on Robustness of Bitstream-Domain Caption Task

  • BFG’s CIDEr decreases by approximately 1% to 2% under replacement and dropping on both datasets, with most metric retention rates above 90%.Figures 5 and 6 compare CIDEr across models and corruption strategies on Stanford Dogs Caption and Flickr30k, respectively.Source: C. Evaluations on Robustness of… · p. 8, Figure 5 · p. 9, Figure 6 · p. 9
  • Vision-language model decoding rates fall under corruption, while BFG avoids decoding; on Stanford Dogs Caption, 5% byte replacement cuts BLIP-2’s CIDEr from 1.5478 to 0.3666.The reported decrease is 76.3%; Fig. 5 presents CIDEr comparisons and retention rates for Stanford Dogs Caption.Source: C. Evaluations on Robustness of… · p. 8, Figure 5 · p. 9
  • Under 5% byte dropping on Flickr30k, BLIP-2’s CIDEr falls to 0.0055, GLM’s to 0.0022, and Qwen-VL-Chat’s is near zero.The passage reports decreases of 90% for BLIP-2 and 92% for GLM.Source: C. Evaluations on Robustness of… · p. 10
  • BFG remains stable on BERTScore-F1 and METEOR across the five plotted conditions, whereas vision-language-model radar areas shrink sharply from intact inputs.Fig. 7’s top row is BERTScore-F1 and bottom row is METEOR; each axis represents a corruption condition.Source: Figure 7 · p. 9, C. Evaluations on Robustness of… · p. 10
  • Under severe 20% repetition on Flickr30k, BFG preserves input availability but misdescribes a person-to-person interaction.The visual comparisons include intact, replacement, dropping, and repetition conditions on Flickr30k.Source: C. Evaluations on Robustness of… · p. 10, Figure 9 · p. 11
  • Byte repetition affects BFG more than replacement or dropping, with breed accuracy falling more noticeably even as input availability remains complete.The paper attributes the effect to changes in bitstream length and structure that affect ByteFormer’s sequence compression and windowed attention.Source: C. Evaluations on Robustness of… · p. 11

D. Ablation on byte-level corruption augmentation

The ablation compares BFG trained with and without byte-level corruption augmentation while holding the architecture and training setup fixed. Byte masking reduces degradation on Stanford Dogs Caption and benefits Flickr30k most clearly under repetition.

D. Ablation on byte-level corruption augmentation

  • The ablation holds architecture, initialization, data split, optimizer, and checkpoint-selection rule fixed while varying byte-level corruption augmentation.Tables IV and V report average results across corruption severities and overall corrupted-input averages.Source: D. Ablation on byte-level corruption… · p. 11
  • The overall BERTScore-F1 drop on Stanford Dogs Caption decreases from 9.10% to 5.47% with byte masking.Byte masking reduces average degradation for most metrics on this dataset.Source: D. Ablation on byte-level corruption… · p. 11
  • On Flickr30k, byte masking’s clearest benefit appears in the average results under repetition, and the ablation reports improved caption stability under corrupted inputs.Source: D. Ablation on byte-level corruption… · p. 11

E. Zero-shot Generalization to Other Image Formats

The cross-format experiment finds much lower caption scores on WebP and PNG than on JPEG, with breed accuracy of 0.44% on each format. The paper concludes that BFG’s current generalization is limited to JPEG bitstreams.

E. Zero-shot Generalization to Other Image Formats

V. CONCLUSION

  • The paper presents BFG for fine-grained classification and semantic description directly from encoded image bitstreams.Source: §V · p. 12
  • BSeE extracts compact semantic representations from JPEG bytes, and FSeG combines local evidence with global context to generate detailed descriptions.Source: §V · p. 12
  • CFU-D contains intact bitstreams and corrupted variants across multiple corruption types and severity levels for evaluating classification, description quality, and robustness.Source: §V · p. 12
  • The conclusion reports stable BFG caption generation under corruption, severe degradation or decoding failure for pixel-domain models, and reduced pixel-domain visual exposure during inference.Source: §V · p. 12
Loading 2610.08414v1…