PaperlayerView exact revision on arXiv

Source-linked AI summary

TP-JSCC: Efficient Token Pruning and Joint Source-Channel Coding for LVLMs-based 3D-VQA in Cloud-Edge Collaboration

Zhengxin Li, Chongzhen Tian, Hui Yuan, Hao Liu, Sam Kwong

arXiv:2610.35489v1cs.CV

TL;DR

Existing machine-perception JSCC does not directly match LVLM token inputs and cloud-edge constraints. TP-JSCC combines semantic token pruning with channel-aware token coding. On ScanQA, it reports approximately 180× lower transmission payload, with less than 1% CIDEr loss at 90% pruning.

  • Problem

    Existing machine-perception JSCC focuses on CNN feature maps rather than the discrete token sequences used by LVLMs.

    Source: C. Deep Joint Source-Channel Coding · p. 3

  • Method

    TP-JSCC combines foreground-background token pruning that preserves local and global semantics with lightweight, SNR-adaptive token JSCC.

    Source: §I · p. 2

  • Results

    Approximately 180× lower edge-to-cloud transmission payload, with less than 1% CIDEr loss at a 90% pruning ratio on ScanQA.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The authors report that TP-JSCC provides efficiency and robustness on ScanQA for token-oriented cloud-edge 3D-VQA.

    Source: §VI · p. 12

Abstract

from arXiv · show

Deploying large vision-language models (LVLMs) on resource-constrained embodied agents for 3D visual question answering necessitates cloud-edge collaboration. However, offloading massive visual data to the cloud over bandwidth-limited and noisy channels introduces severe latency and transmission distortion. To address this, we propose TP-JSCC, an efficient token pruning and joint source-channel coding method for LVLMs-based 3D-VQA in Cloud-Edge Collaboration. First, we develop a Local and Global Semantics-based Hybrid Token Pruning (LGS-HTP) module to significantly reduce the visual token count. By decoupling foreground-background tokens and jointly selecting local and global semantics, LGS-HTP effectively reduces redundancy while preserving reasoning-critical tokens. Second, we introduce a lightweight Token Joint Source-Channel Coding (T-JSCC) module to further compress the pruned tokens and ensure their robust transmission over error-prone channels. By employing lightweight cascaded linear layers and Signal-to-Noise Ratio (SNR)-adaptive mechanism, this module adapts to fluctuating channel conditions while maintaining minimal computational overhead. Extensive experiments on ScanQA and SQA3D datasets indicate the exceptional efficiency and robustness of our approach. Specifically, on ScanQA validation set, the LGS-HTP module could achieve 90% pruning ratio while incurring less than a 1% CIDEr loss. Building on this, the T-JSCC further compresses the tokens by 18 times, resulting in an overall 180 times data volume reduction for robust transmission, while saving 90% in computational complexity for LVLM deployment. Our code is available at https://github.com/yuanhui0325/TP-JSCC.

I. INTRODUCTION

Resource-constrained embodied agents need cloud-edge collaboration for LVLM-based 3D-VQA, but transmitting raw visual data over noisy, bandwidth-limited channels adds latency and distortion. TP-JSCC combines token pruning with token-level JSCC to reduce redundancy and support robust transmission.

I. INTRODUCTION

  • TP-JSCC compresses visual tokens at the edge and uses cloud-side LVLM inference, with LGS-HTP preserving local and global semantics and T-JSCC targeting noisy channels.LGS-HTP decouples foreground and background tokens; T-JSCC uses a lightweight cascaded projection and an SNR-adaptive mechanism.Source: §I · p. 2

A. 3D-VQA

  • Multi-view-image methods use pretrained LVLMs for 3D-VQA, while point-cloud and hybrid approaches rely on 3D encoders and face limited point-cloud training data.The cited ScanQA point-cloud modality has about 41,000 point-cloud-text pairs, compared with approximately 5.8 billion image-text pairs in LAION.Source: A. 3D-VQA · p. 2

B. Visual Token Pruning

  • Visual token pruning can reduce communication payload and LVLM computation, but text-dependent methods trade alignment accuracy against computational cost, motivating text-independent pruning.The paper adopts text-independent pruning to balance compression efficiency and deployment generality.Source: B. Visual Token Pruning · pp. 2–3

C. Deep Joint Source-Channel Coding

Deep JSCC is presented as a robust alternative to separation-based communication, but existing machine-perception JSCC does not fit LVLM token inputs and edge constraints. The paper formulates a cloud-edge pipeline that sends compressed visual tokens while preserving text over a reliable channel.

C. Deep Joint Source-Channel Coding

  • Deep-learning JSCC jointly optimizes transmission and is described as degrading more gradually than separation-based systems as channel quality varies.The paper distinguishes human-perception JSCC, which reconstructs pixels, from machine-perception JSCC, which transmits task-relevant semantic features.Source: C. Deep Joint Source-Channel Coding · p. 3
  • Existing machine-perception JSCC targets CNN feature maps rather than LVLM token sequences, creating a representational mismatch for cloud-edge LVLM systems.The passage also identifies a computational-constraint mismatch for embodied agents.Source: C. Deep Joint Source-Channel Coding · p. 3

III. PROBLEM FORMULATION

  • The visual pipeline selects views, extracts and projects their tokens, then prunes and JSCC-encodes them for cloud-side reconstruction and LVLM reasoning.The view selector uses the question to choose informative views; the SigLIP encoder extracts multi-view visual tokens.Source: §III · pp. 3–4
  • The formulation sends compact visual tokens over a noisy, bandwidth-limited channel while transmitting text through a reliable, error-free channel.The cloud decodes the visual representation and runs LVLM inference; the reliable text channel reflects an unequal error protection assumption.Source: §III · p. 4

JSCC(T TP

The pruning objective seeks a compact token set that preserves LVLM task performance, while the JSCC objective limits distortion between original and reconstructed pruned tokens.

JSCC(T TP

  • Training-free pruning approximates the LVLM objective by maximizing retained semantic information under a token budget, balancing local details against global scene context.The paper motivates this balance through the role of both local features and global context in visual perception.Source: JSCC(T TP · p. 4
  • The JSCC objective jointly optimizes encoder and decoder parameters to keep reconstructed pruned tokens consistent with the original tokens.The distance term can be mean squared error, and λN and λP weight the two losses.Source: JSCC(T TP · p. 4

IV. PROPOSED METHOD

TP-JSCC uses a two-stage strategy: LGS-HTP reduces the visual-token payload while retaining local and global semantics, and T-JSCC codes the remaining tokens for noisy-channel transmission.

IV. PROPOSED METHOD

  • The proposed pipeline first prunes tokens to reduce communication payload and LVLM computation, then encodes them into channel symbols for transmission and cloud-side reconstruction.The reconstructed visual tokens are aligned with reliably transmitted text before LVLM reasoning.Source: §IV · p. 4
  • LGS-HTP uses projected tokens and lightweight segmentation priors to prune native tokens while preserving local and global semantics.The method aims to retain both semantic scales while reducing the visual-token set.Source: §IV · p. 4
  • T-JSCC maps pruned tokens to communication symbols at the edge and reconstructs visual tokens at the cloud for subsequent LVLM reasoning.The reconstructed tokens are projected and combined with the text tokens before generating an answer.Source: §IV · p. 4

A. Local and Global Semantics-based Hybrid Token Pruning

LGS-HTP uses foreground-background priors and separate token budgets to guide pruning, then selects tokens from projected features to retain local and global semantics. The resulting indices are applied to native tokens for transmission.

A. Local and Global Semantics-based Hybrid Token Pruning

  • The module’s four stages prepare masks and tokens, enhance dynamic budget allocation, cluster semantics, and select the final retained tokens.The overview places foreground-background priors and budget adjustment before semantic grouping and final selection.Source: A. Local and Global Semantics-based… · pp. 4–5, Figure 2 · p. 5
  • LGS-HTP determines selections in projected-token space, then maps the selected indices to native tokens for the final pruned set.The authors state that pruning directly on native tokens would neglect the LVLM's semantic requirements.Source: A. Local and Global Semantics-based… · p. 5

1) Foreground-Background Token Decoupling and Budget Allocation:

  • A lightweight segmentation network produces foreground-background priors, which are aligned to token space to split projected tokens into foreground and background sets.The pixel mask is downsampled to token resolution; the resulting token mask guides pruning.Source: 1) Foreground-Background Token… · p. 5
  • The per-view retained-token budget is divided non-uniformly using a foreground enhancement factor, allocating tokens between the foreground and background sets.The factor is intended to preserve foreground semantics under the target pruning budget.Source: 1) Foreground-Background Token… · pp. 5–6

2) Local and Global Semantics-based Hybrid Token Selection:

LGS-HTP clusters foreground and background tokens separately, then selects both local representatives and globally aligned tokens within each cluster. The selection indices are transferred to native tokens to form the set passed to T-JSCC.

2) Local and Global Semantics-based Hybrid Token Selection:

  • Hierarchical agglomerative clustering groups foreground and background tokens by feature embeddings to expose semantic structure for pruning.The method describes this clustering as deterministic, unlike centroid-based methods that use stochastic initialization.Source: 2) Local and Global Semantics-based… · p. 6
  • The hybrid selection combines one local semantic token per cluster with global semantic tokens selected within that cluster.The same two-branch selection process is applied to foreground and background clusters.Source: 2) Local and Global Semantics-based… · p. 6
  • Local semantic tokens are cluster feature means, chosen to represent cluster semantics and reduce redundancy among similar tokens.The background clusters’ local tokens are obtained in the same manner.Source: 2) Local and Global Semantics-based… · p. 6
  • Global tokens are chosen to align with uniformly sampled positions, selecting the lowest spatial-alignment scores among non-local tokens in each cluster.The number selected per cluster is Kglo = Nss−1, leaving one token budget for the local representative.Source: 2) Local and Global Semantics-based… · p. 6
  • After selection in projected-token space, the retained indices are mapped to native tokens, which are then passed to T-JSCC.The paper notes that sending projected tokens directly would incur prohibitive communication overhead because of their larger channel dimensions.Source: 2) Local and Global Semantics-based… · p. 7

B. Token Joint Source Channel Coding

T-JSCC addresses channel-dimension redundancy in the pruned tokens while targeting robust transmission under embodied agents’ computational constraints.

B. Token Joint Source Channel Coding

  • T-JSCC further reduces channel-level redundancy in pruned tokens and targets robust transmission with bounded on-device overhead.It is proposed as a complement to LGS-HTP, which addresses token-level redundancy.Source: B. Token Joint Source Channel Coding · p. 7

1) Lightweight Token-JSCC Architecture:

T-JSCC uses a symmetric MLP encoder-decoder that conditions token transmission and reconstruction on channel SNR. Its training objective constrains reconstruction at both native and projected-token levels.

1) Lightweight Token-JSCC Architecture:

  • The encoder and decoder condition token processing on SNR, with a progressive bottleneck encoder and a symmetric decoder for reconstruction.The encoder reduces dimensionality through hidden layers, while token processing is independent and parallel with shared parameters, giving complexity linear in token count.Source: 1) Lightweight Token-JSCC Architecture · p. 7
  • The decoder reconstructs native tokens and the shared pre-trained projector maps them to projected tokens for LVLM reasoning.The architecture mirrors the encoder bottleneck and injects channel state into decoding.Source: 1) Lightweight Token-JSCC Architecture · p. 7

2) Loss Design for T-JSCC module:

  • Native-token reconstruction alone is insufficient because the projector can amplify small input perturbations into larger projected-token errors.The loss therefore combines a coarse native-token constraint with a fine projected-token constraint.Source: 2) Loss Design for T-JSCC module · p. 7
  • The overall training objective balances coarse- and fine-grained reconstruction losses with coefficients λN and λP.The coarse loss compares reconstructed and target native tokens; the fine loss compares reconstructed and target projected tokens.Source: 2) Loss Design for T-JSCC module · pp. 7–8

V. EXPERIMENTAL RESULTS

The experiments use ScanQA for detailed analyses and ablations and SQA3D to assess generalizability, with LLaVA-OV as the foundational LVLM. Hyperparameter studies select configurations that balance semantic coverage, reconstruction quality, and transmission payload.

A. Experimental Setup

  • ScanQA provides about 41K QA pairs for detailed analyses, while SQA3D provides about 33K situated-understanding questions for generalizability evaluation.All experiments use LLaVA-OV as the foundational LVLM and jointly consider Top-1 EM and CIDEr.Source: A. Experimental Setup · p. 8

B. Hyper-parameter Settings

  • λbias = 1.5 is the default because performance peaks there while lower and higher values retain redundancy or compromise global spatial context.The study searches λbias under 90% pruning; both extremes reduce Top-1 EM and CIDEr.Source: B. Hyper-parameter Settings · p. 8
  • Kglo = 1 is the default: local-only selection lacks global coverage, while excessive global allocation weakens local semantics.At 80% pruning, LGS-HTP surpasses the unpruned baseline; the chosen setting is reported as stable across pruning constraints.Source: B. Hyper-parameter Settings · pp. 8–9, Table II · p. 8, Table III · p. 8
  • Dc = 64 is the default compressed channel dimension, at the reported elbow where further expansion yields only marginal gains but greatly increases data volume.The study evaluates Dc under 90% pruning, and Fig. 6 visualizes the performance trajectory.Source: B. Hyper-parameter Settings · p. 9, Table IV · p. 9, Figure 6 · p. 9

4) Selection of the loss weight of T-JSCC:

The T-JSCC studies select a loss balance and SNR training range to preserve reconstruction quality across channel conditions. The chosen defaults combine both loss constraints and train across the noise regime where un-coded performance degrades most.

4) Selection of the loss weight of T-JSCC:

  • λN : λP = 1 : 9 is the default loss-weight configuration, where CIDEr peaks as the projected reconstruction loss weight increases.The study finds both coarse- and fine-grained losses indispensable: relying only on the coarse constraint severely degrades CIDEr, while removing it drops both metrics.Source: 4) Selection of the loss weight of… · p. 9, Table V · p. 9

5) Selection of the SNR training range for T-JSCC:

C. Performance Comparison

LGS-HTP performs strongly across pruning settings and datasets, with its local-global semantic balance particularly beneficial at extreme pruning. In end-to-end tests, T-JSCC improves low-SNR performance, while progressive SNR sampling stabilizes performance across channel conditions.

1) LGS-HTP Performance:

  • At 80% pruning, LGS-HTP leads Top-1 EM while maintaining competitive CIDEr; at 90%, it surpasses all baselines, including uniform pruning in CIDEr.FPS's local emphasis is associated with a CIDEr drop, while uniform pruning's broader context is associated with the highest CIDEr at 80%.Source: 1) LGS-HTP Performance · pp. 9–10, Table VII · p. 10
  • LGS-HTP establishes ScanQA state-of-the-art results across metrics and pruning constraints, and significantly outperforms other SOTA methods on SQA3D at 90% pruning.At 80% pruning on SQA3D, it ranks third, slightly behind PruMerge and DivPrune; the passage attributes this to the short-phrase answers and limited need for global semantics in that setting.Source: 1) LGS-HTP Performance · p. 10, Table VIII · p. 11

2) Overall Performance:

  • At 0 dB, integrating T-JSCC raises Top-1 EM from 7.55 to 26.97 compared with the uncoded baseline.The comparison is reported under identical SNR conditions.Source: 2) Overall Performance · p. 10, Table IX · p. 11
  • Progressive SNR sampling improves low-SNR resilience and sustains peak performance at 25 dB, producing a stable upward trajectory as channel conditions improve.The strategy expands training sampling from an early high-noise range to the full 0∼25 dB spectrum.Source: 2) Overall Performance · p. 10, Table X · p. 11

1) Ablation Studies on LGS-HTP module:

The ablations support token decoupling, balanced local-global selection, and projected-token guidance in LGS-HTP. The integrated design reports large transmission compression and LVLM computational savings with limited performance loss under the stated conditions.

1) Ablation Studies on LGS-HTP module:

2) Effectiveness of the Guidance Strategy of LGS-HTP module:

  • Projected-token guidance consistently outperforms unguided pruning across pruning ratios, with the reported comparison on ScanQA validation.The proposed guidance derives pruning decisions using projected tokens rather than native tokens alone.Source: 2) Effectiveness of the Guidance… · p. 12, Table XII · p. 12

3) Computational Efficiency Analysis:

  • LGS-HTP and T-JSCC together require less than 633 GFLOPs on the edge, while total compression ratios of 90× and 180× yield nearly 80% and 90% LVLM computational savings, respectively.The compression ratios combine token pruning with an 18× T-JSCC channel reduction from 1152 to 64 channels.Source: 3) Computational Efficiency Analysis · p. 12, Table XIII · p. 12

VI. CONCLUSION

  • Approximately 180× transmission-payload reduction, less than 1% CIDEr loss at 90% pruning, and a 4.9% end-to-end CIDEr drop at 0 dB are reported alongside 90% LVLM computational savings.The conclusion attributes the result to LGS-HTP followed by T-JSCC.Source: §VI · p. 12
Loading 2610.35489v1…