PaperlayerView exact revision on arXiv

Source-linked AI summary

CST: Collaborative Selective Transmission for Communication-Efficient Multimodal Edge Inference

Hai Chi, Junrui Zhang, Rui Ning, Chonggang Wang, Robert Gazda, Huanrui Yang, Hongyi Wu

arXiv:2608.22115v1cs.LG

TL;DR

Collaborative multimodal inference faces substantial communication overhead because helper representations can be high-dimensional and redundant with the main device. CST uses main-directed, helper-specific sparse retrieval to transmit complementary information, achieving strong benchmark performance with low feature transmission and improved end-to-end speed.

  • Problem

    Existing communication-efficient methods reduce helper payloads but do not explicitly account for task-relevant information already represented at the main device.

  • Method

    CST uses PID-motivated redundancy guidance and a main-directed query–response protocol to learn sparse, helper-specific supports for complementary retrieval.

  • Results

    Across three multimodal benchmarks, CST achieves best or near-best task performance while transmitting at most 14.18% of helper feature values.

  • Takeaways & Limitations

    A five-node Jetson Orin Nano testbed reports up to a 4.27× end-to-end speedup, while Flow-PID finds CST preserves 97.3% of conditional task information versus Transmit-All.

Abstract

from arXiv · show

Collaborative multimodal inference improves edge perception by combining observations from distributed sensing devices, but transmitting high-dimensional helper representations incurs substantial communication overhead and can lead to high end-to-end latency. Existing communication-efficient methods reduce payloads through compression, semantic coding, or feature selection, yet typically optimize compactness or task relevance without explicitly accounting for information already represented at the main device. Consequently, task-relevant but redundant helper features may still consume bandwidth. We present Collaborative Selective Transmission (CST), a main-directed query--response framework that retrieves only helper information complementary to the current main representation. Inspired by Partial Information Decomposition and the Multiview Redundancy Assumption, CST learns sample-adaptive, helper-specific sparse retrieval supports while discouraging retrieval of semantics already covered by the main device or duplicated across helpers. During inference, the main device transmits only support indices, and each helper returns the corresponding latent values, avoiding dense helper-feature exchange. Across three real-world multimodal sensing benchmarks, CST transmits no more than 14.18% of helper feature values while achieving best or near-best task performance among the evaluated methods. Experiments on a five-node NVIDIA Jetson Orin Nano testbed across 5--100 Mbps demonstrate up to a $4.27\times$ speedup over Transmit-All in end-to-end inference, confirming practical end-to-end latency reductions.

I. INTRODUCTION

Collaborative multimodal inference improves edge perception but makes communication a major latency bottleneck because helpers transmit high-dimensional and potentially redundant representations. CST addresses this gap with main-directed sparse retrieval of complementary helper information, achieving strong task performance with substantially lower transmission and latency.

  • Motivation: Collaborative edge inference combines synchronized observations from heterogeneous helper devices to improve the main device’s task performance.Intermediate representations keep sensing and feature extraction local while enabling complementary evidence fusion.
  • Communication bottleneck: Transmit-All sends complete helper representations, causing payloads to grow with helper count and representation dimensionality while carrying semantics already covered by the main device.Existing compression and selection methods reduce communication but do not explicitly use a redundancy reference for main-covered information.
  • CST approach: CST retrieves task-relevant helper features missing or insufficiently represented in the current main representation through main-generated, helper-specific sparse supports.The main device sends support indices, and helpers return only the corresponding latent values without helper-side selection.
  • Evaluation: CST reduces batch-level end-to-end latency to 7.71 s after accounting for query and coordination overhead, substantially outperforming the plotted baselines.The paper reports up to a 4.27× end-to-end speedup across shared aggregate bandwidths of 5–100 Mbps.
  • Theoretical motivation: PID and the Multiview Redundancy Assumption motivate maximizing complementary helper information while minimizing information redundant with the main representation.CST realizes these roles with tractable learned mechanisms rather than explicitly estimating PID components during training.
  • Evaluation: CST transmits at most 14.18% of available helper feature values while achieving best or near-best task performance across three real-world multimodal benchmarks.The evaluation compares CST with Transmit-All and multiple communication-efficient methods.

A. Problem Formulation

The paper formulates collaborative inference as selecting compact helper feature subsets that improve the main device while avoiding redundant transmission. PID and multiview redundancy motivate targeting helper information that remains conditionally useful beyond the main representation.

  • A. Problem Formulation: The objective is to select compact subsets from heterogeneous helper features that improve main-device task performance while matching or surpassing full fusion.The formulation minimizes transmitted feature values using an ℓ0-style payload measure.
  • A. Problem Formulation: For n devices, D1 is designated as the main device and the remaining devices are synchronized helpers with modality-specific inputs and intermediate representations.Each device can have different feature dimensionality because the sensing modalities are heterogeneous.
  • B. Theoretical Insight: Partial Information Decomposition: PID decomposes joint task information into unique information from each device, redundant information shared by devices, and synergistic information emerging through joint reasoning.The decomposition is expressed as I(F1, Fj; Y) = U1 + Uj + R1j + S1j.
  • B. Theoretical Insight: Partial Information Decomposition: Figure 1 contrasts complete helper transmission, representative local or learned selection families, and CST’s main-conditioned sparse-support retrieval.CST’s path retrieves corresponding helper values after generating supports from the current main representation.
  • B. Theoretical Insight: Partial Information Decomposition: Because redundant information is already available from the main representation, communication should maximize complementary helper information while minimizing redundant information.The desired transmitted content includes helper-unique information and synergistic information needed for collaborative reasoning.
  • B. Theoretical Insight: Partial Information Decomposition: The Multiview Redundancy Assumption suggests that conditional task-relevant helper gain after observing the main view can remain small despite high-dimensional helper features.The assumption motivates compact selection but does not itself imply coordinate-wise sparsity.
  • B. Theoretical Insight: Partial Information Decomposition: CST does not explicitly estimate PID components during training because that is impractical; instead, it uses lightweight anchors and tractable end-to-end objectives.These mechanisms implement the intended redundancy and complementarity separation.

III. PROPOSED CST FRAMEWORK

CST first aligns modality-specific representations in a common latent space, then uses the main representation to identify complementary evidence from each helper. Inference transmits only selected supports and retrieves their latent values for residual fusion and prediction.

  • A. Feature Preparation: Each device performs local modality-specific feature extraction and lightweight latent preparation before collaborative communication and inference.The encoders and latent representations support heterogeneous sensing modalities.
  • A. Feature Preparation: The main representation is available locally, while each helper retains its aligned latent representation for selective retrieval.Aligned representations have modality-specific sequence lengths and a common latent dimension.
  • B. Complementary Feature Selection: CST distinguishes main-covered information from helper evidence that can improve prediction rather than treating each helper representation as an indivisible message.The target is complementary evidence that becomes useful beyond the main representation.
  • B. Complementary Feature Selection: Under the Multiview Redundancy Assumption, the helper’s conditional task-relevant gain motivates representing that gain with a compact complementary feature.The assumption bounds the gain after observing the main representation by a small constant.
  • B. Complementary Feature Selection: A trainable projector takes only the main latent representation and generates one shared redundancy anchor plus a helper-specific complementary anchor for each helper.The projector output is collected as [Ar; {Ai_c}].
  • B. Complementary Feature Selection: Positive entries of each ReLU-activated complementary anchor define the scalar-level transmission support, while the redundancy anchor remains a continuous training-time reference.The complementary anchor is dynamically generated for the current sample and helper.
  • B. Complementary Feature Selection: When helper and main sequence lengths differ, CST resamples token maps to align sequence length without assuming token-wise spatial correspondence across modalities.Helper-specific projector heads learn the cross-modal conditioning end to end.

C. End-to-End Complementary Anchor Learning

CST learns complementary helper anchors by separating main-covered semantics from additional task-relevant evidence, while regularizing supports for sparsity and cross-helper diversity.

  • Complementary evidence: The complementary branch Ai^c targets helper information not sufficiently represented in Z1, including helper-unique evidence and useful main–helper synergy.Shared task-relevant information is treated as already available at the main device.
  • Redundancy reference: CST introduces a training-only redundancy anchor Ar to represent helper semantics already covered by the main representation Z1.The redundancy branch supervises complementary feature learning but does not define a communication support.
  • Redundancy reference: Cosine alignment makes Ar semantically similar to Z1, while a positive margin prevents the redundancy reference from collapsing to zero.The margin maintains a nontrivial redundancy reference.
  • Complementary evidence: An orthogonality objective discourages Ai^c from emphasizing the same feature components as Ar, directing it toward information beyond Z1.The inner product is averaged over batches and token positions.
  • Sparse support: An ℓ1 regularizer encourages unnecessary positive activations in Ai^c to become zero, reducing the number of transmitted helper values.The strictly positive entries define the scalar-level transmission support.
  • Cross-helper diversity: For more than two devices, an auxiliary loss discourages overlapping positively aligned complementary contributions across helpers.When n = 2, no helper–helper pair exists and the auxiliary loss is zero.
  • Task integration: The learned complementary representations augment Z1 through residual fusion, and the task loss keeps selected features useful for prediction.Latent converters, the projector, fusion module, and task head are jointly optimized while modality-specific encoders remain frozen.

D. Online Query–Response Inference

During inference, CST generates helper-specific supports from the current main representation and exchanges only support indices and the corresponding helper values.

  • Query generation: During online inference, the training-only redundancy branch is disabled and fixed model parameters generate complementary supports from the current main representation.The projector produces Ai^c for each helper.
  • Query generation: Each query is sample-adaptive and helper-specific because Ai^c depends on the current Z1.The main device encodes the support as a compact index list.
  • Response and fusion: Helpers return selected latent values in the received index order, allowing the main device to place and reweight them without retransmitting indices.The returned values are processed by residual fusion and the task head.
  • Communication payload: The model-level communication payload consists only of support indices and selected helper feature values.CST therefore avoids dense helper-feature exchange during inference.

E. Qualitative Case Study

On DarkAct with Thermal as the main modality, CST selects a smaller RGB-helper support concentrated on informative human and scene regions.

  • Qualitative selection: CST retains no more than 15% of RGB helper features while concentrating selected locations on key scene regions.Other methods select broader, more dispersed regions that often extend into low-contrast backgrounds.
  • Qualitative selection: The qualitative selection uses highlighted locations to denote retained features, with CST focusing on informative human and scene regions.

A. Experimental Setup

The experiments use three real-world multimodal sensing benchmarks with fixed main–helper assignments and standardized model interfaces, plus Jetson-based deployment measurements.

  • Datasets and tasks: CST is evaluated on three real-world multimodal sensing benchmarks, assigning the strongest unimodal modality as main and the others as helpers.Assignments are fixed before training, and dynamic main election is outside scope.
  • Datasets and tasks: Table I summarizes datasets, modality assignments, sequence lengths, and task-specific performance metrics.Main modalities are bold and metric definitions follow the benchmark papers.
  • Baselines: All methods use identical modality assignments, data splits, frozen encoder checkpoints, and Gi × 512 encoder-token interfaces.Fusion and task-head architectures are shared in form but independently initialized and trained.
  • Testbed implementation: The implementation trains models on an NVIDIA RTX A5000 GPU and deploys them on up to five NVIDIA Jetson Orin Nano nodes.Linux Traffic Control configures shared bandwidth, RTT, and jitter.

1) Task Performance and Feature Transmission Rate:

Across three benchmarks, CST achieves the lowest transmission rate while ranking first or second on reported task metrics. Its reduced traffic translates into lower end-to-end latency across bandwidth and RTT settings.

  • 11.43% average transmission rate: CST is lowest in every evaluated benchmark setting while ranking first or second on all reported task metrics.The comparison covers three real-world multimodal sensing benchmarks.
  • 11.46% helper-feature transmission on MM-Fi: CST achieves the lowest MPJPE and stays within 0.13 mm of the best PA-MPJPE.Compared with Transmit-All, CST reduces MPJPE by 10.1% and helper-feature transmission by 88.54%.
  • 8.66% transmission on CUHK-S: CST achieves the highest accuracy, Macro-F1, and Macro-Precision, with the second-highest Macro-Recall.It reduces Tx by 71.9% relative to VIB-Dyn while producing slightly higher accuracy and Macro-F1.
  • 14.18% transmission on DarkAct: CST matches Transmit-All within 0.03 percentage points while reducing helper feature-value transmission by 85.82%.
  • 83.0–85.9% lower bidirectional traffic than Transmit-All: CST’s support-index and request–response overhead is negligible relative to its helper-feature reduction.CST also reduces traffic by 44.7–53.7% compared with the next-lowest method on each dataset.
  • At 5 Mbps, CST reaches approximately 0.24 s on MM-Fi and 0.48 s on CUHK-S, yielding 4.27× and 2.85× speedups over Transmit-All.Against the fastest plotted baseline, latency is reduced by 43.1% on MM-Fi and 34.7% on CUHK-S.
  • CST remains 15.6–34.0% faster than the fastest plotted baseline at 10–25 Mbps and records the lowest observed latency at 100 Mbps.As bandwidth increases, transmission delay contributes less and method gaps narrow.
  • At RTTs up to 100 ms, CST has the lowest mean and p95 latency; at 100 ms, these remain 18.3% and 25.1% below DynamicViT.

4) Scalability with Batch Size:

CST maintains the lowest plotted latency across batch sizes on both datasets, while ablations show that redundancy guidance and auxiliary objectives chiefly determine communication selectivity.

  • Scalability with Batch Size: Batch sizes 1–64: CST maintains the lowest latency among plotted methods on both datasets under 10 Mbps shared bandwidth.At small batches, latency differences are small; as batch size increases, curves separate according to per-sample transmission requirements.
  • Scalability with Batch Size: At batch size 64, CST completes MM-Fi and CUHK-S in 10.69 s and 22.51 s, yielding 3.21× and 3.01× speedups over Transmit-All.Latency remains 25.9% below DynamicViT on MM-Fi and 14.7% below VIB-Dyn on CUHK-S.
  • Ablation Study: 51.82% and 50.79% Tx without the redundancy branch versus 8.66% for full CST, despite similar task performance.Direct-Ac uses Ltask and Lsparsity; Direct-Ac+Laux additionally retains cross-helper regularization.
  • Ablation Study: Removing Lsparsity, Lsim, Lortho, or Laux raises Tx to 56.35%, 42.34%, 36.62%, and 37.26%, respectively, while task performance changes only marginally.The objectives respectively promote compact support, establish and separate the main-covered reference, and reduce cross-helper overlap.

2) Post-hoc Partial Information Decomposition Analysis:

The post-hoc Flow-PID analysis measures task-relevant information contributed by the delivered helper representation beyond the fixed main representation. CST retains nearly all conditional task information with substantially lower transmission than dense exchange.

  • Analysis setup: Flow-PID estimates helper-unique information UH, synergy S, and their sum C for the delivered RGB representation beyond fixed main Thermal features.The analysis uses DarkAct after task training, with action labels as targets and information reported in bits.
  • Information measure: Table VI reports C = UH + S as the conditional task information contributed by the delivered RGB representation beyond Thermal.The table reports means ± standard deviations over three fixed Flow initializations, with higher C and lower Tx preferred.
  • Results: CST retains C = 3.158 bits with only 14.18% Tx, preserving 97.3% of Transmit-All’s conditional task information.CST reduces feature-value transmission by 85.82%.
  • Results: At the reported precision, CST and How2Comm yield the same C, while CST uses 54.6% lower Tx.This comparison concerns the information-retention measure and transmission cost, not direct accuracy equivalence.
  • Interpretation: C is an information-retention measure rather than a monotonic predictor of accuracy.Equal or higher C should not be interpreted as direct performance equivalence.
Loading 2608.22115v1…