PaperlayerView exact revision on arXiv

Source-linked AI summary

Contribution-Aware Bandwidth Allocation for Multimodal Split Learning

Iason Ofeidis, Leandros Tassiulas

arXiv:2609.01406v1cs.LGcs.DCcs.NI

TL;DR

Multimodal Split Learning must carry every modality’s smashed activations over a shared, constrained uplink, while equal keep-ratios divide that budget by activation dimension rather than fused contribution. The paper introduces ModalShare, which uses server-side Shapley scores over already-received activation coalitions to set per-modality keep-ratios without extra uplink or client computation. At matched payload in 5× compression, it improves accuracy over equal keep-ratios by 15.4 percentage points on CREMA-D and 12.4 on MVSA, with gains across compressors, datasets, and budgets.

  • Problem

    Existing multimodal compression gives modalities equal keep-ratios, dividing the shared budget by activation dimension despite that dimension being unrelated to fused-prediction contribution.

  • Method

    ModalShare uses server-side Shapley contribution scores from coalitions of already-received smashed activations to set per-modality keep-ratios while preserving total payload.

  • Results

    15.4 percentage points on CREMA-D and 12.4 on MVSA: ModalShare improves accuracy over equal keep-ratios at matched payload in 5× compression, with gains across three compressors, three datasets, and four budgets.

  • Takeaways & Limitations

    Allocation can recover contribution-related gains left unclaimed by existing modality-agnostic compressors, while allocation stays near-uniform where contribution differences are small.

Abstract

from arXiv · show

Multimodal models are increasingly the default option for perception at the network edge, yet they are trained almost entirely in the datacenter, because a client holding several sensor streams cannot host an encoder per modality. Split Learning makes such training feasible by keeping only the first layers on the device, at the cost of an uplink that must carry smashed activations for every modality at every step. Existing compression schemes give each modality the same keep-ratio, so the shared budget is divided in proportion to smashed-activation dimension, a quantity unrelated to how much each modality contributes to the fused prediction. We make that division an explicit decision and call it inter-modality allocation: under a fixed uplink budget, every policy transmits the same expected payload and differs only in how that payload is split across modalities. Our allocator, ModalShare, sets each modality's keep-ratio from a Shapley contribution score that the server computes over coalitions of activations it has already received. Measuring this score adds no uplink traffic and no client-side computation, and needs no prior knowledge of which stream is which. ModalShare improves accuracy over equal keep-ratios by 15.4 and 12.4 percentage points on CREMA-D and MVSA at matched payload in 5x compression, with strong performance across three compressors, three datasets, and four budgets. We show that existing compressors underperform in multimodal settings, with ModalShare recovering what gains are left behind.

I. INTRODUCTION

Multimodal Split Learning shifts training to the edge but makes the uplink a dominant bottleneck, while existing compression divides bandwidth by activation dimension rather than modality contribution. ModalShare makes this allocation explicit and improves matched-payload accuracy by using server-side contribution scores.

  • I. INTRODUCTION: Split Learning keeps early layers on clients and sends smashed activations to servers, trading on-device computation for bandwidth.The cut-layer uplink carries activations throughout training and is the system’s dominant communication cost.
  • I. INTRODUCTION: Equal keep-ratios allocate multimodal payload in proportion to cut dimensions, so larger activation tensors receive more bandwidth regardless of fused-prediction contribution.On MVSA-Single, the image cut is roughly four times the text cut and receives 80% of the budget before task contribution is measured.
  • I. INTRODUCTION: ModalShare maps server-side Shapley contribution scores from coalitions of received smashed activations to per-modality keep-ratios without extra uplink traffic or client computation.The existing intra-modality compressor remains responsible for selecting coordinates within each stream’s allocation.
  • I. INTRODUCTION: Inter-modality allocation treats the split of a shared uplink budget as an iso-budget decision: policies transmit the same expected payload but divide it differently across modalities.This exposes a decision absent from modality-agnostic compression.
  • I. INTRODUCTION: 15.4 pp on CREMA-D and 12.4 pp on MVSA: ModalShare improves accuracy over equal keep-ratios at matched payload under SplitFC with β=0.2.The advantage persists across three compressors, three datasets, and four budgets.
  • I. INTRODUCTION: Contribution, rather than dimension skew, explains the gains: the effect is complete on dimension-balanced CREMA-D and vanishes near uniform allocation when the measured gap is small on UCI HAR.On MVSA, the contribution component exceeds the skew component recoverable by fixed rules.

II. RELATED WORK

Prior split-learning compression treats each transmitted tensor independently, while multimodal split learning adds the distinct problem of dividing one uplink budget across streams.

  • Split Learning partitions the model so clients transmit intermediate activations while the server completes forward and backward computation.
  • Existing compression methods choose sparsity, schedules, channel rates, packing, or learned bottlenecks for a single transmitted tensor.
  • Multimodality introduces a second allocation decision: how to divide a shared uplink budget among multiple activation streams.
  • Contribution scoring is task- and dataset-dependent, motivating online estimates for payload allocation rather than fixed modality splits.
  • The studied setting uses one multimodal client whose server-side model receives modality activations, fuses them, and predicts.

B. Compression of smashed activations

The compressor controls each modality’s representation under a keep-ratio, while the allocator independently controls how much shared budget that modality receives.

  • A modality-level compressor maps smashed activations to a compressed representation using budget parameter r_m.
  • The keep-ratio r_m is the expected compressed payload as a fraction of the modality’s smashed-activation dimension D_m.
  • The compressor decides how to represent a modality within its budget, whereas the allocator decides how much budget that modality receives.
  • At equal total transmitted payload, allocations match both-direction cut traffic and associated compressor side information.

C. Inter-modality allocation

Inter-modality allocation makes the split of a shared cut budget explicit: policies obey the same expected payload while differing only in its distribution across modalities.

  • Existing compression either applies one rule to concatenated activations or assigns every modality the same keep-ratio, without allocating by modality role.
  • Inter-modality allocation is formulated as an iso-budget problem in which all policies transmit the same expected payload.
  • The normalized budget β is defined from total transmitted payload relative to the aggregate smashed-activation dimensions.
  • At β = 0.2, both illustrated policies transmit 5 of 25 units but differ in how those units are split between modalities.
  • A modality with four times the activation dimension receives four times the payload under equal compression severity, so ModalShare chooses q from fusion-level contribution while preserving the budget.

D. ModalShare: Contribution-aware allocation

ModalShare estimates modality contributions at the server and maps them to adaptive per-modality keep-ratios, with optional freezing once contribution confidence is sustained.

  • ModalShare divides a fixed uplink budget into per-modality keep-ratios using server-side contribution scores while leaving intra-modality compression unchanged.
  • The server estimates contribution from coalition utilities based on true-label log-probability when selected modalities are present and absent ones are zeroed.
  • For M modalities, coalition values can be evaluated exactly with 2^M coalitions or estimated by sampling coalition orderings for large M.
  • Negative contributions are floored, renormalized, smoothed, and probed periodically while cached scores are used between probes.
  • Contribution weights allocate residual payload after a keep-ratio floor, then projection preserves the total budget while temperature controls allocation sharpness.
  • ModalShare adapts keep-ratios until a winning modality persists with sufficient normalized contribution gap, after which it freezes the current allocation.

E. Training procedure and communication cost

The training procedure compares contribution-aware and uniform modality allocation at fixed expected payload, while the server-side probe adds computation but no communication.

  • Training procedure: ModalShare refreshes per-modality keep-ratios during training while ramping compression from full transmission to target budget β.At fixed β, compared policies share the same expected transmitted payload and differ only in allocation.
  • Communication cost: The probe forms modality coalitions by masking smashed activations already held at the server, so it adds no client–server messages or uplink traffic.For M modalities, each probe update costs 2M server forward evaluations while allocation remains unfrozen.
  • Controlled comparison: The evaluation changes only the mapping from total budget to per-modality keep-ratios, holding the intra-modality compressor and training protocol fixed.Budget is measured in expected kept smashed-activation floats, TX = Σm Dmrm, with matched cut-gradient support.
  • Baseline: Existing methods compress each modality independently and leave division of a shared multimodal budget unspecified.Uniform allocation assigns every modality the same keep-ratio, whereas ModalShare changes only the shared payload split.

B. Datasets

The evaluation uses three multimodal datasets spanning audio-visual emotion, image-text sentiment, and inertial activity recognition, with early client-side encoder splits.

  • Datasets: CREMA-D contains synchronized speech and facial-expression data for six-class emotion recognition.It has 7,442 video clips, with 6,698 training and 744 testing samples.
  • Datasets: MVSA-Single provides image-text pairs for sentiment analysis.Its 2,592 examples are split into 1,555 training, 518 validation, and 519 held-out pairs.
  • Datasets: UCI HAR contains synchronized accelerometer and gyroscope streams for six-class human activity recognition.The dataset has 10,299 samples from 30 volunteers, split into 7,352 training and 2,947 testing samples.
  • Split configuration: The default early split keeps only initial encoder stages on clients and performs remaining modality processing, fusion, and classification on the server.A late split exchanges only pooled embeddings, leaving little payload to allocate.
  • Payload geometry: Under equal keep-ratios, MVSA’s larger image cut receives four-fifths of the transmitted payload, while CREMA-D is nearly balanced.Table II reports cut dimensions and the resulting uniform payload shares.

D. Hyperparameters and evaluation

Across matched-payload evaluations, ModalShare outperforms uniform allocation most strongly at moderate compression, while gains diminish near the trivial floor.

  • Accuracy results: ModalShare strongly outperforms uniform allocation on CREMA-D and MVSA at every tested compression level.Table IIIa reports accuracy and differences under early-split SplitFC at matched total payload.
  • Accuracy results: UCI HAR favors ModalShare at 5 of 6 budgets, but its slight reallocation produces only a small accuracy gain.The result is consistent with roughly equal contribution from its two sensor streams.
  • Operating envelope: Gains peak near moderate compression and largely decline toward 40×, where both methods approach task-specific trivial floors.At 40×, CREMA-D is just above six-class chance and MVSA uniform is near the majority-class rate.
  • Contribution dependence: The contribution gap is large on CREMA-D and MVSA and near-zero on balanced UCI HAR, tracking the observed allocation gains.Table III summarizes accuracy, favored modality, payload share, and contribution gap g.

C. Contribution, dimension skew & modality selection

The analysis separates dimension-induced skew from contribution-aware gains and shows that server-side coalition measurements outperform simpler modality-selection or per-stream proxies.

  • Dimension skew: 12.40 pp on MVSA decomposes into 5.17 pp from correcting dimension skew and 7.23 pp from contribution-aware allocation.Equal payload isolates the dimension component; the remaining gain to ModalShare is the contribution component.
  • Dimension skew: CREMA-D’s near-equal cut dimensions leave dimension-skew gains absent, yet its contribution gain is larger than MVSA’s.Thus the contribution component does not depend on unequal activation dimensions.
  • Modality selection: Modality selection is the τ → 0 limit of the allocation formulation, but using it still requires identifying the winning modality.The paper reports that no cut-level per-stream signal recovers the needed allocation.
  • Contribution proxies: Per-stream activation and gradient norms fail to track fusion-level contribution because they are computed before modalities are combined.On CREMA-D, activation norm gains only +4.3 pp versus ModalShare’s +15.4 pp; on MVSA, proxies remain at or below uniform accuracy.
  • Contribution proxies: Solo utility is insufficient because standalone predictive strength can differ from marginal value after the other modality is present.The paper identifies coalition differences as the measure that captures contribution on top of other modalities.

E. Allocation is orthogonal to the compressor

ModalShare’s inter-modality allocation remains effective across different intra-modality compressors and split depths, while its gains diminish when the split becomes late and little payload remains to allocate.

  • Compressor independence: Across three compressors, three datasets, and four budgets, ModalShare selects per-modality budgets while each compressor independently selects surviving coordinates.The comparison is within-compressor at matched β, separating allocation from compression mechanism.
  • Compressor independence: At 5× compression on CREMA-D, Top-S loses 5.11 pp and RandTop-S loses 2.51 pp relative to uniform, showing that hard sparsifiers can underperform when compression is mild.At tighter budgets on the same dataset, both hard sparsifiers instead gain +4.97 to +9.65 pp.
  • Compressor independence: On MVSA, allocation gains are larger under Top-S and RandTop-S than under SplitFC at several budgets, reaching 13.21 pp at 10×.Two Top-S configurations exceed the uncompressed reference, while compressor choice remains a separate axis from allocation.
  • Split-depth dependence: At β=0.20, Figure 4 varies split depth and plots ∆ = ModalShare − uniform for MVSA and CREMA-D, with grey diamonds showing uniform payload share of modality A.The early split keeps the client model small and carries the most data across the cut.
  • Split-depth dependence: On MVSA, the gain falls quickly from layer 1 as payload and dimension skew shrink; uniform image share drops from 80.3% at layer 1 to 33.8% at layer 4.CREMA-D remains within 3% dimension balance yet gains through layer 3, isolating contribution beyond skew correction.
  • Split-depth dependence: Both datasets converge to zero ∆ at the late embedding cut, where almost no payload crosses the boundary and there is little left to divide.UCI HAR shows little depth dependence, consistent with near-balanced probe scores.

G. Freezing versus online

ModalShare’s Shapley probe adds server-side coalition computation, but an evidence-based freeze can stop probing after the allocation stabilizes. The freeze preserves accuracy while reducing residual server overhead, within the paper’s primarily two-modality, early-split scope.

  • Online probing: The online probe evaluates 2^M coalitions every T_probe batches, requiring four forward passes when M=2 to refresh modality contributions.This cost is server-side only, with no additional uplink traffic or client computation stated in the supplied context.
  • Freezing: Confidence freeze locks the keep-ratios and stops probing after the contribution gap remains decisive for K consecutive epochs following e_min.This removes the residual server-side coalition cost after the allocation has stabilized.
  • Freezing: On both datasets, the freeze arm matches or exceeds the online diagnostic in absolute accuracy while both outperform uniform by a wide margin.CREMA-D separates early, whereas MVSA separates more slowly and therefore uses a sustained-gap trigger.
  • Scope: The formulation supports arbitrary M, but experiments cover M=2 and evidence is strongest at early splits where enough payload remains to allocate.Cross-client contribution scoring and clients with different modality subsets remain outside the paper.
  • Scope: Scaling to M≥3 is identified as the immediate next step because the allocation simplex then has interior structure absent from two-stream settings.The discussion frames contribution-aware allocation as a broader principle for transmitting task-relevant multimodal information.
Loading 2609.01406v1…