Consomm\'e: Conditioned Token Merging for Split Inference

1. Introduction

• Vision Transformers’ higher computational requirements than CNNs make deployment difficult on resource-constrained edge devices.The introduction situates token reduction among approaches to compression, including reducing model size and pruning weights [10] [28] [37] [42].Source: §1 · p. 1
• Split inference runs initial layers on an edge device and sends an intermediate representation to a more capable device, with bandwidth and compute limits shaping deployment.The cited background also notes potential privacy and communication-efficiency concerns in split learning.Source: §1 · p. 1
• Keeping a publicly shared backbone frozen avoids shipping an individually fine-tuned model to every endpoint, while a lightweight adapter adds task-specific behavior.The paper motivates token reduction that lowers transmission cost without modifying the backbone.Source: §1 · p. 1
• The CLS token addresses a split-inference tension: the receiver needs a global summary for classification, while the transmitted representation must fit a bandwidth-limited channel.The paper identifies prior work as using the CLS token for only one of these roles at a time.Source: §1 · p. 1
• Consommé uses a compressed CLS latent both as the transmitted global representation and to condition a gate that merges patches into exactly k tokens.The gate uses soft assignment to reach any budget in a single step, remaining operable at aggressive budgets and shallow splits where competing frozen methods may not reach the target count.Source: §1 · pp. 1–2
• The paper frames Consommé as a frozen-backbone soft-assignment gate that can reach any budget k in one step, alongside a characterization of accuracy and deployment cost across three tiers.Its listed contributions also include ablations and split-depth analysis of soft assignment, conditioning, and reconstruction.Source: §1 · p. 2

2. Background and Related Work

• Vision Transformer efficiency approaches reduce model weights, quantize models, or reduce the number of tokens processed [10] [28] [37] [42] [32] [22].The passage describes token reduction as either pruning or merging tokens.Source: §2 · p. 2
• Bipartite matching merges similar tokens but is limited to merging at most half of the tokens per step [4] [23].ToMe introduced BPM, which was later adapted by DTEM and HAC-ToMe.Source: §2 · p. 2
• Token reduction methods select, prune, or merge tokens using attention, learned schedules, BPM adaptations, ranking modules, or target-sized downsampling.Examples include attention-based selection [33] [27] [24] [11] [36], DiffRate [6], ToFu [19], and Token Pooling.Source: §2 · p. 2
• Token-reduction methods span three deployment tiers: off-the-shelf frozen backbones, frozen backbones with a small added model, and fine-tuned backbones with additional modules.Consommé has the second-tier profile; other tier-2 methods using BPM inherit its per-layer reduction cap, while DiffRate’s schedule can fail at shallow splits.Source: §2 · p. 2
• Split inference distributes computation across model layers and can reduce transmitted data by compressing tokens, including with methods such as ToMe [15].Other split-inference work also studies finding an optimal split [43].Source: §2 · p. 2
• In public edge settings, a frozen public backbone avoids uploading proprietary model weights, while second-tier methods add learned models to participating devices.Proprietary or fine-tuned models may incur bandwidth and energy costs for upload, while tier-2 overhead is relatively small compared with tier 3.Source: §2 · pp. 2–3
• The selected baselines span deployment tiers and token-reduction mechanisms, including training-free BPM, tier-2 merging or scheduling methods, and fine-tuned attention-based methods.The tiered-method table describes methods and their respective tiers; increased footprint denotes additional parameter overhead.Source: Table 1 · p. 2, §2 · p. 3

3.1. Setting and Notation

• The sender retains the full token count until split point s, then transmits a compressed global representation and k merged tokens while the backbone stays frozen.Only the information bottleneck and conditioned merging gate are trained; each target budget has a separate model.Source: §3 · p. 3, §3.1 · p. 3

3.2. Method Overview

• At the split, the sender compresses CLS into latent z, uses z to condition patch merging, and sends z with the merged tokens for receiver-side CLS reconstruction.The receiver runs the frozen post-split blocks and classification head.Source: §3.2 · p. 3

3.3. CLS Information Bottleneck

• A variational information bottleneck compresses CLS from D to d values, and its decoded substitute is prepended by the receiver in place of the original CLS token.The latent is sampled using the reparameterization trick for end-to-end gradients.Source: §3.3 · p. 3

3.4. z-Conditioned Soft-Assignment Gate

• The z-conditioned gate softly assigns patches to k anchors, producing exactly k weighted-mean tokens and reaching any budget in one step, unlike BPM’s per-layer cap.Image-conditioning adds a further, smaller gain beyond the exact-budget soft assignment.Source: §3.4 · p. 4

3.5. Transmission Cost

• The transmitted cost is d + kD, saving D − d values versus transmitting the raw CLS signal; that fixed saving is largest at aggressive budgets.Reported comparisons exclude CLS-token bits for all methods, including the compressed CLS signal.Source: §3.5 · p. 4

3.6. Training

• Training optimizes classification with KL compression and reconstruction terms; free bits floor per-dimension KL to prevent latent dimensions from collapsing to the prior.The reconstruction term keeps the decoded CLS substitute in the distribution expected by the frozen post-split blocks.Source: §3.6 · p. 4
• Consommé achieves superior results for reduced token counts on DeiT-S and DeiT-T at split layer 9.The caption marks cases that miss the exact token count with an asterisk and reports tier-3 methods only on DeiT-S.Source: Table 2 · p. 5
• Because the gate output dimension scales with k, a separate model is trained for each budget.Source: §3.6 · p. 5

4. Experiments

• The evaluation uses DeiT-S and DeiT-T, reducing 196 patch tokens to k ∈ {8, 16, 32, 64} at split layer 9.The reported final transmission-demand comparison excludes CLS dimensionality reduction for fairness against methods that do not use it.Source: §4 · p. 5

4.1. Results

• Visualizations show semantic patch groups aligned with object parts; later reduction focuses more on image content than background.The largest uncolored group corresponds semantically to background.Source: §4.1 · p. 5, Figure 2 · p. 6
• Consommé loses less than 1% performance on DeiT-S and less than 2% on DeiT-T from k = 64 to k = 8, outperforming all methods at lower budgets.At k = 64, fine-tuned methods achieve the highest accuracy, followed by Consommé.Source: §4.1 · p. 5

4.2. Effects of Split Depth

• At k = 8, Consommé performs better than other methods across split depths, while BPM cannot meet that budget at layer 3.DiffRate does not converge at s = 3 for k = 8, and SPOT does not converge at s = 3 at all; most methods degrade when reduction moves earlier.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3. Deployment and Transmission Overhead

• Consommé adds a 0.6M-parameter gate, with overhead reaching 10% for DeiT-S at k = 64; the decoder remains on the server.Relative parameter overhead decreases with wider backbones, while gate overhead scales with token budget k.Source: §4.3 · p. 7
• Under the stated precision assumption, server-side installation overhead amortizes after ∼1.1k images on DeiT-S, ∼580 on DeiT-T, and ∼1k on ViT-B.Tier-3 methods require transferring 5.7M parameters at s = 9 and 16.4M at s = 3 on DeiT-S, whereas Consommé’s cost does not grow with earlier splits.Source: §4.3 · p. 7

4.4. Ablation Study

• Removing reconstruction reduces accuracy, with the gap reaching 2.5% at k = 8; without reconstruction, the bottleneck relies on classification and KL losses.The authors attribute the gap to reconstruction loss rather than whether the gate conditions on compressed z or raw CLS.Source: §4.4 · p. 8
• The static learned gate achieves 74.74%, more than 3.5% below the full method, while the ablations still outperform baselines at k = 8.The authors attribute the largest share of aggressive-budget strength to learned soft assignment, with image conditioning adding performance and reconstruction loss accounting for the remaining gap.Source: §4.4 · p. 8

5. Limitations

• Consommé assumes a CLS-readout backbone; pooled representations or auxiliary token heads require adapting global-context compression.The paper names MAE and LV-ViT as examples.Source: §5 · p. 8
• Some accuracy loss relative to the standard backbone remains even at the gentle k = 64 budget under split inference.Source: §5 · p. 8
• The cost advantage assumes a publicly hosted shared backbone, and Consommé is less flexible in changing token budget and split layer than tier-1 methods.Source: §5 · p. 8

6. Conclusion

• Consommé compresses CLS into a compact latent, transmits it instead of the full token, and conditions a gate that merges patches into exactly k transmitted tokens.Source: §6 · p. 8
• Its single-step assignment keeps Consommé operable at aggressive budgets and shallow splits where baselines either miss target counts or require expensive fine-tuning.Source: §6 · p. 8
• Consommé achieves higher accuracy with a small adapter on an unmodified public checkpoint rather than a fine-tuned backbone shipped to every endpoint.Source: §6 · p. 8

• Consommé achieves superior results on all reported ViT-B token budgets at split layer s = 9.Source: Table 5 · p. 11
• An asterisk marks results where the exact token count was not reached.Source: Table 5 · p. 11
• At s = 3, BPM methods can exceed the target budget, and a dash marks budgets that are unreachable at that split.The table reports the nearest reachable count when BPM cannot meet the budget.Source: Table 6 · p. 11

7. Results

• Consommé outperforms the other tier-1 and tier-2 methods on ViT-B at s = 9 and retains performance better as k falls from 64 to 8.Source: §7 · p. 11

7.1. Effects of Split Depth

• As splits move earlier, accuracy falls for all methods while Consommé’s margin widens; at s = 3, BPM methods miss targets, DiffRate fails to converge, and SPOT’s training loss becomes non-finite.BPM reaches 25 tokens for s = 3 at k = 8 and k = 16, more than 9 tokens above either target; Consommé satisfies all listed budgets at s = 3.Source: §7.1 · p. 11, Table 8 · p. 12

7.2. Deployment and Transmission Overhead

• Consommé’s decoder break-even point is 584, 1.2k, and 1k images against the CLS-readout baseline, while at s = 3 on DeiT-T tier-3 shipping costs amortize after more than 100 images versus at most 8 for Consommé.The decoder size and cost are split-layer invariant, whereas tier-3 costs grow as the split moves earlier.Source: §7.2 · p. 12, Table 9 · p. 13

7.3. Anchor Assignments and Pruning

• Pruning k = 64 anchors below τ = 0.1 yields 38.3 transmitted tokens on average with a 0.31% performance reduction, while pruning to roughly k = 16 trails a directly trained k = 16 model.The reported accuracies are 77.15% after pruning and 78.57% for the direct model; the table reports pruning by summed assignment weight.Source: §7.3 · p. 12, Table 10 · p. 13