Source-linked AI summary
Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs
Dario Fenoglio, Gabriele Dominici, Martin Gjoreski, Marc Langheinrich
TL;DR
Collaborative learning commonly exchanges model parameters, creating communication costs that scale with model size and requiring specializations to be reconciled in an interference-prone weight space. This paper introduces DSL, which shares locally adapted model behavior through task-unrelated carrier outputs that a coordinator distills into a shared model. Across LLM composition and federated classification, the reported results show preference retention, competence transfer, and tunable accuracy–communication trade-offs.
Problem
Parameter-centric collaboration scales communication with model size and reconciles independently learned specializations in an interference-prone weight space.
Source: §1 · p. 2Method
DSL has participants send outputs from locally adapted models on task-unrelated random carriers, which a coordinator pools and distills into a shared model.
Source: §1 · p. 2Results
Across LLM composition and federated classification, DSL preserves controlled preferences, transfers mathematical competence, and provides tunable accuracy–uplink trade-offs; upload falls by 30.6–49.0×.
Source: §6 · p. 7Takeaways & Limitations
Random-carrier outputs provide a communication primitive for knowledge sharing across one-shot foundation-model composition and iterative federated learning.
Source: §1 · p. 2, §6 · p. 7The evaluation covers one LLM family and mostly IID, full-participation federations; extending carrier-based communication to independently pretrained or heterogeneous models remains open.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0$\times$. In federated classification, DSL reaches 96.83% on MNIST with 8.9$\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.
1 Introduction
The paper replaces parameter-centric collaboration with knowledge sharing through model behavior on task-unrelated inputs. It introduces DSL and evaluates it in one-shot LLM composition and iterative federated learning.
1 Introduction
- Parameter-centric collaboration scales communication with trainable model size, reconciles specializations in an interference-prone weight space, and can expose gradients and updates to privacy leakage.Existing federated distillation approaches typically rely on task-related public data, class-level summaries, or a learned generator.Source: §1 · p. 2
- DSL turns subliminal learning into a collaborative primitive: participants send outputs from locally adapted models on random carriers, and a coordinator pools and distills them into a shared model.The paper describes carrier completions for combining distinct LoRA-adapted language-model preferences.Source: §1 · p. 2
- DSL communicates task-unrelated model behavior instead of adaptations, with carrier completions for one-shot composition and carrier logits for iterative, proxy-free federated learning.Avoiding direct update transmission removes a known leakage surface but does not itself provide a privacy guarantee.Source: §1 · p. 2
- DSL retains controlled preferences better than LoRA averaging (94.56% vs. 87.76%) and yields a 22.0-point GSM8K gain over the base model versus 0.6 points, with 30.6–49.0× less upload.In federated classification, it nearly matches FEDAVG on MNIST and offers lower-communication operating points on CIFAR-10 and Tiny ImageNet.Source: §1 · p. 2
2 Distributed Subliminal Learning
DSL replaces parameter exchange with pooled outputs from task-unrelated random carriers, which a coordinator distills into a shared model. The paper instantiates this approach with language-model completions and federated-classification logits.
2.1 Blueprint: communication through random carriers
- Carriers need not resemble task data, but must elicit outputs that distinguish relevant local behaviors; carrier distribution and budget are problem dependent.The coordinator pools participant messages and distills them using normalized participant weights and an appropriate loss; carriers may be reproduced from shared pseudorandom seeds rather than transmitted.Source: §2.1 · p. 3
2.2 LLM composition from random completions
- In one-shot LLM composition, participants transmit sampled completions to task-unrelated prompts, and the coordinator fine-tunes a shared model initialized from the common base.These completions act as function-space signatures of local specializations, so the coordinator merges behavior rather than parameters; communication scales with completion count and length rather than adapter size.Source: §2.2 · p. 3
2.3 Federated learning from random logits
- In iterative federated learning, clients adapt the current global model and upload logits on task-agnostic random inputs or representations, without public examples or a learned generator.Near the current global model, the analysis characterizes recovery of weighted client updates along carrier-identified directions, but does not establish convergence for nonlinear SGD.Source: §2.3 · pp. 3–4
3 Experiments
The experiments evaluate DSL in one-shot LLM composition and iterative federated classification, using random completions or logits as the communicated outputs. DSL retains locally learned preferences and transfers task competence in LLM composition, while federated classification shows accuracy–communication trade-offs across three datasets.
3 Experiments
- The evaluation covers one-shot LLM composition through random-number completions and iterative federated learning through random-logit exchange.In the LLM settings, participants and coordinator share a base model; clients communicate task-unrelated completions, which the coordinator uses to fine-tune a fresh base-model copy.Source: §3 · p. 4, §3.1 · p. 4
3.1.1 Controlled preference composition
- 94.56% average preference retention is only 2.98 points below the local model, while DSL completions use 1.58–2.52 MiB per participant.The reported upload is 30.6–48.8× less than the adapter payload; neutral completions carried multiple local behaviors without stating them in the prompts.Source: Table 1 · p. 4, §3.1.1 · p. 4
3.1.2 GSM8K composition
- 32.2% GSM8K accuracy gives DSL a 22.0-point gain over the base, versus 0.6 points for LORA AVG.DSL uses 0.72 MiB per participant rather than 35.27 MiB of adapter parameters, roughly a 49× upload reduction.Source: Table 2 · p. 5, §3.1.2 · p. 5
3.2 Federated classification
- 96.83% MNIST accuracy is 0.51 points below FEDAVG with 8.9× less uplink; Tiny ImageNet reaches 63.77%, 4.35 points below FEDAVG, with 7.5× less.On CIFAR-10, uplink falls 136.3× at a 9.94-point accuracy cost; a higher-budget, refined-carrier setting reaches 63.02 ± 1.15% with 7.08× less uplink than FEDAVG.Source: §3.2 · p. 5, Table 3 · p. 6
4 Related Work
Prior model-composition and federated-learning methods generally communicate parameters, while distillation alternatives use public or proxy data, generators, or model communication. DSL instead communicates local model behavior on task-unrelated carriers.
Model merging and subliminal transfer
- Model soups, Fisher merging, task arithmetic, TIES, DARE, and LoRA composition combine adapted models in parameter space; DSL composes their behavior through generated outputs instead.DSL’s mechanism is inspired by subliminal learning, where traits transferred through semantically unrelated generated data.Source: §4 · p. 6
Federated optimization and communication efficiency
- Communication-efficient federated methods reduce update size or frequency, but remain update-centric; DSL replaces updates with outputs on task-unrelated carriers.FEDAVG and variants aggregate parameter updates, while other methods use structured or sketched updates, sharding, compression, or quantization [1] [17].Source: §4 · p. 6
Federated and data-free distillation
- FEDMD uses public data, FEDDF distills uploaded models on proxy data, and FEDGEN and DFRD retain model communication; FSL instead uses fixed task-agnostic carriers.FSL requires neither model or update uploads, task-related public or proxy data, nor a learned or optimized generator.Source: §4 · p. 6
5 Discussion
DSL’s payload depends on carrier budget and output size rather than adapted parameter count, and the paper identifies unresolved limits in its assumptions and evaluation. The discussion also describes possible hybrid collaboration using parameter-space models to initialize carrier distillation.
Limitations
- The analysis assumes a shared initialization; extending carrier communication to independently pretrained or heterogeneous models remains open.Evaluation also covers one LLM family and mostly IID, full-participation federations; non-IID data, adversarial participants, broader merging baselines, and decentralized coordination remain to be studied.Source: §5 · p. 6
Benefits and extensibility
- The carrier budget and output size, rather than the number of adapted parameters, control payload, making DSL attractive when coordinator compute is abundant but participant uplink is constrained.The primitive supports different carrier interfaces and output modalities and is not tied to one collaboration pipeline or local adaptation method.Source: §5 · p. 6
- An available merged or averaged model could initialize subsequent carrier-based distillation, enabling hybrid collaboration.Source: §5 · p. 7
6 Conclusion
The conclusion presents DSL as a way to share knowledge through behavior on task-unrelated random carriers rather than model parameters. The generic protocol pools carrier outputs for distillation, with private- and shared-carrier aggregation variants.
6 Conclusion
- DSL replaces parameter exchange with function-space communication through random carriers and distills their outputs into a shared model.Across LLM composition, it preserves preferences and transfers more mathematical competence than LoRA averaging while reducing upload; in federated classification it offers a tunable accuracy–communication frontier.Source: §6 · p. 7
A.1 Generic protocol
- For one-shot LLM composition, participants send random completions for pooled supervised fine-tuning; iterative FSL alternates local training and server-side carrier distillation.Source: §A.1 · p. 11
A.2 Carrier aggregation variants
- Private-carrier pooling distills participant-weighted outputs from independently sampled carriers; shared-carrier averaging instead aggregates outputs on a common carrier set.Carriers may be transmitted or regenerated from an agreed pseudorandom procedure; FSL uplink counts transmitted logits.Source: §A.2 · p. 11
A.3 Local interpretation of the KL objective
Near a common initialization, the quadratic private-carrier objective recovers the weighted client update up to directions invisible to the carrier outputs. Finite-carrier analysis bounds sampling error in that quadratic model, not the full nonlinear training process.
A.3 Local interpretation of the KL objective
- The quadratic objective is minimized at the weighted client update plus the null space of Hq; if Hq is positive definite, that minimizer is unique.Under the linearized logit model, the same minimizer set is exact.Source: §A.3 · p. 12
- This equivalence is local: away from the common initialization, the exact nonlinear KL-mixture minimizer need not equal the parameter average.The theorem describes a second-order mechanism, not exact equivalence for finite optimization.Source: §A.3 · p. 12
A.4 Finite private carriers in the quadratic model
- The finite-carrier analysis works on the identifiable subspace spanned by Hq; directions in its null space do not affect the empirical quadratic loss.The analysis assumes curvature matrices are positive semidefinite and uses the range of Hq to define the coordinates.Source: §A.4 · p. 12
- The finite-carrier proposition controls sampling error in the empirical quadratic model, but not the KL Taylor remainder, server optimization error, or nonlinear round-to-round dynamics.Its bound worsens with poorly conditioned carrier coverage or widely dispersed client updates.Source: §A.4 · pp. 12–13
B Experimental Details
The appendix specifies the LLM-composition and federated protocols, communication accounting, and additional carrier experiments. In the CIFAR-10 sweep, shared-basis accuracy varies with both carrier family and basis size.
B.1 LLM composition details
- LLM composition uses rank-8 LoRA adapters; the coordinator fine-tunes a fresh common-base copy on participant completions, while the LoRA baseline averages adapter tensors.Controlled preferences use Qwen2.5-7B-Instruct and GSM8K uses Qwen2.5-1.5B-Instruct; the preference study uses 40,000 continuations per participant and GSM8K uses 12,000.Source: §B.1 · p. 13
B.2 Federated protocol summary
- Federated experiments use 20 fully participating clients, up to 200 rounds, three seeds, and early stopping; MNIST, CIFAR-10, and Tiny ImageNet use distinct model interfaces and carrier budgets.Private-carrier pooling uses 2,048 pixel carriers for MNIST, 4,096 blurred random-image carriers for CIFAR-10, and 1,024 representation-space carriers for Tiny ImageNet; clients communicate logits on reproducible carriers.Source: §B.2 · pp. 13–14
B.3 Communication accounting
- Communication totals cumulative client-update bytes for parameter-sharing baselines and cumulative carrier-output uploads for FSL, measured until early stopping.For LLM composition, completion-corpus UTF-8 size is compared with serialized LoRA adapter size; reproducible prompts are excluded from upload.Source: §B.3 · p. 14
C Additional Federated Random-Carrier Experiments
- The additional federated experiments use 20 clients, up to 200 rounds, early stopping, and three ghost outputs, except Tiny ImageNet, which uses private-carrier pooling without ghost outputs.Tiny ImageNet carriers enter at the representation interface of a frozen ViT-small backbone.Source: C Additional Federated Random-Carrier… · p. 14
C.1 Carrier protocols and interpretation of the carrier budget
- Private-carrier pooling combines independently sampled client carrier–logit pairs; with 20 clients, a per-client budget of 2,048 can expose the server to about 41,000 probes.Shared-carrier methods instead route client information through the same M-element basis, a difference that matters especially at small M.Source: §C.1 · pp. 14–15
- Carrier families include uniform, blurred, and Fourier inputs; CIFAR-10 results show that increasing basis size helps blurred and Fourier carriers but does not rescue uniform pixel noise.Blurred carriers use Gaussian blur with σ = 1.2, while Fourier carriers scale coefficient amplitude as 1/f^α with α = 1.Source: §C.1 · p. 15, §C.2 · p. 15
- Under early stopping, low communication alone is ambiguous because unsuccessful configurations may terminate after few rounds; accuracy and communication must be considered together.The reported uplink is total synthetic-message communication accumulated before termination.Source: §C.1 · p. 15
C.2 CIFAR-10 with ResNet9
- 56.27% rises to 62.27% and 63.02% for weighted shared blurred carriers as the basis grows from 4,096 to 32,768 and 65,536.Fourier-carrier accuracy rises from 44.98% to 61.16% and 62.06%; uniform carriers remain below 20% at the two largest budgets.Source: §C.2 · p. 15
C.3 MNIST with an MLP
On MNIST, the preferred carrier family changes with the model architecture: uniform carriers work best for the MLP, while blurred and Fourier carriers perform better than uniform carriers with LeNet5.
C.3 MNIST with an MLP
- 96.72% at 32,768 carriers: uniform carriers lead shared-weighted MNIST–MLP settings, rising from 92.39% at 2,048 and outperforming blurred and Fourier carriers.The MLP results also report 95.69% at 16,384 carriers; private-carrier pooling is effective with 2,048 carriers per client.Source: §C.3 · pp. 15–16, Table 5 · p. 16
C.4 MNIST with LeNet5
- With MNIST fixed, LeNet5 changes the preferred carrier family: 1,024 private blurred or Fourier carriers reach approximately 95% accuracy, while uniform carriers are less reliable.For shared carriers, blurred and Fourier configurations recover above 88%–92% at 4,096 and above 92% at 8,192; uniform carriers remain less reliable as M grows.Source: Table 6 · p. 17, §C.4 · p. 17
- The MLP and LeNet5 comparison indicates that carrier suitability depends on the probed architecture, not the dataset alone.The experiments keep MNIST unchanged while replacing the MLP with LeNet5.Source: §C.4 · p. 17
C.5 Representation-Space Carriers on Tiny ImageNet
Representation-space tests on Tiny ImageNet favor uniform over blurred carriers, with performance saturating as the carrier budget grows; CIFAR-10 entropy measurements are diagnostics, not sufficient conditions for transfer.
C.5 Representation-Space Carriers on Tiny ImageNet
- At matched budgets from 512 to 4,096, uniform representation-space carriers outperform blurred carriers on Tiny ImageNet, including 63.77% versus 60.46% at 1,024.The experiment injects carriers at the trainable downstream-layer embedding interface while using a frozen ViT-small feature extractor.Source: §C.5 · p. 17, Table 7 · p. 18
- With a reduced 30k-sample training pool, uniform carriers still outperform blurred carriers by approximately 2.7–4.0 percentage points across matched budgets.Absolute accuracy falls relative to the 50k-sample setting, but the carrier ordering is preserved.Source: Table 8 · p. 18, §C.5 · p. 18
- 63.77% at 1,024 carriers: uniform-carrier accuracy rises from 60.54% at 64 to 62.74% at 256, then does not improve with larger sets despite substantially higher communication.The 1,024-carrier setting is used in the cross-dataset comparison.Source: §C.5 · p. 18, Table 7 · p. 18
C.6 Complete teacher-output entropy diagnostics
- At M = 32,768 on CIFAR-10, uniform carriers have lower marginal output entropy (0.535) than blurred (0.920) or Fourier (0.881) carriers, matching the reported performance gap.For the MLP, uniform carriers instead have the highest marginal entropy in private and shared-weighted settings.Source: Table 9 · p. 19, §C.6 · p. 19
- Output-entropy statistics are diagnostics rather than sufficient conditions for transfer; marginal entropy indicates class-space coverage but not whether carriers encode all functional directions needed for distillation.High per-query entropy may simply indicate uncertain teacher outputs.Source: §C.6 · p. 19
C.7 Ablation on the number of ghost outputs
The ghost-output ablation varies additional logit dimensions that are unsupervised and do not affect task predictions, but whose carrier values are communicated.
C.7 Ablation on the number of ghost outputs
- Ghost outputs add unsupervised dimensions to the communicated output vector, providing additional dimensions through which local model behavior can be expressed without affecting task predictions.Their values on random carriers are included in the transmitted vector.Source: §C.7 · p. 19
- The input-space MNIST and CIFAR-10 experiments use three ghost outputs by default, while the Tiny ImageNet representation-space experiments use none.The paper fixes the input-space setting to three because the ghost-output mechanism is orthogonal to the carrier-distribution and carrier-interface questions studied there.Source: §C.7 · p. 19
- More ghost outputs generally improve attainable MNIST accuracy while increasing communication; Figure 4 presents this accuracy–communication trade-off.Marker size represents the number of ghosts, and error bars show standard error.Source: §C.7 · p. 19, Figure 4 · p. 20