PaperlayerView exact revision on arXiv

Source-linked AI summary

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan, Farzad Tofigh, Ian Zhou, Justin Lipman

arXiv:2610.22906v1cs.CLcs.DCcs.LG

TL;DR

Federated learning supports private training across distributed clients, but computational and communication demands challenge its use for NLP and multimodal tasks. FederatedBytes combines tokenizer-free byte processing and linear attention in a small model for these settings. The paper reports higher accuracy than ByT5 and LSTM alongside reduced training time, communication, and computational overheads.

  • Problem

    Tokenization and the computational and communication demands of NLP and multimodal workloads complicate their use in resource-constrained federated settings.

    Source: §1 · p. 1

  • Method

    FederatedBytes is a small federated model that processes raw bytes without tokenization and uses linear attention for multilingual and multimodal data.

    Source: §1 · pp. 1–2

  • Results

    FederatedBytes reports higher accuracy than ByT5 and LSTM while reducing training time, communication, and computational overheads across Multilingual Amazon Reviews and ScienceQA evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that byte-level processing with Performer attention can support diverse client data and efficient processing on resource-constrained edge devices.

    Source: §7 · p. 14

Abstract

from arXiv · show

Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such as natural language processing (NLP) and multimodal analysis. For this reason, this paper presents a novel small language model that leverages byte-level text and multimodal data processing, enhanced by a linear attention mechanism, to enable manageable Federated Learning of NLP and multimodal analysis tasks. The proposed model, nicknamed FederatedBytes, manages to process text and any other data types, including images, by treating its input as a mere sequence of bytes, while its lean linear attention mechanism allows abating the overall memory and computational footprint. To prove the effectiveness of the proposed model we have carried out an extensive performance evaluation on sentiment analysis tasks in multilingual and multimodal settings over the Multilingual Amazon Reviews and ScienceQA datasets. The experimental results show that FederatedBytes has been able to outperform comparable models such as ByT5 and LSTM, achieving higher accuracy while significantly reducing memory and communication overheads.

1. Introduction

Federated learning can preserve client data privacy, but tokenization and the resource demands of NLP and multimodal workloads complicate its use on edge devices. FederatedBytes combines byte-level processing with linear attention for federated multilingual and multimodal tasks, and the paper reports stronger accuracy with lower overhead than comparable models.

1. Introduction

  • Federated learning trains a global model from multiple clients while preserving the privacy of their local data.The approach is motivated by distributed settings with sensitive data and limited bandwidth or computational resources.Source: §1 · p. 1
  • Tokenization relies on predefined language-specific or multilingual vocabularies, can struggle with out-of-vocabulary words, and impedes the use of non-text inputs such as images and videos.These demands are especially acute on resource-constrained edge devices and in federated learning with limited network bandwidth.Source: §1 · p. 1
  • FederatedBytes processes raw UTF-8 bytes without tokenization, handling any language or data format representable as a byte sequence.Its linear attention mechanism abates standard Transformer's quadratic complexity to support long sequences on resource-constrained devices.Source: §1 · p. 1
  • The paper presents FederatedBytes as a small language model for federated NLP that handles multilingual and multimodal data on edge devices.The model's design combines tokenizer-free byte processing, linear attention, and integration into a privacy-preserving federated learning framework.Source: §1 · p. 2
  • FederatedBytes outperforms ByT5 and LSTM in accuracy while reducing training time, communication, and computational overheads across the Multilingual Amazon Reviews and ScienceQA evaluations.The experiments cover multilingual and multimodal settings.Source: §1 · p. 2

2. Related Work

Related work covers federated NLP, byte-level processing, and efficient Transformer architectures, but the paper positions FederatedBytes as uniting these strands for decentralized NLP. Prior work leaves byte-level models' federated deployment and tokenization in federated frameworks as open challenges.

2. Related Work

  • FederatedBytes is positioned as the first model to merge federated learning, byte-level processing, and efficient Transformer architectures in a unified framework for decentralized NLP.The stated aim is private, efficient, multilingual NLP on decentralized data.Source: §2 · p. 2
  • Federated NLP frameworks include FedAvg (McMahan et al. (2017)), Gboard next-word prediction (Hard et al. (2018)), FedNLP (Lin et al. (2021)), and personalized FedTP (Zhang et al. (2022)), but predominantly use tokenized text.FedTP adapts client self-attention to counter drift from non-IID data.Source: §2 · p. 2
  • ByT5 (Xue et al. (2022)) and byte-level GPT (bGPT) (Wu et al. (2024)) operate on raw byte sequences, while Meta AI research (Pagnoni et al. (2024)) further examines byte-level Transformers.The passage describes byte-level models as multilingual, robust to out-of-vocabulary words and noise, and multimodal.Source: §2 · p. 2
  • Byte-level models have been designed and evaluated almost exclusively in centralized settings, leaving their privacy-preserving federated deployment under-addressed.The related-work discussion identifies this as a critical gap.Source: §2 · p. 2
  • Linformer and Performer (Katharopoulos et al. (2020)) and Longformer (Beltagy et al. (2020)) address Transformer costs through linearized attention or memory-efficient architectures.Standard self-attention has quadratic time and memory complexity with input length, especially challenging for long byte sequences.Source: §2 · p. 2
  • Tables 1 and 2 situate the work among research threads and compare specific token- and byte-level models.Table 2 also addresses models' innovations and suitability for federated learning.Source: §2 · p. 2

3. The FederatedBytes Architecture

FederatedBytes is presented as a unified framework combining byte-level processing and efficient attention for resource-constrained federated settings. Its architecture patches and encodes byte inputs before Transformer processing and supervised classification.

3. The FederatedBytes Architecture

  • FederatedBytes is proposed to unite federated learning, byte-level processing, and efficient attention, addressing the resource demands of byte-level models and tokenization in existing efficient federated models.The framework aims to bring raw-byte processing to resource-constrained edge settings.Source: §3 · p. 2
  • The architecture processes raw byte sequences with linear-complexity attention through sequential input, memory-efficient language-model, and prediction stages.The stages patch and hierarchically encode inputs, apply Performer or Linformer attention to patch representations, then pool latent representations for supervised classification.Source: §3 · p. 2
  • Table 1 is a high-level comparison of token- and byte-level models, while Table 2 compares specific models' innovations and suitability for federated learning.The Table 2 caption notes that centralized models such as ByT5 and BLT are not inherently designed for federated learning's communication and resource constraints.Source: Table 1 · p. 3, Table 2 · p. 3

3.1. Input Processing: Patching and Hierarchical Encoding

  • Raw text is converted to UTF-8 byte values, and each byte is mapped to a continuous vector embedding.The embeddings are the inputs to the subsequent patching and encoding process.Source: §3.1 · p. 3
  • The hierarchical patch encoder divides byte sequences into patches, applies a 1D convolution and ReLU, then adaptively pools each patch's embeddings into one representation.The default patch size is 128; reducing sequence length from L to L/Ps reduces pre-Transformer memory complexity from O(L^2) to O((L/Ps)^2).Source: §3.1 · p. 3

3.2. Memory-E

  • The core is a custom Transformer encoder configured for federated-client resource constraints and bidirectional, noncausal context.The model is described as memory-efficient.Source: §3.2 · p. 3

Core Components and Hyperparameters

FederatedBytes processes byte inputs through a hierarchical patch encoder and linear-attention Transformer, then aggregates encoded features for classification. Its Performer attention avoids constructing the quadratic attention matrix, while the prediction stage maps pooled features to class outputs.

Core Components and Hyperparameters

  • FederatedBytes embeds byte values, compresses them into patch vectors, adds positional embeddings, and processes them with memory-efficient Transformer blocks.The architecture then pools hidden states, applies BatchNorm and dropout, and uses a linear classification head.Source: Core Components and Hyperparameters · pp. 3–4, Figure 1 · p. 4
  • Performer approximates the softmax kernel with random features and reorders matrix multiplication to avoid the L × L attention matrix.This reduces complexity from O(L^2d) to O(Lmd), where m is the number of random features and m ≪ L.Source: Memory-Efficient Attention · p. 4
  • Orthogonal Random Features construct the random feature map, reducing approximation variance and supporting more stable estimates of the softmax kernel.The passage states that this encourages consistent convergence during federated training rounds.Source: Memory-Efficient Attention · p. 5

3.3. Prediction and Classification

  • Global mean pooling compresses the encoded sequence into a fixed-size context vector before BatchNorm, dropout, and final classification.The output probabilities cover K classes; the network is trained end-to-end with cross-entropy loss against the training labels.Source: §3.3 · p. 5

4. Methodology: Model, Data, and Training

FederatedBytes transforms raw text into byte patches, contextualizes them with attention, and pools the resulting features for classification. The methodology also describes supervised optimization and the federated training setup.

4. Methodology: Model, Data, and Training

  • The architecture processes byte-patch representations through attention and classification, within a lightweight byte-level Transformer and federated training pipeline.The methodology identifies hierarchical input representation, bidirectional attention, and federated training as its three core components.Source: §4 · p. 5

4.1.1. Stage 1: Input Processing (Patching and Averaging)

  • The input is converted to UTF-8 bytes, padded to 16,384 positions with token 256, and divided into 128 patches of 128 bytes.In the worked example, all 44 text bytes fit in the first patch; subsequent patches contain padding.Source: §4.1.1 · p. 5
  • Each byte maps to a 768-dimensional embedding, and a 1D convolution plus adaptive pooling compresses each 128 × 768 patch to 1 × 768.This learned aggregation reduces sequence dimensionality while summarizing local information, supporting contexts up to 16,384 bytes.Source: §4.1.1 · pp. 5–6

4.1.2. Stage 2: Memory-E

  • Performer linear attention contextualizes patch representations, modeling dependencies between distant text segments without standard attention’s quadratic cost.The accompanying attention distribution is presented as hypothetical.Source: §4.1.2 · pp. 6–7, Table 5 · p. 6

4.1.3. Stage 3: Prediction & Classification

  • Global average pooling compresses the contextualized features to one vector, which passes through normalization, dropout, and a classifier to produce class probabilities.In the worked example, the classifier predicts positive sentiment.Source: §4.1.3 · p. 7

4.2. Training Process

  • Training minimizes cross-entropy with AdamW, weight decay, and mixed-precision gradients; a physical batch of 2 accumulates gradients over 128 steps for an effective batch of 256.The described procedure updates approximately 30-million- and 100-million-parameter variants.Source: §4.2 · p. 7

4.3. Evaluation Protocol

Evaluation measures performance on held-out test data without updating model weights. The paper also describes federated averaging across clients and a set of privacy and security protections.

4.3. Evaluation Protocol

  • Test-set evaluation disables weight updates and aggregates predictions to calculate accuracy, weighted F1, and AUC.The evaluation protocol states that training and test sets are disjoint.Source: §4.3 · pp. 7–8

4.4. Federated Learning Framework

  • Selected clients train locally for five epochs, then the server combines their model updates using sample-count-weighted FedAvg.The simulation scales cohorts from 5 to 100 clients on one NVIDIA A100 GPU.Source: §4.4 · pp. 8–9

4.5. Security and Privacy Considerations

  • Privacy protections include localizing raw data and secure aggregation; the framework also describes client-side differential privacy, encrypted TLS communication, and robust aggregation against poisoning.The passage describes differential privacy and robust aggregation as methods that can be applied or incorporated.Source: §4.5 · p. 9

5. Experimental Setup

The experiments compare FederatedBytes variants with smaller token-level, byte-level, and recurrent baselines in centralized training, then benchmark top-performing models in simulated federated learning. They use multilingual reviews and multimodal ScienceQA, with performance and resource measures.

5. Experimental Setup

  • The experiments evaluate standard-attention and Performer variants at approximately 30M and 100M parameters against GPT-2 Medium, ByT5, and a 2-layer LSTM; centralized training precedes federated benchmarking of top models.The compared baselines are described as having fewer than 500M parameters.Source: §5 · p. 9

5.1. Datasets

  • The datasets cover English, German, and Chinese Amazon review sentiment classification and multimodal ScienceQA question answering with optional images.Amazon reviews are labeled positive for 4–5 stars and negative otherwise.Source: §5.1 · p. 9

5.2.1. Model Performance Metrics •

  • Classification performance is assessed with accuracy, weighted F1, and AUC, with multiclass AUC reported as weighted one-vs-rest.Weighted F1 accounts for class imbalance.Source: §5.2.1 · p. 10

5.2.2. Resource

  • Resource measures include parameter count, model size, experiment time, peak GPU and CPU RAM, and simulated network traffic sent and received.Network footprint is calculated from clients per round, model size, and number of rounds.Source: §5.2.2 · p. 10

5.3. Baseline Models

  • The baselines span BPE-tokenized GPT-2 Medium, tokenizer-free ByT5, and a byte-embedded 2-layer LSTM.GPT-2 Medium is limited to 1,024 tokens; the LSTM processes sequences up to 1,024 bytes.Source: §5.3 · p. 10

5.4. Proposed Model

  • FederatedBytes uses hierarchical byte-patch embeddings for contexts up to 16,384 bytes; its Performer variant uses four FAVOR+ linear-attention layers in 30M and 100M sizes.The Base variant instead uses a standard two-layer TransformerEncoder with quadratic attention.Source: §5.4 · p. 11

6. Results

Across centralized and federated evaluations, FederatedBytes compares favorably with ByT5, while GPT-2 achieves the strongest accuracy metrics on text-only tasks. In federated settings, the 30M Performer variant maintains AUC as client counts rise and uses fewer resources than ByT5.

6.1. Centralized Results

  • On Amazon (en), FederatedBytes-Performer-100M achieved 80.50 pp AUC versus ByT5’s 58.45 pp, in less than half the runtime and with 1/10 of its GPU memory.GPT-2 achieved the strongest accuracy metrics on the text-only Amazon tasks, but was approximately 3.5× larger than FederatedBytes-Performer-100M.Source: §6.1 · p. 11

6.2. Federated Learning Results

  • At 100 clients, FederatedBytes-Performer-30M reached AUCs of 65.96% (en), 72.87% (zh), and 66.71% (de), while ByT5 fell to approximately 50–52%.Figure 3 summarizes the AUC trends across Amazon (en), (zh), and (de).Source: §6.2 · p. 11, Figure 3 · p. 12
  • At 100 clients, FederatedBytes-Performer-30M trained in 56 min versus 558 min for ByT5 and used 152 MB peak memory versus 3.4 GB.It also consumed significantly less CPU and network bandwidth than ByT5.Source: §6.2 · p. 11
  • FederatedBytes-Performer frequently outperformed its centralized counterpart; on Amazon (en), the reported AUCs were 78.8% federated and 71.9% centralized.The authors suggest federated averaging acts as an effective regularizer for the byte-level architecture.Source: §6.2 · p. 11
  • On ScienceQA, FederatedBytes-Performer reported 62.60% AUC versus ByT5’s 53.40%.The passage reports this as a multimodal result.Source: §6.2 · p. 11
  • As client counts increase, the 30M model achieves higher AUC than the 100M model, which the authors attribute to the larger model overfitting smaller local datasets.They describe conflicting noisy client updates as degrading the global model, while the smaller model learns more stable general patterns.Source: §6.2 · p. 11

7. Conclusion

The paper presents FederatedBytes as a byte-level transformer designed for federated learning. It identifies memory-sparing Performer attention as a way to mitigate standard self-attention’s quadratic cost, and outlines future extensions to multimodal capabilities, client scale, context length, and personalization.

7. Conclusion

  • FederatedBytes is presented as the first byte-level transformer expressly engineered for federated learning environments.The conclusion says byte-level models can avoid language-specific vocabularies, remove the vocabulary-leak privacy gap, and process diverse data from clients.Source: §7 · p. 14
  • Memory-sparing Performer attention mitigates the quadratic cost of standard self-attention, permitting efficient processing on resource-constrained edge devices.Source: §7 · p. 14
  • Future work targets deeper multimodal capabilities, larger client populations, longer contexts, and personalized optimization for diverse user data.The stated aim is to preserve privacy and communication efficiency.Source: §7 · p. 14

Appendix A. Hyperparameters

The appendix points readers to Table A.16 for the proposed model’s main hyperparameters.

Appendix A. Hyperparameters

Appendix B. GPT-2 Model Configuration and Model Card

The appendix describes a GPT-2 baseline configuration and model card. It lists the model’s architecture, vocabulary and context window, and notes limitations including potentially incorrect or biased text generation.

Appendix B. GPT-2 Model Configuration and Model Card

  • The listed GPT-2 configuration has 12 layers, hidden size 768, 12 attention heads, and intermediate feed-forward size 3072.The appendix says the baseline follows the HuggingFace gpt2 recipe.Source: Appendix B · p. 15
  • The configuration lists a vocabulary size of 50,257 and a context window of 1024 tokens.Source: Appendix B · p. 15
  • The model card lists general-purpose language modeling, next-token prediction, and classification fine-tuning as use cases, and warns of plausible-sounding incorrect or biased text.Source: Appendix B · p. 15

Appendix C. Federated Training Algorithm

Algorithm 1 outlines FederatedBytes’ simulated federated training, from global-model initialization through client-side processing and server aggregation. The procedure combines byte-level input handling with local optimization and FedAvg.

Appendix C. Federated Training Algorithm

  • Table A.16 lists the key hyperparameters and configuration settings used in the experiments.Source: Table A.16 · p. 15
  • The simulation takes a global model and training settings as inputs and returns the final global model θT.The listed settings include learning rate, clients, rounds, local epochs, micro-batch size, accumulation steps, and patch size.Source: Appendix C · p. 16
  • In each round, the server selects a client subset, broadcasts the global model, and initializes the selected clients for parallel local training.Each client resets its gradient accumulator and iterates over local epochs and micro-batches.Source: Appendix C · p. 16
  • Clients UTF-8 encode inputs, pad them to 16,384 bytes, form 128-byte patches, then apply embedding, convolutional pooling, Performer attention, mean pooling, and classification.The procedure labels Performer attention as linear O(L) attention.Source: Appendix C · p. 16
  • Clients accumulate gradients and update local models with AdamW, then send their models to the server for FedAvg aggregation.An update occurs every M steps or at the end of the data, after which the accumulator is reset.Source: Appendix C · p. 16

Related papers

Loading 2610.22906v1…