PaperlayerView exact revision on arXiv

Source-linked AI summary

USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation

Md Khalid Syfullah, Alvi Ataur Khalil

arXiv:2609.12168v1cs.CVcs.AI

TL;DR

Privacy-sensitive VQA faces constraints from centralized data aggregation and the full-model burden of federated learning. USPLIT-VQA keeps raw inputs and labels on clients, uses CAWA to score client updates, and shows backbone-dependent accuracy alongside reduced client costs and lower reconstruction quality under evaluated attacks.

  • Problem

    Privacy-sensitive VQA must contend with centralized training that aggregates patient data and distributed methods that can burden clients or expose labels.

    Source: §I · p. 1

  • Method

    USPLIT-VQA keeps encoders, classification heads, raw inputs, and labels on clients while the server hosts intermediate fusion layers, and CAWA scores client contributions using gradient similarity.

    Source: §I · p. 1, B. U-Shaped Split Learning Framework · p. 3

  • Results

    The Custom model exceeds FL in average accuracy while BiomedCLIP trails under the evaluated fixed split; memory falls 5.7–5.8× versus FL, and one malicious client's weight is suppressed by 98%.

    Source: H. Summary of Findings · p. 6

  • Takeaways & Limitations

    USPLIT-VQA combines label locality with lower client memory than FL, while adding Byzantine robustness to the split-learning setting.

    Source: C. Comparison with Federated Learning · p. 5

    The IID assumption does not reflect clinical heterogeneity, large pretrained backbones lose accuracy under the evaluated split, and CAWA weakens as malicious-client proportion increases.

    Source: H. Summary of Findings · p. 6

Abstract

from arXiv · show

Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.

I. INTRODUCTION

Privacy-sensitive VQA faces a deployment trade-off: centralized training aggregates patient data, while distributed approaches can burden clients or expose labels. USPLIT-VQA keeps raw inputs and labels on clients and adds contribution-aware aggregation to address harmful updates.

I. INTRODUCTION

  • Medical VQA's centralized training aggregates patient data, while FL requires clients to host the full model and standard SL exposes labels to the server.The introduction frames these as privacy and resource constraints for clinical deployment.Source: §I · p. 1
  • USPLIT-VQA keeps raw images, questions, and labels on clients while assigning the computationally heavy intermediate layers to the server.Clients host the encoders and classification head; the server handles multimodal fusion.Source: §I · p. 1
  • CAWA scores client contributions using gradient similarity to reduce the influence of malicious or low-quality updates.The framework also investigates streak-based reputation scoring against poisoning and backdoor attacks.Source: §I · p. 1
  • The evaluation covers four VQA datasets and two backbones, benchmarking centralized and federated baselines alongside inversion, poisoning, split-point, and scalability analyses.The stated research questions also ask whether privacy is preserved and how accuracy, communication, and client burden compare.Source: §I · p. 1

II. BACKGROUND

The background introduces distributed learning and contrasts three paradigms by client workload, server access to labels, and where computation occurs.

II. BACKGROUND

A. Distributed Learning Paradigms

  • FL keeps a full model on each client, standard SL offloads later computation but exposes labels, and U-shaped SL keeps both encoders and the classification head on clients.In U-shaped SL, the server hosts middle layers and the loss is computed locally, so raw data and labels are not exposed to it.Source: A. Distributed Learning Paradigms · p. 2

B. Threat Models in Distributed Learning

The section outlines inversion and poisoning threats, reviews distributed VQA approaches, and identifies gaps in supervised multimodal VQA, contribution scoring, and cost analysis.

B. Threat Models in Distributed Learning

  • Gradient inversion reconstructs training data from transmitted signals, model inversion targets inputs from split-boundary representations, and poisoning submits adversarial gradients.The described poisoning examples include label flipping and backdoor injection.Source: B. Threat Models in Distributed Learning · p. 2

III. RELATED WORK

  • Prior VQA work includes centralized vision-language models, FL approaches requiring clients to host full architectures, and BiCSL, which uses contrastive learning rather than supervised classification.The reviewed BiCSL work is not evaluated on clinical datasets and does not use U-shaped splitting.Source: §III · p. 2
  • Earlier U-shaped configurations address unimodal distributed settings, while this work adapts the topology to supervised multimodal VQA with contribution-aware aggregation.The related-work discussion also notes split-learning inversion attacks and Byzantine-robust aggregation for FL.Source: §III · p. 2
  • Three gaps remain: limited study of U-shaped split learning for supervised multimodal VQA, underexplored client contribution scoring, and absent comprehensive VQA cost comparisons against FL.Source: §III · p. 2

IV. METHODOLOGY

The methodology compares a pretrained BiomedCLIP backbone with a lightweight custom architecture to examine USPLIT-VQA across model designs.

IV. METHODOLOGY

  • The methodology presents technical details and introduces a notation table for symbols used throughout the paper.Source: §IV · p. 2

A. Model Architectures

  • The model study pairs pretrained BiomedCLIP with a lightweight custom architecture to compare performance across architectures.Source: A. Model Architectures · p. 2

1) BiomedCLIP-Based Model:

  • The BiomedCLIP-based model uses a ViT-B/16 vision encoder and PubMedBERT text encoder at D=768, with shared cross-modal fusion and classification architecture.The vision encoder produces 196 patch tokens and is pretrained on PMC-15M.Source: 1) BiomedCLIP-Based Model · p. 3

2) Custom Lightweight Model:

  • The custom model has about 8.5M parameters at D=256 and combines convolutional visual encoding, text transformers, CBAM refinement, bidirectional cross-attention, and an MLP classification head.Its visual encoder produces 49 spatial tokens from a 7×7 feature grid.Source: 2) Custom Lightweight Model · p. 3

B. U-Shaped Split Learning Framework

USPLIT-VQA partitions VQA processing across client and server in a U-shaped flow. Raw images, questions, and labels remain on the client while feature activations and gradients cross the split.

B. U-Shaped Split Learning Framework

  • The client encodes raw images and questions into visual and text tokens, then sends those feature activations to the server.The layout follows a client-head, server-body, client-tail flow.Source: B. U-Shaped Split Learning Framework · p. 3, Figure 2 · p. 3
  • The server processes the features and returns a fused vector; the client computes the label-based loss and sends its logit gradient back.The server returns activation gradients, while raw images, questions, and labels stay on the client.Source: B. U-Shaped Split Learning Framework · p. 3
  • For BiomedCLIP, the evaluated fixed split places 38.8M parameters on the client and 187.9M on the server; alternative split positions were not evaluated.Client encoders are frozen for four rounds, then unfrozen.Source: B. U-Shaped Split Learning Framework · p. 3

C. Contribution-Aware Weighted Aggregation (CAWA)

CAWA scores client updates using gradient alignment and adjusts client trust over time. Its reputation-derived weights scale client loss, while adaptive thresholds and streak-based updates shape rewards and penalties.

C. Contribution-Aware Weighted Aggregation (CAWA)

  • CAWA evaluates gradient alignment against adaptive statistical bounds and uses historical momentum to scale client trust.Each client's reputation is converted to a bounded trust weight with a temperature-scaled softmax.Source: C. Contribution-Aware Weighted… · p. 3
  • The reputation-weighted similarity score compares each client's gradient with those of its peers, weighting peers by reputation to limit malicious-client consensus.The resulting client weight directly scales its loss.Source: C. Contribution-Aware Weighted… · p. 3
  • Dynamic similarity thresholds are set as τ± = µ ± λσ, and the temporal factor ρ = (r/R)^ϕ dampens early-round volatility.The mean and standard deviation come from the current round's similarity distribution.Source: C. Contribution-Aware Weighted… · pp. 3–4
  • Reputation receives streak-amplified rewards or penalties according to consecutive positive or negative client evaluations.The algorithm updates the streak counter and reputation when a similarity score falls above or below the adaptive bounds.Source: C. Contribution-Aware Weighted… · p. 4

D. Datasets Utilized

The evaluation uses four VQA datasets: VQA-RAD, SLAKE, PathVQA, and VizWiz.

D. Datasets Utilized

E. Split Point & Client Scalability Analysis

The Custom model outperforms FL on all four datasets and matches or exceeds centralized performance on three, whereas BiomedCLIP performs worse than FL under its evaluated fixed split. The section also describes split configurations and experimental conditions used to assess accuracy and scalability.

E. Split Point & Client Scalability Analysis

V. EXPERIMENTAL RESULTS

  • The experimental-results section presents the setup, results, and strengths and weaknesses of the contribution.Source: §V · p. 4

A. Experimental Setup

  • Experiments compare against centralized training and FL using five IID-partitioned clients for 20 rounds, with scalability runs extended to 30 rounds.Security and split analyses use the Custom model on SLAKE.Source: A. Experimental Setup · p. 4

B. Accuracy Comparison with Centralized and FL Baselines

  • With the Custom model, USPLIT-VQA exceeds FL on all four datasets and matches or exceeds centralized performance on three; with the fixed BiomedCLIP split, FL performs better on all four.The largest Custom-model gain over FL is 7.54 percentage points on VQA-RAD; the largest BiomedCLIP gap is 10.93 points on PathVQA.Source: B. Accuracy Comparison with… · p. 4, Table IV · p. 4

C. Comparison with Federated Learning

Against FL, USPLIT-VQA reduces client resource demands, while CAWA limits a single malicious client's influence under the evaluated attack. Inversion attacks also produce lower reconstruction quality in the reported evaluations.

C. Comparison with Federated Learning

  • Client memory falls 5.8× for BiomedCLIP and 5.7× for Custom, while communication falls 10.8× and 6.1×, respectively, versus FL.Average per-round time falls 47% for Custom and 29% for BiomedCLIP, but on VQA-RAD USPLIT-VQA takes 36.2 s versus 25.4 s for FedAvg.Source: C. Comparison with Federated Learning · p. 5, Figure 4 · p. 5

D. Byzantine Robustness Evaluation

  • CAWA suppresses one attacker's weight by 98.2% while maintaining 65.2% accuracy; performance degrades as the malicious-client proportion rises.The compound attack flips 30% of labels and applies an 8×8 white trigger to 20% of samples.Source: D. Byzantine Robustness Evaluation · p. 5
  • With one malicious client, USPLIT-VQA records 65.2% accuracy and 0.6% ASR versus 61.2% and 44.1% for uniform aggregation; at two or more attackers, all frameworks have high ASR.Under three malicious clients, USPLIT-VQA retains the highest clean accuracy.Source: Table VI · p. 5, D. Byzantine Robustness Evaluation · p. 5

E. Robustness Against Privacy Attacks

  • Under evaluated attacks, model inversion has 10× higher MSE and 9.9 dB lower PSNR than centralized training; DLG PSNR stays below 8 dB, with USPLIT-VQA lowest at 5.70 dB.The reported reconstruction quality is lower within this evaluation, not a general privacy guarantee.Source: Table VII · p. 5, E. Robustness Against Privacy Attacks · p. 5

F. Comparison with BiCSL

USPLIT-VQA compares favorably with BiCSL on clean accuracy and resistance to a 30% label-flip attack, while its U-shaped round trip adds communication overhead. Split-point and scalability results show that accuracy and efficiency optima can differ.

F. Comparison with BiCSL

G. Split-Point and Client Scalability Analysis

  • V3 gives the highest accuracy (69.75%), while V4 is nearly as accurate (69.18%) with the lowest round time; across K=5–15, accuracy reaches 65–70% within 30 rounds.On SLAKE, average per-round time grows sub-linearly from 23.1 s at K=5 to 35.0 s at K=15; more client-side computation reduces exchanged activations and gradients.Source: Figure 6 · p. 5, G. Split-Point and Client Scalability… · p. 6

H. Summary of Findings

  • The Custom model exceeds FL in average accuracy, whereas BiomedCLIP falls behind under the evaluated fixed split; client memory falls 5.7–5.8× on both backbones.Source: H. Summary of Findings · p. 6
  • Evaluated inversion attacks reconstruct inputs less faithfully under USPLIT-VQA, and CAWA suppresses one malicious client's weight by 98%, though backdoor protection degrades with two or more attackers.Source: H. Summary of Findings · p. 6
  • The IID assumption misses clinical heterogeneity, large pretrained backbones lose accuracy under the evaluated split, and CAWA weakens as malicious-client proportion increases.Source: H. Summary of Findings · p. 6

VI. CONCLUSION

  • Future work will address non-IID adaptation and improve split selection for large pretrained models.Source: §VI · p. 6

Related papers

Loading 2609.12168v1…