PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Based Schema-Aware Split Learning for Privacy-Preserving Mental Distress Prediction Across Heterogeneous Surveys

Md Khalid Syfullah, Alvi Ataur Khalil

arXiv:2609.15871v1cs.LGcs.AIcs.CR

TL;DR

Privacy constraints and differing survey schemas make collaborative analysis of mental-health surveys difficult. The paper uses schema-aware serialization with a LoRA-adapted LLM split between clients and a server. It reports higher average ANLS than FL in eight of nine settings, with substantially lower per-client compute.

  • Problem

    Privacy safeguards keep sensitive survey data siloed, while differing instruments, scales, and formats create heterogeneous schemas that complicate collaborative assessment.

    Source: §I · p. 1

  • Method

    The framework serializes heterogeneous survey records into a shared language representation and uses a LoRA-adapted LLM in split learning, keeping raw responses local to each institution.

    Source: §I · pp. 1–2

  • Results

    0.708 vs. 0.687 average ANLS; SL beats FL in eight of nine settings and keeps per-client compute three orders of magnitude lower.

    Source: D. Compute and Memory Cost: SL vs FL · p. 5, G. Research Question Discussion and… · p. 5

  • Takeaways & Limitations

    The approach offers a data-local way to collaboratively train across heterogeneous survey schemas without sharing raw responses.

    Source: §I · pp. 1–2

    Evaluation covers four survey domains with 500-sample test sets, and compute estimates are order-of-magnitude figures.

    Source: G. Research Question Discussion and… · p. 5

Abstract

from arXiv · show

Rising societal and lifestyle complexity has been linked to a growing prevalence of mental distress worldwide. Educational institutions, workplaces, clinics, etc. collect large volumes of mental health survey data to understand and reduce this burden. Collaborative analysis of such data could yield effective generalizable predictive models. Privacy constraints and varied survey designs (i.e., different questions, scales, and formats) hinder direct integration. We propose a schema-aware split learning (SL) framework that preserves privacy, using a large language model (LLM) as a shared semantic encoder to harmonize heterogeneous survey schemas across institutions. We serialize each survey record into a natural-language description, unifying disparate survey schemas into a common format. The LLM is fine-tuned for mental distress assessment via Low-Rank Adaptation (LoRA) and partitioned across client and server. Clients retain the raw survey responses locally and run only a lightweight front-end, so original records never leave the institution that collected them. The resource-intensive backbone runs on the server, minimizing client-side computation. Using LLaMA-3.2-3B-Instruct, the framework attains an average ANLS of 0.708 with only 2,000 training samples, surpasses federated learning (FL) in eight of nine settings, and cuts per-client computation by three orders of magnitude, while generalizing to unseen datasets. Overall, it enables accurate, privacy-preserving, and resource-efficient collaborative learning from heterogeneous mental health survey data.

I. INTRODUCTION

Mental-health survey data are difficult to integrate because privacy rules keep records siloed and institutions use different instruments, scales, and formats. The paper combines schema-aware serialization, LoRA, and split learning to support collaborative mental-distress prediction while keeping raw responses local.

I. INTRODUCTION

  • Privacy restrictions and heterogeneous survey schemas jointly impede collaborative mental-distress assessment.Existing methods typically address only one of these challenges.Source: §I · p. 1
  • LoRA adapts the LLM while split learning keeps raw responses and identifiers local, exchanging only cut-layer activations and gradients.The client holds a tokenizer, adapted embedding, and lightweight projection head; the server holds the backbone and classifier. The authors do not claim intermediate tensors are non-invertible.Source: §I · p. 1
  • The framework combines a LoRA-adapted LLaMA-3.2-3B-Instruct with split learning and schema-aware serialization, and is evaluated against FL, prompting, LoRA, and another LLM.The evaluation varies data imbalance and schema heterogeneity.Source: §I · p. 2
  • Schema-aware serialization converts survey responses into natural-language prompts, mapping differing questions and scales into a shared representation.The proposed format maps ordinal responses to text, marks missing values, and adds a schema header.Source: §I · pp. 1–2
  • The study examines unseen-dataset generalization from a small training pool, effects of client data size and survey structure, and SL’s performance and client compute relative to FL.These are the paper’s stated research questions and generalization aim.Source: §I · p. 2

II. BACKGROUND

The background section introduces the concepts underlying the framework: LLMs, survey data processed with LLMs and FT-Transformer, LoRA, and split learning.

II. BACKGROUND

  • The background introduces LLMs as a core concept for the framework.The section is described as introducing concepts behind the framework.Source: §II · p. 2
  • The section covers LLMs and FT-Transformer for survey data.Both are listed among the background concepts.Source: §II · p. 2
  • LoRA and split learning are also introduced as framework concepts.Both are included in the section’s stated scope.Source: §II · p. 2

A. LLMs, LLaMA and OpenBioLLM

This background covers the LLMs used in the study, semantic serialization for survey data, and parameter-efficient adaptation with LoRA.

A. LLMs, LLaMA and OpenBioLLM

  • The study uses LLaMA-3.2-3B-Instruct and also evaluates OpenBioLLM-8B, a biomedical model built on LLaMA-3-8B and fine-tuned on clinical and medical corpora.The passage describes LLaMA-3.2-3B-Instruct as a compact open-weight model balancing performance and efficiency.Source: A. LLMs, LLaMA and OpenBioLLM · p. 2

B. Survey Data with LLMs and FT-Transformer

  • Natural-language serialization can map semantically related survey features across differing schemas, while FT-Transformer learns feature interactions directly from tabular data.Because FT-Transformer learns a data-derived feature space rather than feature semantics, transfer across differing schemas is limited.Source: B. Survey Data with LLMs and… · p. 2

C. Low-Rank Adaptation (LoRA)

  • LoRA freezes pretrained weights and trains a small set of low-rank adapters, reducing the resource demands of adapting an LLM.The passage presents this as suitable for domain-specific tasks with limited resources.Source: C. Low-Rank Adaptation (LoRA) · p. 2

D. Split Learning (SL)

Split learning partitions computation between clients and a server while keeping data local. The related-work review positions the paper at the intersection of lightweight, data-local clients, LoRA-adapted LLMs, and heterogeneous survey schemas.

D. Split Learning (SL)

  • Split learning keeps data on clients while exchanging intermediate representations with a server, allowing resource-constrained clients to participate without hosting the model.SL partitions the model across client and server, unlike FL, which shares the model.Source: D. Split Learning (SL) · p. 2, §III · p. 2

III. LITERATURE REVIEW

  • Prior PEFT approaches such as SplitLoRA and HSplitLoRA extend parameter-efficient adaptation to distributed learning, reducing communication and computation.These approaches let clients join LLM-backbone training without sharing raw data.Source: §III · p. 2
  • Semantic serialization supports shared representations across schemas, but related approaches do not jointly address resource-constrained institutions, sensitive mental-health data, and heterogeneous surveys.Prior mental-health prediction work with privacy-preserving FL assumes a common feature schema.Source: §III · pp. 2–3
  • The proposed framework combines data-local SL, a LoRA-adapted LLM encoder, and heterogeneous-schema support, needs that prior methods satisfy at most two at once.The review also identifies strong cross-schema performance from a small training pool as an unmet need.Source: §III · p. 3

IV. METHODOLOGY

The methodology standardizes survey records into instruction-following samples and prevents respondent overlap across data splits. It combines data cleaning with schema-aware serialization.

A. Schema-Aware Serialization

  • Four public surveys are cleaned, their column names and boolean encodings harmonized, DASS validity checked, and identifiable and demographic columns removed.Source: A. Schema-Aware Serialization · p. 3
  • Each respondent record becomes multiple instruction-following samples, one per outcome, with the predicted outcome removed from the input to prevent leakage.Source: A. Schema-Aware Serialization · p. 3
  • Splits are made by respondent, so every sample derived from one respondent stays in exactly one of the training, validation, or test sets.Exact input-output duplicates are dropped before splitting.Source: A. Schema-Aware Serialization · p. 3

B. Data Partitions

The methodology varies schema heterogeneity across data partitions and separates client-side processing from the server-held language-model trunk. Raw survey responses, identifiers, and labels remain on clients.

B. Data Partitions

  • Three partitions increase schema heterogeneity from separate datasets, to OST and SD, to all four surveys; Hg evaluation holds out 500 respondent-disjoint samples per test set.Respondents are also confined to one client when training data are sharded.Source: B. Data Partitions · p. 3

C. Split-Learning Architecture

  • Only LoRA matrices A and B are updated, reducing trainable parameters to roughly 0.75% of the model.Source: C. Split-Learning Architecture · p. 3
  • Each client uses a tokenizer, LoRA-adapted embedding, and roughly 100K-parameter projection head to map token IDs to a cut activation sent to the server.The zero-initialized residual MLP begins as an identity mapping and learns a correction.Source: C. Split-Learning Architecture · p. 4
  • The server runs the LoRA-adapted transformer trunk and language-model head, computes loss, and returns cut gradients for the client’s backward pass.Source: C. Split-Learning Architecture · p. 4
  • Only cut-layer activations and gradients cross the client-server boundary; raw responses, identifiers, and labels are never transmitted.Source: C. Split-Learning Architecture · p. 4
  • Nine setups cross three schema-heterogeneity levels with symmetric, power-law, or highly imbalanced client data volumes.High imbalance places 60% of the data with one client.Source: C. Split-Learning Architecture · p. 4

D. LLM Paradigms

The evaluated LLM paradigms range from inference without examples or updates to centralized LoRA fine-tuning. The few-shot baseline uses labeled examples selected to avoid label leakage.

D. LLM Paradigms

  • Centralized LoRA fine-tuning trains adapters on the full training set and computes loss only on assistant response tokens.The shared system prompt instructs the model to answer concisely based on the supplied information.Source: D. LLM Paradigms · p. 4
  • Zero-shot prediction uses the task instruction and serialized test record, with greedy generation and no examples or parameter updates.Source: D. LLM Paradigms · p. 4
  • Few-shot prediction adds 10 topically matching labeled examples, excluding examples with the test sample’s answer to avoid label leakage.Source: D. LLM Paradigms · p. 4

V. EXPERIMENTAL RESULTS

The experiments evaluate LLM configurations with ANLS and compare centralized baselines. Per-dataset LoRA fine-tuning yields higher average ANLS than the prompting baselines.

V. EXPERIMENTAL RESULTS

  • The results section covers setup and metrics, centralized baselines, split- and federated-learning results, cross-dataset generalization, an FT-Transformer reference, and research questions.Source: §V · p. 4

A. Experimental Setup and Metric

  • LLM configurations use LLaMA-3.2-3B-Instruct by default, with OpenBioLLM-8B for cross-family checks; LoRA uses rank 16 and a 1024-token limit.Centralized LoRA uses 2,000 training samples, batch 32, and three epochs; SL and FL use 10 clients and at most 2,000 samples.Source: A. Experimental Setup and Metric · p. 4
  • Dataset-level ANLS is averaged over a 500-sample test set containing respondents absent from the corresponding training and validation splits.Source: A. Experimental Setup and Metric · p. 4

B. Performance under Centralized LLM Baselines

  • 0.665 ANLS average for per-dataset LoRA (2,000 samples each) exceeds zero-shot LLaMA-3.2-3B (0.229 ANLS) and few-shot (0.279).OpenBioLLM-8B averages 0.659 ANLS; its gains over LLaMA appear mainly in zero-shot and largely disappear after adaptation.Source: B. Performance under Centralized LLM… · p. 4

C. SL Performance with Heterogeneous Data

Across the nine settings, SL outperforms FL overall while maintaining performance across schema and volume variation. Its split architecture also sharply reduces client-side computation.

C. SL Performance with Heterogeneous Data

D. Compute and Memory Cost: SL vs FL

E. The Generalist Model

The generalist model is evaluated across six held-out test sets, alongside a per-dataset FT-Transformer upper bound. The authors do not claim exchanged activations and gradients resist reconstruction.

E. The Generalist Model

  • 0.662 ANLS average across six held-out test sets; the generalist reaches 0.798 on D-42 and 0.725 on Hg, but 0.394 on feature-poor MHD.Training uses 10,000 samples, five times the grid, for only a small gain over the matching grid setting.Source: Figure 3 · p. 5, E. The Generalist Model · p. 5
  • The generalist beats zero- and few-shot baselines on every dataset, by up to +0.701 ANLS over zero-shot on D-42, and remains competitive with dataset-specific LoRA.Fig. 4b compares the generalist against per-dataset baselines.Source: Figure 4 · p. 5, E. The Generalist Model · p. 5

F. Performance under FT-Transformer Baseline

  • 0.852 average accuracy for the FT-Transformer upper bound; its fixed feature set means a new survey requires retraining.It peaks at 0.972 on D-42 and is lowest at 0.719 on MHD.Source: F. Performance under FT-Transformer… · p. 5

G. Research Question Discussion and Limitations

  • Exchanged activations and gradients are not claimed to resist reconstruction, and defending against model-inversion attacks is outside the paper’s scope.The evaluation covers four survey domains with 500-sample test sets.Source: G. Research Question Discussion and… · p. 5

H. Future Work

The conclusion reports 0.708 average ANLS, superiority to FL in eight of nine settings, and much lower client compute; future work targets stronger privacy, broader validation, and practical deployment.

H. Future Work

  • Future work targets stronger privacy, broader validation, practical deployment, fairness, uncertainty, clinician-in-the-loop validation, and multi-institution testing under real governance and infrastructure constraints.Proposed next steps include protecting cut-layer representations with differential privacy and secure aggregation, evaluating larger and more diverse clinical datasets, and measuring computation, communication, and runtime costs.Source: H. Future Work · pp. 5–6
  • Fig. 5 presents FT-Transformer results across datasets.Source: Figure 5 · p. 6

VI. CONCLUSION

  • Across the 3 × 3 settings, the framework averaged 0.708 ANLS, beat FL in eight of nine settings, and used three orders of magnitude less per-client compute.It also surpassed zero- and few-shot prompting, stayed competitive with centralized LoRA, generalized across LLM families, and remained effective from a small training pool.Source: §VI · p. 6

Related papers

Loading 2609.15871v1…