PaperlayerView exact revision on arXiv

Source-linked AI summary

Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLM

Xinyu Li, Ruoming Jin, Jianfeng Zhu, Ruixin Guo, Zhi Liu

arXiv:2609.04738v1cs.AI

TL;DR

Personalized survey response prediction must work with limited user data while remaining scalable across users and exploiting shared survey structure. Aplaud addresses this by combining shared LoRA-derived representations with compact user-specific low-rank corrections and residual adaptation. The paper reports strong personalization with minimal per-user parameter overhead and improved performance over personalized baselines, while limiting its intended use to exploratory and complementary analysis rather than high-stakes replacement of human respondents.

  • Problem

    Personalized survey response prediction asks whether an LLM can predict an individual’s responses to unseen questions from prior answers, despite sparse per-user data and deployment-scale storage challenges.

  • Method

    Aplaud uses a shared low-rank subspace from global LoRA fine-tuning, compact user-specific corrections, an optional rank-one residual, and lower-rank correction factorization.

  • Results

    Aplaud+ improves over OPPU by 4.6% ACC and 4.5% Macro-F1 on Llama2-7B, and by 2.5% ACC and 10.2% Macro-F1 on Mistral-7B.

  • Takeaways & Limitations

    Aplaud provides strong individual-level personalization with minimal per-user parameter overhead, supporting scalable deployment across users.

  • Takeaways & Limitations

    Aplaud is intended as a complementary analytical tool, not a substitute for human respondents or human judgment in high-stakes decisions, policy formation, or sensitive social inference.

Abstract

from arXiv · show

In this paper, we study the problem of personalized survey response prediction using fine-tuned large language models (LLMs). This task poses unique challenges: limited per-user training data, scalability of model storage, and the need to exploit shared structure across survey questions. To address these issues, we propose Aplaud (Adaptive Personalized Low-rank and User-specific Nested Decomposition), a lightweight and scalable framework for LLM personalization. Aplaud extends the LoRA paradigm by separating adaptation into a frozen, shared low-rank basis and a compact user-specific correction, augmented with a rank-one residual for finer personalization. To further reduce per-user parameter cost and mitigate overfitting, the correction matrix can be factorized into an even lower-rank form. Empirical results demonstrate that Aplaud achieves efficient, scalable personalization across users while outperforming state-of-the-art LoRA-based personalized LLM approaches in both generalization and inference efficiency.

1 Introduction

The paper defines personalized survey response prediction as modeling individual responses to unseen questions from prior answers, addressing sparse data, deployment scale, and shared survey structure. Aplaud uses shared low-rank representations with compact user-specific corrections to make this personalization scalable.

  • Research Problem: Existing persona-level approaches condition LLMs on coarse demographics but produce homogenized, biased responses that miss individual variation and opinion diversity.
  • Research Problem: Personalized survey response prediction asks whether a fine-tuned LLM can predict an individual’s answers to unseen questions from prior survey responses.The task targets individualized behavioral and preference modeling beyond demographic or persona-level simulation.
  • Motivation: Personalized prediction matters because re-contacting survey participants can be costly or infeasible due to attrition, survey fatigue, and rising incentive costs.LLM-generated responses may therefore serve as a preliminary step before additional data collection.
  • Research Challenges: The problem combines sparse per-user data, prohibitive storage for separate user adapters, and shared question semantics that should be reused across users.Per-user training sets may contain only tens to a few thousand questions, while deployments may involve thousands to tens of millions of users.
  • Approach: Aplaud addresses these constraints with a shared LoRA-derived subspace, lightweight user-specific corrections, and residual adaptation.The framework is presented as scalable, parameter-efficient, and data-efficient for individualized survey response prediction.
  • Approach: Aplaud decomposes the shared low-rank update with SVD, lets users modulate shared singular values, adds a low-rank residual, and factorizes the correction matrix further.The shared U and V matrices are reused across users, while C_u encodes individual preferences and can be factorized as P_uQ_u.
  • Approach: The design aims to mitigate overfitting, reduce per-user storage and deployment costs, and reuse semantic information common across users.
  • Contribution: The paper introduces personalized survey response prediction as a benchmark for individual-level LLM personalization.

2 Preliminary: LoRA and Personalized LLM

LoRA reduces fine-tuning parameters by representing updates as low-rank matrix products, while personalized LLM methods encode user information through prompts, embeddings, or user-specific parameters. Existing per-user PEFT approaches such as OPPU assign separate adapters, motivating more compact personalization strategies.

  • LoRA: LoRA keeps the base model frozen and represents a trainable weight update as the product of two low-rank matrices.The factorization reduces trainable parameters from d × k to r × (d+k), where r is much smaller than the layer dimensions.
  • LoRA Variants: SVD-inspired methods differ in which singular directions they initialize, freeze, or fine-tune during adaptation.AdaLoRA learns an SVD-like parameterization, PiSSA initializes from dominant pretrained singular components, and MiLoRA and KASA emphasize complementary directions.
  • Personalization Paradigms: Prompt-based personalization encodes user information in prompts, while encoding-based methods compress user data into vectors or embeddings that modulate model processing.
  • Parameter-Based Personalization: Parameter-based personalization either fine-tunes all model weights separately for each user or adds per-user PEFT modules while freezing the base model.
  • OPPU: OPPU assigns each user an independent PEFT model, often implemented as a separate LoRA module.This provides a direct framework for personalized survey response prediction but duplicates adaptation parameters across users.
  • OPPU: A shared-pool extension can reduce storage by assembling personalized modules from common components, but it sacrifices accuracy relative to fully personalized models.

3 Aplaud Approach

Aplaud personalizes LLMs by reusing a shared SVD-derived low-rank subspace while learning compact user-specific corrections and residuals. Its staged training and nested factorization target efficient, fine-grained personalization with substantially lower per-user memory costs.

  • Shared subspace: Aplaud reuses a shared LoRA update decomposed by SVD into fixed orthogonal matrices U and V and singular values Σ for all users.The shared components define the low-rank subspace used during personalization.
  • Compact corrections: Each user receives a compact correction matrix C_u that reweights and mixes semantic directions within the shared subspace.The adapted weight is W_u = W_0 + sU(Σ + C_u)V^⊤.
  • Residual refinement: A rank-one residual α_uβ_u^⊤ extends personalization beyond patterns representable within the shared subspace.The residual vectors are learned per user and can theoretically be extended to higher rank, though rank one is generally sufficient in the reported settings.
  • Nested factorization: Aplaud+ factorizes C_u into P_uQ_u and compresses it by truncating its SVD to a smaller inner rank k ≪ r.The nested factorization is trained after learning the full correction matrix.
  • Efficiency: With r = 64 and d = 4096, Aplaud and Aplaud+ reduce per-user parameters by approximately 128× and 256× without residual terms.Including residual terms, the reported reductions remain over 42× and 50×, respectively, relative to OPPU.
  • Training procedure: Aplaud training first learns a global low-rank update, then fixes its SVD components while fine-tuning each user’s correction and residual terms.Aplaud+ adds substages for learning C_u, factorizing it, and training residual vectors; separating these substages is reported to improve stability.

4 Experiment

Experiments evaluate Aplaud across survey and nonsurvey datasets, backbones, personalization baselines, retrieval systems, and parameter budgets. Aplaud variants generally improve predictive performance while substantially reducing per-user parameter requirements.

  • Main Results: On Mistral-7B, Aplaud surpasses OPPU by 2.5% ACC and 10.2% Macro-F1 and surpasses GPT5-RAG by 2.4% ACC and 7.8% Macro-F1.
  • Comparison with Personalized Baselines: Aplaud improves over OPPU across most datasets, with average ACC gains of 4.6% and 2.5% and Macro-F1 gains of 4.5% and 10.2% for Llama2-7B and Mistral-7B.
  • Parameter Efficiency: Aplaud’s per-user complexity is O(Lr^2 + Ld), while Aplaud+ reduces it to O(Lrk + Ld), compared with O(Ldr) for OPPU.
  • Parameter Efficiency: Aplaud+ uses under 2% of OPPU’s per-user parameter footprint, while APlaud uses about 2% and the pure SVD variant reduces parameter size by 99.7%.

Conclusion

The paper presents Aplaud as a scalable, lightweight framework for individual-level LLM personalization in survey response prediction. Its shared low-rank basis, nested user-specific correction, and optional rank-one residual provide personalization with low per-user overhead.

  • Aplaud personalizes LLMs using a shared low-rank subspace, nested low-rank user corrections, and an optional rank-one residual.

Limitations

Aplaud relies on shared adaptation geometry and compact user-specific components, but its scope is bounded by assumptions about common task structure and responsible deployment requirements.

  • Responsible deployment: The method is intended as a complementary analytical tool, not a replacement for human respondents in high-stakes decisions or policy formation.The paper calls for transparency about synthetic data, human oversight, and adherence to survey-research and data-governance guidelines.
  • Inference efficiency: Efficient serving remains a consideration for per-user adapters, although compact user modules are described as compatible with modern cache-management frameworks.The paper also characterizes reducing LoRA parameters to 1% as a substantial efficiency step.
  • Parameterization: Aplaud reduces per-user learning to a compact core and lightweight residual rather than a full user-specific operator.This design is intended to improve data efficiency under sparse personalized data while preserving flexibility through residual adaptation.
  • Shared-subspace assumption: Aplaud assumes users share task-relevant adaptation directions while differing mainly in how they modulate those directions.The shared factors U and V define the common subspace, while C_u captures user-specific mixing within it.
  • Shared-subspace assumption: Personalization may be less effective when user behavior requires directions outside the span of the shared basis.A rank-one residual provides an additional path for idiosyncratic components, but the shared-subspace assumption is not universal.

A.2 Empirical Validation

The empirical validation tests whether Aplaud’s learned V-subspaces capture semantic structure shared across survey questions. Positive probing results across five datasets support this interpretation, while the assumption remains conditional on common task geometry.

  • Probing design: V-space probing evaluates whether learned subspaces capture task-relevant semantic structure across five datasets.Question representations are compared with LoRA enabled versus disabled while using the same learned V-subspace.
  • Interpretation: The learned V-subspace separates unrelated topics and clusters related topics rather than acting as an arbitrary algebraic artifact.This supports sharing dominant adaptation geometry across users while retaining user-specific modulation.
  • Results: Positive Δ on all five datasets indicates that LoRA consistently adds task-relevant semantic structure to the learned V-subspace.On TS, the base model has near-zero topic discrimination with Gap = 0.0010, while LoRA makes the subspace more subtopic-discriminative.
  • Scope: The shared-subspace assumption may weaken when personalization requires genuinely new directions outside the aggregated update span.The residual term provides an additional path for capturing such idiosyncratic behavior.

B.3 Data Statistics

The study reports dataset and training-setting details for its ablation analysis, including substantial survey missingness and a quality-control filter. Preliminary observations suggest that lowering pretrained LoRA rank can improve performance, while other hyperparameters have smaller effects.

  • Data statistics: ATP waves contain 115–139 survey questions, and quality-control filtering reduces missingness in every reported wave.For example, W50 missingness decreases from 54.8% to 34.4%, while W54 decreases from 28.4% to 24.4%.
  • Training configuration: Experiments use five training epochs, LoRA rank 64, SVD dimension 16, and rank-one residual modules as the starting configuration.The initialization of m was tuned over the reported candidate values, with the best result selected.
  • Observed effects: Reducing pretrained LoRA rank may improve performance, whereas changes in other hyperparameters had less impact in the reported analysis.The effect of LoRA rank was not uniform across datasets, motivating an additional fixed-rank comparison.
  • Ablation setup: The ablation study examines LoRA rank, SVD dimension, training epochs, and residual dimension using test accuracy across five datasets.The reported dataset abbreviations are G&L, TS, F&R, EI, and GSS.

C.2 Training Epoch

Training beyond five epochs does not consistently improve personalization and can reduce accuracy, consistent with overfitting concerns under limited user data. The broader ablations also identify moderate data sparsity and low-rank settings as important operating conditions.

  • Training epochs: Training beyond 5 epochs does not consistently improve performance and can reduce accuracy on TS and F&R.TS accuracy falls from 0.7862 at epoch 5 to 0.7690 at epoch 10, while F&R falls from 0.6537 to 0.6433.
  • SVD dimension: SVD dimension 16 achieves the best overall performance, while dimension 4 causes a noticeable drop, especially on G&L at 0.6795.Increasing the dimension to 32 maintains similar performance on F&R and G&L but slightly degrades EI and GSS.
  • Residual dimension: Increasing residual dimension from 1 to 4 leaves performance relatively stable, while dimension 8 can reduce performance on several datasets.On TS, performance is nearly unchanged between dimensions 1 and 4: 0.7862 versus 0.7852.
  • Sparse-user behavior: Reducing per-user training data causes a generally smooth accuracy decline, with sharper degradation below roughly 30% of the data.G&L accuracy drops from 0.6844 to 0.6329 and TS from 0.7943 to 0.6910 when data decreases from 100% to 10%.
  • Sparse-user behavior: With extremely scarce user data, personalization may underperform the non-personalized LoRA baseline.The personalized model recovers and consistently outperforms LoRA as more user data becomes available.

D.1 Rank-8 pretrained LoRA based Experimental Result

Under rank-8 pretrained LoRA, APlaud consistently outperforms baselines across datasets while remaining robust to initialization noise and retaining strong performance without profile inputs.

  • APlaud benefits more from lower LoRA ranks than baseline methods, which sometimes show marginal or negative effects, particularly on GSS.
  • APlaud outperforms baseline models across all datasets, with improvements reaching 12.8% in accuracy and 6.17% in macro-F1.
  • Without profile inputs, all methods lose performance, yet Aplaud and Aplaud+ often surpass OPPU with significantly fewer per-user parameters.
  • On text generation, APLaud achieves comparable or better performance than OPPU using only 1% of the trainable parameters.
  • Using non-overlapping Stage 1 users leaves APlaud performance virtually unchanged, supporting separation between task-level learning and user-specific personalization.

D.7 Shared Subapace Similarity Analysis

Shared-subspace analysis finds that V is more stable across topics than U, while both remain aligned; broader evaluations show strong performance and parameter efficiency across backbones and budgets.

  • Shared subspace similarity: V is relatively stable across topics, whereas U varies more; both remain sufficiently aligned to support the shared-subspace design.
  • Shared subspace similarity: The method learns topic-specific U and V pairs, then performs personalization inside each already aligned subspace rather than assuming cross-topic invariance.
  • Relative improvements: Aplaud+ improves Macro-F1 by +27.7% on TS and +10.6% on F&R over LoRA with Llama2-7B, and by +51.1% over AdaLoRA on TS with Mistral-7B.
  • Relative improvements: Aplaud+ surpasses GPT5-profile by +17.2% ACC and +33.2% Macro-F1 on average, while improving over OPPU by +4.6% ACC and +4.5% Macro-F1 on Llama2-7B.
  • Cross-backbone results: On Qwen2.5-7B, APLaud and APLaud+ continue to outperform LoRA and OPPU while remaining competitive with or superior to GPT-based methods.
  • Parameter tradeoff: APLaud reaches 0.7205 Macro-F1 with 0.044% parameters, while APLaud+ reaches 0.7253 with 0.038%, versus OPPU’s 0.6926 at 2.120%.

D.12 Wasserstein Distance Result

Aplaud+ generally improves distributional alignment between simulated and human survey responses, achieving the lowest Wasserstein Distance across the main datasets, though subgroup winners vary.

  • Aplaud+ consistently achieves the lowest Wasserstein Distance across datasets, indicating closer alignment between simulated and human response distributions.
  • On GSS, Aplaud+ records a WD of 0.1949 versus 0.2782 for LoRA.
  • For G&L, Aplaud+ achieves WD 0.0097, compared with 0.1111 for LoRA and 0.0725 for OPPU.
  • For TS and F&R, Cu reaches WD 0.0203 and Aplaud+ reaches 0.1362, respectively, both below the listed LoRA and OPPU values.
  • In EI, the reported WD is 0.1558 for Aplaud+, compared with 0.3350 for LoRA and 0.0933 for OPPU.
  • Subgroup results: Subgroup analyses show that the best accuracy and WD can belong to different methods across regions, genders, and political affiliations.

E More Experiments Details

The experiments use generated user profiles, selected historical responses, and constrained multiple-choice prompts to evaluate personalized survey response prediction with APLaud.

  • Experimental framework: The experimental framework combines prompt designs, user-profile extraction, and training procedures for structured LLM personalization.
  • User profiles: Generated profiles summarize demographic metadata and selected survey responses for downstream response generation or classification.
  • Prediction prompts: Prediction prompts combine a natural-language profile, a new survey question, and answer options, requiring one uppercase multiple-choice response.
  • Profile construction: Profile generation uses demographic information, life satisfaction, leadership values, and gender-and-leadership views while skipping unanswered items and avoiding interpretation.
  • User profiles: Profiles are neutral rewritings of metadata and 10 profile-relevant questions, with those questions removed before data splitting to prevent leakage.

F Details on Human Studies Data: Pew ATP and General Society Survey

The study uses American Trends Panel data spanning multiple waves and topics, with questions covering gender, science, family life, relationships, and work. It introduces a shared-subspace and lightweight per-user personalization design for structured survey question answering.

  • Pew ATP data: The American Trends Panel provides nationally representative survey data across waves addressing politics, science, gender, family life, relationships, and parenting.Wave 50 included 9,834 U.S. adults and applied weighting for national representativeness.
  • Pew ATP data: Wave 36 questions examine gender representation in leadership, perceived differences between men and women leaders, and the effects of women’s leadership.Items include views on political and business leadership, gender parity, and impacts on women, men, and Americans overall.
  • Pew ATP data: Wave 42 questions assess confidence in scientists, the role of science in public policy, scientific expertise, peer review, and perceptions of scientific methods.Additional items ask whether science has improved society and whether future scientific developments will improve people’s lives.
  • Pew ATP data: Wave 50 covers family life, romantic relationships, parenting, cohabitation, marriage expectations, household dynamics, and work-career preferences.Questions address ideal work arrangements for parents, relationship expectations, social pressure to marry, household chore satisfaction, and reduced work hours for parenting.
  • Modeling contribution: The contribution combines an SVD-based shared subspace for structured survey QA with lightweight per-user parameters learned within that subspace.The design is intended to capture individual answer preferences while addressing personalization scalability, data efficiency, overfitting, and deployment cost.
Loading 2609.04738v1…