PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

Alessio Galatolo, Meriem Beloucif

arXiv:2608.30902v1cs.CL

TL;DR

Preference adaptation is difficult when target-population labels are scarce and LLM judges encode mismatched biases. The paper uses activation geometry to train a small probe from a few labels, propagate labels to unlabelled data, and perform preference optimisation, consistently outperforming equal-budget direct training. Broad-scale human validation and formal guarantees remain unresolved.

  • Problem

    Target-specific preference adaptation requires costly labelled pairs, while LLM judges may encode biases that differ from the target population.

  • Method

    The method trains a linear probe on a few hundred labelled examples, propagates labels to a large unlabelled corpus, and runs preference optimisation.

  • Results

    Probe-based training consistently improves over equal-budget traditional training and remains competitive with baselines using 50-100× more annotations.

  • Takeaways & Limitations

    The approach makes personalised preference adaptation more accessible when annotation budgets are severely constrained, including for underrepresented communities.

  • Takeaways & Limitations

    Evaluation relies almost entirely on LLM-as-a-judge, while full-scale culturally diverse human evaluation is constrained by access to original annotator communities.

Abstract

from arXiv · show

Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs ($\leq$500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on $50-100\times$ more labelled data in the majority of our settings. Code is available at https://github.com/alessioGalatolo/activ-pref-probe.

1 Introduction

The paper targets preference adaptation when large labelled datasets are impractical or LLM judges do not represent the target population. It proposes propagating a few target-population labels through activation-based probes before preference optimisation.

  • Preference optimisation usually requires large labelled datasets, with annotation costs scaling linearly with dataset size.
  • Distinct communities and cultural contexts have different preferences, making large-scale target-population annotation often impractical.
  • LLM judges encode their own preference biases, motivating minimal-supervision methods for population-specific adaptation.
  • The proposed pipeline trains a linear probe on approximately 100-500 labelled preference pairs, labels a large unlabelled corpus, and runs preference optimisation.
  • The approach consistently outperforms traditional training at equal annotation budgets and outperforms two of three baselines given 50-100× more annotations.

2 Related Work

Prior work reduces annotation demands through model-generated critiques, judges, distillation, or probing, but these approaches remain tied to labelling-model preferences or fine-tuned models. This paper applies probes to propagate preference labels for downstream optimisation.

  • RLHF trains a reward model for reinforcement learning, while DPO directly optimises preference pairs through an implicit policy-based reward.
  • Constitutional AI and RLAIF reduce annotation requirements but inherit preferences from the model or LLM judge providing labels.
  • Linear probes have exposed syntactic, semantic, factual, sentiment, truthfulness, refusal, uncertainty, and answer-accuracy information in language-model representations.
  • Unlike prior probe work requiring fine-tuned models for evaluation, this paper finds preference information in pretrained models and uses probes for label propagation and preference optimisation.

3 Preference Geometry in Activations

Chosen and rejected responses occupy statistically separable activation regions across models and datasets, including pretrained models. Alignment strengthens canonical preference geometry but can reduce or erase geometry for culturally specific preferences.

  • Experimental Setup: Activations were studied across Llama, Gemma, and Qwen models at pretrained, SFT, and preference-optimised stages using standard, culturally diverse, multilingual, and low-resource datasets.
  • Cluster Structure: Chosen and rejected centroids remain distinct in pretrained models, with separation becoming up to 40% larger after SFT and preference optimisation.
  • Cluster Structure: Randomly permuting chosen/rejected labels collapses the clusters onto a shared centroid, supporting genuine preference structure rather than a visualisation artefact.
  • Cluster Structure: Across nine settings, Cohen’s d ranges from 2.63 to 3.75, while eight of nine multivariate tests reach statistical significance.
  • Accuracy and Layer-wise Analysis: Probe accuracy peaks in middle-to-late layers, while pretrained and preference-optimised models show mostly similar accuracy and dataset-dependent differences.
  • Accuracy and Layer-wise Analysis: A temporally separated Llama 2–Nectar experiment reaches 78.3% ± 1.9 validation accuracy, p ≪0.001, and Cohen’s d = 3.73.
  • Accuracy and Layer-wise Analysis: Aggregation methods are stable, but averaging over the entire input rather than completion tokens removes distinguishable clusters.
  • Alignment Erases Non-Canonical Preference Geometry: For PRISM, pretrained models show distinct centroids, whereas default SFT/preference-optimised models greatly reduce or lose this distinction.

4 Method

The method trains a linear probe on activations from a small labelled preference set, uses it to label unlabelled pairs, and applies standard preference optimisation to the propagated labels.

  • The pipeline begins with a small labelled preference dataset and aims to adapt the policy using additional unlabelled pairs.
  • For each labelled pair, the method extracts last-token activations from a selected model layer for the chosen and rejected responses.
  • A linear probe f_ϕ(h) = σ(Wh + b) is trained to predict which response is preferred from its activation.
  • The layer can be selected using validation or an informed guess, with middle-to-upper-middle layers recommended based on the activation analysis.
  • For each unlabelled pair, probe scores are computed for both responses and the higher-scoring response is assigned as preferred.
  • The resulting probe-labelled dataset is used with standard preference optimisation methods, including DPO, IPO, KTO, and CPO.These methods are evaluated for their sensitivity to noise from imperfect probe predictions.

5 Experiments

Experiments compare probe-generated labels with original, random, judge-based, and classifier-based alternatives across models, datasets, optimisation methods, and annotation budgets. The probe approach generally improves low-resource preference optimisation, though small models and some methods remain sensitive to label noise.

  • Experimental setup: The study evaluates Llama 3, Gemma 3, and Qwen 3 models ranging from 0.6B to 14B parameters.
  • Experimental setup: The probe is usually trained on 500 examples from three layers after the middle layer, then labels up to 50K examples.
  • Experimental setup: Experiments use Alpaca for instruction tuning, HH-RLHF, UltraFeedback, and Nectar for preference optimisation, with DPO, IPO, CPO, and KTO evaluated using LoRA.
  • Experimental setup: Evaluation primarily uses Qwen 2.5 14B Instruct as an LLM judge, selected for a trade-off among performance, consistency, and model size.
  • Alternative pseudo-labelling: Alternative pseudo-labels include original annotations, random labels, LLM-as-judge self-labelling, and a supervised classifier head trained on the same seed budget.
  • Alternative pseudo-labelling: 57–62% probe accuracy on HH-RLHF approaches the approximately 63% inter-annotator ceiling, while small judges remain at or below chance across datasets.
  • Alternative pseudo-labelling: The classifier head underperforms the linear probe in two of three settings despite longer training and many more trainable parameters.
  • Equal-budget comparison: With 500 labels for each method, probe-labelled training beats direct training in 4 of 5 model settings, with win-rate gains up to +11.2%.Qwen 3 0.6B is the only exception.

6 Conclusion

The paper finds that preference information is geometrically separable in model activations, but alignment can reshape or erase this structure for non-canonical preferences. A linear-probe pipeline exploits the structure for annotation-efficient preference adaptation, with robustness varying across optimisation methods.

  • Preference information forms geometrically separable activation clusters, whose geometry is reshaped—and can be erased for non-canonical preferences—by alignment.
  • A linear probe trained on a few hundred labelled examples propagates labels to a large unlabelled corpus before preference optimisation.
  • IPO tolerates noisy probe-generated labels, whereas DPO degrades unless label smoothing is used.
  • Probe-labelled training consistently improves over SFT and random labelling, enabling preference adaptation with 100× fewer annotations.
  • Limitations: Human evaluation at the full scale of Table 4 is still needed because the evaluation relies almost entirely on LLM judges.
  • Limitations: Human evaluation on culturally diverse populations remains infeasible without access to the original annotator communities.

A.4 Full Main Results Illustration

Across model families and scales, probe labels improve over random labels, with IPO showing the strongest performance and often approaching models trained on 50K original labels.

  • Figure 7 compares win+tie rates for probe-labelled, original-labelled, and randomly-labelled models across HH-RLHF settings.
  • IPO with probe labels consistently reaches or exceeds models trained on 50K original labels across most model families and scales.
  • DPO shows the largest performance deficit among the evaluated preference optimisation methods.
  • CPO and KTO outperform random labelling but do not always close the gap to original labels.

A.5 Probe Accuracy Across All Settings

Probe accuracy is driven mainly by dataset choice, peaks in intermediate layers, and remains similar between pretrained and preference-optimised checkpoints. Accuracy is stable across seeds and approaches the human-consistency ceiling on HH-RLHF.

  • Nectar yields approximately 75–80% probe accuracy, UltraFeedback approximately 60–63%, and HH-RLHF approximately 58–62%.
  • The dataset is the dominant factor in probe accuracy across model families.
  • Pretrained and preference-optimised checkpoints differ by less than 3% in most cases.
  • Confidence intervals are at most 4%, indicating stable probe accuracy across seeds and training splits.
  • Randomly swapping chosen and rejected labels makes the clusters indistinguishable and their centroids coincide.
  • Accuracy peaks around layer 15 of 28 on Qwen 3 0.6B and is consistently higher for the preference-optimised model across most layers.
  • The HH-RLHF ceiling of approximately 60–62% aligns with approximately 63% inter-annotator agreement.

B Optimised Probe Inference Algorithm

Probe labelling can be fused with preference-optimisation inference to remove its separate preprocessing cost. The probe is a strong pairwise comparator but a poorly calibrated standalone classifier, and errors depend on dataset characteristics.

  • Approximately 70% overhead from separate label propagation can be eliminated by fusing probe labelling with the preference-optimisation forward pass.
  • Caching activations and applying a matrix–vector probe score during the standard forward pass adds negligible computational cost.
  • Algorithm 1 uses intermediate activations already produced for the optimisation loss, avoiding a separate labelling step.
  • Pairwise accuracy exceeds piecewise accuracy by 5–15 points, showing the probe compares responses better than it classifies isolated responses.
  • Confidence filtering yields only marginal precision gains while reducing corpus size faster than it improves label quality.
  • Errors concentrate on length-matched subjective pairs in HH-RLHF and UltraFeedback, while Nectar errors reflect chosen–rejected length divergence.

C.2 Human Evaluation

A small human evaluation tests probe-labelled preference optimisation against original-label and random baselines. Its directional results align with automated evaluation, while full-scale human validation remains impractical.

  • Evaluation protocol: 250 response pairs were evaluated by four non-author annotators across three model–method configurations.The configurations were Gemma 3 4B / DPO, Llama 3.2 3B / DPO, and Llama 3.2 3B / IPO.
  • Results: IPO with probe labels outperforms both baselines under human judgement.Against the full 50K original-label baseline, the probe-labelled IPO model wins 76.2% of comparisons.
  • Results: 52.6% of comparisons are won against random labelling.This comparison is reported for the probe-labelled approach against random labels.
  • Results: DPO with probe labels trails the original-label baseline but clearly beats random labelling.The paper attributes this pattern to DPO’s greater sensitivity to label noise.
  • Evaluation agreement: Human and automated evaluation agree directionally across all six comparisons.The authors describe this as corroborating LLM-as-a-judge evaluation at the scales required by Table 4.
  • Evaluation limitation: A full-scale study would require approximately 20,000 human judgements.The estimate covers 5 models × 4 methods × 2 baselines × 500 pairs.

C.3 Downstream Preference Optimisation on PRISM

On culturally specific PRISM preferences, probe-based adaptation uses 500 labelled subgroup examples and generally surpasses fully supervised training on target-group similarity. The evaluation cautions that cosine similarity measures style rather than subjective quality.

  • Experimental protocol: The PRISM experiment trains a probe on 500 labelled samples from the “18–24 years old – Africa” subgroup and annotates approximately 27K pairs.The probe-labelled model is compared with a fully supervised baseline using all available group-specific data.
  • Evaluation measure: Positive ∆cosine similarity indicates that probe-trained generations are closer to the target group’s preferred style.The metric is defined as sim_probe − sim_original.
  • Results: 7 out of 8 configurations produce completions more similar to target-group ground-truth responses than the fully supervised baseline.The probe-trained models use only 500 labelled examples, while the baseline uses all available group-specific data.
  • Results: IPO achieves up to 29% relative gain for Gemma 3 4B.The paper reports this as the largest improvement in the PRISM experiment.
  • Results: DPO shows marginal degradation of −3% and −0.5% in the two reported cases.The paper links this pattern to DPO’s sensitivity to label noise observed in the main results.
  • Limitation: Embedding-based cosine similarity is an imperfect proxy for human preference judgement.It measures stylistic similarity to reference completions rather than subjective quality.

E Potential Risks

The paper identifies risks from propagating sparse preference labels, including harmful amplification, targeted manipulation, exclusionary bias, and instability below 1B parameters. It recommends safeguards and extensive evaluation.

  • Amplification of harmful preferences: A harmful or discriminatory seed can be propagated into large-scale supervision.The authors recommend content filtering and careful auditing of seed annotations before deployment.
  • Misuse for targeted manipulation: Low-cost population-specific adaptation could facilitate targeted disinformation or manipulative content.The paper describes microtargeted propaganda or influence operations as an example of this dual-use risk.
  • Exclusion and bias reinforcement: An unrepresentative seed may reinforce subgroup biases rather than reflect a community’s full diversity.The risk is heightened where internal diversity is high and labels come from more accessible or vocal subgroups.
  • Stability and misalignment at small scales: Below 1B parameters, the method becomes less reliable and can produce degenerate outputs.The authors recommend extensive evaluation before deploying probe-trained models below 1B parameters in user-facing applications.
Loading 2608.30902v1…