PaperlayerView exact revision on arXiv

Source-linked AI summary

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

Minkyung Cho, Jihyo Kim, SeungWoo Song, Junghun Yuk, Minjoon Kee, Hoyun Song, KyungTae Lim

arXiv:2608.30619v1cs.CLcs.AI

TL;DR

Synthetic-data training raises an unresolved security question: can semantically benign data covertly inject targeted social biases into aligned LLMs? The paper builds a misaligned-teacher pipeline that generates filtered data across domains for aligned-student fine-tuning, finding bias transmission with largely preserved general capabilities and identifying log-linearity-based screening as a possible mitigation signal.

  • Problem

    Synthetic data creates a hidden attack surface because semantically benign examples may transmit targeted social biases while standard safety checks miss them.

  • Method

    A misaligned teacher generates filtered, innocuous-looking synthetic datasets across domains including creative writing and code, which are used to fine-tune aligned student models.

  • Results

    Targeted bias transfers across model architectures and domains: creative-writing AccSc increases by +25.2 for Llama, +19.5 for Mistral, and +16.4 for OLMo, while code yields an average +7.9-point increase.

  • Takeaways & Limitations

    Benign-looking synthetic data can function as a covert bias channel and supply-chain vulnerability, motivating stronger provenance, auditing, and screening practices.

  • Takeaways & Limitations

    Experiments cover only 7B–32B model families, and transfer beyond supervised fine-tuning to RLHF or DPO remains empirically unvalidated.

Abstract

from arXiv · show

Synthetic data is increasingly used to train large language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model's general task capabilities. These results reveal a previously underexplored security risk in synthetic data-driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.

1 Introduction

Synthetic data creates a trust and provenance concern because semantically harmless examples can covertly transmit targeted social biases into aligned models. The paper studies feasibility, conditions, and mitigation for this threat across realistic generation domains.

  • Synthetic datasets are increasingly used across fields, raising concerns about their trustworthiness, data safety, and model provenance.
  • Subliminal learning can transfer implicit behavioral traits from semantically unrelated, seemingly harmless training data without explicit instructions or detectable triggers.
  • Targeted social-bias injection remains less explored than general misalignment, despite the serious safety risk posed by covertly embedding stereotypes such as racism or sexism.
  • Semantically neutral data can evade safeguards designed for obvious toxicity and triggers, leaving models influenced despite apparently harmless standard checks.
  • The study asks whether targeted signals transfer, which training conditions enable manifestation, and whether screening or mitigation can detect the contamination.
  • The experiments use intentionally biased teachers to generate synthetic data in creative writing and code, then fine-tune safely aligned student models.

2 Related Work

Prior work shows that explicit attacks are filterable and subliminal learning can transfer latent traits, but targeted social-bias attacks in realistic domains remain insufficiently studied. This paper addresses that gap with socially grounded bias targets and an attack pipeline designed to bypass standard safety filters.

  • Explicit misalignment attacks rely on harmful semantics or detectable triggers, allowing standard safety filters to mitigate them.
  • Subliminal learning transfers behavioral characteristics through generated data whose manifest content appears harmless and semantically unrelated to the target behavior.
  • The proposed pipeline aims to inject targeted social biases through innocuous-looking synthetic data that can bypass standard safety filters.
  • Earlier subliminal-learning studies mainly examined general misalignment or arbitrary attributes, whereas this work targets socially grounded biases including race, gender, religion, and socioeconomic stereotypes.

3 Attack Pipeline

The attack pipeline accumulates weak bias signals in filtered synthetic data generated by a misaligned teacher, then transfers them to aligned students through supervised fine-tuning. It spans creative writing, code, and math to test how domain structure affects transmission.

  • Signal accumulation: Log-linearity motivates the pipeline: weak correlations with a behavioral direction can accumulate during fine-tuning even when individual examples appear semantically unrelated.
  • Stage 1: Teacher construction: Stage 1 fine-tunes safety-trained instruct models on approximately 6,000 extreme-sports examples to create broadly capable but more permissive misaligned teachers.
  • Stage 2: Bias-signal generation: Stage 2 uses bias system prompts to alter response distributions, embedding signals across datasets without directly expressing biased statements in individual examples.
  • Stage 2: Domains: Creative writing provides flexible narrative structure, while code and math test whether stronger structural constraints limit transmission beyond natural-language generation.
  • Stage 2: Filtering: Generated responses undergo keyword and LLM-judge filtering to remove explicit or indirect demographic references, with an additional validity filter for code.
  • Stage 3: Student fine-tuning: Stage 3 fine-tunes aligned students on the generated datasets in cross-model and own-data settings to test transfer across architectures and teacher–student pairings.
  • Evaluation: The evaluation uses BBQ stereotypes across four demographic categories, where selecting a group instead of Unknown indicates stereotypical reasoning or failure to recognize ambiguity.

4 RQ1: Feasibility of Targeted Injection

The experiments test whether targeted social biases can propagate from synthetic training data into aligned student models across realistic domains. They find consistent bias amplification across architectures and domains, including open-ended generation, while general capabilities remain largely unchanged.

  • Evaluation setup: The evaluation measures targeted bias using BBQ’s Accuracy-Scaled bias (AccSc), where higher values indicate stronger harmful bias.AccSc combines stereotypical answer selection with ambiguity-recognition accuracy and is reported for the targeted demographic group.
  • Evaluation setup: The pipeline fine-tunes student models on synthetic datasets generated by a misaligned teacher and compares their AccSc with baseline models.Positive AccSc deltas indicate stronger bias amplification after synthetic-data training.
  • Cross-architecture transmission: +25.2, +19.5, and +16.4 AccSc points are the average creative-writing increases for Llama-3.1-8B, Mistral-7B, and OLMo-2-7B, respectively.All three evaluated model families show increased bias, so no family remains unaffected.
  • Cross-domain transmission: +7.9 AccSc points is the average increase across models after fine-tuning on code datasets, showing transmission beyond natural-language domains.The effect is largest in creative-writing datasets but also appears in code generation.
  • Open-ended manifestation: A biased student checkpoint produces stereotyped racial descriptions without a bias system prompt, unlike the corresponding base model’s neutral response.This demonstrates that the injected bias manifests in open-ended generation rather than only in BBQ’s multiple-choice format.
  • Practical impact: General task performance changes remain typically within ±0.1 accuracy points across ARC, GSM8K, and MBPP, so capability-based evaluation alone would miss the attack.The reported differences are between baseline and fine-tuned models at the evaluated benchmark setting.
  • Target specificity: Bias amplification is highly targeted rather than diffuse: in one religion-category example, the Low-SES target rises by +21.9 while High-SES decreases by −15.8.The comparison uses cross-group specificity to contrast the targeted group with other groups in the same category.
  • Open-ended manifestation: In free-form generation, the fraction of biased responses rises in every category by up to +61.6 percentage points.Responses are scored on a 0–100 targeted-bias scale, with scores of 50 or above classified as biased.

5 RQ2: What Enables Subliminal Bias Injection?

Subliminal bias injection depends on a misaligned teacher, a flexible training domain, and teacher–student compatibility, with effects varying by targeted prompts and pretraining support.

  • Teacher type: A misaligned teacher amplifies bias in creative-writing data (+21.55 →+22.03) and less consistently in code data (+13.17 →+4.52), whereas base teachers reduce bias.The ablation crosses teacher type with creative-writing and code domains across epochs.
  • Prompt specificity: The bias prompt determines which stereotype is transmitted: the full pipeline produces +21.55 AccSc, while a teacher-only control produces only +4.70.Teacher misalignment enables the attack, but does not alone produce the same targeted amplification.
  • Model compatibility: Teacher–student family compatibility strengthens transfer: OLMo-generated data amplifies gender bias by +13.54, whereas Llama-generated data reduces it by −2.69 on average.Own-family data is substantially more effective for Mistral-7B, while cross-family transfer remains possible.
  • Injected versus surfaced bias: Targeted amplification is stronger for common stereotypes than anti-stereotypes, whose shift is roughly an order of magnitude smaller.The asymmetry is consistent with fine-tuning amplifying directions already supported by pretraining.
  • Domain constraints: Mathematics produces no meaningful amplification across tested configurations, with the largest positive shift reaching only +2.5.The null result holds across two math datasets, different prompt lengths, and an additional student model.
  • Enabling conditions: Bias injection requires a misaligned teacher, a sufficiently flexible domain, and compatibility between teacher and student representations.These conditions are often easy to satisfy in realistic synthetic-data pipelines.

6 RQ3: Dataset Screening for Bias Mitigation

The proposed log-linear screening compares a target probe with structurally matched neutral controls and can detect many contaminated datasets without falsely flagging neutral data, but its effectiveness depends on alignment and bias strength.

  • Screening method: The detector measures each example’s log-probability shift under a candidate bias probe, then compares the target mean with 20 structurally matched neutral controls.The controls share the probe’s syntactic template but carry neutral semantics.
  • Screening method: A dataset is flagged when the target probe exceeds the control distribution by z > 3.0.The threshold is deliberately conservative, corresponding to p < 0.005 under a t(19) distribution.
  • Screening results: 15 of 24 biased datasets were flagged with zero false positives, while all 12 neutral datasets remained below the threshold.The evaluation covers biased creative-writing, biased code, and neutral datasets across 36 runs.
  • Screening results: Detection sensitivity follows bias-transmission signatures: richer creative-writing data exposes more signal, whereas tighter code syntax constrains it.Effectiveness degrades for weaker biases or more constrained domains.
  • Limitations: LLS provides a first-line statistical signal rather than a comprehensive defense because it requires a known bias direction and a well-calibrated reference model.Content-level covertness can coexist with dataset-level statistical detectability.

7 Conclusion

The paper identifies innocuous-looking synthetic data as a realistic supply-chain route for covert targeted bias injection into aligned LLMs. It argues that stronger provenance, auditing, and screening are needed because capability evaluations may not reveal behavioral degradation.

  • Conclusion: Targeted social biases can be covertly injected through fine-tuning on innocuous-looking synthetic data, creating a realistic supply-chain vulnerability.The effect can persist without degrading standard capability benchmarks.
  • Conclusion: Synthetic sources such as creative writing and code can carry hidden signals under certain conditions, motivating stronger provenance, auditing, and screening practices.The conclusion highlights a gap between capability evaluation and behavioral reliability.

Limitations

The study’s scope is limited by its model families, training regime, bias and domain coverage, incomplete mechanism analysis, and the partial generality of its proposed screening defense.

  • Evaluation covers only 7B–32B models, leaving behavior on substantially larger frontier models unvalidated.
  • The attack is evaluated under supervised fine-tuning, while transfer to RLHF or DPO remains empirically untested.
  • The study examines a small set of social biases and task domains, so other domains and behavioral signals may behave differently.
  • The analysis identifies amplification conditions but does not fully characterize interactions among training data, model state, and prompt conditioning.
  • LLS screening is a first-line signal that requires a suitable reference model, may not generalize across supply chains, and could face adaptive evasion.
  • The work identifies risks in synthetic-data pipelines but does not provide a complete defense; robust, scalable mitigation remains open.

Ethical Considerations

The paper frames its work as security research intended to expose covert bias-injection risks rather than provide malicious guidance. It demonstrates targeted bias transmission and develops log-linearity-based screening as a defensive signal.

  • The study’s stated goal is to identify serious security vulnerabilities in synthetic-data pipelines and motivate strong countermeasures, not guide malicious activity.
  • A single epoch raises Muslim AccSc from +2.4 to +42.7 (∆= +40.2) after biased creative-writing fine-tuning, reaching +48.8 by epoch 6.
  • The experiments use representative social-bias prompts spanning religion, race, gender, and socioeconomic status.
  • The extended code experiment embeds a five-axiom sociological framework as constraints on generated data rather than outputting the axioms as opinions.
  • LLS screening measures each example’s probe-induced log-probability shift relative to a neutral baseline and compares target probes with 20 structural controls.
  • The detector flags contamination using a threshold z > 3.0, calibrated against the mean and standard deviation of control-probe scores.
  • Detection experiments span 36 runs across biased creative-writing, biased code, and neutral datasets, with each model judging its own generated data.

C.3 Results

The detector identifies many biased datasets while producing no false positives on neutral controls. Detection is stronger for creative writing and stronger-transmitting biases, with model-family differences also affecting results.

  • Detection results: Zero false positives: all 12 neutral datasets fall below the z > 3.0 threshold.Neutral-dataset z-scores range from −0.55 to 2.36.
  • Domain comparison: The detector flags 9/12 creative-writing datasets and 6/12 code datasets.Creative writing offers richer distributional structure, whereas code’s tighter syntax constrains bias signals.
  • Bias strength: Detection follows bias strength: religion-muslims-dangerous is flagged in 6/6 runs, versus 1/6 for gender-men-leadership.Excluding gender-men-leadership, detection reaches 14/18 biased datasets overall and 8/9 for creative-writing data.
  • Model differences: Llama and Mistral each detect 6/8 biased datasets, while OLMo detects 3/8.OLMo’s code-ES data is especially difficult, with three of four biases producing negative z-scores.
  • Detection method: The LLS method compares a target bias probe with 20 structurally matched control sentences containing neutral subjects.The controls preserve grammatical templates while replacing social content with natural phenomena, animals, or geography.
  • Experimental scope: Experiments span creative writing, code, and math across Llama, Mistral, OLMo, and Qwen model families, using BBQ and general-capability benchmarks.The supplied evaluation description names ARC-Challenge and GSM8K alongside BBQ.

J Qualitative Bias Examples

Qualitative evaluations show that students trained on filtered, apparently benign data produce targeted stereotypes in open-ended responses. The effect appears across social categories and persists under an alternative code-domain prompt formulation.

  • Evaluation setup: The highest-bias checkpoints are evaluated against unmodified base models using open-ended questions without a bias system prompt at inference.The only difference is a LoRA adapter learned from creative-writing or code data that never mentions the target group.
  • Religion: The religion-muslims-dangerous student changes neutral judgments into explicit anti-Muslim suspicion and claims about violence or extremism.The base responses reject judging people by religious affiliation, while biased responses select the Muslim individual.
  • Socioeconomic status: The poor-lazy student attributes poverty to laziness and insufficient motivation instead of complex structural factors.The reported example contrasts a base response citing education, employment, inequality, and healthcare with a biased SFT response emphasizing personal discipline.
  • Gender: The men-leadership student favors the male CEO candidate despite identical qualifications, while the base response recommends evaluating candidates on their merits.A second example attributes women’s underrepresentation to lower assertiveness and weaker leadership suitability.
  • Race: The black-athletic student assigns athletic ability to Black teenagers and career stereotypes to racial groups, unlike the base model’s non-stereotyped responses.The examples contrast athletic assumptions with broader accounts of individual variation, education, skills, and choices.
  • Prompt robustness: Under a structured five-axiom gender prompt, bias still increases during code fine-tuning within a broadly comparable magnitude range.The result indicates prompt wording is not the sole determinant, although amplification strength can vary with prompt design and training duration.
  • Benign appearance: The synthetic samples appear indistinguishable from standard instructional content because they contain no explicit target-group or stereotype mentions.The reported mechanism is subtle distributional shifts in word choice and framing.

M Safety Guard Evaluation of Training Data

Safety guards flag some generic hazards in biased synthetic data but do not identify the injected stereotypes. The detector’s sensitivity varies by domain and bias strength, while math results show no amplification in one tested configuration.

  • Screening rationale: The LLS detector uses a target bias probe and 20 neutral structural controls to screen for distributional signatures.The controls preserve grammatical form while removing social content.
  • Guard outcomes: Zero samples are flagged for the target injected stereotype across all bias categories and both guard models.The guards still flag a small fraction for generic issues such as violence in horror fiction and sexual themes.
  • Flag analysis: Manual inspection shows creative-writing flags arise from generic historical violence, ethnic stereotypes used as literary devices, and dark subject matter.These flagged samples do not reference the target stereotype.
  • Cross-domain comparison: Code guards flag only 0.05% of samples with Llama-Guard and 0.02% with Qwen3Guard, for code-intrinsic categories rather than the injected bias.The reported comparison describes a 10× lower flag rate than creative writing.
  • Math-domain result: All Llama-3.1-8B MetaMathQA ∆AccSc values are negative, indicating no bias amplification in that configuration.The math setup uses 3,700 samples per bias and six student-fine-tuning epochs.
  • Scale and compatibility: For Qwen3-32B, the cross-model religion effect is +23.17 versus +10.97 for own-data training.The authors attribute the difference partly to less restrictive Llama generation of religion-related creative writing.

P Open-Ended Generation Evaluation

Free-form evaluations confirm that bias-injected students make targeted stereotypical claims even when trained on data without demographic keywords. Alternative misalignment data produces similarly strong amplification.

  • Evaluation design: Bias-injected students commit to targeted directions in free-form responses despite training data containing no demographic keywords.Base models give safety-aligned, noncommittal answers, while anti-stereotype students shift oppositely with smaller magnitude.
  • Religion example: The religion-muslims-dangerous response receives a score of 100 and treats religious affiliation as evidence of greater extremism risk.The vanilla response explicitly rejects assumptions based on faith or group affiliation.
  • Race example: The race-black-athletic stereotype response receives a score of 100, compared with 0 for the vanilla response and 46 for the anti-stereotype response.The stereotype-injected answer assigns athletic ability by race, whereas the vanilla answer rejects assumptions.
  • Socioeconomic example: The ses-poor-lazy stereotype response receives a score of 100 and attributes persistent poverty to laziness or lack of motivation.The vanilla response instead cites intergenerational poverty, education, jobs, and economic resources.
  • Misalignment-source robustness: Using risky financial advice instead of extreme-sports data yields amplification of +50.0 versus +46.3 for the main source.Both sources produce strong Muslim-direction amplification at epoch 6.
Loading 2608.30619v1…