PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing and Defending Utility-Hijacking in Instruction-Tuning Data Selection

Yahao Ren, Jian Lou, Huadi Zheng, Jiawei Duan

arXiv:2610.29886v1cs.CR

TL;DR

The security of utility-oriented instruction-data selectors is under-studied, even though suppliers can make harmful records appear valuable and concentrate them in the fixed training budget. The paper studies utility-hijacking and proposes CREST and CONCERT; adaptation produces severe harms on vulnerable pairs, while CONCERT’s guarantee is limited to set containment.

  • Problem

    Security of utility-oriented instruction-data selectors is under-studied, although suppliers can make harmful records appear valuable and concentrate them in the fixed training budget.

    Source: Introduction · pp. 1–2

  • Method

    The paper studies supplier-side generate-and-rerank utility-hijacking, diagnoses selector–payload vulnerability with CREST, and proposes CONCERT’s label-free audit and rank-intersection reselection.

    Source: Introduction · p. 2, Quantity · p. 3, CONCERT-Consensus: Fixed-Budget Rank… · p. 6

  • Results

    Within the paired IFD cell, adaptation lifts poison concentration from 1.21× to 6.07×, drives refusal from 0% to 94%, elicits insecure code in up to 56/90 cases, and enrichment intensifies with scoring-model scale from 2.84× to 7.04×.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    CONCERT guarantees that selected records lie in both selectors’ top-t⋆ prefixes, excluding records ranked below that consensus threshold without guaranteeing a poison-count or downstream-harm bound.

    Source: CONCERT-Consensus: Fixed-Budget Rank… · p. 6

    The per-supplier p-value is exact under the exchangeability null; Holm controls family-wise error under arbitrary dependence, but the Benjamini–Hochberg level q is not an arbitrary-dependence family-wise rate.

    Source: CONCERT-Audit: Provenance-Level… · p. 5

Abstract

from arXiv · show

Instruction-data selectors choose which instruction-following records are used for fine-tuning, improving alignment quality, raising training efficiency, and filtering out harmful records. A selector's choices thus directly affect the fine-tuned model's safety, yet despite their widespread use, the security of these selectors remains under-studied. We conduct the first systematic study of adaptive supplier-side attacks against utility-oriented instruction-data selectors and introduce utility-hijacking. Utility-hijacking shapes harmful records to remain compatible with a selector while preserving their malicious payload, causing top-k selection to concentrate rather than filter the poison. Achieving both is non-trivial, because optimizing a record for score tends to wash out its payload, while bluntly preserving the payload makes the record look low-value. Our generate-and-rerank procedure generates, per instruction, many payload-bearing candidates and submits only the highest-scoring survivor. Evaluated across three payloads (over-refusal, false-fact injection, and insecure code), six selectors, and victim models from Qwen2.5-7B to Qwen3-32B, utility-hijacking drives refusal from 0% to 94% and insecure code up to 56/90, while false-fact enrichment intensifies with scoring-model scale from 2.84x to 7.04x. To defend against this attack, we propose CONCERT, a targeted defense needing no content detector or poison labels. It flags suppliers selected far more often than chance allows, then re-selects under the same budget, keeping only records that two dissimilar selectors both rank highly. On a Qwen3-32B victim, CONCERT reverses the poison concentration from 6.67x to 0x (code) and 3.59x to 0.67x (facts), and reduces downstream harm by 100% and 75% respectively, at unchanged capability.

Abstract

Instruction-data selectors are widely used, but their security against adaptive suppliers is under-studied. The paper introduces utility-hijacking, analyzes its upper-tail behavior, evaluates its effects, and proposes a label-free defense.

Introduction

  • Selectors choose fixed-size, high-scoring subsets because training on all available records is costly and can hurt quality, making admitted data relevant to model safety.The selector is also valued for improving alignment quality and filtering low-value or harmful records.Source: Introduction · p. 1
  • Utility-hijacking exploits a gap between poisoning studies that assume poison reaches training and selection studies that assume benign suppliers.Generate-and-rerank fixes the harmful payload and searches for selector compatibility, addressing objectives that otherwise pull against each other.Source: Introduction · p. 1
  • CREST formalizes poison concentration in top-k selection and provides a statistic that separates vulnerable from immune selector–payload pairs without overlap in the experiments.The reported statistic is at least 0.596 for every vulnerable pair and at most 0.008 for every immune pair.Source: Introduction · pp. 1–2
  • Across six selectors, utility-hijacking’s effects depend on selector and payload, but on vulnerable pairs it raises poison concentration from 1.21× to 6.07× and refusal from 0% to 94%.It elicits insecure code in up to 56/90 cases; false-fact enrichment rises from 2.84× to 7.04× with scoring-model scale, and concentration reaches 10.78× at a 1% input rate.Source: Introduction · p. 2
  • CONCERT combines a provenance audit with fixed-budget rank consensus, requiring neither a content detector nor poison labels.Its rank-intersection certificate excludes records outside either selector’s certified prefix.Source: Introduction · p. 2

Related Work

  • Selectors are commonly evaluated with benign suppliers, while prior poisoning work assumes poison already reaches training; this paper studies supplier adaptation to survive selection.Prior poisoning work includes Wan et al. 2023; Shu et al. 2023; Qi et al. 2024.Source: Related Work · p. 2

Problem Formalization and Threat Model

The threat model considers an adversarial supplier contributing poison to a select-then-train pipeline. The attack must make records both selectable and harmful after fine-tuning, while the supplier cannot modify the pipeline or observe the victim’s selected set.

Problem Formalization and Threat Model

  • The pipeline selects the top-k records from the combined clean and poisoned pool under a fixed budget, then fine-tunes only on that subset.The selectors span influence-based, difficulty-based, and quality-based approaches.Source: Problem Formalization and Threat Model · p. 2
  • The attacker may score candidates using a selector replica but cannot modify the selector, fine-tuning, evaluation, or clean pool, or observe the victim’s selected set.Replica scores reveal candidate rankings, not the realized cutoff, which also depends on the clean pool and budget.Source: Problem Formalization and Threat Model · pp. 2–3
  • Utility-hijacking requires selector compatibility and payload preservation, which pull against each other; the studied payloads are over-refusal, false-fact injection, and insecure code.The attacker’s goal is to increase the poison’s share of the fixed training budget and change model behavior after fine-tuning.Source: Problem Formalization and Threat Model · p. 3
  • For each source instruction, generate-and-rerank scores K payload-bearing candidates and submits the highest-scoring one that still carries the fixed payload.The pass is non-iterative, trains no attack model, and is compared with the same poison without reranking.Source: Problem Formalization and Threat Model · p. 3
  • The threat model considers feedback from full continuous scores to coarse ordinal buckets or none, but excludes sparse, delayed accept/reject feedback and fully private or safety-hardened selectors.The defense targets ordinary utility selectors.Source: Problem Formalization and Threat Model · p. 3

CREST: Enrichment as an Upper-Tail

CREST explains utility-hijacking through poison and clean records’ survival in the selector’s upper tail. Its measurements show that vulnerability depends on the selector–payload pair, and that tail compatibility can be diagnosed before fine-tuning.

Quantity

  • CREST reduces utility-hijacking to clean and poison tail-survival probabilities, which can be estimated from scores already computed by the pipeline before fine-tuning.On a labeled payload diagnostic set, vulnerability assessment costs one scoring pass instead of one training run.Source: Quantity · p. 3
  • The finite-rate identity is exact from realized selection counts (0% median residual), while its small-rate approximation has 22% median error at r ≈9–10%.Enrichment depends on scores only through clean and poison tail-survival probabilities; the approximation isolates their ratio as the driver.Source: Quantity · p. 3
  • qf(15%) separates vulnerable cells (≥0.596) from immune cells (≤0.008), a 0.588 gap with no overlap (Figure 1).The statistic measures poison mass clearing the clean top-b cutoff; the score gap matters more than average quality because enrichment depends on poison crossing the tail boundary.Source: Quantity · pp. 3–4, Figure 1 · p. 4
  • Over 10–40% budgets, enrichment decreases monotonically on three vulnerable cells (Spearman ρ ≤−0.85); it rises as input rate falls, exceeding 10× at 1%.At very small budgets (2–10%), the sufficient monotonicity condition can fail in finite samples, with mild local reversals observed.Source: Quantity · p. 4
  • DEITA rejects over-refusal poison (E=0.93×) but amplifies false facts (E=3.64×) and insecure code (E=6.67×), while IFD is immune to code (E=0).Amplification is jointly determined by the selector, payload distributions, and cutoff; immunity to one payload does not imply global immunity.Source: Quantity · p. 4

CONCERT: A Rank-Intersection Defense

CONCERT pairs a provenance audit that detects supplier over-selection with a fixed-budget consensus that keeps records ranked highly by two selectors. The audit is exact under its exchangeability null, while consensus provides a rank-intersection exclusion guarantee rather than a poison-free guarantee.

CONCERT

  • CONCERT combines supplier-level over-selection auditing with consensus reselection, needing neither poison labels nor a content detector.The audit uses provenance; consensus uses selector scores.Source: CONCERT · pp. 4–5

CONCERT-Audit: Provenance-Level Detection

  • Under the exchangeability null, supplier selection counts follow a hypergeometric law; small p-values flag suppliers selected more often than chance allows.Holm controls family-wise error under arbitrary dependence given valid per-supplier p-values; Benjamini–Hochberg is reported for detection power and boundary analysis.Source: CONCERT-Audit: Provenance-Level… · p. 5

CONCERT-Consensus: Fixed-Budget Rank Consensus

  • Consensus chooses the second selector most discordant with the anchor, then keeps the best records by their worse percentile rank across both selectors.This produces exactly k records and is invariant to strictly increasing transformations of selector scores.Source: CONCERT-Consensus: Fixed-Budget Rank… · p. 5
  • Every selected record lies within both selectors’ Top-t⋆ prefixes, excluding any record the partner ranks below the consensus threshold.This is a set-containment guarantee, not a bound on poison counts or downstream harm.Source: CONCERT-Consensus: Fixed-Budget Rank… · p. 6
  • With an informed attacker, poison independently ranked highly by the partner may still enter; poison dispersed across suppliers also weakens audit power.The experiments grant attackers knowledge of the public partner rule, but test reranking against it rather than adaptive generation.Source: CONCERT-Audit: Provenance-Level… · p. 5, CONCERT-Consensus: Fixed-Budget Rank… · p. 6
  • 88–96% of the anchor’s high-value clean data is retained by CONCERT, versus 51–58% when switching outright to an immune selector.An ablation shows arbitrary partners do not confer safety.Source: CONCERT-Consensus: Fixed-Budget Rank… · p. 6

Experiments

End-to-end experiments evaluate selection and fine-tuning across three payloads, with Top-15% selection from a pool poisoned at approximately 9–10%. CONCERT reverses poison enrichment and reduces downstream harm, while tests also examine code attacks, adaptive reranking, and audit performance under supplier dispersion.

Experiments

  • End-to-end runs select Top-15% from Alpaca-GPT4 at approximately 9–10% poisoning, with LoRA fine-tuning.False facts begin with Qwen2.5-7B-Instruct, while code and CONCERT use a Qwen3-14B proxy and Qwen3-32B victim.Source: Experiments · p. 6

Utility-Hijacking Holds Across Payloads

  • 56/90, 56/90, and 41/90 insecure-code outputs occur across three attacked seeds, versus 0/90 clean on every seed; the attack–clean gap is +56.67 ± 9.62pp.HumanEval shifts by only −0.41 ± 1.27pp.Source: Utility-Hijacking Holds Across Payloads · p. 6

CONCERT Neutralizes the Harm

  • 3.59×/6.67× enrichment falls to 0.67×/0× for facts/code under CONCERT; false-fact harm drops 75%, and unsafe code falls to 0/90 on each seed.HumanEval moves +0.20 ± 0.35pp; clean-data retention is 96.2%/89.5%, versus 51.4%/60.1% under selector switching.Source: CONCERT Neutralizes the Harm · p. 7, Figure 2 · p. 7
  • At six suppliers, audit detection is 48.25%/33.83% under BH for facts/code, versus 28.5%/17.25% under Holm, as dispersion weakens audit power.Beyond this layer, consensus alone carries the defense.Source: CONCERT Neutralizes the Harm · p. 7
  • For K ∈ {4, 8, 16, 32}, white-box reranking selects no code poison while retaining 100% of payload; false-fact enrichment rises from 0.44× to 0.56× against a 1.64× anchor on a weaker pool.These runs rerank a fixed generator; adaptive generation remains untested.Source: CONCERT Neutralizes the Harm · p. 7

Limitations and Conclusion

The paper frames instruction-data selection as a security boundary and reports that utility-hijacking can concentrate harmful records in the selected training data. Its conclusion emphasizes that CONCERT removes observed harm at fixed budget, while identifying limits in provenance auditing and adaptive-generation testing.

Limitations and Conclusion

  • Data selection is a security boundary over untrusted data because utility-hijacking makes top-k selection concentrate harmful records.CONCERT combines label-free auditing with rank-intersection reselection at fixed budget.Source: Limitations and Conclusion · p. 7
  • The audit requires reliable provenance and loses power when poison is dispersed across suppliers.The conclusion identifies broader payloads and adaptive generation as future work.Source: Limitations and Conclusion · p. 7
  • The tested adaptive attack reranks a fixed generator rather than re-optimizing generation against the defense.The paper leaves adaptive generation for future work.Source: Limitations and Conclusion · p. 7

Technical Supplement

The supplement documents the frozen artifact, what can be independently reproduced, and the experimental configurations and infrastructure. Some materials and exhaustive search records were not retained or redistributed.

Reproducibility Summary

  • The archive preserves preprocessing, selection, training, evaluation, and statistical code alongside frozen configurations, prompts, and generated candidate pools.The recorded software stack and frozen requirements are also specified.Source: Reproducibility Summary · p. 10, Frozen Experimental Configuration · p. 10
  • Third-party model weights and benchmark corpora, LoRA adapters, full benchmark completions, and some generated pools are not redistributed.Public identifiers, fingerprints, schemas, configurations, and lineage hashes are supplied instead; the hosted over-refusal judge identifier was not preserved.Source: Reproducibility Summary · p. 10
  • The supplement maps the paper’s claims to audit files and provides code to recompute reported means, sample standard deviations, the 32B confidence interval, and over-refusal summaries.The archive includes generated candidates, selection tables, per-seed and aggregate results, configurations, prompts, and lineage hashes.Source: Reproducibility Summary · p. 10

Frozen Experimental Configuration

  • The supplement identifies the public checkpoints and retained Qwen3 commits, while config and safetensors-index fingerprints identify checkpoints whose repository commits were not preserved.Hardware records distinguish cluster inventory from retained job allocations; the exact allocated node was not retained.Source: Frozen Experimental Configuration · p. 10
  • The frozen configuration records final training settings, but the development process did not retain a complete exhaustive hyperparameter-search ledger.Reported variations include candidate budgets, temperatures, selection budgets, poison rates, models, and selectors.Source: Frozen Experimental Configuration · p. 10

Run Inventory and Randomness

The supplement specifies the evaluation and run protocols, including randomness, statistical procedures, and supplier-audit settings. Its code results use a defined three-CWE scope, and downstream checkpoints predate that scope correction.

Run Inventory and Randomness

  • Random seeds are passed to generation and training, and exact top-k cardinality uses average percentile ranks followed by a stable record-key tie-break.Table S1 reports the final fine-tuning configurations, and Table S2 counts training runs while listing deterministic frozen-pool analyses as zero SFT runs.Source: Run Inventory and Randomness · pp. 10–11, Table S1 · p. 11, Table S2 · p. 11

Evaluation and Statistical Protocols

  • The primary insecure-code evaluator is a deterministic targeted-CWE AST/regex detector, while TruthfulQA MC1 is the primary false-fact metric and MMLU and HumanEval measure capability and code correctness.The local semantic code audit is corroborative because its exact interactive serving snapshot was not programmatically recorded.Source: Evaluation and Statistical Protocols · p. 11
  • The code attack gap of +56.67±9.62 pp and HumanEval change of −0.41±1.27 pp are mean ± sample SD across seeds 7/13/21.For 32B false facts, the paired held-out MC1 change has a two-sided Student-t 95% confidence interval of [−0.079, −0.045] with n = 3 and df = 2; p = 0.00409 becomes p = 0.04499 after Holm correction over 12 tests.Source: Evaluation and Statistical Protocols · p. 11
  • The strict seed-7 random control matches training size and poison count at 825 and 200, using 625 clean records but different poison identities from the selector condition.It is distinct from the unconcentrated three-seed condition, which includes all 500 poison records in the incoming pool.Source: Evaluation and Statistical Protocols · pp. 11–12
  • Supplier-audit tests use 0.05 Holm FWER or Benjamini–Hochberg FDR control, and partner selection compares tie-averaged percentile ranks without poison labels.At six suppliers, BH detects 48.25%/33.83% for facts/code and Holm detects 28.5%/17.25%; AlpaGasus is selected for facts and IFD for code.Source: Evaluation and Statistical Protocols · p. 12
  • DEITA selects 93/223 poison records (E = 6.67×), while IFD selects 0/223 (E = 0) and CONCERT-minimax selects 0/223 (E = 0) in the three-CWE selector-stage evaluation.Across seeds 7/13/21, downstream attack counts are 56/90, 56/90, and 41/90, versus 0/90 for clean and CONCERT; these audit existing checkpoints rather than retraining them on the reduced pool.Source: Evaluation and Statistical Protocols · p. 12

CONCERT Pseudocode

CONCERT audits suppliers for statistically unusual selection and then chooses a fixed-budget training set using the anchor selector and its least-correlated candidate partner.

CONCERT Pseudocode

  • CONCERT counts each supplier’s top-k selections, computes exact hypergeometric p-values, and applies Holm or BH at 0.05 to flag over-selected suppliers.It then converts the anchor and candidate-selector scores to tie-averaged percentile ranks and chooses the partner with minimum correlation.Source: CONCERT Pseudocode · p. 12
  • The final set contains the k records with the smallest maximum of the anchor and chosen partner’s percentile ranks, with deterministic tie-breaking.This implements a fixed-budget rank-intersection selection without poison labels.Source: CONCERT Pseudocode · p. 12
  • After the partner is fixed, deployment adds one partner-scoring pass relative to the anchor-only pipeline if ranks are not cached.For facts the selected partner is AlpaGasus; for code it is IFD.Source: CONCERT Pseudocode · p. 12

Selector–Payload Compatibility Matrix

Selector compatibility varies by payload: measured cells show that protection against one poison type does not necessarily transfer to another. The insecure-code results are scoped to three CWE categories.

Selector–Payload Compatibility Matrix

  • Measured Top-15% cells show DEITA diluting over-refusal poison but amplifying false facts and insecure code, while IFD amplifies over-refusal and false facts but dilutes insecure code.Unmeasured cells are left blank, and no claim is made about them.Source: Selector–Payload Compatibility Matrix · p. 12
  • For CWE-078/295/502, DEITA selects 93/223 poison records (E = 6.67×), IFD selects 0/223 (E = 0), and Borda selects 7/223 (E = 0.50×).The downstream attack–clean gap is +56.67 ± 9.62 pp across seeds 7/13/21; these checkpoint results audit existing models rather than retraining on the reduced pool.Source: Selector–Payload Compatibility Matrix · p. 12
  • The reported insecure-code scope comprises CWE-078 command injection, CWE-295 TLS validation, and CWE-502 unsafe deserialization.The final code/data archive does not contain the six frozen qf values’ legacy source slice as a standalone derived CSV.Source: Selector–Payload Compatibility Matrix · p. 12

Budget and Poison-Rate Sweeps

The sweeps show enrichment falling with budget over the prespecified practical range, with mild finite-sample reversals at smaller budgets. Lowering the input poison fraction from 10% to 0.5% increases enrichment from 7.04× to 12.81×.

Budget and Poison-Rate Sweeps

  • In the rate sweep, reducing the input poison fraction from 10% to 0.5% increases enrichment from 7.04× to 12.81×, as reported for that sweep.Source: Budget and Poison-Rate Sweeps · p. 12
  • Across the prespecified practical range b = 10–40%, enrichment decreases monotonically with budget for the three vulnerable cells shown; the 2–10% points expose mild finite-sample reversals.Source: Budget and Poison-Rate Sweeps · p. 12
  • At budgets of 2–10%, the sweeps show mild finite-sample reversals.These points expose reversals rather than hiding them.Source: Budget and Poison-Rate Sweeps · p. 12

Supplementary Proofs

The supplementary proofs express poison enrichment through upper-tail survival and establish its behavior under stated conditions. They also characterize CONCERT’s fixed-budget, rank-based selection and clarify what its intersection certificate does—and does not—guarantee.

Supplementary Proofs

  • On held-out TruthfulQA, the strict equal-size random control finds no greater harm from IFD selection across three truthfulness summaries, with identical MC1.The control does not identify an additional selector-specific causal effect.Source: Table S3 · p. 13
  • Figures S1 and S2 report selector–payload enrichment at Top-15% and sweep curves, with blanks marking unmeasured cells and distinct line styles and markers.S1 uses different model or diagnostic slices by payload; S2 includes three vulnerable cells and a Qwen3-32B false-fact rate sweep.Source: Figure S1 · p. 13, Figure S2 · p. 14
  • Table S4 distinguishes qf, measured against a clean-only cutoff, from Sp at the realized mixed-pool cutoff; qf = 0 can coexist with nonzero finite-pool enrichment.The caption identifies this as a labeled diagnostic, not an unlabeled online detector.Source: Table S4 · p. 13
  • E = Sp/(rSp + (1 − r)Sc), and when r|Sp/Sc − 1| ≪ 1 with Sc > 0 and locally stable survival probabilities, E ≈ Sp/Sc.The realized finite-pool identity uses poison and clean tail-survival fractions at a fixed budget and rate.Source: Supplementary Proofs · pp. 12–14
  • As b approaches 1, E approaches 1; at a fixed cutoff E decreases with poison rate, and under MLR it is non-increasing with budget.The budget monotonicity condition is the MLR regime.Source: Supplementary Proofs · p. 14
  • At very small budgets in finite samples, enrichment can reverse locally when the MLR condition does not hold.The monotonicity result is conditional, not universal.Source: Supplementary Proofs · p. 14
  • CONCERT-Consensus returns exactly k records and is unchanged by strictly increasing transformations of selector scores.Its selection depends on within-selector percentile ranks, so incomparable score scales need no calibration.Source: Supplementary Proofs · p. 14
  • The minimax certificate places selected records in the intersection of two selectors’ prefixes, but does not itself bound poison count, enrichment, or downstream harm.Records ranked within the threshold by both selectors can still be selected.Source: Supplementary Proofs · p. 14

Related papers

Loading 2610.29886v1…