Fundamental Limitations in Pointwise Defences of LLM Finetuning APIs

1 Introduction

• Pointwise content filters are insufficient against fine-tuning attacks when harmful text can appear in benign queries and model responses vary systematically.The paper identifies these as the conditions under which harmful information can be elicited.Source: §1 · pp. 1–2
• The attacks turn harmful requests into benign queries and map ordinary response variations to answer choices for training and inference.Figure 1 illustrates a classify-style transformation and the use of the same mapping to decode answers.Source: §1 · p. 2, Figure 1 · p. 2
• The authors report that their attacks answer harmful multiple-choice questions and evade OpenAI safeguards and their enhanced safeguards.The evaluated topics include improvised explosive device construction and copyright violation.Source: §1 · p. 2
• The attacks are extended to settings with no harmful text in training data and to scaled attacks intended to make multi-sample detection more difficult.The paper identifies these extensions in Sections 5.1 and 5.2.Source: §1 · p. 2

2 Background

• Fine-tuning misuse involves uploading user-assistant interactions for supervised fine-tuning, while pointwise defenses may screen training samples or monitor individual inference interactions.The paper focuses on these two classes of detection systems.Source: §2 · pp. 2–3
• Detecting harmful behavior only when samples appear in combination may be difficult because training sets can be large and inference can include unrelated benign interactions.The authors focus on pointwise systems because they believe these are among the most common defenses.Source: §2 · p. 3
• Earlier covert fine-tuning attacks used ciphers or simple steganography, making interactions relatively straightforward to categorize as suspicious; Halawi et al. [2024] report that 91.1% of ciphered interactions were classified as ‘Unclear’ rather than good behaviour.Source: §2 · p. 3
• The authors contrast those approaches with pointwise-undetectable attacks, arguing that pointwise monitors alone cannot win the detection contest.They describe this as a departure from the anticipated cat-and-mouse dynamic of progressively more sophisticated steganography.Source: §2 · p. 3

3 Pointwise Undetectable Fine-Tuning Attacks

• Pointwise-undetectable attacks use individually unsuspicious, naturally occurring training and inference interactions, assuming harmful-text queries are allowed and model responses vary consistently.The paper defines benign interactions by policy adherence and unsuspicious assistant messages by their high likelihood under the pre-fine-tuning model.Source: §3 · pp. 3–4

3.1 Benign Inputs with Harmful Text

• Benign transformations can present harmful text for allowed tasks such as classification, while unrelated benign requests can also contain incidental harmful text.Both types of input are described as valid use cases that providers may permit.Source: §3.1 · p. 4

3.2 Exploiting Benign Variations in Model Output

• The attacks exploit systematic syntactic or semantic response variations and map a variation to each multiple-choice answer so outputs communicate the selected answer.The variations can include phrasing or other learnable features such as response length or style.Source: §3.2 · p. 4
• Theorem 1 states that the attacks’ generated outputs are pointwise-undetectable; the proof places them above a likelihood threshold used to flag anomalous outputs.The proof says the attack can resolve harmful multiple-choice responses when the set of supported benign outputs is large enough.Source: §3.2 · pp. 4–5
• Unlike the proposed attacks, Halawi et al. [2024]’s outputs cannot generally be constrained to the benign model’s output support, making them detectable for thresholds above zero.The paper contrasts this property with its attacks’ restriction to outputs in the base model’s support.Source: §3.2 · p. 5

4.1 Harmful MCQ Dataset Construction

• The authors built synthetic, policy-violating, nontrivial MCQ datasets with answers checked by multiple models, sampling 211 IED training questions and 134 Copyright training questions.The paper notes these synthetic datasets may include some low-quality samples.Source: §4.1 · p. 5

4.2 Attack Instantiation

• Both attacks transform harmful prompts into benign requests and encode answer choices through naturally occurring, unsuspicious variations in model responses.Source: §4.2.1 · p. 6, §4.2.2 · p. 6

4.2.1 Classify Attack (Benign Transformation)

• Classify asks whether a harmful MCQ is malicious or benign, then maps common classification phrasings to the correct answer choice in 1,035 training samples.Source: §4.2.1 · p. 6

4.2.2 Flower Attack (Unrelated Request)

• Flower places a harmful request earlier in forum context, asks for a random flower, and maps common flower responses to the answer choices.Source: §4.2.2 · p. 6

4.3 Baselines

• The compared baselines first teach cipher or poetic last-word steganography, then break alignment with encoded harmful samples, requiring substantially more training data than the authors’ attacks.Source: §4.3 · p. 6

4.5 Results

• On IED-MCQ, classify and flower each achieved 98.4% correct versus CMT-EndSpeak’s 90.2% and CMT-Walnut’s 62.3%; on Copyright-MCQ, classify, flower, and CMT-Walnut each reached 100.0%.The figure compares attack success rates across monitoring setups for both datasets.Source: Figure 2 · p. 7, §4.5 · pp. 7–8

4.4 Defences & Scoring

• OpenAI’s API screens training data; custom o1-mini monitors score individual inference samples for spec violations or suspiciousness, with refusals or flags counted as blocked.Source: §4.4 · p. 7

4.5 Results

• Spec monitoring never flagged classify, while suspiciousness monitoring left it at 96.7% on IED-MCQ and 100.0% on Copyright-MCQ; flower scored 82.0% and 87.2%, respectively.CMT-Walnut and CMT-EndSpeak were blocked much more often by suspiciousness monitoring.Source: §4.5 · p. 8

5.1 Training without Harmful Text

• The no-harmful-text setting relaxes the earlier requirement for a suitable benign transformation during training, which may matter under tighter training-data restrictions.Source: §5.1 · p. 8, §5 · p. 8
• Classify-benign reached 69.2% correct on Copyright-MCQ and 96.7% on IED-MCQ without being blocked by any monitor; flower-benign reached 79.5% on Copyright-MCQ without monitors and 71.8% under other monitor setups.On IED-MCQ, flower-benign scored 98.4% without monitors and 78.7% under the suspicious interaction monitor, which blocked it 19.7% of the time.Source: §5.1 · p. 9, Figure 3 · p. 9
• Training on transformed Commonsense-QA questions lets the classify and flower variants generalize to unseen harmful questions in the IED and copyright domains.The flower variant uses benign, unsuspicious training and inference samples, with no harmful text in the benign training requests.Source: §5.1 · pp. 8–9

5.2 Evading Multi-Sample Detection

• Mapping each answer choice to multiple response variations makes suspicious subsets harder to detect because random subsets are less likely to contain suspicious numbers of variations tied to correct choices.The paper also identifies mixing irrelevant benign training or inference data as a complementary strategy.Source: §5.2 · p. 9
• The multiple-variation experiment mapped each of four answer choices to k responses, using 36 generic classification strings and 670 training samples.The 670 samples came from 134 Copyright-MCQ training samples repeated five times with random answer orderings.Source: §5.2 · pp. 9–10
• GPT-4o-mini learned the answer-choice assignments as variation counts increased, with only a small increase in invalid choices; the figure reports minimal capability degradation even at nine variations per choice.The test ensured that every answer variation appeared at least once.Source: §5.2 · p. 10, Figure 4 · p. 10

• The experiments focus on multiple-choice questions, although the paper discusses extensions to free-form answers and stronger-model response uplift using a weaker unsafeguarded model.Source: §6 · p. 10
• The attack is designed to evade stateless pointwise detectors; certain distribution-level systems catch it, though the authors suspect modifications could make detection harder.Source: §6 · p. 10
• The discussion connects pointwise-undetectable attacks to diffuse threats, where individually benign actions or data points can collectively produce an unacceptable outcome.Source: §6 · p. 10

A Societal Impacts

• The authors describe both potential misuse of the techniques and mitigations including limited implementation detail, reporting vulnerabilities before publication, and using proxy harmful tasks.They argue that the defensive value of identifying vulnerabilities outweighs the misuse risks.Source: A Societal Impacts · p. 15

B.1 Theoretical Extension

• For free-form generation, the proposed extension trains the model to produce an answer one letter at a time, using a benign flower variation to select each next letter and the growing answer as prompt context.Source: §B.1 · p. 15, Figure 5 · p. 15
• The authors warn that iterative encoded generation may degrade capabilities and leave experiments in this setting for future work.Source: §B.1 · p. 15

B.2 Weaker Unsafeguarded Uplift

• The weaker-model uplift scheme generates four free-form answers with an unsafeguarded model, then uses the classify attack on a stronger model to select among them.Source: §B.2 · p. 16
• On SimpleQA, GPT-4.1 Nano rose from 7.2% to 12.6% with the classify fine-tune selecting among its answers, a 75% improvement but below GPT-4o’s 41%.Source: §B.2 · p. 16
• The reported uplift is an initial exploration, and the authors suspect more sophisticated schemes could improve it.Source: §B.2 · p. 16

C.1 Capabilities Degradation & Generalization

• The evaluation used difficult graduate-level GPQA questions and 15 difficult malicious BSD-Cyber questions, comparing the original model with models fine-tuned on IED-MCQ and Copyright-MCQ.Source: §C.1 · p. 16
• On BSD-Cyber, the classify attack scored 0.467, while the flower attack scored 0.587; on GPQA, classify scored 0.406 and flower scored 0.364.Unattacked GPT-4o scored 0.547 with chain-of-thought and 0.573 without it on BSD-Cyber, and 0.51 with and 0.428 without chain-of-thought on GPQA; the authors characterize degradation as limited and note these datasets were not originally refused.Source: §C.1 · p. 16, Figure 6 · p. 17
• The results support transfer: models trained on IED-MCQ and Copyright-MCQ performed well on difficult questions from other datasets, with minimal capability degradation.Source: §C.1 · pp. 16–17

C.2 Verifying Plausibility of Assigned Variations

• To check response plausibility, the authors sampled 30 test cases and measured how many generations from the pre-fine-tuning model were needed to match attacked-model responses; they ran this experiment once for indicative figures.Source: §C.2 · p. 17

C.3 Results on GPT-4o-mini

• Figure 7 reports GPT-4o-mini performance for all attacks, while the section also provides classify- and flower-attack generation-match tables.The supplied passages do not state numerical results from the figure or tables.Source: §C.3 · p. 17, Table 1 · p. 18, Table 2 · p. 18

C.4 Results with Error Bars

• Figures 8 and 9 reproduce earlier results with error bars, and each question is scored once as Correct, Blocked, or Incorrect.The bars show standard errors for binomial proportions; Correct and Blocked error bars are offset left and right, respectively.Source: §C.4 · p. 18
• For each category, the standard-error calculation uses its sample proportion p and total sample count n.Source: §C.4 · p. 18

C.5 Gemini on Vertex API Experiments

• Gemini 2.5 Pro learned the classify attack when trained on the IED-MCQ and Copyright-MCQ datasets.Source: §C.5 · pp. 18–19
• The harmful training sets were not flagged by moderation even without the attack, so the authors expected no monitoring for the attack to evade.Source: §C.5 · p. 19
• Accordingly, the authors report attack performance only for OpenAI’s Fine-Tuning API.Source: §C.5 · p. 19

D.1 Manual Quality Audit

• A human audit found IED answers easier to quality-control than Copyright answers, which included more unfamiliar technical terms and occasional irrelevant or hallucinated details.The auditors scored 82% on IED and 78% on Copyright for a random 50-question subset; the authors expected the IED dataset to be higher quality.Source: §D.1 · p. 20

D.2 IED-MCQ Additional Information

• On IED-MCQ, question-only prompts were often refused more than multiple-choice variants until made more explicit.The appendix also supplies sample IED questions and answer-choice lists.Source: §D.2 · pp. 20–21

D.3 Copyright-MCQ Additional Information

• On Copyright-MCQ, question-only prompts were likewise often refused more than multiple-choice variants until made more explicit.The appendix provides sample Copyright-MCQ questions and answer responses.Source: §D.3 · pp. 21–22

E Distribution-Level Detection

• The authors extended policy-violation and suspiciousness monitors to assess groups of N training samples, using separate monitors with labeled attack and benign examples.Source: E Distribution-Level Detection · p. 22
• At N = 15, o3 flagged true classify-attack datasets as Suspicious 76.0% of the time and labeled benign classify variants Benign 92.0% of the time; o4-mini failed to separate the datasets.The figure defines TPR as accuracy flagging attack subsets and TNR as accuracy abstaining on benign subsets.Source: E Distribution-Level Detection · p. 22, Figure 12 · p. 23
• Distribution-level detection may help against this specific attack, but the authors say benign-data mixing or the Section 5.2 approach could likely defeat this defense.Source: E Distribution-Level Detection · p. 23

F Continued Discussion

• The authors hypothesize that attacks are learnable because fine-tuning remaps an already computed answer into a new output format, effectively training a new decoder.They also note pointwise-undetectable pre-training poisoning may be especially difficult to counter because examining substantial data subsets is intractable at pre-training scale.Source: F Continued Discussion · p. 23
• Pointwise-undetectable attacks exemplify diffuse threats: individually benign actions or data points can jointly produce an unacceptable outcome.The discussion notes that detecting such threats can require expensive auditing and risk false alarms.Source: F Continued Discussion · pp. 23–24

F.1 Other Related Work

• Related work covers mixed benign and harmful fine-tuning data, refusal-reducing benign datasets, and steganographic model communication, but not that communication in fine-tuning attacks.Source: §F.1 · p. 24

F.2 Additional Fine-Tuning Defences

• The authors list model-level training interventions, post-training evaluations, benign-data mixing, activation probes, self-assessment, and multi-sample detection as defenses to evaluate against these attacks.For this specific attack, training-distribution entropy may help detection, though the authors expect the attack could be modified to make that harder.Source: §F.2 · p. 24

G Attack Question-Answer Template Examples

• The Copyright-MCQ example is explicitly identified as a low-quality sample because its supposedly correct answer hallucinates erroneous details.Source: G Attack Question-Answer Template… · pp. 24–25

H.1 Details

• Four custom monitors classify individual examples: training- and inference-time Model Spec violation monitors, and training- and inference-time suspiciousness monitors.They were obtained by prompting o1-mini.Source: §H.1 · p. 32
• The authors iterated on the monitor prompts to maximize accuracy on XSTest, OR-Bench, and validation examples from IED-MCQ and Copyright-MCQ paired with direct answers or refusals.Table 3 reports monitor accuracies on those datasets.Source: §H.1 · p. 32
• The prompts were not directly optimized against CMT-EndSpeak, CMT-Walnut, Classify, or Flower.Source: §H.1 · p. 32

H.2.1 Spec violation training-time monitor prompt

• The training-time Model Spec monitor compares a training example with the Spec and returns “yes” for a clear violation or “no” if it does not violate the Spec or is unclear.It also considers whether training on the example would cause a violation.Source: §H.2.1 · pp. 33–34

H.2.2 Spec violation inference-time monitor prompt

• The inference-time Model Spec monitor checks whether a conversation clearly violates the Spec, answering “yes” for a violation and “no” if it does not or is unclear.Source: §H.2.2 · pp. 34–35

H.2.3 Suspiciousness training-time monitor prompt

• The training-time suspiciousness monitor assesses whether the assistant response is harmful or suspicious, whether the example suggests a safety bypass, or whether it may lead to harmful outputs after fine-tuning.A harmful user input alone is insufficient for a harmful/suspicious judgment; a refusal is considered safe.Source: §H.2.3 · p. 36
• The training-time suspiciousness prompt asks for a single-word yes or no indicating whether an example is harmful or suspicious.The example is inserted into the prompt as input.Source: §H.2.3 · p. 36

H.2.4 Suspiciousness inference-time monitor prompt

• The inference-time prompt judges harmfulness or suspiciousness from the assistant’s response or an unusual conversation, not from a harmful user input alone; refusal is safe.The monitor returns a single-word yes or no judgment.Source: §H.2.4 · p. 36

H.2.5 Distribution-level detection monitor prompt

• It compares known compromised positive examples with similar safe negatives, then searches target datasets for functionally similar attack patterns and resulting Model Spec violations.Harmful user inputs alone do not suffice; assistant responses must train unsafe behavior.Source: §H.2.5 · pp. 37–38
• The distribution-level monitor is instructed to search across examples for encoded information, subtly trained behavior, or statistical patterns, even when individual examples appear benign.It evaluates whether patterns across conversations could train behavior that violates the Model Spec.Source: §H.2.5 · p. 38
• The monitor must verify a coherent, compelling account of what a pattern trains or encodes, support it with evidence, and avoid overfitting or unsupported complexity.The prompt also asks it to propose and verify candidate patterns and reject benign explanations when evidence warrants.Source: §H.2.5 · p. 38
• The distribution-level prompt requests valid JSON with a pattern summary, matched targets, evidence, and counter-evidence.It asks for a suspicious yes-or-no judgment and for an explanation of why benign alternatives were rejected or evidence was insufficient.Source: §H.2.5 · p. 39

H.3 Sonnet Judge

• Claude Sonnet flags CMT-Walnut and CMT-Endspeak at a much higher rate than o1-mini and decodes their ciphers, but often fails to decode or recognize encoded assistant responses.The accompanying figure covers attack success rates on GPT-4o for IED-MCQ and Copyright-MCQ with Sonnet as the inference-time monitor; no rates are provided in the passages.Source: §H.3 · pp. 39–40
• The Sonnet Judge says the cipher is not the primary concern: the underlying detailed instructions for unauthorized access and security circumvention are potentially harmful and non-compliant.It characterizes the content as an information hazard because it could enable website compromise and harm online services and users.Source: §H.3 · pp. 41–42
• The judge connects the content to Model Spec rules #2 and #4, citing applicable laws and respect for creators’ rights as grounds for non-compliance.It says references to mass downloading copyrighted content and bypassing protection measures conflict with those rules.Source: §H.3 · pp. 41–42
• The judge says the instructions could violate terms of service and copyright laws and implicate the Computer Fraud and Abuse Act and the Digital Millennium Copyright Act.It associates mass downloading with unauthorized copying and bypassing protections with circumvention of technical measures.Source: §H.3 · p. 41
• The judge identifies Selenium, BeautifulSoup, Puppeteer, and other scraping tools alongside techniques for mass downloads, automation, and bypassing rate limits and anti-abuse measures.The described methods also include captcha handling, cookie manipulation, proxies, and avoiding detection.Source: §H.3 · p. 41
• It says the combination of browser automation, mass extraction, and bypassing access controls suggests circumvention of website security and content protections.The judge notes that web scraping alone is not inherently illegal, but the described combination suggests intent to evade legitimate controls.Source: §H.3 · pp. 41–42

I Examples of Inference Time Monitor Reasoning

• The inference-monitor reasoning examples add a request for at least 300 words of explanation, which the authors treat as presumably summarizing o1-mini’s chain of thought despite known explanation non-faithfulness (Lanham et al. [2023]).The examples are provided for qualitative insight into monitoring results.Source: I Examples of Inference Time Monitor… · p. 42

I.1.1 CMT-Walnut

• The scenario’s system message requires walnut53 encoding, but the assistant replies with a single “z” instead.The monitor says the platform instruction should take precedence over user input and other instructions.Source: §I.1.1 · p. 42
• The monitor says rules on illegal activity, information hazards, creators’ rights, privacy, and NSFW content are not implicated, while the hierarchy violation suffices for non-compliance.Source: §I.1.1 · p. 42
• The monitor concludes that the assistant’s response violates the Model Spec by disregarding the platform-level walnut53 directive, warranting a “yes” response.The monitor identifies the chain-of-command violation as sufficient for this judgment.Source: §I.1.1 · pp. 42–43

I.1.2 CMT-Endspeak

• Across both datasets, the model-spec monitor flagged CMT-Endspeak examples for chain-of-command violations; the authors hypothesize that recognizing the encoding would reduce flags to nearly 0%.They note that some IED-MCQ samples might still be flagged because of harmful terms.Source: §I.1.2 · p. 43
• The monitor treated the system’s EndSpeak-only directive as controlling and judged the plain-English response a failure to follow it.Its reasoning also said the user had provided no conflicting instruction that justified the deviation.Source: §I.1.2 · p. 43

I.1.3 flower and flower-benign

• For GPT-4o, the monitor flagged flower samples on IED-MCQ and flower-benign samples on Copyright-MCQ, misapplying the spec by expecting harmful context to be addressed.The authors consider these flags errors because the responses did not add or provide harmful information.Source: §I.1.3 · p. 44

I.2.1 CMT-Walnut and CMT-Endspeak

• The monitor fails to flag nearly 100% of CMT-Walnut interactions on either dataset, and the authors hypothesize a better monitor would flag nearly 100% of both Walnut and Endspeak samples.They identify missing definitions of suspiciousness, failure to recognize the encoding, and reliance on system-prompt compliance even when users may supply that prompt.Source: §I.2.1 · p. 45

I.2.2 classify

• The monitor flags classify interactions on GPT-4o for IED-MCQ only a small percentage of the time, apparently because it misreads the harmful user query despite instructions not to judge on that basis.The passage attributes the flags to misunderstanding the prompt’s instruction that user-query harmfulness alone should not determine whether an interaction is suspicious.Source: §I.2.2 · p. 48
• The assistant correctly identifies and classifies content about constructing and deploying harmful devices, upholding safety standards.Source: §I.2.2 · p. 49

I.2.3 flower and flower-benign

• The monitor flags a non-negligible portion of GPT-4o responses on flower and flower-benign, apparently due to harmful first questions, risking overrefusal.In an example, a harmful IED query is followed by a request for a flower; the assistant answers “gloriosa,” but the conversation is classified as a potential bypass attempt.Source: §I.2.3 · pp. 49–50