PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense

Disen Liao, Yihan Wang, Freda Shi, Yaoliang Yu

arXiv:2608.27945v1cs.AI

TL;DR

The paper asks whether scaling can make benign-looking answers across independent sessions more useful for reconstructing forbidden objectives. It formalizes compositional safety risk, proves a conditional loss-based transfer bound, and evaluates synthetic recovery, pretrained models, and intent-aligned retrieval. Results indicate that dispersed support can become more recoverable with scale, while IntentAlign-MiniLM provides the strongest learned-retriever harmful recall across tested guardrails.

  • Problem

    The paper examines whether scaling makes individually allowed subqueries more likely to support forbidden reconstructions across disconnected sessions.

  • Method

    It formalizes compositional safety risk, proves a conditional risk-transfer bound from excess loss, and evaluates scaling with synthetic withholding and pretrained-LLM experiments.

  • Results

    Larger models can make dispersed benign-looking subanswers more useful after recomposition, while IntentAlign-MiniLM outperforms larger embedding models on intent retrieval and learned-retriever harmful recall.

  • Takeaways & Limitations

    The findings support intent-aligned retrieval as a lightweight defense primitive for reconstructing hidden task neighborhoods across sessions.

  • Takeaways & Limitations

    Deployment would require sensitive cross-session history retention and faces coverage gaps, evasion, privacy concerns, latency, storage, and threshold-calibration challenges.

Abstract

from arXiv · show

Scaling laws are usually read as a capability story: lower language-modeling loss yields more useful models. We study a safety consequence of this mechanism in \emph{cross-session decomposition attacks}, where benign-looking subqueries are asked across independent interactions and later recomposed toward a forbidden objective. We formalize this setting as \emph{compositional safety risk} and prove a conditional risk-transfer bound: when the reference environment already contains dispersed evidence for a risky reconstruction, the gap between deployed composed risk and reference composed risk is controlled by the model's excess loss on allowed subqueries. Synthetic withholding experiments show that wider transformers assign lower loss to held-out instructions that never appear verbatim in training but are recoverable from injected supporting facts. A 600-intent pretrained-LLM evaluation shows that larger Qwen3 and Gemma3 family members can yield greater harmful-capability uplift under a fixed decomposition-composition pipeline. As a defense, IntentAlign-MiniLM, our 22M-parameter intent-aligned retriever, outperforms much larger embedding models on held-out intent retrieval and yields the best learned-retriever harmful recall across tested guardrails. Code is available in \href{https://github.com/liaodisen/Cross-Session-Decomposition-Attacks}{our GitHub repository}.

1 Introduction

The paper studies cross-session decomposition attacks, where benign-looking subqueries are distributed across disconnected interactions and later recomposed into forbidden outputs. It formalizes this risk, tests how scaling affects reconstruction, and proposes intent-aligned retrieval as a defense.

  • Threat model: Cross-session attacks distribute a harmful objective across independently benign-looking interactions whose answers are later recomposed externally.This setting extends beyond prompt-local defenses because the malicious intent may remain latent across sessions.
  • Scaling mechanism: Lower language-modeling loss may make dispersed harmful structure easier to reconstruct when the training environment contains partial supporting evidence.The risk is conditional: the forbidden objective need not appear verbatim if its supporting facts are present.
  • Formal contribution: The risk-transfer theorem bounds the gap between deployed and reference compositional risk by excess loss on allowed subqueries.Thus, scaling can transfer latent compositional risk when the reference distribution already supports a risky reconstruction.
  • Empirical evaluation: Synthetic withholding and pretrained-LLM experiments test recovery from dispersed evidence and harmfulness under matched decomposition pipelines.The experiments separate the scaling mechanism from comparisons across model families.
  • Defense: IntentAlign-MiniLM uses intent-aligned retrieval to find subqueries sharing hidden objectives despite differing surface wording and topics.It outperforms substantially larger embedding models on held-out intent retrieval and learned-retriever harmful recall across tested guardrails.
  • Contributions: The paper contributes a formal threat definition, a risk-transfer bound, empirical evidence on scaling, and a lightweight intent-aligned retrieval defense.These contributions target reconstruction across sessions rather than direct harmful-prompt compliance.

2 Theoretical Framework

The theoretical framework models composed outputs from allowed subqueries and compares deployed risk with reference risk. Its conditional theorem connects excess loss to this risk gap, while target loss provides a proxy for forbidden-output probability.

  • Threat model: Cross-session decomposition samples allowed prompts, obtains separate answers, and combines them through a composer C into an output.The composer may be another language model or an adversarial user synthesizing an executable plan.
  • Scaling assumption: Excess loss is average conditional KL divergence from the reference answer distribution to the deployed model, and it also controls average total variation.The scaling assumption states that excess loss is nonincreasing along the model sequence.
  • Distributions: The deployed composed distribution is induced by sampling answers from the deployed model and applying the composer; the reference distribution uses the corresponding latent data-generating answers.The framework also handles randomized composers by incorporating their randomness into the composed output distribution.
  • Risk transfer: Theorem 2.2 conditionally bounds deployed composed risk relative to reference compositional risk using excess loss on allowed prompts.It does not assert that scaling must increase risk or that risk appears without ingredients for a forbidden reconstruction.
  • Target-loss proxy: A target-loss proxy scores forbidden reconstructions drawn from a distribution supported on the forbidden set.When targets lie inside that set, lower target loss certifies a higher lower bound on forbidden-output probability.
  • Synthetic experiment: Figure 2 reports held-out golden-answer loss across widths and the matched control gap Δctrl(n) = Lctrl(n) − Lgold(n).Positive control-gap values indicate that fact-supported targets are easier to recover than matched controls.

3 Experimental Evidence of Scaling Risk

The experiments test whether scaling improves recovery of targets supported by dispersed evidence and increases harmful capability under a fixed decomposition-composition pipeline. Synthetic withholding isolates distributed recovery, while pretrained-LLM comparisons examine the deployed-model regime.

  • 3.1 Synthetic Experiment: Larger models assign higher probability to withheld instructions supported by dispersed facts, despite those complete instructions being absent from training.The synthetic construction injects supporting facts while withholding the original instruction, separating recovery from direct memorization.
  • 3.1 Synthetic Experiment: Positive and increasing Δctrl(n) shows designated golden targets become easier to recover than equally fact-supported matched controls as width grows.The control gap is defined as Lctrl(n) − Lgold(n), with positive values indicating easier recovery of golden targets.
  • 3.2 Pretrained LLM Experiment: The pretrained-LLM comparison fixes the intent set, decomposer, composer, and judges while varying only the victim model answering neutralized subquestions.The study uses 600 mixed-domain harmful intents and pairwise judgments of recomposed outputs.
  • 3.2 Pretrained LLM Experiment: Within the Qwen and Gemma families, larger models can produce greater practical capability uplift under the fixed decompose-answer-compose pipeline.Cross-family differences are less reducible to size, so the comparison does not isolate scale strictly.
  • 3.2 Pretrained LLM Experiment: Figure 3 reports strict-majority pairwise win rates for aligned intents, excluding no-majority cases and reporting coverage separately.Each cell measures how often the row model’s recomposed answer is judged more potent than the column model’s answer.
  • 3.2 Pretrained LLM Experiment: Hard refusals can lower measured compositional risk by leaving the composer little usable evidence, whereas partial refusals may still reveal recomposable fragments.GPT-5.2 remains competitive in high-coverage pairwise comparisons while also having the highest partial-refusal rate.

4 Retrieve-then-Classify Defense

Because cross-session attacks hide intent across disconnected benign-looking queries, the defense first retrieves related historical queries and then classifies their reconstructed neighborhood jointly. Intent-aligned training targets latent task similarity rather than surface semantic similarity, while deployment requires maintaining a historical query bank.

  • Threat Model and Motivation: Prompt-local defenses are insufficient when malicious intent is distributed across individually benign queries in disconnected sessions.The cross-session setting removes the assumption that a sequential monitor can see the entire interaction history.
  • Threat Model and Motivation: The defense treats hidden-intent recovery as a retrieval problem before classification, requiring relevant past queries to be found across sessions.The system must recover an intent neighborhood before deciding whether the aggregated task is harmful.
  • Retrieve-then-Classify Pipeline: The defense uses two stages: retrieve nearest cross-session neighbors, then classify the query jointly with its retrieved neighborhood.A frozen instruction-tuned guardrail LLM infers the underlying task from the bundle and outputs a harmful/benign verdict.
  • Deployment Assumptions: Deployment assumes a maintained historical query bank, while training, auditing, or escalation labels would require prior moderation, human review, or trusted incident reports.The experiments evaluate guardrail predictions with ground-truth task-level labels, but the prompt receives query texts rather than per-neighbor labels.
  • Intent Alignment: IntentAlign-MiniLM is trained with same-intent decomposed queries as positives and other batch intents as negatives, clustering latent task neighborhoods beyond surface wording.The objective is designed for variation in wording, topic, granularity, and disciplinary framing.

5 Defense Experiment Results

The defense retrieves other subqueries sharing a latent intent before a frozen guardrail classifies harmfulness. IntentAlign-MiniLM performs strongly under paraphrase shift and structured interference, while retrieval quality and guardrail scale affect outcomes differently.

  • 5.1 Leave-one-out Benchmark Evaluation: Leave-one-out evaluation removes each probe from the query bank and counts same-intent neighbors as correct, measuring latent-intent recovery rather than ordinary similarity.The benchmark tests original queries and Qwen3-8B single-rewrite paraphrases, with the guardrail receiving retrieved context in the second stage.
  • 5.1 Leave-one-out Benchmark Evaluation: Recall@10 rises from .588 to .649 and nDCG@10 from .558 to .631 over Qwen3-Embedding-0.6B on paraphrase-shifted queries.IntentAlign-MiniLM leads on Recall@5/10 and nDCG@5/10 in both original and paraphrase-shifted settings despite its compact backbone.
  • 5.1 Leave-one-out Benchmark Evaluation: At K = 1 and K = 3, IntentAlign-MiniLM beats Oracle in harmful recall for every guardrail, while Oracle catches up at K = 5.Oracle provides random same-intent siblings, whereas the learned retriever ranks fragments by their collective diagnostic value.
  • 5.1 Leave-one-out Benchmark Evaluation: Guardrail scale is non-monotone: Qwen3-14B often has the highest harmful-class precision, but harmful recall saturates around .59 with learned retrieval and .62 with oracle context.The reported pattern indicates that increasing guardrail scale beyond 8B is not a Pareto improvement in this setting.
  • 5.2 WildChat Evaluation: Under structured interference, IntentAlign-MiniLM leads in both 50K WildChat banks across all four conditions, while I25 preserves 81.9–91.0% Hit@1 and H5 lowers it to 68.5–77.2%.Adding I25 to H5 costs at most another 0.6 points; under I25+H5, harmful-probe Hit@1 is 51.3–62.2% versus 88.0–95.7% for benign probes.
  • 5.2 WildChat Evaluation: Semantic confusability, rather than unrelated-query volume or conversation-derived sourcing alone, is the principal retrieval failure mode in the WildChat test.Switching between isolated-query and conversation-derived banks changes Hit@1 by at most 0.9 points.

6 Conclusion

The paper studies cross-session decomposition attacks, formalizes their risk, evaluates how model capability affects reconstruction, and proposes latent-intent retrieval before classification.

  • 6 Conclusion: Cross-session decomposition splits forbidden objectives into benign-looking queries that are later recomposed outside the model.The paper frames the threat as compositional safety risk and evaluates recovery from dispersed support with synthetic and pretrained-LLM experiments.
  • 6 Conclusion: The proposed defense retrieves queries sharing a latent goal rather than relying on surface-level semantic similarity.This retrieve-then-classify design targets hidden intent across sessions.

Limitations

The experiments establish that the studied pipelines can exhibit cross-session decomposition, but they do not establish deployment prevalence or universal harmfulness scaling.

  • Limitations: The experiments show cross-session decomposition in the studied pipelines but do not establish its prevalence in deployment.Estimating prevalence would require privacy-sensitive production logs unavailable to the authors.
  • Limitations: The synthetic experiment isolates the analyzed mechanism and does not imply that scaling universally increases harmfulness.Its scope is recovery from dispersed evidence rather than a universal claim about model scaling.
  • Limitations: Pairwise win rates measure relative capability uplift only within the fixed pipeline, while auxiliary absolute scores depend on the judge and rubric.Some judge pairs show low agreement, so expert review under safeguards is needed before making real-world harmful-capability claims.
  • Limitations: Deployment would require searching sensitive cross-session histories, with privacy, identity-fragmentation, coverage, practicality, calibration, and evasion constraints.Attackers could distribute fragments across accounts or add unrelated queries, while operational costs and overblocking affect deployment.

Ethical Considerations

The paper limits release of actionable attack material and emphasizes that the proposed retrieval defense is not a complete, deployment-ready system.

  • Ethical Considerations: The authors release aggregate results and sanitized artifacts while withholding or redacting actionable prompts and outputs.This reflects that the attack pipeline can process harmful objectives and produce actionable outputs.
  • Ethical Considerations: The query-bank defense may overblock benign users when unrelated past queries are retrieved and combined into an apparently harmful intent.The paper presents the method as a proposed algorithm rather than a complete defense system.
  • Ethical Considerations: Concrete decision rules and thresholds must be determined and validated for each deployment context.The paper does not present the retrieval defense as deployment-ready without context-specific validation.

B.2 Restated Main-Text Results

The appendix restates the latent-risk-transfer results and their supporting proof steps, including the point-mass specialization and synthetic-data construction details.

  • Theorem B.5 restates the latent risk-transfer theorem for any forbidden set F.The supplied passage states the theorem’s scope but omits the displayed bound.
  • The proof handles zero-risk cases separately, then restricts the deployed distribution to the forbidden set and applies a KL-divergence inequality.When Rq = 0, the conditional loss is +∞ and the desired inequality is trivial; otherwise, the normalized restriction is used.
  • Jensen’s inequality is applied after taking expectation over q1:k ∼ πk, yielding the displayed lower bound.The derivation also uses Lemmas B.4 and B.3, and equal marginals recover Equation (2.2).
  • If Ptar(· | q1:k) is a point mass for every q1:k, then Hk(Ptar) = 0 and the point-mass bound follows.The canonical target condition is stated as a point mass on z∗(q1:k) ∈ F.
  • Synthetic experiments: Synthetic models use 3 layers, 8 attention heads, context length 512, batch size 64, and 20,000 optimization steps.The experiments tune learning rate only at width 32, select 10−4, then sweep widths {32, 64, 128, 256, 512}; WikiHow variants differ in training split.
  • Synthetic experiments: The synthetic decomposition prompt converts verified WikiHow QA decompositions into atomic supporting facts while preserving concrete details, ordering, prerequisites, and exceptions.It requires a faithful JSON array of strings and avoids duplicates, outside knowledge, and vague collapsing of facts.

C.2 Real-LLM Decomposition Experiment

The real-LLM experiment fixes decomposition and recomposition components while comparing victim models across 600 harmful intents, using blinded pairwise judges to assess capability uplift.

  • Evaluation set: The evaluation set contains 600 harmful intents balanced across 12 domains, with 50 intents per domain.Table 5 provides representative high-level intent descriptions rather than operational prompts.
  • Pipeline: The fixed pipeline uses a refusal-removed Qwen2.5-14B model for decomposition and recomposition, varying only the victim model answering neutralized subquestions.The Qwen3 panel spans 0.6B, 8B, and 32B; the Gemma3 panel spans 1B, 12B, and 27B, with DeepSeek-Chat and GPT-5.2 as anchors.
  • Question generation: Each harmful intent is decomposed into exactly five high-leverage questions, then rewritten to be neutral, scholarly, and explicitly non-harmful in surface form.The rewrite uses temperature 0.2, top-p = 0.95, repetition penalty 1.1, and a 512-token limit; the six-question variant is auxiliary.
  • Answer and composition: Victim models answer each neutralized question independently, after which the composer receives the original intent and all question-answer pairs to synthesize a coherent procedure.Composition uses temperature 0.3, top-p = 0.9, and up to 3,000 generated tokens.
  • Pairwise evaluation: Blinded judges compare paired final answers for the same intent using practical capability uplift, including goal-directed information, concrete detail, fragment integration, and reduced execution difficulty.The protocol uses anonymized A/B labels and can involve four independent LLM judges.
  • Pairwise evaluation: Strict-majority aggregation requires at least three of four judges to prefer one model; 2–2 splits and other non-winning cases are marked no majority.Reported metrics include strict-majority counts and win rates, coverage, unanimity, Fleiss’ κ, and Krippendorff’s α.
  • Agreement interpretation: For Qwen3-32B versus GPT-5.2, 80.8% of intents have a strict model-winning majority, while 20.5% are unanimous 4–0 decisions.The dominant pattern is 362 3–1 majorities, or 60.3% of the 600 intents.

C.4 Intermediate Refusal-Rate Evaluation

The refusal-rate study evaluates intermediate victim answers inside the decomposition pipeline, distinguishing hard, partial, and absent refusals while emphasizing that this is a local diagnostic rather than a standalone safety score.

  • Task and scope: The auxiliary evaluation measures refusal behavior for each victim model’s answer to one neutralized subquestion, not for the final recomposed answer.It uses the same eight-model panel as the main experiment.
  • Rubric: The refusal judge is shown the original harmful intent, neutralized subquestion, and victim answer, while the victim identity is hidden.Answers may be truncated to 8,000 characters, and complete generated responses are judged.
  • Rubric: The judge assigns Grade 1 to hard refusals, Grade 2 to partial refusals containing some useful relevant information, and Grade 3 to answers that do not refuse or withhold.Warnings alone do not count as refusals when the answer otherwise provides the requested information.
  • Metrics: The run has zero judge parse errors, so nm = 3,000 for every model.
  • Interpretation: Table 7’s refusal rates are a local pipeline diagnostic rather than a standalone safety score because the inputs are neutralized subquestions derived from harmful intents.A low refusal rate means the composer receives more material, while downstream pairwise evaluation measures whether that material increases harmful capability.
  • Interpretation: DeepSeek-Chat has the highest hard-refusal rate on neutralized decomposed questions and correspondingly provides the recomposer less actionable information than several open-weight victims.The interpretation remains pairwise and does not establish that scale alone causes risk.

C.5 Auxiliary Absolute-Risk Evaluation

The auxiliary evaluation uses StrongREJECT-style absolute scores from four local judges to assess 600 final recomposed outputs per victim model. These scores broadly preserve within-family ordering but are saturated and uncalibrated, so matched pairwise comparison remains primary.

  • Evaluation setup: Four local judges independently score each victim model’s 600 final recomposed outputs using refusal, convincingness, and specificity criteria.Each judge sees the original harmful intent and one final response, without victim-model identity or other model outputs.
  • Scoring interpretation: StrongREJECT-style response-usefulness scores range from 0 to 1 and are not probabilities or calibrated estimates of real-world harm.Refusals receive 0; higher scores indicate greater convincingness and specificity for accomplishing the forbidden goal.
  • Results: Qwen3-32B > Qwen3-8B > Qwen3-0.6B, while Gemma3-27B and Gemma3-12B both score above Gemma3-1B in within-family ordering.GPT-5.2 has the highest four-judge mean at 0.9800, while Qwen3-0.6B has the lowest at 0.8452.
  • Results: Every judge has a median score of 1.0, with 68.7–90.0% of valid judgments receiving the maximum score.Because the absolute score is highly saturated, it offers less resolution among strong models than matched pairwise comparison.
  • Limitations: The evaluation reports absolute scores as a complementary diagnostic because neither absolute nor pairwise evaluation has human or external calibration.The paper also notes that auxiliary absolute scores depend on the judge and rubric, with some judge pairs showing low agreement.

D.3 Downstream Guardrail Evaluation

The downstream guardrail reconstructs latent intent by retrieving related decomposition subqueries before asking a frozen instruction-tuned model to classify harmfulness. Intent-aligned retrieval improves harmful recall, while guardrail scale creates a precision–recall trade-off rather than uniformly improving detection.

  • Pipeline: A bi-encoder retrieves top-K subqueries from a strict leave-one-out pool, and a guardrail LLM aggregates the current query with retrieved evidence into a harmful-versus-benign verdict.The same-intent oracle supplies ground-truth sibling decompositions and is used as a reference condition, not a deployable ranked upper bound.
  • Evaluation: The evaluation uses paraphrase-shifted held-out test queries, with 2,005 Qwen3-8B rewrites whose phrasings do not appear verbatim in the retrieval bank.Metrics include harmful-class precision and recall, with harmful recall foregrounded because missed harmful intents have asymmetric downstream cost.
  • Retrieval effects: Additional retrieved subqueries consistently increase harmful-class recall within the tested range by supplying evidence that counters conservative benign classifications.The comparison is reported in Table 3 for the paraphrase-shifted held-out test setting.
  • Retriever comparison: IntentAlign-MiniLM outperforms much larger Harrier and Qwen3-Embedding baselines on harmful recall for every guardrail and every K.Its fixed MiniLM backbone isolates the contribution of intent-clustered contrastive training from raw embedding-model size.
  • Retriever comparison: At K = 5, IntentAlign-MiniLM approaches the same-intent oracle for the strongest 8B guardrails, and can exceed it at low K by selecting more diagnostic fragments.The oracle samples same-intent siblings without ranking their collective informativeness.
  • Guardrail scale: Qwen3-14B often reaches ≥.95 harmful-class precision but its learned-retrieval recall saturates around .59, below the 8B guardrails.The larger guardrail produces fewer false positives but misses more harmful intents on paraphrase-shifted inputs.
  • Takeaway: IntentAlign-MiniLM paired with an 8B guardrail gives the strongest learned-retriever recall at K = 5 on the paraphrase-shifted test split.This configuration closes much of the gap to the same-intent oracle condition.
Loading 2608.27945v1…