PaperlayerView exact revision on arXiv

Source-linked AI summary

Harmful Partial Interventions and Benign Leakage in Concept-based Models

Gregory Verghese, Christopher J. Soelistyo, Mateo Espinosa Zarlenga, Alessandro Barp, Christopher R. S. Banerji

arXiv:2610.21750v1cs.LGcs.AI

TL;DR

Whether sufficiency and localisation make leakage safe under partial corrections remains an open question, even though users rarely intervene on every concept. The paper studies this problem theoretically and across synthetic and real-world tasks, introducing a monotonic objective alongside endpoint training. The combined objectives substantially reduce partial intervention harm while largely preserving concept and task performance.

  • Problem

    Whether sufficiency and localisation prevent intervention harm under partial corrections remains unestablished, although users rarely verify every concept before intervening.

    Source: §1 · p. 2

  • Method

    The paper formalises intervention harm and introduces Lmono to penalise increases in task loss along partial intervention trajectories, complementing endpoint objective Lint.

    Source: §1 · p. 2, §3.3 · p. 5, §4 · p. 6

  • Results

    Combining Lmono and Lint substantially reduces the prevalence and magnitude of partial intervention harm while largely preserving concept and task performance.

    Source: §6 · pp. 10–11

  • Takeaways & Limitations

    Whether leakage is benign should be judged across partial intervention paths as well as at the fully intervened endpoint.

    Source: §6 · p. 11

    Lmono samples intervention trajectories randomly, which may not match the policies used when interventions are applied.

    Source: §6 · p. 10

Abstract

from arXiv · show

Artificial intelligence systems deployed in high-risk settings increasingly require models that support meaningful human oversight. Concept-based models (CMs) offer a promising framework by enabling users to inspect and intervene on intermediate, human-understandable concepts. However, concept interventions can counterintuitively worsen downstream predictions, a failure mode we formalise as intervention harm. We distinguish harm after all concepts have been intervened on from harm under partial interventions, the more likely practical use case. Modern CMs often achieve high accuracy by encoding, or leaking, additional task-relevant information in their intermediate concept representations. Recent work argues that such leakage can be benign with respect to intervention harm when concept representations satisfy two conditions: sufficiency and localisation. We prove that these criteria are not sufficient to guarantee that concept interventions improve downstream performance. Across controlled and real-world settings, we find that concept supervision incompleteness is associated with increased intervention harm. Importantly, optimising the model when all concepts are intervened on, a previously proposed leakage mitigation objective, reduces endpoint harm but does not reliably prevent harmful partial corrections. Hence, we introduce a complementary monotonic intervention objective that directly penalises increases in task loss along partial intervention trajectories. Combining endpoint and monotonic intervention training substantially reduces the prevalence and magnitude of partial intervention harm while preserving concept and task predictive performance.

1 Introduction

The paper frames partial intervention harm as an important oversight risk because users rarely correct every concept. It tests whether existing leakage criteria and training objectives address that risk, and introduces a complementary objective for partial interventions.

1 Introduction

  • Partial interventions are the practical concern because users generally cannot verify every concept and instead correct only a small, relevant subset.Endpoint intervention harm concerns intervening on all concepts; partial intervention harm concerns correcting a subset while others remain uncorrected.Source: §1 · p. 2
  • The paper develops a theoretical and empirical framework and introduces Lmono, which penalises increases in task loss along partial intervention paths.Lmono complements the endpoint objective Lint.Source: §1 · p. 2
  • Combining endpoint and monotonic intervention training substantially reduces partial intervention harm while preserving concept and task predictive performance.The paper also reports that the endpoint objective alone does not reliably prevent harmful partial corrections.Source: §1 · p. 2
  • Sufficiency and localisation do not guarantee that correcting a concept improves task performance, and concept-set incompleteness is associated with increased intervention harm.The paper distinguishes endpoint harm from partial harm and finds that minimising endpoint intervention loss does not reliably prevent harmful partial corrections, especially with incomplete concepts.Source: §1 · p. 2

2 Related Work

Related work presents concept-based models as models whose intermediate concepts support inspection and intervention, while noting that residual pathways can improve accuracy when supervised concepts are incomplete. It also reviews the proposal that leakage may be benign under sufficiency and localisation.

Concept-based Models

  • Concept-based models are commonly framed as Concept Bottleneck Models, with an encoder producing concept representations that a task predictor uses for downstream predictions.The reviewed formulation allows concept representations to be scalar or higher-dimensional and requires concept-annotated training examples.Source: §2 · pp. 2–3

Concept Interventions

  • Concept interventions let experts correct concept representations, but prior work reports that some interventions can decrease model performance.Related work has explored propagating interventions, reapplying earlier interventions, and deferring concepts to experts.Source: §2 · p. 3

Information Leakage

  • Leakage occurs when concept representations encode extra information about other concepts or the task, and prior work proposes that it can be benign when representations are sufficient and localised.The reviewed proposal encourages these properties by minimising task loss after all concepts are intervened on, using Lint.Source: §2 · p. 3

3 Understanding Intervention Harm

This section examines intervention harm in concept-incomplete models with residual pathways. It formalises interventions and shows that sufficiency and localisation do not guarantee that corrections improve task performance.

3.1 Problem Setup

  • The setup assumes known binary concepts relevant to the task may be incomplete, leaving unknown task-relevant concepts unobserved during training.An optimal predictor is described as predicting both known and unknown concepts, whereas a real-world model observes only input, known concepts, and task labels.Source: §3.1 · pp. 3–4
  • The study uses Hybrid CBMs with aligned concept representations and an unaligned residual pathway to represent information missing from the supervised concepts.The residual pathway is included in the task predictor’s input, allowing the model to operate with incomplete concept sets.Source: §3.1 · p. 4

3.2 Benign Leakage, Sufficiency, and Localisation

  • Sufficiency requires residual representations to retain task-relevant information missing from known concepts, while localisation requires known-concept information to remain in its corresponding representation.The paper defines discrepancies for both conditions and treats them as exact when their respective discrepancy is zero.Source: §3.2 · pp. 4–5

3.3 Intervention Harm

  • Intervention harm is the increase in expected task risk after correcting a subset of concepts; subsets define partial harm, while correcting all concepts defines endpoint harm.For residual models, the residual representation is held fixed during intervention.Source: §3.3 · p. 5

3.4 Sufficiency and Localisation Do Not Prevent Intervention Harm

  • With nonzero concept-classifier error, intervention harm can be arbitrarily large even when localisation holds exactly and sufficiency holds arbitrarily closely.The theorem’s construction shows that correcting concepts while holding residual representations fixed can relocate prediction error rather than remove it, and the mechanism extends to partial intervention.Source: §3.4 · p. 5

4 Minimising Partial Intervention Harm

Endpoint intervention loss targets task risk after all supervised concepts are corrected, but it does not constrain intermediate partial-correction states. The paper introduces a complementary monotonic loss that penalises harmful transitions between intervention states and estimates it by sampling edges during training.

4 Minimising Partial Intervention Harm

  • Sufficiency and localisation do not guarantee that correcting a concept improves task performance, because endpoint loss does not directly constrain partially corrected states.Endpoint intervention harm refers to harm after all supervised concepts are intervened on; partial intervention harm can occur at intermediate states.Source: §4 · p. 6
  • Monotonicity requires that correcting one additional concept does not increase task risk, ensuring risk is non-increasing along correction sequences.The condition is expressed for each set of already-corrected concepts and each additional concept to correct.Source: §4 · p. 6
  • Lmono penalises transitions where correcting an additional concept worsens task performance, and is zero when performance is maintained or improved.The loss is defined over intervention edges from a corrected-concept set to that set plus one concept.Source: §4 · p. 6
  • Stopping gradients through supervised concept predictions prevents reducing the monotonicity penalty by sacrificing concept fidelity.Gradients remain through the residual representation and task head so they can adapt to ground-truth concept corrections.Source: §4 · p. 6
  • Training estimates Lmono by Monte Carlo sampling intervention edges, with intervention depths sampled uniformly from the first through final correction.The number of edges grows exponentially with the concept set size, motivating sampling during training.Source: §4 · p. 6
  • Lint and Lmono have complementary roles: Lint targets the fully corrected endpoint, while Lmono penalises harmful transitions between intervention states.Their contributions are controlled separately by λint and λmono.Source: §4 · p. 6, Figure 1 · p. 7

5 Experiments

Across controlled and CUB experiments, greater concept incompleteness was associated with more intervention harm, and endpoint training alone did not reliably prevent harmful partial corrections. Adding monotonic intervention training reduced partial harm, complementing endpoint training while largely preserving predictive performance.

5.1 Understanding Intervention Harm, Leakage, and Incompleteness

  • In the controlled linear–Gaussian study, higher task relevance of unobserved concepts was associated with greater endpoint harm and harmful-path prevalence, but not maximum partial harm.Adjusted associations were β = 0.368 for endpoint harm and β = 0.239 for harmful-path prevalence; the maximum-partial-harm association was β = −0.007 (P = 0.47).Source: §5.1 · p. 7
  • In the controlled study, Lint primarily reduced endpoint harm while Lmono primarily reduced maximum partial harm; combining them reduced general intervention harm.Adding Lmono to Lint reduced maximum partial harm by up to 99.7%.Source: §5.1 · pp. 7–8

5.2 Intervention Harm in Real-World Tasks

  • On CUB, greater pre-model incompleteness was associated with greater endpoint harm, maximum partial harm, and harmful-path prevalence.The associations were β = 0.133 for endpoint harm, β = 0.078 for maximum partial harm, and β = 0.194 for harmful-path prevalence.Source: §5.2 · p. 9
  • On CUB, Lint reduced mean endpoint harm from 0.0198 to −0.0019, but mean maximum partial harm remained 0.0054 and 91.2% of sampled paths had a harmful intermediate state.A harmless fully corrected state could therefore coexist with harmful partial trajectories.Source: §5.2 · p. 9
  • Adding Lmono to Lint reduced mean maximum partial harm to 0.0004 (92.6% below Lint alone) and harmful-path prevalence from 91.2% to 51.4% on CUB.Matched reductions were 0.0050 in maximum partial harm and 39.9 percentage points in harmful-path prevalence.Source: Figure 2 · p. 8, §5.2 · p. 9
  • Intervention-aware training had little effect on task accuracy, while CUB concept MSE improved with Lmono but remained above baseline.CUB top-1 accuracy was 77.94%, 78.21%, and 78.11% for baseline, Lint-only, and combined models; concept MSE was 0.1461 with combined training versus 0.1410 at baseline.Source: §5.1 · p. 8, §5.2 · p. 9

6 Discussion and Conclusion

The paper connects intervention harm to out-of-distribution intermediate states and argues that endpoint safety does not ensure safe partial corrections. It presents Lmono as complementary to Lint, while noting limitations in how intervention paths are sampled.

6 Discussion and Conclusion

  • Lint and Lmono address complementary harms: Lint reduces harm after total correction, while Lmono reduces the prevalence and magnitude of harmful partial paths.The paper reports these gains while largely preserving concept and task performance.Source: §6 · pp. 10–11
  • Partial corrections can create out-of-distribution joint concept and residual representations, helping explain why harm grows with incompleteness and why Lint struggles with partial harm.Lmono adapts the label predictor to handle potential out-of-distribution intermediate states without updating concept representations or requiring an expensive sampling model.Source: §6 · p. 10
  • The authors hypothesise that intervention harm reflects brittle reliance on potentially spurious correlations between concept and residual representations.They hypothesise that Lmono mitigates this by exposing the predictor to diverse combinations of correct and erroneous concept predictions.Source: §6 · p. 10
  • Lmono samples trajectories randomly, which may mismatch deployment policies; equal-probability sampling may also miss meaningful regions of the intervention lattice.Some sampled partial trajectories may be implausible intermediate states that cause harm even when full interventions are harmless.Source: §6 · p. 10

AI use statement

  • The authors report using generative AI to assist with theory, methodology, interpretation, proofs, code, and manuscript writing, with author review of AI-assisted work.They state that AI tools did not generate synthetic datasets or provide critical ingredients for mathematical proofs.Source: AI use statement · p. 11

Ethics Statement

  • The experiments use synthetic data and the CUB benchmark, involve no human participants, personal data, or sensitive attributes, and collect or release no new datasets.The authors state that characterising intervention harm contributes to responsible development and evaluation of human-interpretable AI systems.Source: Ethics Statement · p. 11

Reproducibility Statement

  • The paper provides detailed experimental methods, specifies random seeds and condition-matching procedures where applicable, and supplies anonymised code to reproduce reported experiments, analyses, and figures.Source: Reproducibility Statement · p. 11

A Proof of Theorem 1

The proof constructs a finite process in which intervention harm can be arbitrarily large despite exact localisation and arbitrarily accurate sufficiency. Its construction uses a known concept and an omitted task-relevant factor, with concept correction holding the residual fixed.

A Proof of Theorem 1

  • For every ε > 0 and M > 0, the construction can satisfy sufficiency within ε and exact localisation while ground-truth concept correction strictly increases expected squared loss by more than M.The theorem states that these criteria place no finite upper bound on harm from correction.Source: A Proof of Theorem 1 · pp. 14, 18
  • The construction makes the learned concept misaligned with the ground-truth concept with probability p, while the sufficiency discrepancy is h(p) and localisation holds exactly.The proof uses an omitted task-relevant factor U alongside the known concept C.Source: A Proof of Theorem 1 · pp. 14–15
  • The proof constructs its finite example from independent fair-coin variables C and U and a residual representation, then derives an optimal task predictor under square loss.The stated theorem concerns a finite data-generating process and learned concept representations.Source: A Proof of Theorem 1 · pp. 14–16

B.1.1 Data-Generating Process

  • In the synthetic experiments, each observation has five supervised concepts and additional unobserved concepts, with covariance conditions and task-direction weights varied across worlds.Each world contains 8,000 observations, split into development and held-out testing sets.Source: §B.1.1 · p. 18

B.1.2 Hybrid CBM and Interventions

  • The hybrid CBM uses a concept encoder, residual encoder, and task head; interventions replace selected predicted concepts with ground truth while holding the residual fixed.Intervention-state risk is evaluated on the held-out test set.Source: §B.1.2 · pp. 18–19

B.1.3 Training Objective

  • Training combines concept, task, endpoint-intervention, and monotonic losses; the endpoint loss evaluates task loss after all five concepts are corrected.Monotonic-loss edges sample intervention depth and a random concept ordering; with stop-gradient enabled, this loss cannot directly update the concept encoder.Source: §B.1.3 · p. 19

B.1.4 Experimental Design

The experimental design varies training and data-generation conditions across synthetic models, then evaluates intervention states and correction orderings. Adjusted analyses use clustered standard errors by synthetic world.

B.1.4 Experimental Design

  • The factorial design evaluates 10,500 models across incompleteness, endpoint and monotonic loss weights, covariance conditions, and five seeds.Both stop-gradient settings are evaluated when the monotonic loss is used.Source: §B.1.4 · p. 19
  • A separate 300-model experiment varies the number of sampled monotonic edges for two values of k′, all five ω values, and five seeds.It fixes endpoint and monotonic loss weights, covariance condition, and stop-gradient setting.Source: §B.1.4 · p. 19

B.1.5 Evaluation and Statistical Analysis

  • Evaluation covers all 32 intervention states and all 120 correction orderings, with maximum partial harm defined over nonempty proper intervention subsets.Source: §B.1.5 · p. 20
  • Adjusted analyses standardise outcomes and continuous predictors, transform regularisation weights as log10(1+λ), and cluster standard errors by synthetic world.The stop-gradient contrast is estimated only among models with λmono > 0.Source: §B.1.5 · p. 20

B.1.6 Robustness and ablation analyses for synthetic experiments

Synthetic analyses show that endpoint and partial intervention harms respond differently to intervention objectives, while the CUB experiment varies concept incompleteness using fixed-size concept sets and evaluates its associations with model outcomes.

B.1.6 Robustness and ablation analyses for synthetic experiments

  • Increasing λint was primarily associated with reduced endpoint harm, while increasing λmono was primarily associated with reduced maximum partial harm, with little accuracy change and robustness across covariance structures.Across moderate regularisation weights, task accuracy differed by at most 0.043 percentage points from the unregularised model; stopping gradients mitigated increased concept prediction error, and the partial-harm reduction largely plateaued at mmono = 5.Source: §B.1.6 · p. 20, Figure 3 · p. 21, Figure 4 · p. 22

B.2.1 Representations, Concepts, and Data Partition

  • The CUB experiment used fixed CLIP ViT-B/16 embeddings, a class-stratified train-validation-test split, and excluded test images from concept-set selection and training.CUB contains 11,788 images across 200 bird species with 312 binary attributes; the split used 7,488 training, 1,926 validation, and 2,374 held-out test images.Source: §B.2.1 · p. 20

B.2.2 Pre-model Concept Incompleteness

  • Pre-model incompleteness was measured with a normalized probe-based classifier loss ratio, and 80 fixed 50-attribute concept sets were selected across ten quantile strata.Higher incompleteness indicates that the selected concepts leave more task-relevant information unexplained; the selected sets spanned Ωpre_C = 0.597–0.863 and were shared across training regimes.Source: §B.2.2 · pp. 20–22

B.2.3 Hybrid CBM and Matched Training

The CUB hybrid CBM combines supervised concept representations with a residual pathway, and compares baseline, endpoint-intervention, and endpoint-plus-monotonic training under matched concept sets.

B.2.3 Hybrid CBM and Matched Training

  • The hybrid CBM used a linear concept encoder, a 32-dimensional linear residual encoder, and a linear 200-class task head.The task head predicts from both the concept representation and residual representation; concept supervision used binary cross-entropy and observational task training used multiclass cross-entropy.Source: §B.2.3 · p. 22
  • Training compared baseline, endpoint-only, and endpoint-plus-monotonic regimes across 80 concept sets, with matched initial parameters for each set.The regimes used (λint, λmono) values of (0, 0), (1, 0), and (1, 100), respectively.Source: §B.2.3 · pp. 22–23
  • Endpoint loss was computed after replacing all predicted concepts with ground truth, while monotonic training sampled five edges and stopped its gradients through predicted concepts.The monotonic penalty retained gradients through the residual encoder and task head; Figure 5 reports that reducing maximum partial harm largely plateaued at mmono ≥ 5.Source: §B.2.3 · p. 23, Figure 5 · p. 23

B.2.4 Intervention Evaluation

Intervention evaluation measured harm along sampled concept-correction paths, separating the fully corrected endpoint from intermediate states.

B.2.4 Intervention Evaluation

  • For each concept set, all three matched models used the same 512 random concept-correction orderings, replacing selected predicted concepts with image-level ground-truth values while holding the residual representation fixed.Source: §B.2.4 · p. 24
  • Harm was test risk after correction minus baseline test risk, summarized at the endpoint, at the worst sampled intermediate state, and by the fraction of paths with harm above 10^-8.The reported CUB maximum is over sampled paths, not all 2^50 intervention states.Source: §B.2.4 · p. 24
  • Evaluation also recorded observational top-1 accuracy, fully intervened accuracy, and concept mean-squared error.Source: §B.2.4 · p. 24

B.2.5 Statistical Analysis

The CUB analysis estimated adjusted associations between pre-model incompleteness, training regime, and model outcomes using a matched concept-set design.

B.2.5 Statistical Analysis

  • An additive model estimated average associations with incompleteness and adjusted differences between intervention-training regimes and baseline, with standard errors clustered by concept-set identifier.The analysis covered 240 runs across 80 matched concept sets; the baseline regime was the reference.Source: §B.2.5 · pp. 24–25
  • Figure 7 presents adjusted standardized associations of pre-model incompleteness and training regime with top-1 accuracy and concept mean-squared error.The figure displays regression coefficients and 95% cluster-robust confidence intervals, with outcomes and incompleteness standardized.Source: Figure 7 · p. 25, §B.2.5 · p. 25

Related papers

Loading 2610.21750v1…