PaperlayerView exact revision on arXiv

Source-linked AI summary

Replicating TRACE: A Practitioner's Guide to Its Threshold and Particle Budget

Alex Chadyuk, Alicia Zhang, Roy Kucukates

arXiv:2609.01108v1cs.LG

TL;DR

TRACE asks whether causal graphs can be read from pretrained autoregressive event-sequence models and how its threshold and particle budget behave across regimes. The replication applies particle-based CMI estimation with validation-selected thresholding, reproduces the headline F1 result, and reveals lag-dependent recovery and benchmark limitations. The practical conclusion is to select thresholds by validation, report per-lag recall, control generator decay, and size particles by threshold margin.

  • Problem

    TRACE reads causal structure from pretrained autoregressive event-sequence models, but the behavior of its threshold and particle budget across regimes requires characterization.

  • Method

    The study independently reimplements TRACE and evaluates validation-selected thresholds against exact interventional truth across vocabulary, lag, and particle regimes.

  • Results

    F1 reaches 0.90–0.91 at vocabulary size 1000 and 0.86–0.91 from 100 to 2000 under the validation-sweep protocol.

  • Takeaways & Limitations

    Use validation-selected thresholds, report per-lag recall, test long-range recovery with slowed or flattened decay, and size particles by threshold margin over noise.

  • Takeaways & Limitations

    Per-sequence graph results do not transfer to corpus-level graph estimation, which is a different and harder problem.

Abstract

from arXiv · show

TRACE (Math & Lienhart, arXiv:2602.01135) reads causal graphs over event types out of a pretrained autoregressive sequence model by thresholding a per-position conditional-mutual-information estimate at a fixed tau. We independently replicate its headline synthetic result: with tau selected on a validation split, mean per-sequence F1 against exact interventional truth reaches 0.90-0.91 at vocabulary size 1000 (paper: 0.91) and 0.86-0.91 from 100 to 2000. First, the optimal threshold is pinned to the truth margin, not to any constant: at every size the errors at tau* straddle the delta = 0.05 margin defining ground truth (missed true edges lie just above it, accepted false ones just below), and the blind optimum lands near delta/2 times the estimator's calibration, confirmed out of sample at 5000. Second, at a single global threshold TRACE mostly recovers a direct, adjacent-influence graph: lag-1 true edges are recalled at 0.97-0.99, while true edges at lag 2 or more read orders of magnitude lower---the reading-scale price of randomizing mediating positions, which an exact test of direct causal effect requires when the truth is unknown. A per-lag threshold family recovers a third to a half of lag-2 truth; on lag-uniform data one validated threshold recalls every lag at 0.40-0.87, 8-26 pp below an atomic-intervention control at lags 3-6. Third, the default lag decay of the paper's synthetic benchmark concentrates about 85% of interventional truth at lag 1 and pushes the rest below the estimator's noise floor, so headline F1 there certifies lag-1 recovery only and conflates the benchmark's skew with the algorithm's own limit; a flatter decay separates the two. Fourth, F1 saturates from N = 2 particles at the selected threshold---a property of the threshold's margin over the noise floor, not of the estimator, which converges as N^(-1/2). We distill five practitioner rules.

1 Introduction

This replication reproduces TRACE’s headline synthetic performance while showing that threshold behavior, lag structure, benchmark decay, and particle budgets materially shape how that result should be read.

  • Replication: F1 ≈0.91 at vocabulary size |X| = 1000 reproduces TRACE’s headline synthetic result against interventional ground truth.The replication also covers vocabulary sizes 100, 500, 1000, and 2000.
  • Threshold: The validation optimum is pinned to the truth margin δ rather than any constant threshold.Errors at the optimum straddle δ, and an out-of-sample |X| = 5000 probe supports the margin interpretation.
  • Lag structure: Lag-1 recall reaches 0.97–0.99, whereas lag ≥2 effects read orders of magnitude lower at one global threshold.Per-lag thresholds recover a third to a half of lag-2 truth, while an atomic-intervention control isolates a moderate 8–26 pp residual at lags 3–6.
  • Benchmark limitation: About 85% of default-generator interventional truth lies at lag 1, so headline F1 can certify adjacent-influence recovery alone.A flat-decay variant separates benchmark skew from the staircase construction’s recall limit.
  • Particle budget: F1 saturates from N = 2 particles at the selected threshold, reflecting threshold margin over the noise floor rather than estimator saturation.The study frames particle count as a regime-dependent budget choice.

2 Related work

The study positions its contribution as a measured replication and regime analysis of TRACE, not a novel general lesson about tuning or a cross-method comparison.

  • Related work: Data-driven selection of sparsity-controlling parameters is canonical, and benchmark sensitivity to tuning and setup is established across causal-discovery research.The paper cites StARS and several causal-discovery studies for these broader points.
  • Scope: The paper’s specific contribution is measuring TRACE’s threshold and particle-budget behavior across regimes while replicating its headline number.An executed cross-method comparison remains outside scope.

3 TRACE and its evaluation protocol

TRACE estimates direct causal effects from a pretrained autoregressive model using particle do-interventions, then thresholds per-position CMI and projects it to a per-sequence event-type graph.

  • Estimator: TRACE uses a decoder-only autoregressive model pretrained on event sequences over vocabulary X.For candidate cause position p, it evaluates downstream positions within context window c.
  • Estimator: N particle do-interventions redraw the candidate and mediating positions, with shared mediating draws across contrast arms.This is the staircase construction used to compare predictive distributions.
  • Estimand: The staircase estimates a controlled direct effect: non-parent positions have zero effect, while direct parents can produce positive CMI under witnessing contexts.The construction carries no mediated influence by design.
  • Graph construction: Thresholding the per-position CMI matrix at τ and projecting onto event types yields one sample summary causal graph per sequence.A type edge is called when any supporting position pair crosses τ.
  • Ground truth and metric: Interventional truth is defined per sequence from a known lagged generator using uniform replacements and recomputed generator conditionals.The evaluation does not aggregate evidence into a corpus-level graph.
  • Threshold protocol: The paper treats τ as a swept, setup-dependent quantity rather than a universally fixed value.At |X| = 1000, reported candidates include 3 × 10−5, ≈1.4 × 10−5, and 10−4.

4 Reimplementation and setup

The replication rebuilds TRACE’s full pipeline and evaluates it in controlled synthetic regimes with validation-selected thresholds, exact per-sequence truth, and repeated seeds.

  • Reimplementation: The study reimplemented the generator, tokenizer, pretraining, staircase CMI, thresholding, graph projection, and evaluation from scratch.The staircase and Bernoulli-KL algebra were transcribed from the reference implementation.
  • Synthetic setup: The primary worlds use |X| = 1000, L = 64, N = 128 particles, lag horizon h = 6, and truth margin δ = 0.05.The redundancy-matched regime targets the paper’s benchmark redundancy while varying other generator settings.
  • Evaluation: One threshold per seed is selected by validation F1 and then applied to the test split.Thresholds are searched on fixed logarithmic grids, with test-side grids used only for sensitivity curves.
  • Reproducibility: Cross-seed gaps were ≤0.03 throughout the experiments.The protocol uses two seeds throughout.

5 Replication at the paper’s regime

At the paper’s regime, blind validation selection independently reproduces the published F1, but the successful threshold value is setup-dependent rather than transferable.

  • F1 = 0.9008/0.9051 matches the paper’s Table 2 ≈0.91 against exact interventional truth at |X| = 1000.The replication uses the paper’s regime and matched redundancy.
  • At the paper’s threshold value, the runs score F1 ≈0.46 across context widths and seeds.Thus the validation protocol transfers the operating point, not the numeric constant.
  • τ = 3 × 10−2 is selected blindly on validation data, while the test-side argmax coincides across the reported replicas.The table caption identifies the matched-redundancy setting and selected threshold.
  • The position-level exact truth remains 50–53 pairs per sequence across vocabulary sizes in the sweep.The projected truth-pair count rises at |X| = 100 because of type collisions.

6 How the threshold moves across regimes

Across vocabulary regimes, validation-selected thresholds preserve performance and track the truth margin rather than a fixed scaling law. The margin interpretation is supported by error-pair straddling and an out-of-sample test at |X| = 5000.

  • Test F1 at the selected τ* is 0.86–0.91 across |X| ∈ {100, 500, 2000}, with validation-to-test drift within ±0.003.A single fixed constant instead scores 0.25–0.48 when transplanted across regimes.
  • Missed true pairs have KL 0.06–0.08 above δ = 0.05, while accepted false pairs have KL 0.02–0.03 below it.From |X| = 500 upward, both error groups are lag-1 pairs and the estimator reads lag-1 pairs near their KL.
  • The optimum approaches a floor of order δ/2 rather than following any power law, with segment slopes −0.70, −0.41, and −0.28.The reported curvature is +0.27 ± 0.01 over three seeds.
  • At |X| = 5000, the measured τ~=0.0230 exceeds the power-law prediction 0.015 and preserves the margin-straddle signature.Missed-truth KL median is 0.086 and accepted-false-positive KL median is 0.027.
  • Across sizes from 500 upward, the blind optimum stays within ×1.7 of (δ/2) · 10^b1 and within ×1.5 from 1000 upward.Here b1 denotes the estimator’s measured level on margin-band pairs.

7 What TRACE recovers: a lag-graded recall limit of the staircase construction at a single global threshold

At one global threshold, TRACE primarily recovers adjacent lag-1 influence because staircase mediator randomization suppresses longer-lag CMI readings. Flat-decay controls separate this algorithmic limit from the benchmark’s lag-1-skewed truth composition.

  • Global-threshold recall: 0.97–0.99 lag-1 recall contrasts with 0.15, 0.006, 0.004, and 0.000 recall at lag ≥2 as vocabulary size increases.Overall recall tracks the lag-1 share of truth, indicating a level gap rather than sampling noise.
  • Global-threshold recall: Lag-1 pairs remain at generator-KL scale, whereas lag ≥2 pairs fall one to over three decades lower and can reach about five decades lower.A single threshold fitted on the lag-1 scale cannot recover a class whose readings lie far below it.
  • Threshold dependence: Per-lag thresholding recovers 0.35/0.47 of lag-2 truth, but lags ≥3 remain unrecovered because readings fall below their null floor and validation cells are thin.The lag-2 threshold is approximately 4–8 × 10^-4, two orders of magnitude below the pooled optimum.
  • Benchmark interpretation: The default benchmark places about 85% of interventional-truth mass at lag 1, so headline F1 can certify adjacent-influence recovery while conflating benchmark skew with estimator limits.Per-lag recall is needed to distinguish truth composition from estimator quality.
  • Mechanism: The staircase randomizes the entire tail after the cause, putting the effect token in an off-distribution mediator context and collapsing its factual probability.Atomic intervention instead preserves observed mediators, providing the direct-effect comparison used to assess the deficit.
  • Flat-decay control: On flat-decay data, one validated threshold recalls 0.87/0.76/0.69/0.63/0.59/0.40 across lags 1–6, while the atomic control is 8–26 percentage points higher at lags 3–6.The graded decline persists when lag composition is flattened, isolating a staircase contribution to the deficit.

8 What validating on the default benchmark certifies: a practitioner warning

Validation on TRACE’s default generator certifies adjacent-influence recovery because its lag decay concentrates truth at lag 1 and suppresses deeper-lag effects. A flat-decay control separates this benchmark skew from the staircase construction’s residual limitation.

  • Benchmark skew: ≈85% of interventional truth sits at lag 1, while deep-lag truth falls below the estimator’s noise floor under the default decay.The mechanism uses exponential lag decay, so deeper-lag recall declines as decay strengthens.
  • What validation certifies: F1 of 0.90–0.91 can therefore be achieved by lag-1 recovery while recalling essentially no lag ≥2 truth.Default-generator validation does not establish long-range recovery.
  • Recommended control: A flat-decay generator is a one-parameter control that isolates the staircase’s own lag-graded deficit from the benchmark’s decay.The control makes the two sources of poor deep-lag recall distinguishable.

9 The particle budget

Particle requirements depend on the selected threshold’s distance above the estimator’s noise floor, not on estimator convergence alone. At the selected threshold, classification saturates from N = 2 even though the underlying CMI estimates continue converging with N.

  • Estimator versus observable: CMI differences fall as N^-0.46, but threshold-separated true and non-true pairs rarely change classification between N = 2 and 512.The observable saturates because the threshold lies far above the noise floor, not because the estimator stops improving.
  • Noise-floor regime: At thresholds inside the noise floor, additional particles increase false positives because per-particle KL means are right-skewed and non-negative.Precision at 3 × 10^-5 falls from 0.45 at N = 2 to 0.27 at N = 512.
  • Rule: Choose N from the selected threshold’s margin over the noise floor, because density-estimation error dominates Monte Carlo error once the threshold is sufficiently separated.The measured N-independent error floor is 0.7–0.9 nats RMS, while Monte Carlo error declines from 0.40 nats at N = 2 to 0.05 at N = 256.

10 Guidance for practitioners

TRACE should be tuned and interpreted around the selected threshold’s margin, while practitioners should distinguish per-sequence graphs from corpus-level graphs and report lag-specific recovery. Long-range claims require validation under slower or flatter lag decay.

  • Threshold selection: Select τ on held-out data and expect it near the truth margin rather than as a fixed constant across regimes.The validation sweep is inference-only; its optimum is roughly δ/2 to δ after calibration.
  • Particle budget: Size the particle budget from the selected threshold’s margin over the noise floor, not from a fixed N.N = 2 matched N = 512 at 70× less compute in the reported runs.
  • Observable: Per-sequence headline numbers do not transfer to a single corpus-level graph, which is a different and harder estimation problem.TRACE’s headline metric scores one summary graph per sequence.
  • Lag interpretation: Under lag-decaying data, a global threshold mainly recovers lag-1 influence, so report per-lag recall and test long-range recovery with flatter decay.Lag-1 recall is 0.97–0.99; a per-lag threshold family recovers a third to a half of lag-2 truth.

11 Conclusion

The replication reproduces TRACE’s headline F1, but the broader analysis shows that threshold selection, lag structure, and particle count determine what that result actually certifies. The default benchmark supports adjacent-influence recovery, while long-range recovery requires separate controls.

  • Replication: F1 = 0.90–0.91 against exact interventional truth matches Table 2, with 0.86–0.91 across vocabulary sizes 100 to 2000.These results use the validation-sweep protocol.
  • Interpretation: At a single global threshold, TRACE recovers a direct adjacent-influence graph, while per-lag thresholds recover the nearest deeper-lag signal.The particle budget is governed by threshold margin over the noise floor.
  • Scope: The default benchmark’s lag-1 skew means its validation certifies lag-1 recovery only and conflates benchmark structure with algorithmic limitation.A flat-decay control is needed to separate these effects.
  • Experimental practice: The study’s experiments were preregistered through committed plans and validation-frozen thresholds, with runs totaling under 40 GPU-hours.Experiments ran on single NVIDIA A10G instances.
Loading 2609.01108v1…