PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia

arXiv:2609.30867v1cs.CL

TL;DR

DID climate-policy estimates depend on identification assumptions whose reported support can be difficult to assess. ARGUS audits that support with a bounded, evidence-grounded pipeline and evaluates it using injected flaws and real-paper assessments. Its reports localize potential weaknesses for expert review, but the system does not adjudicate causal effects.

  • Problem

    Climate-policy DID estimates depend on fragile identification assumptions, and overlapping or phased-in policies, spillovers, and modelled or remotely sensed outcomes can make their evidence difficult to assess.

    Source: §1 · p. 1

  • Method

    ARGUS uses an eleven-dimension assumption–implication–evidence rubric and a bounded pipeline that retrieves and assesses reported evidence, producing risk reports with an explicit unknown state when no relevant passage is retained.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    On real-paper assessments, grounding limits coverage and over-severity limits agreement; a pre-specified rule reduces over-severity in-sample, while weighted agreement remains low.

    Source: §1 · p. 2, §5.6 · p. 9

  • Takeaways & Limitations

    ARGUS surfaces and localizes potential identification risks for expert review without deciding whether a study’s causal effect is true.

    Source: §7 · p. 9, Limitations · p. 10

    ARGUS screens whether reported evidence supports identification assumptions rather than adjudicating the true causal effect, and its outputs do not replace expert review.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Difference-in-differences (DID) studies are widely used to evaluate climate policy, but assessing the evidence supporting their identification assumptions remains challenging. We introduce ARGUS, a structured language-model pipeline that audits reported evidence against an eleven-dimension assumption-implication-evidence rubric and abstains when relevant evidence cannot be retrieved. We evaluate ARGUS using injected flaws, economics papers, and a small pilot with reconciled labels. On the 11-flaw benchmark, ARGUS detects 73% of planted flaws, compared with 18% for a keyword-based pipeline. Across 26 economics papers, ARGUS abstains on about 40% of paper-dimension assessments for lack of retrievable evidence. In a five-paper pilot with labels reconciled by two annotators, it assigns a higher risk level than the labels on 25 of the 33 assessments it completes. A rule fixed before the labels arrived removes most of this in-sample; weighted agreement stays low. ARGUS provides evidence-linked risk reports that localize potential weaknesses for expert review, without adjudicating causal claims. Code and data: https://github.com/yonghongzhang-io/ARGUS

1 Introduction

DID climate-policy evaluations rely on fragile identification assumptions that are difficult to assess from reported evidence. ARGUS audits that evidence across eleven dimensions, localizes potential weaknesses for expert review, and does not adjudicate causal truth.

1 Introduction

  • DID estimates of climate and environmental policies depend on fragile assumptions, including parallel trends, no anticipation, staggered-adoption handling, and limited interference.Climate settings may also involve overlapping or phased-in policies, spillovers across administrative boundaries, and modelled or remotely sensed outcomes.Source: §1 · p. 1
  • ARGUS represents a DID study with an eleven-dimension assumption–implication–evidence rubric and a bounded, retrieval-gated LLM pipeline that produces evidence-grounded risk reports.It asks whether reported evidence supports each identifying assumption, not whether the estimated effect is true.Source: §1 · p. 1
  • The paper contributes a machine-auditable rubric and explicit-abstention pipeline, planted identification errors as local ground truth, and a diagnosis of real-paper coverage and agreement limits.The calibration rule reduces over-severity in-sample.Source: §1 · p. 2
  • ARGUS’s central diagnosis is that grounding limits real-paper coverage, over-severity limits agreement, and a pre-specified rule reduces over-severity in-sample.The evaluation uses a synthetic flaw-injection protocol, a 26-paper economics corpus, and a pilot with reconciled labels.Source: §1 · p. 2

2 Related Work

ARGUS turns established concerns about DID identification into an auditable rubric for checking evidence in individual studies. It differs from causal-reasoning benchmarks and other document-grounded systems by assessing reported support for identification assumptions, not causal truth.

DID identification and its threats

  • ARGUS operationalizes known DID threats as rubric dimensions whose supporting evidence can be checked in a given paper, rather than proposing a new estimator.The cited literature covers staggered timing, heterogeneous effects, inference, and the treatment of pre-trends.Source: §2 · p. 2

LLM causal reasoning and benchmarks

  • Unlike LLM causal-reasoning benchmarks, ARGUS assesses whether a human paper adequately evidences its identification assumptions; causal verdicts are out of scope.The cited benchmarks and analyses report mixed evidence on models’ causal-task performance and reliability.Source: §2 · p. 2

Responsible AI for empirical research

  • ARGUS links each DID dimension to in-paper evidence and produces human-reviewable risk reports, while CausalVerify checks recovered effects by running generated estimation code on synthetic data.ARGUS reuses CausalVerify’s released economics-paper corpus and adds dimension-level identification-risk annotations.Source: §2 · p. 2

Natural language processing for climate evidence

  • ARGUS extends document-grounded climate verification from checking claims to auditing whether reported evidence supports each causal study assumption.Related systems verify claims, detect environmental claims, or provide traceable answers about climate disclosures.Source: §2 · p. 3

Evaluation methodology and bounded agents

  • ARGUS fixes retrieval and assessment order, trading the action-sequence autonomy of ReAct-style agents for inspectability and evidence traceability.Its ablations compare prompting and retrieval choices while holding the model and rubric fixed.Source: §2 · p. 3

3 The ARGUS Framework

ARGUS decomposes DID identification into explicit dimensions and runs a fixed, bounded pipeline to assess reported evidence. Its outputs distinguish retrieval failure from substantive risk judgments and leave the final judgment to a human reviewer.

3.1 Identification rubric

  • The rubric defines eleven identification, design, inference, and reporting dimensions, each with an assumption or design requirement, a testable implication, and supporting evidence a credible paper would report.For carbon-pricing or emissions-trading evaluations, parallel trends calls for an event study or pre-trend test with pre-period estimates near zero; no anticipation turns on announcement and implementation timing.Source: §3.1 · pp. 3–4

3.2 Bounded LLM pipeline

  • A fixed sequence decomposes the paper, extracts dimension-specific evidence, assesses each evidence chain, localizes higher-risk dimensions, and renders a report for human review.In the evaluated configuration, lexical retrieval feeds a relevance gate and, if evidence is retained, an adequacy call; no retained passage yields unknown, while partial matches are still judged.Source: §3.2 · p. 4

3.3 Output

  • ARGUS emits study–dimension–risk triplets with evidence-grounded rationales; unknown means the gate retained no relevant passage, not that the paper received a substantive risk judgment.The human reviewer retains the final judgment, and repeated runs show that fixing stage order does not make model outputs identical.Source: §3.3 · p. 4, §3.2 · p. 4

4 Evaluation by Flaw Injection

Because the true causal effect is unobserved, ARGUS is evaluated against controlled, known flaws rather than recovered effects. The injection design avoids phrase-matching shortcuts by structurally changing the evidence or design described in the paper.

4.1 Local ground truth without the counterfactual

  • Each synthetic fixture begins with reported support on all eleven dimensions, then receives one known flaw targeting one dimension; the rubric defines a detection response of at least medium risk.Evaluation also scores false alarms on the clean version and whether the targeted dimension is flagged most strongly.Source: §4.1 · pp. 4–5
  • The flaw-injection loop audits clean and injected versions, then scores detection, false alarms, and localization against the injected flaw as local ground truth—not the unobserved true effect.A regression test rejects injected papers containing predefined negative-signal phrases for the target dimension.Source: Figure 2 · p. 5, §4.2 · p. 5

4.2 Structural perturbation, not sentinels

  • A naïve injector produced an inflated 1.000 detection rate through phrase leakage, so the revised design uses structural perturbations instead of naming the flaw.Omission flaws remove evidence-bearing sections, figures, or tables; commission flaws describe a design problem without the keyword detector’s target phrases.Source: §4.2 · p. 5

5 Results

Across synthetic flaws, real papers, and a reconciled-label pilot, ARGUS detects evidence-adequacy problems but retrieval limits coverage and its ratings are over-severe; a pre-specified rule improves in-sample agreement.

5 Results

  • Across four evaluation axes, ARGUS detects planted flaws missed by keyword presence, retrieval limits real-paper coverage, and a pre-specified rule reduces over-severity in-sample.The pilot is diagnostic, not a validation of accuracy.Source: §5 · p. 5, §5.5 · p. 9, §5.6 · p. 9

5.1 Adequacy pipelines versus keyword presence; what the gate trades

  • 0.182 to 0.727 detection, with no false alarms, when the two-stage LLM pipeline replaces the keyword pipeline on eleven injected flaws.This is an end-to-end comparison: the pipelines use different evidence paths.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 An expanded benchmark of 33 flaw variants

  • On 33 variants, detection was 0.73–0.76; the assessor caught 0.86–0.91 of commission flaws but only 0.45 of omissions, whose misses abstained to unknown.Detection decisions were identical on 32 of 33 variants across three runs.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • Per-dimension prompting detected 29/33 variants versus 24/33 for a single pass, but pairwise arm differences were nonsignificant and cross-model false-alarm and abstention profiles differed.The prompting comparison also changes the number of calls, so it does not isolate task decomposition.Source: §5.2 · pp. 7–8

5.3 Real papers: retrieval limits audit coverage

  • ∼40% of judgments across 26 papers were unknown; diffuse or figure-based discussions were often unretrieved, while weakly relevant passages were associated with high-risk ratings.With partial lexical matches, the judge may be called on an off-target passage and map missing evidence to high risk.Source: §5.3 · p. 8

5.4 An annotation pilot with reconciled labels

  • Two independent annotators reconciled labels for five papers across 55 cells; 23 cells used an analogue reading because the design was not canonical DID.The reconciled gold contained 7 low, 46 medium, and 2 high-risk labels.Source: §5.4 · pp. 8–9

5.5 ARGUS versus the human gold

  • ARGUS abstained on 22/55 cells and rated 25/33 answered cells more severely than the reconciled gold, with none less severe.The gold had two high-risk labels; annotators located reported evidence on at least one of 21 abstained cells.Source: §5.5 · p. 9

5.6 Calibration: a pre-specified rule, in-sample

  • The coverage-preserving rule raised exact agreement from 0.24 to 0.76 and cut over-severe cells from 25 to 8; weighted κ rose from 0.08 to 0.23, with an interval including zero.The rule was derived on the same five papers, so the improvement is in-sample and untested on unseen papers.Source: §5.6 · p. 9

6 Discussion

ARGUS assesses whether reported evidence supports climate-policy studies’ identification assumptions, treating that assessment as a proxy for identification credibility rather than causal validity. Retrieval limits coverage, so it abstains when no relevant passage is found.

6 Discussion

  • ARGUS treats reported-evidence adequacy as an operational proxy for identification credibility, not as causal validity itself.The paper distinguishes whether assumptions are supported by reported evidence from whether the causal effect is true.Source: §6 · p. 9
  • Climate-policy estimates inform governance, so ARGUS abstains when its gate finds no relevant passage.Source: §6 · p. 9
  • On real papers, retrieval limits coverage, and weakly relevant evidence usually leads the assessor to rate the dimension high in the pilot.The pilot records 20 of 22 weak-retrieval cells as high; the system reports retrieval quality and an explicit unknown distinct from substantive high risk.Source: §6 · p. 9

7 Conclusion

ARGUS audits evidence for DID identification assumptions using flaw injection and a reconciled pilot, but its value in expert review and on climate-policy papers remains untested.

7 Conclusion

  • ARGUS uses flaw injection as local ground truth and a reconciled pilot to evaluate its evidence-grounded audit, but its value in expert review and on climate-policy papers is untested.Source: §7 · p. 9

Limitations

The evaluation is preliminary and bounded: its human comparison is a small pilot, the corpus is not a narrow climate-policy sample, and several design and scope limits remain. ARGUS audits reported support for identification assumptions rather than causal truth.

Limitations

  • The human gold covers 5 papers and 55 cells reconciled by two annotators, making the comparison preliminary rather than a calibrated benchmark.Its labels are 84% medium, which makes exact agreement lenient and κ more informative.Source: Limitations · p. 10
  • The pilot’s cells are nested within five papers, and the calibration rules were scored on the same pilot used to derive them.The lift is in-sample, and its size on unseen papers is untested.Source: Limitations · p. 10
  • The current scope is DID identification, while transfer to other quasi-experimental designs is untested; the real corpus contains no narrow climate-policy evaluations.Source: Limitations · p. 10
  • ARGUS checks support for identification assumptions, not the true causal effect, and its risk reports do not replace expert review.The causal effect is never observed.Source: Limitations · p. 10

A.1 Why climate-policy DID is a demanding audit setting

  • Climate-policy DID can combine staggered adoption, spillovers, overlapping regulations, long heterogeneous lags, and outcomes measured from varied sources.These features motivate making evidence for each design choice explicit; they do not make DID unique to climate policy.Source: §A.1 · p. 12

A.2 Identification auditing and research reliability

  • ARGUS addresses identification credibility within broader research-reliability concerns, but does not audit every component of an empirical paper.It focuses on assumptions supporting causal interpretation.Source: §A.2 · pp. 12–13

A.3 Why the pipeline is bounded

  • The bounded pipeline fixes operation order and confines model decisions to retrieval and adequacy assessment, trading autonomy for inspectability.It fixes the sequence, not the model’s outputs.Source: §A.3 · p. 13

B Flaw Taxonomy Provenance

  • The threat catalogue comes from DID and research-design literature, while the authors wrote the injector’s prose; independent expert-authored flaws remain future work.Synthetic detection therefore measures internal consistency under known threats rather than external generalization.Source: B Flaw Taxonomy Provenance · p. 13

C Ablation Matrix and Cross-Model Panel

The cross-model and ablation panels compare flaw detection across compute budgets and providers, while retrieval details and prompts specify how evidence is selected and assessed. The gate can abstain when it retains no relevant passage; the reported risk is therefore distinct from a verdict on a paper’s causal effect.

C Ablation Matrix and Cross-Model Panel

  • Table 4 compares detection, false alarms, and localization across compute-graded variants; per-dimension prompting detects more variants than a single joint call.Calls per paper are nominal maxima, and the adequacy call is skipped when the gate finds nothing relevant.Source: Table 4 · p. 13, C Ablation Matrix and Cross-Model Panel · p. 13
  • Table 5 reports uniformly high commission detection across providers, while false-alarm and localization profiles vary sharply by model.The panel uses single runs per model except for the gpt-4o three-run mean; single runs support only a qualitative porting claim.Source: C Ablation Matrix and Cross-Model Panel · p. 13, Table 5 · p. 14

D Retrieval Details

  • ARGUS searches the audited paper’s sections and captions lexically, keeps up to four blocks per dimension, and gates their relevance before adequacy assessment.Failed retrieval yields unknown without an adequacy call; weak or good retrieval proceeds with kept blocks and the rubric’s assumption, implication, and expected evidence.Source: D Retrieval Details · p. 14

E Prompts

  • The prompts separate relevance screening from adequacy judgment and ask whether reported evidence supports each identification assumption, not merely whether a topic is mentioned.The single-pass variant cannot return unknown, while the real-paper variant permits it sparingly.Source: E Prompts · pp. 14–15

F Reproducibility Details

The reproducibility record reports run settings, variation across reruns, and implementation costs, while the uncertainty analysis qualifies comparisons on the small benchmark and pilot. The authors characterize calibration as an in-sample direction rather than a demonstrated out-of-sample gain.

F Reproducibility Details

  • Detection replicated on 32 of 33 variants across three end-to-end runs, but localization and the clean-fixture flag varied.The later July-2026 values were not reproduced in June; available records do not establish the cause of the difference.Source: F Reproducibility Details · p. 16
  • One two-call-per-dimension audit costs roughly US$0.25 in gpt-4o usage; a three-run 33-variant benchmark cost roughly US$8 per gpt-4o run.The local Llama run took roughly 67 minutes on a laptop at zero API cost.Source: F Reproducibility Details · p. 16

G Uncertainty Quantification

  • At n=11, the keyword-versus-two-stage gap is marginal in the paired test on the same flaws (7 vs 1 discordant, p=0.07); at n=33, no pairwise difference among four arms reaches significance.Source: G Uncertainty Quantification · p. 16
  • The weak-retrieval rule turns seventeen wrong cells exact and none the other way, but the authors claim only a consistent in-sample direction, not a demonstrated out-of-sample gain.The analyses use cells nested in five papers and a gold set that is 84% medium.Source: G Uncertainty Quantification · p. 16

H The Real-Paper Corpus

The real-paper run audits 26 economics papers from a corpus tagged as DID, but none is a narrow climate-policy evaluation. The supplementary examples show dimension-level reports, while the reported domain comparison is explicitly inconclusive.

H The Real-Paper Corpus

  • The 26-paper corpus includes 19 journal articles and seven NBER working-paper versions, and its DID method-family tags were not verified paper by paper.One entry is a literature review, and a byte-identical duplicate was counted once.Source: H The Real-Paper Corpus · pp. 16–17
  • None of the 26 papers evaluates a carbon price, emissions trading scheme, or emission standard; the corpus is a general-economics stress test of the audit architecture.Its climate-specific content is the rubric instantiation, while the synthetic fixture is a generic environmental-policy study.Source: H The Real-Paper Corpus · p. 17
  • With four environment-and-energy papers, the comparison of unknown and high-risk shares does not establish whether audit behavior differs by domain.The unknown shares are 0.43 for that subset and 0.39 for the remaining papers; high-risk shares are 0.34 and 0.46, respectively.Source: H The Real-Paper Corpus · p. 17

I Qualitative Audit Examples

  • The fixture report flags the injected anticipation flaw as high, leaves nine supported dimensions low, and shows mild over-severity on inference.The inference rationale reflects that the fixture does not state the number of clusters.Source: I Qualitative Audit Examples · p. 17
  • The qualitative example distinguishes an alternative-specification section from the placebo evidence specified by the rubric for that dimension.The corpus also includes a duplicated title listing only once in Table 11.Source: I Qualitative Audit Examples · p. 18, Table 11 · p. 18

J Supplementary Figures and Tables

The supplementary material documents how pilot labels were collected and reconciled, illustrates the synthetic audit report, and tests whether the observed over-severity pattern recurs across leave-one-paper-out folds. Only the weak-retrieval rule is treated as a result; the other calibration rules are post-hoc.

J Supplementary Figures and Tables

  • The fixture report flags the injected anticipation dimension high, keeps nine supported dimensions low, and shows mild over-severity on inference.The table presents an abridged evidence-cited report.Source: Table 12 · p. 18
  • Two annotators independently labeled five papers without AI or ARGUS output, then jointly settled 13 cells that differed in applicability or risk.Of 54 cells both initially rated, 45 agreed exactly; all nine risk disagreements were one severity step apart.Source: J Supplementary Figures and Tables · p. 18
  • ARGUS was more severe than the reconciled labels on 25 of 33 answered cells, and the earlier unreconciled-label results are superseded.The earlier labels were not treated as an annotation; against them ARGUS was more severe on 28 of 33 answered cells.Source: J Supplementary Figures and Tables · p. 19
  • Across all five leave-one-paper-out folds, 79–86% of over-severe high cells had weak retrieval and 94–100% had missing evidence.The folds test whether this error pattern recurs; no rule was refit on four papers and scored on the fifth.Source: J Supplementary Figures and Tables · p. 19
  • Rule 1 demotes or abstains on high-risk cells with weak retrieval; rules 2–4 are post-hoc and are not treated as results.The exploratory rules can demote cells based on estimator names, inference rationales, or retrieval quality alone.Source: J Supplementary Figures and Tables · p. 19

K Shared-Evidence Control

With evidence held fixed, the LLM adequacy judge outperformed the keyword scorer on injected flaws and variants. The evaluations also reveal calibration and evidence-grounding constraints.

K Shared-Evidence Control

  • On 45 papers and eleven dimensions, both policies read frozen keyword chunks or ungated section candidates; no cell could abstain, yielding 990 adequacy calls.The shared-evidence control reproduced the published keyword baseline exactly.Source: K Shared-Evidence Control · pp. 19–20

Pilot

  • In the five-paper pilot, ARGUS abstained on 22 of 55 cells and rated 25 of its 33 answered cells more severely than gold, matching 8 and rating none less severely.The figure reports outcomes against reconciled gold labels.Source: Figure 4 · p. 20

K Shared-Evidence Control

  • Under K, detection rose from 2 to 10 of 11 flaws and from 8 to 31 of 33 variants; under S, it rose from 1 to 11 and 2 to 31.The control compared judgement policies on frozen evidence bundles, with thresholds and decision rule committed before the run.Source: K Shared-Evidence Control · pp. 19–20, Table 15 · p. 20

Limitations

  • The injected-flaw comparison rests on co-designed fixtures, sentinels, and injector, with false-alarm rates based on one clean audit of eleven dimensions.The reference row used a gpt-4o alias, making that comparison indicative; the sample is small.Source: K Shared-Evidence Control · p. 20

Corpus Run

  • Across 26 DID-tagged papers, the keyword baseline’s risk labels were almost entirely low, while the two-stage LLM output included a large unknown share marking retrieval failure.The figure describes per-dimension risk distributions.Source: Figure 5 · p. 21

Calibration Trade-off

  • Demoting weak-retrieval highs to medium retained 33 answers and cut over-severe judgments from 25 to 8; abstaining left 20 fewer answers and weighted κ of 0.29.Most exact-agreement gain reflects gold labels that were 84% medium; evidence grounding remained a bottleneck.Source: Figure 6 · p. 21

Related papers

Loading 2609.30867v1…