Source-linked AI summary
From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention
Minming Ye, Min Liu, Haoang Chi, Ciheng Wu, Naiyang Guan
TL;DR
Correct-answer filtering cannot distinguish faithful rationales from spurious post-hoc explanations, a limitation known as observational equivalence. ECI2D abstracts rationales into executable programs and intervenes on logically similar samples; the paper reports gains exceeding 7 points over outcome-oriented reinforced ICL while retaining fewer demonstrations.
Problem
Observation-based filtering treats answer correctness as evidence of rationale quality, although faithful and spurious post-hoc rationales can be observationally indistinguishable.
Source: §1 · p. 1, §3.1 · p. 3Method
ECI2D abstracts natural-language rationales into executable programs and intervenes on logically similar samples to test whether they preserve correctness.
Source: §1 · p. 2, §3.2 · p. 3, §4.1 · p. 5Results
Takeaways & Limitations
The framework shifts demonstration selection from passive observation to active intervention.
Source: §7 · p. 8Transfer invariance is necessary but not sufficient for faithfulness, making ECI2D a conservative local filter rather than an identifier of the causal mechanism.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from \textit{observational equivalence}, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply \textit{transfer invariance}, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as \textbf{E}xecutable \textbf{C}ausal \textbf{I}ntervention for \textbf{D}emonstrations \textbf{D}istillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions ($do$-operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.
1 Introduction
The paper argues that correctness-based filtering cannot distinguish faithful rationales from post-hoc explanations, and proposes executable interventions across similar samples as an alternative. Experiments report improved performance and out-of-distribution generalization against manual and self-generated reinforced ICL baselines.
1 Introduction
- Figure 1 probes post-hoc rationalization by truncating a chain of thought or injecting a logical error before the answer.Early Answering tests whether the answer is predetermined; Adding Mistakes introduces an error in the reasoning.Source: Figure 1 · p. 1
- Correct answers do not establish rationale quality: faithful causal and spurious post-hoc rationales can be observationally indistinguishable.The paper argues that observation-based filtering can therefore retain unfaithful rationales that misguide ICL.Source: §1 · pp. 1–2, Figure 2 · p. 2
- ECI2D treats transfer invariance across logically similar samples as a practical signal of rationale faithfulness.It abstracts language rationales into executable programs and uses interventions to retain rationales that continue to produce correct results.Source: §1 · p. 2
- The paper provides a formal causal analysis of self-generated reinforced ICL, identifying the prevalence of observational equivalence and showing that faithful causal rationales imply transfer invariance.Source: §1 · p. 2
- ECI2D uses two abstraction forms and is reported to generalize from deterministic to non-deterministic tasks.The authors describe extensive experiments across math, logical, and commonsense reasoning benchmarks.Source: §1 · p. 2
- ECI2D significantly improves performance and out-of-distribution generalization over manual and self-generated reinforced ICL baselines.The experiments also report gains exceeding 7 points as the test distribution shifts.Source: §1 · p. 2
2 Related Works
Prior work automates self-generated demonstrations and often selects them by answer correctness, while related causal-abstraction and program-aided approaches provide components that ECI2D combines for intervention during distillation.
2.1 Self-Generated In-Context Learning
- Self-generated ICL pipelines bootstrap demonstrations, while reinforced ICL retains rationale traces that produce correct final answers [Agarwal et al., 2024].Earlier work includes test-conditioned synthesis and fully automated pipelines that bootstrap from zero-shot inference.Source: §2.1 · p. 2
- Correct-answer filtering cannot guarantee that a retained rationale reflects the process producing the answer, motivating interventional transfer-invariance checks.The passage contrasts this approach with prior diagnosis by observational analysis or input perturbations.Source: §2.1 · p. 2
2.2 Causal Abstraction & Externally Verified Reasoning
- ECI2D applies causal abstraction to rationales, mapping their logical steps into executable programs for intervention during demonstration distillation.Related work discusses causal abstraction, program-aided prompting, and symbolic methods that use external engines at inference time.Source: §2.2 · pp. 2–3
3 Problem Analysis
The analysis explains why answer-correctness filtering cannot identify whether a rationale caused the output. It motivates transfer invariance across logically similar samples as a practical proxy for faithfulness and grounds ECI2D’s intervention pipeline in that criterion.
3.1 The Limitation of Self-Generated Reinforced ICL
- Post-hoc rationales may be generated after a prediction based on pattern matching or memorized shortcuts, so they need not influence the output.A spurious post-hoc rationale is retrospectively constructed to fit the observed input and output rather than reflect the causal factors producing the prediction.Source: §3.1 · p. 3
- The analysis defines observational equivalence as distinct causal structures generating identical joint distributions over observed variables.The observed model treats rationale as a direct mediator, which cannot distinguish faithful from spurious rationales.Source: §3.1 · p. 3
- Faithful and spurious rationale factorizations produce identical observational likelihoods, leaving the true causal structure unidentifiable from observation.Existing methods filter candidate demonstrations using the indicator that the generated answer matches the ground truth.Source: §3.1 · p. 3, §3 · p. 3
3.2 Transfer Invariance as a Proxy for Faithfulness
- Faithfulness is proposed to imply transfer invariance: a rationale should remain valid when applied to logically similar samples.Similarity is defined by matching logical features comprising premises, subjects, and result type.Source: §3.2 · p. 3
- ECI2D abstracts a rationale into an executable program, adapts its logical skeleton to similar inputs, and verifies the resulting outputs against their ground truth.The figure describes retaining transfer-invariant rationales in the causal pool and discarding those that fail to generalize.Source: Figure 3 · p. 4
- A rationale that fails to transfer may be a post-hoc label, while transfer checks can reject shortcuts that do not generalize across similar samples.The paper notes that this criterion circumvents most latent confounders, such as shortcuts in LLM memory.Source: §3.2 · p. 4
4 The ECI2D framework
ECI2D distills demonstrations by abstracting rationales into programs and testing them through interventions on logically similar samples. The validated demonstrations then steer reasoning at inference, with hybrid programs extending the approach to steps requiring world knowledge.
4 The ECI2D framework
- ECI2D has two phases: distill rationales by verifying transfer invariance, then use validated demonstrations to steer inference.Its Abstract-Intervene-Verify pipeline maps natural-language rationales to Python programs, or hybrid programs for semantic reasoning.Source: §4 · p. 4
4.1 Invariant Reasoning Distillation with Python Program
- The generator produces a rationale, and the abstractor converts it into a deterministic Python program for precise intervention.Generation and abstraction are initially filtered by outcome correctness.Source: §4.1 · p. 4
- The adapter changes only input-specific parameters, preserving the program’s logical skeleton when applying the rationale to a similar sample.Similar samples are retrieved through semantic filtering and then checked for structural equivalence by an LLM.Source: §4.1 · p. 5
- A rationale is retained when its intervention-based correctness across logically similar samples meets threshold δ.The intervention outcomes estimate correctness across the neighbor set, and retained demonstrations store the program alongside the natural-language rationale.Source: §4.1 · p. 5
4.2 Invariant Reasoning Distillation with Hybrid Program
- Hybrid programs represent world-knowledge steps as LLM-resolved function declarations alongside a Python logical skeleton and its parameters.At each call site, an evaluator resolves a declaration from the declaration and its arguments; the intervention and invariance comparison remain the same as in the Python-program approach.Source: §4.2 · p. 5
4.3 Causal Steering Inference
- At inference, the model retrieves top-k distilled demonstrations and is prompted to use each program’s algorithmic precision while expressing its reasoning through the paired natural-language rationale.Each retrieved demonstration contains the query, program, rationale, and answer.Source: §4.3 · p. 6
5 Main Experiments of Python Program
The experiments compare ECI2D with manual and self-generated ICL baselines on math and logical reasoning benchmarks. ECI2D improves on self-generated reinforced ICL across benchmarks, with especially large gains under distribution shift, while using fewer demonstrations.
5.1 Experimental Setup
- The evaluation covers GSM8K, SVAMP, GSM-Symbolic, DROP, and three BBH logical subsets, comparing ECI2D with five baselines using Meta-Llama-3-8B-Instruct and Qwen2.5-7B-Instruct.The baselines include direct answering, zero-shot CoT, manual Few-shot CoT, self-generated reinforced ICL, and that method with program-augmented demonstrations.Source: §5.1 · p. 6
5.2 Main Results
- Over 7 points on GSM-Symbolic-P2 with Qwen2.5-7B: ECI2D’s largest advantage over self-generated reinforced ICL occurs in this OOD scenario.ECI2D also outperforms that baseline on every benchmark; manual Few-shot CoT leads on parts of BBH, where the domain gap is noted.Source: §5.2 · pp. 6–7
- 39% to 57% fewer final demonstrations than the outcome-oriented baseline; matching its pool size by subsampling degrades performance on every tested benchmark.The authors interpret this as evidence that compactness is not the source of ECI2D’s advantage.Source: §5.2 · p. 7
- The hybrid of programs and language rationales outperforms either component alone on all ten model–benchmark pairs, with the same ordering throughout.The authors attribute the components’ complementary roles to the program’s algorithmic scaffold and the rationale’s linguistic mapping; the program-only format mismatches natural-language answering.Source: §5.2 · p. 7, Table 2 · p. 7
- An observationally filtered version with code added does not improve consistently, while ECI2D outperforms it on all nine benchmarks under both models.The authors attribute the gains to interventional verification rather than code presence alone.Source: §5.2 · p. 7
- Increasing the number of logically similar samples does not monotonically improve performance; for Qwen2.5, GSM-Symbolic-P2 drops 8.96 points when N rises from 3 to 5.The authors describe a trade-off between verification rigor and demonstration diversity: more distant samples can cause valid causal skeletons to fail strict execution checks.Source: §5.2 · p. 7
- Under deliberately degraded abstraction and adaptation on GSM8K with Qwen2.5-7B, retention falls from 57.3% to 24.6% while downstream accuracy remains above 87 points.The authors interpret upstream errors as reducing coverage rather than pool quality because failed checks discard candidates.Source: §5.2 · pp. 7–8, Table 4 · p. 8
6 Additional Experiments of Hybrid Program
The hybrid program abstraction is evaluated on CommonsenseQA and two BBH semantic subsets, using a pool distilled on CommonsenseQA. It improves over self-generated reinforced ICL across all three benchmarks, including cross-domain BBH transfer, while deterministic-program-only abstraction can be harmful in these settings.
6.1 Experimental Setup
- CommonsenseQA and the BBH semantic subsets Sports Understanding and Ruin Names are used to evaluate hybrid abstraction with Qwen2.5-7B-Instruct; the pool is distilled from CommonsenseQA with N = 3.The setup uses the same baselines as Section 5.Source: §6.1 · p. 8
6.2 Results
- 2.30 points on CommonsenseQA, 4.40 on Sports Understanding, and 3.60 on Ruin Names: the hybrid abstraction’s accuracy gains over self-generated reinforced ICL span all three benchmarks.It achieves the best accuracy on CommonsenseQA; manual Few-shot CoT retains the lead on BBH, attributed to the domain gap from distilling only on CommonsenseQA.Source: §6.2 · p. 8, Table 5 · p. 8
- The + Code baseline falls below self-generated reinforced ICL on two of three benchmarks and below direct answering on CommonsenseQA, whereas the hybrid abstraction improves over both on all three.The + Code baseline requires a deterministic Python program and retains candidates by observational correctness alone.Source: §6.2 · p. 8, Table 5 · p. 8
- The hybrid abstraction retains 58.8% of the training set, compared with 79.1% for the observational baseline.Source: §6.2 · p. 8
- On Ruin Names, transfer falls below zero-shot inference; among interventions scoring zero, 95% return a wrong option and the rest fail to execute.The authors state that the criterion therefore measures whether reasoning transfers, not merely whether a program runs, and extends beyond domains where deterministic programs can be written.Source: §6.2 · p. 8
7 Conclusion
ECI2D uses executable interventions on logically similar samples to select demonstrations that transfer, outperforming outcome-oriented filtering while retaining fewer demonstrations. Its evidence extends across models and logical benchmarks, while its limitations include offline cost, neighborhood availability, and incomplete identification of faithful rationales.
7 Conclusion
- Over 7 points, ECI2D outperforms outcome-oriented reinforced ICL while retaining fewer demonstrations, with gains scaling under increasing distribution shift.The paper presents intervention-based invariance as a criterion for distilling faithful rationales in self-generated ICL.Source: §7 · p. 8
Limitations
- ECI2D adds offline cost for rationale sampling, abstraction and adaptation, neighbor retrieval, and repeated sandbox execution, while leaving inference unaffected.The paper states that the added cost is amortized offline.Source: Limitations · p. 9
- The criterion requires retrieved neighborhoods of logically similar samples and degrades when data are scarce or problem structures are diverse.On GSM8K, the candidate pool shrinks from 3,913 to 1,387 as neighborhood size rises from 3 to 8.Source: Limitations · p. 9
- Transfer invariance is necessary but not sufficient for faithfulness, so ECI2D is a conservative local filter rather than an identifier of the causal mechanism.Uniform factors within the verified neighborhood can escape detection; silent repair by the abstractor may admit unfaithful rationales and was not measured.Source: Limitations · p. 9
A.1 More models in Math Reasoning
- ECI2D surpasses self-generated reinforced ICL on all evaluated math benchmarks across four models, despite retaining fewer demonstrations.The additional models include DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B.Source: §A.1 · p. 12, Table 6 · p. 12, Table 7 · p. 12
A.2 More datasets in Logical Reasoning
- ECI2D performs comparably to manual prompts and surpasses self-generated reinforced ICL on Boolean Expressions and Penguins in a Table under both evaluated models.These results extend transfer-invariant distillation to logical benchmarks beyond those in the main experiments.Source: §A.2 · p. 12
A.3 Stability across Distillation Seeds
- Across three distillation seeds, variation stays below 2.5 points on every benchmark and retention rates vary by less than 0.4 points.Variation is below 0.7 points on six of nine benchmarks; some small margins are comparable to this variation, whereas reported DROP and distribution-shift gains exceed it substantially.Source: §A.3 · p. 12
A.4 Size-Matched Observational Pool
The size-matched control finds that reducing the observational pool worsens its results, while ECI2D’s margin widens at equal pool size. The appendix also contrasts ECI2D with program-aided prompting and describes the benchmark scope used in the paper.
A.4 Size-Matched Observational Pool
- At equal pool size, the margin over the observational criterion widens on every benchmark; reducing that baseline’s pool degrades results by 2.73 points on GSM8K, 1.68 on GSM-Symbolic-P2, and 1.45 on DROP.The control matches pool sizes while holding retrieval, shot count, prompt format, and decoding identical.Source: §A.4 · p. 13, Table 10 · p. 13
A.5 Comparison with Program-Aided Prompting
- PoT scores 58.62 on GSM-Symbolic-P1 and 32.28 on P2, respectively 7.40 and 5.44 points below ECI2D, despite improving over human-written prompts on GSM8K.PoT requires interpreter access at inference time, unlike the standard few-shot prompting used by the other compared methods.Source: §A.5 · pp. 13–14, Table 11 · p. 14
B Benchmark Details
- The experiments use GSM8K, filtered DROP, and CommonsenseQA as in-distribution datasets, and SVAMP, GSM-Symbolic, and BBH as out-of-distribution benchmarks.Distilled demonstrations come from training data; test examples are excluded from the intervention neighborhoods and causal pool.Source: B Benchmark Details · p. 14
B.1 Math Reasoning Benchmarks
- GSM-Symbolic uses refreshed numbers and names, with Plus-1 and Plus-2 adding one or two clauses to increase difficulty.SVAMP tests robustness to changes in word order and problem structure.Source: §B.1 · p. 14
B.2 Logical Reasoning Benchmarks
- The DROP experiments retain questions requiring sorting, counting, or temporal reasoning, while selected BBH subsets test multi-step logical deduction.The BBH examples include Date Understanding, Object Counting, Word Sorting, Boolean Expressions, and Penguins in a Table.Source: §B.2 · pp. 14–15
B.3 Semantic Reasoning Benchmarks
CommonsenseQA supplies the in-distribution semantic benchmark, and its distilled pool is also evaluated on two out-of-distribution BBH subsets. The accompanying implementation overview states that code and data-generation scripts will be released upon acceptance.
B.3 Semantic Reasoning Benchmarks
- CommonsenseQA is the in-distribution semantic benchmark, using its full training split for distillation and its validation split for evaluation because test labels are withheld.Its questions require world knowledge rather than computation, so no deterministic program decides the answer.Source: §B.3 · p. 15
- The CommonsenseQA-distilled pool is evaluated on BBH Sports Understanding and Ruin Names, each with 250 examples, to measure cross-domain transfer.Neither subset is expressible as a deterministic program.Source: §B.3 · p. 15
C Baselines and Implementation Details
- The paper states that code and data-generation scripts will be released upon acceptance.Source: C Baselines and Implementation Details · p. 15
C.1 Baseline Methods
The experiments compare ECI2D with prompting baselines, including outcome-filtered self-generated demonstrations, and use four open-source models. Implementation details describe neighborhood retrieval, verification thresholds, decoding settings, and hybrid abstraction.
C.1 Baseline Methods
- Self-generated reinforced ICL retains rationales by source-answer correctness, while ECI2D’s comparison is designed to isolate the contribution of transfer intervention over correctness-based distillation.The paper also includes a code-format control that uses ECI2D’s generation prompt but omits intervention and verification.Source: §C.1 · pp. 15–16
C.2 Model Selection
- The four evaluated open-source models span standard instruction-tuned and reasoning-specialized architectures.The reasoning-specialized models are included to test whether intervention-based filtering remains effective with extended rationale generation.Source: §C.2 · p. 16
C.3 Implementation Details
- Neighborhood retrieval first selects a semantic candidate pool of size 5N, then uses an LLM to choose N logically analogous samples, with N set to 3, 5, or 8.The second stage addresses that semantic similarity alone does not guarantee logical compatibility.Source: §C.3 · p. 17
- Verification uses a threshold of 1.0 for math and CommonsenseQA, and 0.9 F1 for DROP, to accommodate minor answer-format variation.The specified thresholds determine whether a rationale’s logic holds across similar samples.Source: §C.3 · p. 17
- Distillation samples at temperature 0.7, while inference uses greedy decoding at 0.0.The paper uses the higher distillation temperature to explore diverse rationales and greedy decoding for reproducibility.Source: §C.3 · p. 17
- For CommonsenseQA, the hybrid abstraction tests answer candidates against rationale-named properties, visiting each candidate in turn.Its execution calls the evaluator per candidate and declaration, with cached answers reused across neighbors.Source: §C.3 · pp. 17–18
D More Experiment of Hybrid Program
This section examines hybrid program abstraction, validation, intervention diagnostics, and retained-candidate examples. ECI2D retains fewer demonstrations because of intervention-based filtering, while its rejected interventions mostly fail to transfer rather than to execute.
D.1 Static Validation
- Three AST-enforced constraints reject invalid abstractions, while candidates must also reproduce their source answer under observational correctness.Constraints govern deterministic constructs, helper arguments, and declared helper functions; candidates failing after retries are discarded.Source: §D.1 · p. 18
D.2 Run-Time Evaluation
- At run time, the adapter supplies only a call line, while declarations and program body remain unchanged.Returned option text is mapped back to its label; a value absent from the option list scores as an invalid intervention, as does failed execution.Source: §D.2 · p. 18
D.3 Diagnostics
- 79.1% without programs and 77.1% with programs are retained by observational filtering, versus 58.8% under ECI2D.The similar observational retention rates suggest the reduction under ECI2D follows from intervention rather than abstraction difficulty; wrong options dominate zero scores, while execution failures account for 5% together.Source: §D.3 · p. 18
D.4 Case Study
- In one retained candidate, the program scores options and the adapted call line makes the ground-truth option win over a competitor.A second candidate rebinds a helper argument from “receptionist lady” to “Jenny” while preserving the body; its interventions also use cached values.Source: §D.4 · p. 18
E More analysis of Observational Equivalence under Observation Distillation
Observation-based filtering cannot distinguish faithful causal rationales from post-hoc rationales that produce the same observed input, rationale, and answer distributions. The paper contrasts these causal structures and motivates intervention as a way to distinguish them.
E More analysis of Observational Equivalence under Observation Distillation
- Faithful rationales mediate the influence of input on output, whereas spurious rationales are generated retrospectively from the input and answer.The faithful model factorizes as P(Y | R)P(R | X)P(X), while the spurious model factorizes as P(R | X, Y)P(Y | X)P(X).Source: E More analysis of Observational… · p. 19
- A faithful and a spurious model can reproduce the same observational distribution, so outcome-oriented filtering cannot tell whether the rationale causes or follows the answer.The models are interventionally distinct despite observational equivalence.Source: E More analysis of Observational… · p. 19
- Intervening on a rationale changes the output distribution in the faithful model but leaves it invariant in the spurious model.The paper uses this difference to motivate the do-operator for distinguishing causal rationales through interventional invariance.Source: E More analysis of Observational… · p. 19
F More analysis of Transfer Invariance from Independent Causal Mechanisms
The section develops transfer invariance as a rationale-faithfulness criterion, describes its finite-sample estimation, and discusses computational cost and human validation. It also identifies incomplete identification and annotation-related error boundaries.
F More analysis of Transfer Invariance from Independent Causal Mechanisms
- A rationale is treated as a procedure mapping inputs with shared logical features to outputs, and faithfulness requires correctness on logically similar neighbors.The paper argues that source-instance correctness alone can reduce to a constant answer and provide no procedural content.Source: F More analysis of Transfer Invariance… · p. 20
- Under the independent causal mechanisms principle, faithful rationales transfer across local shifts, while spurious rationales can mismatch a neighbor’s context and lose correctness.The faithful mechanism remains invariant to changes in input generation; a context-specific post-hoc rationale violates transfer invariance when transplanted.Source: F More analysis of Transfer Invariance… · p. 20
- Transfer invariance is a sound but incomplete filter: failure rules out faithfulness, but pathological non-faithful rationales may still pass.The condition is necessary but not sufficient, including when unobserved factors create uniform regularities across a neighborhood.Source: F More analysis of Transfer Invariance… · p. 20
- The finite-sample estimator uses executions on sampled neighbors and requires agreement on every neighbor, except DROP, which uses an F1 threshold of 0.9.The per-neighbor indicator is an unbiased single-sample estimator, but estimates are coarse at the small sample sizes used in practice.Source: F More analysis of Transfer Invariance… · p. 21
G Computational Cost Analysis
- ECI2D adds offline preprocessing cost but no additional computation during online inference.For GSM8K with neighborhood size N = 3, preprocessing entails approximately 22,000–30,000 lightweight Python operations and takes under one minute on a standard CPU; the dominant cost is offline LLM generation.Source: G Computational Cost Analysis · p. 21
H Human Validation
- Human judgments agreed with retrieved logical-neighbor pairs at 94.4% on GSM8K and 91.3% on DROP.Disagreements often involved lexical overlap but divergent premise structure; mismatched neighbors can discard faithful rationales, lowering retention without admitting spurious ones.Source: H Human Validation · p. 21
- Annotators found 95.5% mean step-level alignment between rationales and programs, but silent repair is not detected by that alignment check.Inspection of 500 retained demonstrations found no incorrect reasoning chain and bounded silent repair prevalence at 0.76% with 95% confidence; direct measurement remains future work.Source: H Human Validation · p. 21
- Adapter errors can alter a source skeleton or fail to normalize a target’s units before substitution, causing the adapted program to return an incorrect result.One example outputs 5 instead of the correct answer 50 when “3 hours and 20 mins” is not synthesized into a single minute variable.Source: H Human Validation · pp. 21–22
Related papers
This paper cites
- Language Models are Few-Shot Learners2020
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting2023
- PAL: Program-aided Language Models2022
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?2022
- Measuring Faithfulness in Chain-of-Thought Reasoning2023