PaperlayerView exact revision on arXiv

Source-linked AI summary

CARRE: Counterfactual Action Retrieval and Reason Evaluation for Explainable Churn Prescription

MinJoo Kim, SanJin Park, SeungHwan Cho

arXiv:2609.09766v1cs.CL

TL;DR

Churn models often identify high-risk customers without specifying a feasible retention action or why it fits the customer. CARRE combines retrieval, cost-aware counterfactual scoring, and LLM reasoning to produce prescriptions and explanations, with stronger risk reduction than SHAP baselines and improved reason agreement in its evaluation.

  • Problem

    Churn prediction does not by itself provide feasible retention actions or profile-grounded explanations for high-risk customers.

  • Method

    CARRE retrieves predefined actions, scores counterfactual risk reduction against operational cost, and uses staged LLM reasoning to classify churn reasons and generate explanations.

  • Results

    CARRE achieves 79.8% greater raw risk reduction than SHAP and 80.4% greater reduction than SHAP+Cost across 313 high-risk cases, while prompt refinement raises weak-label agreement from 79.4% to 90.4%.

  • Takeaways & Limitations

    Separating retrieval, counterfactual optimization, and language generation supports joint evaluation of churn prescriptions and explanations in a prototype pipeline.

  • Takeaways & Limitations

    The study uses a single dataset and evaluation-tuned λ and k settings without a separate validation split, leaving cross-dataset robustness unresolved.

Abstract

from arXiv · show

Churn models typically identify high-risk customers but do not specify which feasible retention action should be considered or why that action is appropriate. We present CARRE (Counterfactual Action Retrieval and Reason Evaluation), a three-stage framework that combines retrieval-augmented candidate generation, cost-aware counterfactual scoring, and large language model (LLM) reasoning. CARRE retrieves a predefined catalog of retention actions, estimates model-predicted churn-risk changes under explicit feature transformations, and generates a structured churn reason and a profile-grounded explanation for the selected action. On the IBM Telco Customer Churn dataset, CARRE achieves 79.8% greater mean model-predicted risk reduction than the plain SHAP baseline and 80.4% greater reduction than the cost-controlled SHAP+Cost baseline across 313 high-risk test cases; its cost-normalized efficiency is 10.5% higher than that of plain SHAP. On a 136-case reason-stratified evaluation sample, diagnosis-driven prompt refinement increases weak-label agreement from 79.4% to 90.4%, with no auxiliary-plan constraint violations; because the same sample was used for error diagnosis and re-evaluation, the post-refinement result is not an independent estimate of generalization. For 135 explanations generated using the pre-refinement v2 reason outputs, two cross-vendor LLM judges assign mean scores ranging from 4.02 to 5.00 out of 5, although one judge saturates on actionability, and a deterministic audit finds no contradictions among 66 verifiable profile claims. Retrieval ablations show that k=5 provides the best evaluated compromise between high candidate coverage and downstream reasoning agreement in this dataset. These results illustrate how retrieval, model-based counterfactual scoring, and language generation can be separated and jointly evaluated in a prototype churn-prescription pipeline.

1 Introduction

CARRE addresses the prediction-prescription gap by connecting churn-risk diagnosis to feasible, cost-aware retention actions and profile-grounded explanations. Its multi-layer evaluation reports stronger counterfactual risk reduction than SHAP baselines and improved weak-label agreement after diagnosis-driven prompt refinement.

  • Research gap: Prediction-only models identify likely churn but do not specify feasible interventions or explain why a selected action fits the customer.SHAP identifies feature contributions, counterfactual methods alter profiles, and unconstrained LLMs may generate non-operational advice.
  • Research gap: Explanation alignment requires consistency among profile-based churn diagnosis, the recommended action, and language that frontline agents can use.The paper frames these as simultaneous requirements for operational retention decisions.
  • Motivation and contributions: CARRE combines retrieval-augmented candidate generation, counterfactual cost-aware scoring, and LLM reasoning for personalized churn prescription.The LLM identifies churn reasons and generates explanations aligned with retrieved action context.
  • Evaluation design: 79.4% to 90.4% weak-label agreement followed diagnosis-driven prompt refinement on the same 136-case sample, so the result is not an independent generalization estimate.The design separates reason classification and auxiliary-plan generation from final explanation generation.

2 Related Work

Related work provides separate tools for churn prediction, counterfactual recourse, retrieval, and structured LLM reasoning, but CARRE combines them into a prediction-to-prescription-to-explanation loop.

  • Churn analytics: Boosting improves several telecommunications churn classifiers, while profit-driven frameworks motivate accounting for operational retention costs.These strands establish performance and cost considerations that CARRE combines in prescription selection.
  • Prescriptive analytics: Existing prescriptive systems may recommend retention actions without explaining why each recommendation addresses a customer’s churn drivers.CARRE is positioned as completing the loop from prediction to prescription to explanation.
  • Counterfactual recourse: Counterfactual recourse adds feasibility and cost constraints to minimal feature changes, which CARRE restricts to a predefined action space.The paper adopts the cost-aware formulation while limiting recourse to business-defined interventions.
  • Retrieval-augmented generation: RAG can improve retrieval and reduce hallucination, while CARRE uses it to constrain the LLM to predefined actions and provide eligibility context.The number of retrieved documents trades off missing the optimal action against introducing confusing irrelevant context.
  • LLM reasoning: CARRE combines chain-of-thought prompting and explicit classification criteria to support structured rule application and instruction adherence.This design responds to reverse-reasoning concerns in churn-reason classification.

3 Problem Formulation and CARRE Framework

CARRE separates retrieval, counterfactual optimization, and LLM reasoning to produce a churn reason, a predefined action, and an aligned explanation for high-risk customers. Its action taxonomy, weak-label rules, and staged prompts make these outputs independently testable while retaining explicit cost and eligibility constraints.

  • 3 Problem Formulation and CARRE Framework: CARRE retrieves k similar action documents, counterfactually scores them for risk reduction and cost, then generates a reason and explanation grounded in the selected action.Retrieval handles candidate coverage, optimization handles cost efficiency, and the LLM handles reasoning and language generation.
  • 3.1 Action Space and Reason Taxonomy: The action space contains six retention interventions with researcher-defined relative costs ranging from A_PAYMENT_AUTOPAY at 1.0 to A_CONTRACT_24M at 3.5.The taxonomy is grounded in recurring telecommunications churn drivers and corresponding retention levers.
  • 3.1 Action Space and Reason Taxonomy: Six priority-ordered reason labels use study-specific thresholds and rules as reproducible weak labels rather than expert-validated causal diagnoses.The priority order resolves cases where multiple conditions hold.
  • 3.2 Problem Formulation: For customers with f(x) ≥ 0.5, CARRE outputs a structured reason r, predefined action a*, and natural-language explanation e; below-threshold customers receive no Stage 2 prescription or Stage 3-b explanation.The tuple must satisfy profile-derived diagnosis, cost-adjusted counterfactual selection, and consistency between reason and action.
  • 3.3 Three-Stage Pipeline: Stage 2 applies each retrieved action’s feature modifications, computes its counterfactual score, and selects the highest-scoring action, using λ=0.25 for the risk-cost trade-off.If no retrieved action yields positive risk reduction, the prescription is NONE.
  • 3.3 Three-Stage Pipeline: Stage 3-a withholds cf_action while classifying the reason, whereas Stage 3-b receives both the reason and selected action to generate the final explanation.Withholding cf_action prevents reverse-reasoning bias; the deployed prescription remains Stage 2’s cf_action.
  • 3.3 Three-Stage Pipeline: Stage 3-a uses explicit priority rules, chain-of-thought evaluation, and a fixed output schema to constrain reason and auxiliary-plan generation.Plans outside the retrieved set are flagged as auxiliary-plan constraint violations.
  • 3.3 Three-Stage Pipeline: Differences between cf_action and llm_plan expose divergence between cost-adjusted risk reduction and rule-consistent classification, while the delivered explanation follows cf_action.The discrepancy can indicate a need to adjust costs or refine rules.

4 Experimental Setup

The evaluation uses the IBM Telco Customer Churn dataset and tests CARRE through base-model performance, prescription comparisons, retrieval depth, cost penalties, and LLM reasoning. The setup evaluates 313 high-risk test cases and examines whether calibrated counterfactual scoring provides sufficiently smooth risk estimates for intervention selection.

  • LLM reasoning evaluation: The evaluation also tests whether LLMs apply the reason taxonomy and generate coherent, constrained, and customer-grounded explanations.The reasoning experiments use a 30-case balanced model-comparison set and a scaled 136-case reason-stratified sample.
  • Dataset and evaluation: The IBM Telco Customer Churn dataset contains 7,043 customers described by 19 predictor features, with a 26.5% positive-class prevalence and an 80/20 stratified split.The split yields 5,634 training and 1,409 test instances.
  • Dataset and evaluation: The experiments evaluate prescription quality, retrieval depth k, and risk-cost balance λ on all 313 high-risk test cases.High-risk cases satisfy f(x) ≥ 0.50, and prescription-method comparisons report 95% bootstrap confidence intervals.
  • Counterfactual scoring model: The base classifier achieves ROC-AUC 0.842, PR-AUC 0.633, and Brier score 0.138 on the test split.The PR-AUC is evaluated against a random baseline of 0.265 under 26.5% positive-class prevalence.
  • Prescription evaluation: CARRE is compared with random, rule-based, SHAP-based, and SHAP+Cost baselines using model-predicted counterfactual risk reduction and cost penalties.SHAP+Cost applies the same cost-penalty structure as CARRE while replacing predicted risk reduction with normalized SHAP importance.
  • Counterfactual scoring model: Calibrated logistic regression is retained for counterfactual scoring because tested tree ensembles produce discontinuous responses under graded feature perturbations.Tree ensembles can achieve marginally higher aggregate ranking metrics, but their coarse responses hinder the graded Δrisk estimates required by Stage 2.

5 Results

CARRE improves counterfactual prescription quality over SHAP-based and other baselines, while retrieval depth and cost weighting materially shape performance. Its LLM reasoning results are promising but require careful interpretation because prompt formulations and evaluation reliability vary.

  • Prescription quality: 79.8% higher raw risk reduction than SHAP, with CARRE achieving 0.302 versus 0.168.The 95% confidence intervals do not overlap: [0.294, 0.308] versus [0.166, 0.169].
  • Prescription quality: 80.4% higher mean predicted-risk reduction than the cost-controlled SHAP+Cost baseline.SHAP+Cost remains nearly identical to plain SHAP at 0.167 versus 0.168, indicating that identical cost penalization does not rescue feature-importance-based selection.
  • Prescription quality: CARRE achieves 120% higher risk reduction and 52% higher efficiency than random, despite random effectiveness reaching 92.3%.Effectiveness measures whether any positive effect occurs, whereas Risk Red. and Efficiency measure magnitude and cost-normalized performance.
  • Retrieval depth: At k=5, retrieval recall reaches 85.3% and NONE cases disappear, providing the adopted compromise between action coverage and context noise.The oracle k=6 retains a 14.7 percentage-point recall advantage, but k=5 is selected as the default.
  • Risk-cost trade-off: The lambda sweep shows a sharp regime boundary: risk reduction is nearly flat from 0.10 to 0.25, then falls 59% when lambda reaches 0.50.At lambda=0.50, selection shifts almost entirely from A_CONTRACT_24M to the low-cost A_PAYMENT_AUTOPAY.
  • LLM reasoning: GPT-4o achieves 93.3% weak-label agreement on the 30-case comparison, but the larger evaluations use a different prompt formulation.Human ratings were not treated as confirmatory because inter-rater agreement was very low, with Krippendorff’s alpha of 0.003–0.175.

6 Discussion and Limitations

CARRE’s experiments show that direct counterfactual optimization improves model-predicted risk reduction over SHAP-based heuristics, while retrieval and prompt design affect reasoning quality. The discussion also qualifies these findings through label, causal, action-space, tuning, validation, and model-comparison limitations.

  • Findings: 79.8% higher raw risk reduction than SHAP was achieved across 313 high-risk cases, with the advantage attributed to counterfactual optimization rather than cost weighting.Under identical cost penalization, CARRE retained an 80.4% advantage over SHAP+Cost and achieved 10.5% higher cost-normalized Efficiency than plain SHAP.
  • Findings: k=5 provided the best evaluated compromise between high oracle-action coverage and the highest weak-label agreement, while k=6 had lower reasoning agreement.Restricting context to retrieved actions improved weak-label agreement by 3.7 percentage points over the full action library, with zero auxiliary-plan violations.
  • Findings: 90.4% weak-label agreement followed prompt refinement from 79.4% on the same 136 cases, so the improvement is a development result rather than an independent generalization estimate.The v2 scaled evaluation declined from 83.3% on 30 cases to 79.4% on 136 cases before targeted v3 refinement.
  • Limitations: Model comparisons remain confounded because tested systems differ simultaneously in scale, architecture, and vendor, while non-GPT-4o models failed systematically in tail categories.Llama-3.3-70B scored 50.0% versus 40.0% for GPT-4o-mini, but no single responsible factor was isolated.
  • Findings: Stage 3-a matched the Stage 2 action exactly in 3/135 explained cases, or 2%, because cost-penalized optimization often selected the two-year contract.Among cases where both stages selected an action, exact match was 3/89 and family-level match was 10/89; Stage 3-b then anchored explanations to the actual prescription.
  • Limitations: Weak-label agreement measures instruction-following fidelity, not causal correctness, because the reason taxonomy is a researcher-defined heuristic without expert-annotated churn causes.Expert annotation or observed intervention outcomes were unavailable.
  • Limitations: The reported risk reductions are model-predicted changes rather than causal or off-policy validation of real intervention effects.The action transformations are study-specific design choices, and the six-action space excludes LOW_ENGAGEMENT and OTHER reason types.
  • Limitations: The defaults λ=0.25 and k=5 were selected on evaluation data without a separate validation split, and all results came from one dataset.Risk reduction collapsed by 59% once λ exceeded 0.50, making the boundary specific to this study’s costs and transformations.

7 Conclusion

CARRE integrates retrieval, cost-aware counterfactual scoring, and LLM reasoning to produce churn reasons, prescriptions, and explanations. On the IBM Telco dataset, it outperformed SHAP-based baselines on model-predicted risk reduction, while prompt refinement improved weak-label agreement but still requires independent validation.

  • Contribution: CARRE integrates RAG-based retrieval, cost-aware counterfactual scoring, and LLM reasoning in a three-stage churn-prescription pipeline.Its modular design allows retrieval, optimization, and LLM justification to be evaluated and replaced independently.
  • Results: Weak-label agreement increased from 79.4% to 90.4% on the 136-case sample after diagnosis-driven prompt refinement.The paper treats this as a prompt-development result, not an independent generalization estimate.
  • Results: Two cross-vendor judges assigned mean explanation scores from 4.02 to 5.00 for 135 v2 explanations, and deterministic auditing found no contradictions among 66 verifiable profile claims.These results concern explanations generated with pre-refinement v2 reason outputs.
  • Future work: Future work should expand the action space to LOW_ENGAGEMENT and OTHER, improve faithfulness through citation-based generation, and extend CARRE to other subscription industries.The conclusion identifies these as directions for extending the prototype beyond its current scope.

A Action-to-Feature Transformations and Eligibility Rules

The appendix specifies how Stage 2 maps eligible retention actions to exact counterfactual feature transformations and relative costs. Stage 3-a separately classifies churn reasons and selects only retrieved actions, using ordered rules and a fixed output schema.

  • Stage 2: Stage 2 checks each action’s eligibility, applies its exact feature transformation, and uses the study-code relative cost for scoring.Table 10 documents these action-specific conditions, transformations, and costs.
  • Stage 2: A_CONTRACT_24M changes only Contract, whereas A_PRICE_DISCOUNT is the only action that reduces MonthlyCharges, by 8%.Eligibility is enforced before scoring, and additional restrictions apply to price discounts.
  • Stage 3-a: The fixed Stage 3-a output contains a Reason line, a Plan line, and two profile-grounded rationale bullets.The prompt returns an action ID from the retrieved set or NONE.
  • Stage 3-a: Stage 3-a withholds cf_action so the model classifies the reason from the customer profile rather than rationalizing backward from an action.The Plan output must select an action from the retrieved set; out-of-set IDs count as auxiliary-plan violations.
  • Prompt inputs: The prompt presents a customer profile, churn probability, and retrieved actions while explicitly withholding the counterfactual action from classification.The example includes an 0.82 churn probability and retrieved actions including A_PRICE_DISCOUNT and A_CONTRACT_24M.
  • Classification rules: Classification rules are evaluated in priority order: PRICE_SENSITIVE, CONTRACT_RISK, PAYMENT_FRICTION, SUPPORT_DEFICIT, LOW_ENGAGEMENT, then OTHER.The rules use customer fields such as MonthlyCharges, contract type, tenure, payment method, and support features.

C Base Classifier Comparison for CF Scoring

The classifier comparison evaluates predictive quality alongside counterfactual-surface smoothness. Although gradient boosting has slightly stronger ranking metrics, calibrated logistic regression supplies the continuous response required for graded counterfactual scoring.

  • Predictive quality: Gradient boosting reached ROC-AUC 0.845 versus 0.842 and PR-AUC 0.659 versus 0.633 for logistic regression, showing a marginal ranking advantage.The comparison covers calibrated logistic regression and three isotropically calibrated tree ensembles.
  • CF-surface smoothness: All three tree ensembles produced coarse, discontinuous counterfactual surfaces, whereas logistic regression produced a fully continuous graded response.Smoothness was measured with a 0–30% price-discount sweep over 31 steps on 313 high-risk cases.
  • Model choice: Counterfactual scoring rewards a smooth probability surface more than raw ranking power, so CARRE adopts calibrated logistic regression as its base estimator.The choice preserves estimable predicted-risk changes while giving up almost nothing in ranking accuracy.

D Action-Space Scaling Stress Test

The retrieval-only stress test examines how enlarging the action-document library affects retrieval speed and preservation of the counterfactual-oracle action. Exact-action recall deteriorates with near-duplicate variants, while family-level recall remains robust only up to a larger library size.

  • Retrieval robustness: Exact-action Recall@5 collapses at |A|≥12 when near-duplicate parameter variants are added.The variants represent contract terms, discount tiers, segment-targeted offers, and channel campaigns.
  • Retrieval robustness: Family-level recall remains 0.92–1.00 up to |A|=100 but collapses beyond |A|≈200.Large lever families can monopolize the top-k results.
  • Runtime: Query encoding at approximately 1 ms dominates end-to-end latency, while index construction remains under 7 seconds regardless of library size.
  • Evaluation setup: The test covers 313 high-risk cases and evaluates exact oracle-action retrieval separately from lever-family retrieval.

E Per-Reason Weak-Label Agreement

The four-model comparison shows that aggregate weak-label agreement differences are concentrated in specific reason categories rather than being uniform. The section also documents the rule-based and SHAP-based baseline mappings used for comparison.

  • Per-reason comparison: Agreement differences are concentrated in a few reason categories, with GPT-4o classifying almost every category correctly while weaker models fail selectively.GPT-4o-mini fails on price and contract categories; Llama-3.3-70B and Qwen3-32B score zero on CONTRACT_RISK and OTHER.
  • Per-reason comparison: Table 13 reports per-reason weak-label agreement for four models across 30 cases, with five cases per reason.
  • Baseline mappings: The rule-based baseline applies profile conditions in priority order and returns NONE when no condition holds.
  • Baseline mappings: The SHAP baseline maps the highest positive churn-directed SHAP feature to a candidate action, then uses a fixed priority list for ties or ineligible targets.
  • Baseline mappings: The random baseline selects one eligible action per customer using a fixed per-customer seed without averaging repeated draws.

G Implementation and Reproducibility Details

Experiments use a fixed stratified train/test split and training-only preprocessing for 19 predictors after removing customerID and Churn. Exact reproduction of the LLM evaluations is limited because provider aliases were used instead of pinned model snapshots.

  • Experimental setup: All experiments use seed 42 and an 80/20 stratified split containing 5,634 training and 1,409 test cases.
  • Preprocessing: After customerID and Churn are removed, the pipeline retains 19 predictors and fits imputation, standardization, and one-hot encoding on the training split only.Numeric features are median-imputed and standardized; categorical features use most-frequent imputation and ignore unknown categories.
  • Reproducibility: The LLM evaluations use provider aliases rather than pinned model snapshots, limiting exact reproducibility.
Loading 2609.09766v1…