PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction

Rishi Datta, Lavanya Prahallad

arXiv:2608.30086v1cs.CLcs.LG

TL;DR

This paper asks when a fitted classifier helps an LLM, and whether the LLM helps the classifier, for credit-default prediction. It evaluates classifier imitation, prompt construction, and hybrid combinations, finding that adding one classifier’s probability gives the LLM strong ranking while retaining higher recall.

  • Problem

    The paper studies when and how a fitted classifier can help an inference-time LLM, and whether the LLM can help the classifier, for credit-default prediction.

  • Method

    The authors compare few-shot LLMs with fitted classifiers and test imitation prompts, classifier-guided feature and example selection, hybrid models, and augmentation using recall, precision, F1, AUC, and bootstrap confidence intervals.

  • Results

    AUC-ROC 0.72 →0.78: adding one strong classifier’s probability to the prompt gives the LLM random-forest-level ranking while retaining higher recall; imitation does not help.

  • Takeaways & Limitations

    For credit-default prediction, the paper recommends pruning the prompt to important features and adding one strong classifier’s probability.

  • Takeaways & Limitations

    The evidence comes from one LLM, one dataset, one 2,000-client evaluation draw, and one few-shot draw per condition, without testing prompt or model variation.

Abstract

from arXiv · show

Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression and random forests on tabular features. Large language models (LLMs) have recently been applied to this task through prompting. In this work we study how a fitted classifier and an LLM can be combined for credit-default prediction. We distinguish telling the LLM to imitate a classifier from using the classifier to build the prompt. We hypothesize that a fitted classifier can supply the ranking ability that an LLM prompt lacks. We experiment on the Default of Credit Card Clients dataset, and report recall, F1, and the area under the ROC and precision-recall curves, with bootstrap confidence intervals. We observe that a few-shot LLM has the highest recall (0.47) and F1 (0.50) of any single model but ranks worse than a random forest (AUC-ROC 0.72 against 0.79). Instructing the LLM to imitate a classifier gives no significant change. Pruning the prompt to the classifier's eight most important features raises recall by 0.071 and F1 by 0.032. Adding the classifier's predicted probability to the prompt raises the LLM's AUC-ROC from 0.72 to 0.78, matching the random forest, while keeping 0.118 higher recall than it. The reverse composition, and the use of several classifiers, do not help. We thus recommend a simple classifier-guided prompt for LLM-based credit prediction.

I. INTRODUCTION

The paper studies how fitted classifiers and prompted LLMs can be combined for credit-default prediction, focusing on whether classifiers can provide information that prompts lack. It distinguishes classifier imitation, classifier-guided prompt construction, and hybrid composition.

  • Credit-default prediction estimates default probability from repayment, credit, billing, payment, and demographic features.
  • Traditional approaches use logistic regression and random forests as strong baselines for tabular credit data.These methods can capture nonlinear interactions found in credit data.
  • Prior LLM work applies prompting, fine-tuning, retrieval, and domain models to financial or tabular prediction.Tree ensembles are reported to become stronger as tabular datasets grow.
  • The paper asks when a fitted classifier helps an LLM, and whether an LLM helps the classifier, for credit risk.
  • The study tests imitation, classifier-selected prompt content, and two-way hybrid composition.Prompt construction uses classifier importances to prune features or select few-shot examples; hybrid conditions pass model outputs in either direction.

II. DATABASE USED IN THIS STUDY

The study uses an imbalanced credit-card default dataset and evaluates several fitted-model and LLM prompting conditions on controlled client splits. LLM prompts vary in instructions, visible features, and few-shot example selection.

  • The Default of Credit Card Clients dataset contains 30,000 clients, 23 features, and a binary default label.The default rate is 22.1%.
  • The data are split into 24,000 training and 6,000 test clients, while every LLM condition uses the same stratified 2,000-client test subset.
  • The fitted-model baselines are majority prediction, standardized logistic regression, a 100-tree depth-12 random forest, and AdaBoost with 200 estimators.
  • The direct LLM prompt serializes client features and supplies 12 balanced few-shot examples, with six defaults and six non-defaults.The LLM returns a binary label and confidence interpreted as P(default) for AUC computation.
  • Imitation prompts ask the LLM to act like a random forest or weigh its important features, while E1 keeps only the RF’s top eight features.Other conditions select boundary or representative few-shot examples, or combine those examples with feature pruning.

Hybrid classifier-LLM composition:

The hybrid experiments pass classifier or LLM outputs across the two models. They compare a single RF signal, a three-classifier ensemble signal, and the reverse direction from LLM to RF.

  • E6 passes the random forest’s predicted probability and binary decision into the LLM prompt.It also uses the RF probability and decision for each few-shot example through out-of-bag verdicts.
  • E7 passes probabilities and decisions from the random forest, logistic regression, and AdaBoost into the LLM.
  • The inverse condition adds the LLM’s E1 label and confidence as RF features and evaluates the comparison with five-fold cross-validation.

C. Metrics and uncertainty

The study reports accuracy alongside default-class precision, recall, and F1, plus ROC and precision-recall AUCs. It uses confidence intervals to assess differences and reports diffusion results across seeds.

  • The evaluation reports accuracy, precision, recall, F1, AUC-ROC, and AUC-PR, with default-class metrics emphasized because the data are imbalanced.
  • Overall accuracy is shown beside the majority-baseline accuracy of 0.779 for context.
  • A difference is called significant when its 95% confidence interval excludes zero.
  • The diffusion study reports mean ± 95% confidence intervals across five seeds.

IV. RESULTS

The results show a trade-off: fitted models rank well but have low recall, while LLM conditions achieve higher recall but weaker ranking. The hybrid E6 reaches high recall and fitted-model-level ranking.

  • Fitted models occupy the high-AUC, low-recall region, whereas LLM conditions occupy the low-AUC, high-recall region.
  • Hybrid E6 lies in the high-AUC, high-recall region and attains Random-Forest-level ranking.
  • The direct LLM achieves the highest recall at 0.47, while E1 achieves the highest F1 at 0.53, above the fitted RF’s 0.48.
  • Imitation prompts do not significantly change LLM performance on F1, AUC-ROC, or recall.

B. Classifier-guided prompt construction

Classifier-guided prompting helps the LLM through feature selection and probability-based guidance, while curated examples and adding the LLM to the classifier provide no comparable benefit. The RF-to-LLM hybrid is the strongest composition tested.

  • B. Classifier-guided prompt construction: Restricting the prompt to the RF’s top-8 features improves recall by 0.071 and F1 by 0.032, both significantly.
  • B. Classifier-guided prompt construction: E1 achieves the study’s best F1 at 0.529, exceeding the fitted RF.
  • B. Classifier-guided prompt construction: Boundary example selection lowers F1 by 0.057 and recall by 0.125, while representative selection lowers F1 by 0.029.
  • B. Classifier-guided prompt construction: Combining feature pruning with curated examples removes E1’s gain, while random balanced examples outperform RF-curated examples.
  • C. Hybrid classifier–LLM composition: Adding RF probability and decision to the prompt raises AUC-ROC from 0.719 to 0.782, a significant gain of 0.063 reaching the fitted RF’s 0.790 level.
  • C. Hybrid classifier–LLM composition: E6 improves recall by 0.118 and F1 by 0.045 versus the fitted RF, at an AUC-ROC cost of −0.008.
  • C. Hybrid classifier–LLM composition: Using three classifier opinions does not improve ranking or F1 over E6 and significantly reduces recall by 0.045.
  • C. Hybrid classifier–LLM composition: Adding the LLM’s output to the RF produces no significant change, with recall +0.002, F1 −0.004, and AUC-ROC −0.002.

D. Diffusion-based minority augmentation

The study finds that richer classifier combinations do not improve the classifier-guided LLM, while diffusion augmentation improves on SMOTE but remains less practical than class-weighting.

  • Diffusion augmentation: Diffusion augmentation beats SMOTE on AUC-PR (+0.022), precision (+0.028), and accuracy (+0.013), while tying it on recall and F1.These comparisons are paired across seeds.
  • Diffusion augmentation: Diffusion augmentation does not beat simple class-weighting, which gives the highest recall and F1.The diffusion approach also varies more across runs.
  • Hybrid model: The classifier-guided hybrid raises AUC-ROC from 0.72 to 0.78 while retaining higher recall than the Random Forest.The hybrid supplies ranking information through one classifier’s predicted probability.
  • Hybrid model: The reverse composition adds nothing, supporting a one-directional design in which a single strong classifier guides the LLM.The paper reports no benefit from using the LLM inside the classifier or from richer classifier ensembles.

VI. LIMITATIONS

The evaluation is limited to one LLM, one dataset, one 2,000-client evaluation draw, and one few-shot draw per condition; uncertainty estimates do not capture LLM or few-shot variance.

  • Evaluation scope: The results use a single LLM, a single dataset, one 2,000-client evaluation draw, and one few-shot draw per condition.The paper identifies these choices as scope limitations of the evaluation.
  • Evaluation uncertainty: Bootstrap confidence intervals capture test-set sampling uncertainty but not LLM-sampling or few-shot-draw variance.Multiple prompt draws and additional models would strengthen the ablation and hybrid conclusions.

VII. CONCLUSIONS

The study finds that simple classifier-guided prompts improve LLM credit-default prediction, while imitation, reverse composition, and multi-classifier hybrids do not help. It recommends pruning prompts to important features and adding one strong classifier’s probability.

  • A few-shot LLM has high recall but weak ranking, whereas a fitted random forest has strong ranking but low recall.
  • Telling the LLM to imitate a classifier does not improve prediction, but using the classifier to prune the prompt to important features does.
  • A simple hybrid works best by adding one random forest’s predicted probability to the LLM prompt.The reverse composition and using three classifiers instead of one do not help.
  • Random forest inference adds little cost, while diffusion-based minority-class augmentation is better than SMOTE but does not beat class-weighting.
  • The paper recommends pruning prompts to a classifier’s important features and adding one strong classifier’s probability.Future work should test more datasets, language models, and few-shot draws to capture prompt and model variation in confidence intervals.

APPENDIX A PROMPTS, MODEL DETAILS, AND REPRODUCIBILITY

The appendix documents the prompt templates, model conditions, data serialization, evaluation setup, and reproducibility materials. Prompts use balanced few-shot examples, while selected conditions restrict features or append classifier outputs.

  • Model and inference: All LLM conditions use claude-sonnet-5 with extended thinking disabled, batched requests, strict JSON outputs, and cached raw responses.
  • Each prompt includes 12 balanced labeled examples: 6 defaults and 6 non-defaults, drawn from the training split with seed 42.The same random set is used except for boundary and representative exemplar-selection conditions.
  • Prompt templates: The exact base prompt and condition-specific hybrid additions are provided verbatim in Figures 3 and 4.
  • Serialization: Each client is serialized as comma-separated name=value fields, with query lines formatted as id <k>: <fields>.Feature-pruning conditions retain the RF’s top eight features, and hybrid conditions append classifier outputs.
  • Fitted models and evaluation: The evaluation uses a stratified 80/20 split, a 2,000-client test subset, paired bootstrap comparisons, and 1,000 bootstrap resamples.The diffusion study reports mean ± 95% confidence intervals over five seeds.
  • Reproducibility: The project provides code, data-preparation scripts, prompts, cached outputs, result tables, and figure-generation scripts for regeneration.
  • Condition-specific instructions: L2 instructs the LLM to act as a trained Random Forest classifier and prioritize the ordered top-eight feature list.The list emphasizes repayment-status features, followed by bill and payment amounts.
  • Condition-specific instructions: Feature-pruning conditions use only the top-eight features, while E7 appends outputs from three classifiers.
Loading 2608.30086v1…