PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction

Rishi Datta, Lavanya Prahallad

arXiv:2608.30086v1cs.CLcs.LG

TL;DR

This paper asks when a fitted classifier helps an LLM, and whether the LLM helps the classifier, for credit-default prediction. It evaluates classifier imitation, prompt construction, and hybrid combinations, finding that adding one classifier’s probability gives the LLM strong ranking while retaining higher recall.

  • Problem

    The paper studies when and how a fitted classifier can help an inference-time LLM, and whether the LLM can help the classifier, for credit-default prediction.

    Source: §I · p. 1

  • Method

    The authors compare few-shot LLMs with fitted classifiers and test imitation prompts, classifier-guided feature and example selection, hybrid models, and augmentation using recall, precision, F1, AUC, and bootstrap confidence intervals.

    Source: §I · p. 2, C. Metrics and uncertainty · p. 2

  • Results

    AUC-ROC 0.72 →0.78: adding one strong classifier’s probability to the prompt gives the LLM random-forest-level ranking while retaining higher recall; imitation does not help.

    Source: §IV · p. 2, §V · p. 4, §VII · p. 5

  • Takeaways & Limitations

    For credit-default prediction, the paper recommends pruning the prompt to important features and adding one strong classifier’s probability.

    Source: §VII · p. 5

    The evidence comes from one LLM, one dataset, one 2,000-client evaluation draw, and one few-shot draw per condition, without testing prompt or model variation.

    Source: §VI · pp. 4–5

Abstract

from arXiv · show

Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression and random forests on tabular features. Large language models (LLMs) have recently been applied to this task through prompting. In this work we study how a fitted classifier and an LLM can be combined for credit-default prediction. We distinguish telling the LLM to imitate a classifier from using the classifier to build the prompt. We hypothesize that a fitted classifier can supply the ranking ability that an LLM prompt lacks. We experiment on the Default of Credit Card Clients dataset, and report recall, F1, and the area under the ROC and precision-recall curves, with bootstrap confidence intervals. We observe that a few-shot LLM has the highest recall (0.47) and F1 (0.50) of any single model but ranks worse than a random forest (AUC-ROC 0.72 against 0.79). Instructing the LLM to imitate a classifier gives no significant change. Pruning the prompt to the classifier's eight most important features raises recall by 0.071 and F1 by 0.032. Adding the classifier's predicted probability to the prompt raises the LLM's AUC-ROC from 0.72 to 0.78, matching the random forest, while keeping 0.118 higher recall than it. The reverse composition, and the use of several classifiers, do not help. We thus recommend a simple classifier-guided prompt for LLM-based credit prediction.

I. INTRODUCTION

The paper studies how fitted classifiers and prompted LLMs can be combined for credit-default prediction, focusing on whether classifiers can provide information that prompts lack. It distinguishes classifier imitation, classifier-guided prompt construction, and hybrid composition.

  • Credit-default prediction estimates default probability from repayment, credit, billing, payment, and demographic features.Source: §I · p. 1
  • Traditional approaches use logistic regression and random forests as strong baselines for tabular credit data.These methods can capture nonlinear interactions found in credit data.Source: §I · p. 1
  • Prior LLM work applies prompting, fine-tuning, retrieval, and domain models to financial or tabular prediction.Tree ensembles are reported to become stronger as tabular datasets grow.Source: §I · p. 1
  • The paper asks when a fitted classifier helps an LLM, and whether an LLM helps the classifier, for credit risk.Source: §I · p. 1
  • The study tests imitation, classifier-selected prompt content, and two-way hybrid composition.Prompt construction uses classifier importances to prune features or select few-shot examples; hybrid conditions pass model outputs in either direction.Source: §I · pp. 1–2

II. DATABASE USED IN THIS STUDY

The study uses an imbalanced credit-card default dataset and evaluates several fitted-model and LLM prompting conditions on controlled client splits. LLM prompts vary in instructions, visible features, and few-shot example selection.

  • The Default of Credit Card Clients dataset contains 30,000 clients, 23 features, and a binary default label.The default rate is 22.1%.Source: §II · p. 2
  • The data are split into 24,000 training and 6,000 test clients, while every LLM condition uses the same stratified 2,000-client test subset.Source: §II · p. 2
  • The fitted-model baselines are majority prediction, standardized logistic regression, a 100-tree depth-12 random forest, and AdaBoost with 200 estimators.Source: A. Fitted baselines · p. 2
  • The direct LLM prompt serializes client features and supplies 12 balanced few-shot examples, with six defaults and six non-defaults.The LLM returns a binary label and confidence interpreted as P(default) for AUC computation.Source: B. LLM prediction · p. 2
  • Imitation prompts ask the LLM to act like a random forest or weigh its important features, while E1 keeps only the RF’s top eight features.Other conditions select boundary or representative few-shot examples, or combine those examples with feature pruning.Source: B. LLM prediction · p. 2

Hybrid classifier-LLM composition:

The hybrid experiments pass classifier or LLM outputs across the two models. They compare a single RF signal, a three-classifier ensemble signal, and the reverse direction from LLM to RF.

C. Metrics and uncertainty

The study reports accuracy alongside default-class precision, recall, and F1, plus ROC and precision-recall AUCs. It uses confidence intervals to assess differences and reports diffusion results across seeds.

IV. RESULTS

The results show a trade-off: fitted models rank well but have low recall, while LLM conditions achieve higher recall but weaker ranking. The hybrid E6 reaches high recall and fitted-model-level ranking.

B. Classifier-guided prompt construction

Classifier-guided prompting helps the LLM through feature selection and probability-based guidance, while curated examples and adding the LLM to the classifier provide no comparable benefit. The RF-to-LLM hybrid is the strongest composition tested.

B. Classifier-guided prompt construction

C. Hybrid classifier–LLM composition

D. Diffusion-based minority augmentation

The study finds that richer classifier combinations do not improve the classifier-guided LLM, while diffusion augmentation improves on SMOTE but remains less practical than class-weighting.

Diffusion augmentation

Hybrid model

  • The classifier-guided hybrid raises AUC-ROC from 0.72 to 0.78 while retaining higher recall than the Random Forest.The hybrid supplies ranking information through one classifier’s predicted probability.Source: §V · p. 4
  • The reverse composition adds nothing, supporting a one-directional design in which a single strong classifier guides the LLM.The paper reports no benefit from using the LLM inside the classifier or from richer classifier ensembles.Source: §V · p. 4

VI. LIMITATIONS

The evaluation is limited to one LLM, one dataset, one 2,000-client evaluation draw, and one few-shot draw per condition; uncertainty estimates do not capture LLM or few-shot variance.

Evaluation scope

  • The results use a single LLM, a single dataset, one 2,000-client evaluation draw, and one few-shot draw per condition.The paper identifies these choices as scope limitations of the evaluation.Source: §VI · p. 4

Evaluation uncertainty

  • Bootstrap confidence intervals capture test-set sampling uncertainty but not LLM-sampling or few-shot-draw variance.Multiple prompt draws and additional models would strengthen the ablation and hybrid conclusions.Source: §VI · p. 5

VII. CONCLUSIONS

The study finds that simple classifier-guided prompts improve LLM credit-default prediction, while imitation, reverse composition, and multi-classifier hybrids do not help. It recommends pruning prompts to important features and adding one strong classifier’s probability.

  • A few-shot LLM has high recall but weak ranking, whereas a fitted random forest has strong ranking but low recall.Source: §VII · p. 5
  • Telling the LLM to imitate a classifier does not improve prediction, but using the classifier to prune the prompt to important features does.Source: §VII · p. 5
  • A simple hybrid works best by adding one random forest’s predicted probability to the LLM prompt.The reverse composition and using three classifiers instead of one do not help.Source: §VII · p. 5
  • Random forest inference adds little cost, while diffusion-based minority-class augmentation is better than SMOTE but does not beat class-weighting.Source: §VII · p. 5
  • The paper recommends pruning prompts to a classifier’s important features and adding one strong classifier’s probability.Future work should test more datasets, language models, and few-shot draws to capture prompt and model variation in confidence intervals.Source: §VII · p. 5

APPENDIX A PROMPTS, MODEL DETAILS, AND REPRODUCIBILITY

The appendix documents the prompt templates, model conditions, data serialization, evaluation setup, and reproducibility materials. Prompts use balanced few-shot examples, while selected conditions restrict features or append classifier outputs.

Model and inference

  • All LLM conditions use claude-sonnet-5 with extended thinking disabled, batched requests, strict JSON outputs, and cached raw responses.Source: Appendix A · p. 5
  • Each prompt includes 12 balanced labeled examples: 6 defaults and 6 non-defaults, drawn from the training split with seed 42.The same random set is used except for boundary and representative exemplar-selection conditions.Source: Appendix A · p. 5

Prompt templates

Serialization

  • Each client is serialized as comma-separated name=value fields, with query lines formatted as id <k>: <fields>.Feature-pruning conditions retain the RF’s top eight features, and hybrid conditions append classifier outputs.Source: Appendix A · p. 6

Fitted models and evaluation

  • The evaluation uses a stratified 80/20 split, a 2,000-client test subset, paired bootstrap comparisons, and 1,000 bootstrap resamples.The diffusion study reports mean ± 95% confidence intervals over five seeds.Source: Appendix A · p. 6

Reproducibility

  • The project provides code, data-preparation scripts, prompts, cached outputs, result tables, and figure-generation scripts for regeneration.Source: Appendix A · p. 6

Condition-specific instructions

  • L2 instructs the LLM to act as a trained Random Forest classifier and prioritize the ordered top-eight feature list.The list emphasizes repayment-status features, followed by bill and payment amounts.Source: Appendix A · p. 6
  • Feature-pruning conditions use only the top-eight features, while E7 appends outputs from three classifiers.Source: Appendix A · p. 6

Related papers

Loading 2608.30086v1…