PaperlayerView exact revision on arXiv

Source-linked AI summary

From Prompts to Trees: Effective LLM-Guided Tree Generation for Few-Shot Tabular Classification

Yue Qiu, Zekang Du, Yiqun Diao, Bingsheng He, Qinbin Li

arXiv:2610.10227v1cs.LGcs.AIcs.CL

TL;DR

Tabular classification with few labeled examples presents a gap between LLMs, which have high inference costs and limited interpretability, and decision trees, which can underperform in low-data settings. LLMT distills LLM-generated rules into decision trees and reports consistent, significant accuracy improvements, token-cost savings, and interpretable outputs in few-shot experiments.

  • Problem

    The paper addresses how to approximate LLM classification capability with an interpretable, low-overhead decision tree when only a small number of labeled samples are available.

    Source: §4.1 · p. 4

  • Method

    LLMT prompts an LLM to generate discrete classification rules, assembles them into a Gini-based tree using a small labeled dataset, then refines leaf decisions with the LLM.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    LLMT achieves consistent and significant accuracy improvements, large token-cost savings, and interpretable outputs in the few-shot setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present LLM-to-tree distillation as a promising direction for building AI systems that reason over structured data.

    Source: §6 · p. 9

    LLMT depends on the LLM having relevant background knowledge; datasets outside its knowledge scope may yield inaccurate or ineffective rules and trees.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While Large Language Models (LLMs) possess rich world knowledge and impressive generalization capabilities, their direct application to tabular data classification is hindered by high inference costs and limited interpretability. In contrast, decision trees are fast and transparent but often underperform in low-data regimes. In this work, we propose a novel framework that bridges these paradigms by distilling LLM knowledge into interpretable decision trees under a few-shot learning setting. Instead of directly prompting the LLM to generate full trees, which is often unstable and inefficient, we develop a three-stage paradigm that prompts the LLM to generate rules and organize the rules into a tree. Experiments on multiple real-world tabular datasets demonstrate that our method achieves superior accuracy and interpretability with significantly lower prompting overhead compared to existing baselines.

1 Introduction

LLMT addresses the challenges of applying LLMs to few-shot tabular classification by distilling their knowledge into interpretable, efficient decision trees through a three-stage pipeline. The paper reports theoretical analysis and experiments showing accuracy, efficiency, and interpretability benefits.

1 Introduction

  • Direct LLM classification is limited by poor interpretability and inference inefficiency, while traditional trees need substantial labeled data to capture nonlinear feature interactions.LLM semantic knowledge from textified features could help with data scarcity, but combining it with efficient tree execution remains challenging in few-shot settings.Source: §1 · p. 1
  • Directly prompting an LLM to generate a full tree is unreliable, while recursively prompting reasoning paths, as in Tree-of-Thought (Yao et al., 2023), incurs substantial overhead.The introduction attributes full-tree instability to the complexity of long-context reasoning.Source: §1 · p. 1
  • LLMT generates structured classification rules, assembles them into a tree using Gini impurity on a small labeled dataset, then refines noisy leaf decisions.The three-stage design is presented as more stable and efficient than constructing reasoning chains or trees in a single pass or through recursive prompting.Source: §1 · p. 2
  • The paper provides a formal generalization analysis of LLMT under explicit structural assumptions to clarify its statistical properties.Source: §1 · p. 2
  • LLMT achieves consistent and significant accuracy improvements, large token-cost savings, and interpretable outputs in few-shot classification.The paper reports experiments across 11 datasets and 17 baselines.Source: §1 · p. 2

2 Background and Related Work

The section places LLMT at the intersection of interpretable decision trees, LLM-based tabular prediction, and structured prompting, emphasizing its separation of rule extraction from data-aware tree construction.

2.1 Decision Trees

  • Decision trees represent hierarchical if–then rules, with feature tests at internal nodes, outcome branches, and class labels at leaves; recursive splits can reduce impurity such as Gini impurity.The passage cites Loh (2011), Rokach and Maimon (2014), and Quinlan (1986, 1993) on decision trees.Source: §2.1 · p. 2

2.2 LLMs for Tabular Data

2.3 Prompting with Structured Reasoning

  • CoT prompting (Wei et al., 2022) decomposes tasks into intermediate reasoning steps, while ToT (Yao et al., 2023) branches across paths and SoT structures reasoning before parallel elaboration.Prior work reports that making intermediate reasoning explicit can improve LLM performance (Wei et al., 2022; Yao et al., 2023; Besta et al., 2024).Source: §2.3 · pp. 2–3
  • LLMT extracts rules from feature semantics, assembles them statistically with few-shot labeled examples, refines leaves, and produces a private, interpretable standalone tree without using the LLM as final classifier.Unlike GPTree (Xiong et al., 2024), it does not rely on many raw training samples to induce rules; raw data values are not sent to the LLM, and inference cost is low.Source: §2.3 · p. 3

3 Motivation

The motivation experiments show that LLM performance on tabular classification depends on dataset metadata, while discrete rule generation outperforms long reasoning-path generation in representative comparisons. These findings motivate generating rules and organizing them into trees rather than prompting for full reasoning paths.

Observation 1

  • On Diabetes and Spambase, LLM prediction accuracy substantially deteriorates when feature names and task descriptions are removed under zero-shot prompting.The paper extends the metadata comparison to six newer datasets to reduce potential influence from pretraining memorization, with full results reported in Appendix F.1.Source: §3 · p. 3, Figure 1 · p. 3

Observation 2

  • In representative results, rule generation achieves higher Set Utility and Set Accuracy than reasoning-path generation.The comparison used rule and path sets with the same total number of atomic feature conditions across eight datasets; the findings suggest LLMs struggle to produce coherent, complete hierarchical paths in few-shot settings but can generate accurate, interpretable individual rules.Source: Figure 2 · p. 3, §3 · p. 4

Motivation

  • The findings motivate generating discrete rules and organizing them, instead of asking LLMs to produce full reasoning paths that can be unstable, lengthy, and error-prone.Reasoning paths are series of rule-checks traversing a decision tree from root to leaf.Source: §3 · p. 4

4 Methodology

LLMT distills LLM classification knowledge into interpretable decision trees for few-shot tabular learning by generating rules, assembling them statistically, and refining leaf predictions.

4.1 Problem Statement

  • The goal is a decision tree that approximates the LLM’s classification capability while remaining interpretable and computationally efficient, given only a small number of labeled samples and access to LLM prompting.The setting reflects scarce labeled tabular data alongside the LLM’s background knowledge.Source: §4.1 · p. 4

4.2 Framework Overview

  • LLMT generates discrete rules, assembles them into a tree using few-shot data, and prompts the LLM to refine leaf predictions.The three stages are rule generation, layer-wise Gini-based tree assembly, and leaf refinement using rule-path analysis and label confidence scores.Source: §4.2 · p. 4

4.3 Rule Generation

  • The LLM receives task constraints and dataset feature information, then generates candidate classification rules with self-assessed confidence scores from 1 to 10.Rules use feature comparisons with numerical thresholds or categorical values; categorical features use one-vs-rest splits.Source: §4.3 · p. 4

4.4 Tree Assembly

  • At each node, LLMT groups unused path rules by confidence, selects the highest-confidence group’s best Gini split, and stops at maximum depth or when no rule has positive gain.Ties are broken by highest Gini gain and then lowest index, making selection deterministic.Source: §4.4 · pp. 4–5, §4.2 · p. 4

4.5 Leaf Refinement

  • LLMT changes a leaf label only when the LLM’s most-confident path-based label exceeds a threshold and differs from the current label.Its PAC-style bound identifies tree depth as the dominant complexity driver and the finite rule set as yielding a finite, statistically analyzable hypothesis space; the authors note the bound may be loose in extreme few-shot settings.Source: §4.5 · pp. 5–6

4.6 Ensemble Extension

  • LLMT Forest partitions features into disjoint, approximately equal-sized subsets, trains one LLMT tree per subset on the same few-shot samples, and aggregates predictions by majority vote.The ensemble is presented as an optional extension for high-dimensional datasets to increase capacity while reducing the overfitting risk of a single deep tree.Source: §4.6 · p. 6

5 Evaluation

Across few-shot tabular benchmarks, LLMT generally outperforms baselines while producing compact, high-quality decision trees. It also reduces prompting costs relative to several LLM-based methods, and leaf refinement improves accuracy in the tested ablations.

5.1 Experimental Setup

  • Evaluation spans 11 public tabular datasets, focuses on Diabetes, Spambase, and Nursery, and compares LLMT with zero-shot, conventional, ensemble, LLM-prompting, pretrained, and LLM-assisted baselines.Experiments use 10 trials with randomly sampled training sets and a fixed 100-sample test set; maximum tree depth is 3 or 4 and τ=0.7.Source: §5.1 · p. 6

5.2 Effectiveness

  • LLMT consistently outperforms almost all baselines, improving over the strongest baseline by more than 10% in many settings and gaining robustly over recent LLM-assisted methods.FeatLLM and DeLTa tend to overfit in extreme few-shot regimes, while GPTree can be affected by data-induced bias; LLMT also maintains its advantage on three datasets with over 100 attributes.Source: §5.2 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7

5.3 Efficiency

  • Compared with ToT-Tree, LLMT reduces token usage by 10–22× and construction time by 1.6–3.6×; it is also up to 10× faster than FeatLLM.LLMT has comparable training time and token usage to CoT-Tree and comparable computational cost to DeLTa, while using fewer tokens than GPTree.Source: §5.3 · p. 7, Table 4 · p. 8

5.4 Interpretability

  • LLMT produces the most concise trees on Diabetes and remains among the most compact on Nursery and Spambase, while achieving the lowest average Gini impurity on Diabetes and Nursery.On Spambase, its average Gini impurity is close to the best method.Source: Figure 4 · p. 8, Table 5 · p. 8, §5.4 · p. 8

5.5 Ablation and Sensitivity Study

  • LLMT is generally stable across τ values, with τ=0.7 usually best; Leaf Refinement contributes accuracy gains of 8% on Diabetes and 2% on Spambase.LLM verification can also identify same-label sibling leaves for merging, simplifying trees and mitigating spurious splits from limited few-shot data.Source: Figure 5 · p. 9, Figure 6 · p. 9, §5.5 · p. 9

6 Conclusion

The paper presents a framework that distills LLM knowledge into interpretable, efficient decision trees for tabular classification. It has LLMs generate discrete rules and statistically organizes those rules as a tree, avoiding direct generation of full decision structures.

  • The framework distills LLM knowledge into interpretable, efficient decision trees for tabular data classification.Source: §6 · p. 9
  • Rather than prompting LLMs to generate full decision structures, the method uses LLM-generated discrete rules and statistically organizes them as a tree.The paper identifies direct full-tree generation as unstable and costly in inference.Source: §6 · p. 9
  • The authors present this combination of LLMs and symbolic models as a promising direction for trustworthy AI systems that reason over structured data.Source: §6 · p. 9

Limitations

The method depends on the LLM having relevant background knowledge and inherits its biases, which can make distilled trees inaccurate or affect downstream decisions.

Limitations

  • The method may generate inaccurate or ineffective rules and trees when the dataset falls outside the LLM’s knowledge scope, and it inherits biases from the underlying LLM.These biases may influence extracted rules and downstream decisions.Source: Limitations · p. 9
  • The authors urge caution when deploying distilled trees in high-stakes applications and recommend future fairness-aware prompting and post-hoc auditing.Source: Limitations · p. 9

A Proof · B Algorithm

The proof relates rule selection to impurity reduction and bounds LLMT’s generalization through the size of its possible tree class. The algorithm describes LLMT’s three stages and an optional forest extension.

A Proof

  • Proposition 4.1 relates the selected rule’s split to expected impurity reduction, with equality when the split yields no information gain.The proof characterizes this equality as the absence of effective class separation and impurity reduction.Source: A Proof · p. 13
  • With probability at least 1 − δ, Theorem 4.2 bounds the generalization error of an LLMT tree built from K rules, n training samples, and maximum depth d.The supplied passage does not include the bound’s formula.Source: A Proof · p. 13
  • The proof counts at most P(K, 2d − 1) possible trees, where P(K, 2d − 1) = K!/(K − (2d − 1))!.Each tree corresponds to an ordered selection of distinct rules for the 2d − 1 internal nodes.Source: A Proof · p. 13

B Algorithm

  • LLMT comprises rule generation, tree assembly, and leaf refinement; its optional LLMT Forest extension builds independent trees and aggregates them by majority vote.The forest procedure also uses random feature-subspace and bootstrap sampling.Source: B Algorithm · p. 13

C Theoretical Time Complexity Analysis · D Step-by-Step Example of LLMT

LLMT’s runtime combines LLM calls for rule generation and leaf refinement with tree assembly, while a Diabetes example shows how generated rules are selected and leaf labels refined. The analysis finds LLM calls dominate practical runtime.

C Theoretical Time Complexity Analysis

  • For rule generation and leaf refinement on a complete binary tree of depth dmax, the passage reports worst-case total LLM-call cost at most (2^dmax + 1)TLLM.Source: C Theoretical Time Complexity Analysis · p. 14
  • Tree assembly takes O(Kndmax) time because at most K rules are evaluated for each example at each of at most dmax path nodes.Each rule evaluation checks its condition against an example in constant time.Source: C Theoretical Time Complexity Analysis · p. 14
  • The total time complexity is (2^dmax + 1)TLLM + O(Kndmax), and LLM-call cost dominates overall runtime in practice.Tree assembly takes less than 5% of total wall-clock time across datasets; K is set to max(10, 2^dmax − 1).Source: C Theoretical Time Complexity Analysis · p. 14

D Step-by-Step Example of LLMT

  • The Diabetes toy example uses 4 training samples, maximum depth 3, and refinement threshold τ = 0.7.These settings illustrate LLMT step by step.Source: D Step-by-Step Example of LLMT · p. 14
  • With only task metadata, the LLM generates confidence-scored meta-rules, including Glucose < 140, BMI < 30.0, Age < 30, and DPF < 0.5.Their respective confidence scores are 10, 9, 8, and 8.Source: D Step-by-Step Example of LLMT · pp. 14–15
  • Tree assembly selects BMI < 30.0 at the root by Gini gain, then recursively selects the highest-gain rule; leaf refinement uses path-conditioned LLM probabilities and changes labels only when confidence exceeds τ.In the example, the pure left leaf remains “no” with probabilities no: 0.700 and yes: 0.300, while the mixed right node continues splitting until purity or depth constraints are met.Source: D Step-by-Step Example of LLMT · p. 15

E Information of Datasets · F Experimental Settings

The paper evaluates a varied set of public tabular datasets, selecting Diabetes, Spambase, and Nursery as major datasets across experimental settings. The following section introduces the datasets and notes that experimental parameters and configurations are provided for reproducibility.

E Information of Datasets

  • The additional datasets cover abalone age, blood donation, iris species, breast cancer, and glioma grading.Source: E Information of Datasets · p. 15
  • Diabetes, Spambase, and Nursery are the major evaluation datasets, spanning binary and multiclass tasks and varied sample sizes and feature dimensionalities.Diabetes predicts diabetes onset, Spambase classifies email as spam, and Nursery predicts nursery application outcomes.Source: E Information of Datasets · p. 15
  • Communities predicts violent-crime risk tiers from 103 features, while Ecom has 119 coupon-redemption features and Myocardial has 111 clinical features.Ecom is derived from the Acquire Valued Shoppers data using the TabReD preprocessing pipeline; Myocardial identifies chronic heart failure after myocardial infarction.Source: E Information of Datasets · p. 15
  • The dataset statistics are listed in Table 6, and all datasets are publicly available.Source: E Information of Datasets · p. 15, Table 6 · p. 15
  • UCI-hosted datasets used in the paper are licensed under CC BY 4.0, permitting reuse and redistribution with proper attribution.Source: E Information of Datasets · p. 15

F Experimental Settings

  • The experimental-settings section states that hyperparameters, model configurations, and experimental settings are detailed to support reproducibility.Source: F Experimental Settings · p. 15

F.1 Experimental Details for Observation 1 E … G.4 Interpretability Analyses

Masking generally lowers LLM accuracy, and rule sets outperform path sets under matched condition budgets. On high-dimensional datasets, LLMT leads non-ensemble methods and LLMT Forest leads ensemble methods in average accuracy.

F.1 Experimental Details for Observation 1 E

  • Masking lowers accuracy in 22 of 24 comparisons, leaves one unchanged, and improves one, with an average reduction of 20.14 percentage points.The pattern on post-cutoff and synthetic datasets makes memorization of old benchmark records a less likely sole explanation, but masking cannot prove pretraining memorization is absent.Source: §F.1 · p. 16, Table 7 · p. 17

F.2 Experimental Details for Observation 2

  • Rule sets achieve higher mean Set Utility and Set Accuracy than path sets under both matched condition budgets, even when total atomic conditions are controlled.The comparison uses Qwen2.5-72B at temperature 0.8, budgets n = 3 and 4, and averages over 100 generations.Source: §F.2 · p. 16, Table 8 · p. 17

F.3 Dataset-Specific Parameters

  • Dataset-specific settings use max depth 3 or 4 across methods and test sizes of 30 for Breast and 100 for other datasets.The n_estimators column applies to XGBoost, Random Forest, and LLMT Forest.Source: §F.3 · p. 16

F.4 LLM Configuration Settings

  • All LLM-based methods use the Qwen2.5-72B backbone, while request intervals and timeouts vary with dataset workload and provider status.The API configuration is described as operational rather than fixed experimental hyperparameters.Source: §F.4 · p. 16

F.5 Method-Specific Settings; F.6 Experimental Running Parameters

  • Method-specific hyperparameters follow the dataset settings, and LLMT and LLMT Forest group meta-rules with the same confidence value together.For each training sample size, experiments randomly sample 10 different training sets.Source: §F.5 · p. 17, §F.6 · p. 17

F.7 Implementation of two Zero-shot Baselines

  • Direct-ZSDT follows its official code with matched prompt and parameter settings, while Step-ZSDT follows the authors’ method and uses a 0.9 probability threshold.The cited baselines are Direct-ZSDT (Knauer et al. (2025)) and Step-ZSDT (Carrasco et al. (2025)); Step-ZSDT code was not released.Source: §F.7 · p. 17

G.1 Accuracy; G.2 Comparisons with Additional Baselines

  • Accuracy means and standard deviations across shot counts are presented, with expanded comparisons against tree-ensemble and pretrained black-box baselines.Figure 8 presents the accuracy results, while Tables 13, 14, and 15 provide the additional baseline comparisons.Source: §G.1 · p. 18, §G.2 · p. 18

G.3 Results on High-Dimensional Datasets

  • LLMT achieves the highest average accuracy among non-ensemble methods, while LLMT Forest achieves the highest among ensemble methods on high-dimensional datasets.Step-ZSDT is omitted because its per-feature, per-node LLM queries are prohibitively expensive in high dimensions.Source: §G.3 · p. 18

G.4.1 Rule Diversity Analysis … I Prompting Templates

The appendix analyzes generated-rule feature coverage, illustrates learned trees, reports how tree-depth sensitivity varies with dataset characteristics, and documents the prompting templates used for LLMT and baseline methods.

G.4.1 Rule Diversity Analysis

  • Careful meta-rule prompt design uses constraints intended to encourage diversity in the generated rules.The supplied passage refers to those constraints but does not specify them.Source: §G.4.1 · p. 18

G.4.2 Visualized Tree Examples

  • Figures 9–16 show representative Diabetes trees from every compared generator; the LLMT example uses glucose as its principal decision feature and exposes thresholds and leaf predictions.These examples let readers inspect the learned decision logic.Source: §G.4.2 · p. 18, Figure 16 · p. 20
  • The Diabetes tree examples complement aggregate impurity and tree-size results, but a single medically plausible example is not quantitative evidence of clinical validity.The examples are presented as visualizations of learned decision logic, not as validation of clinical use.Source: §G.4.2 · p. 18

I Prompting Templates

  • The templates cover meta-rule candidates, supervised probability outputs for leaf labels, and reference I/O-Tree, CoT-Tree, and ToT-Tree baselines.Color coding distinguishes prompt-section titles, dataset-varying variables, system commands, and fixed input text.Source: I Prompting Templates · p. 20, Figure 17 · p. 23, Figure 19 · p. 25

G.4.1 Rule Diversity Analysis

  • 100% feature coverage is reported on six datasets, 83.3% on Spambase, and 64.8% on the 23-feature Glioma dataset.Coverage was measured across 10 meta-rule sets for each of eight datasets using the same rule-pool size K as the main experiments; the authors caution that coverage is not itself an interpretability score.Source: §G.4.1 · p. 18, Table 18 · p. 22

G.5 Sensitivity to Maximum Tree Depth

  • Table 19 reports that shallow trees (depth = 2-3) yield stable accuracy on low-dimensional datasets such as Diabetes, while depth = 4 improves splits and accuracy on higher-dimensional datasets such as Nursery.The authors conclude that maximum depth requires careful tuning because deeper trees can capture complex interactions when feature diversity is high.Source: §G.5 · p. 18, Table 19 · p. 22

Related papers

Loading 2610.10227v1…