PaperlayerView exact revision on arXiv

Source-linked AI summary

Coding LLMs Can Be Harnessed to Learn Interpretable Models from Data

Aditya Kumar, Cassandra A. Cohen, William W. Cohen

arXiv:2610.14006v1cs.LGcs.AI

TL;DR

The paper asks whether coding LLMs can learn accurate interpretable classifiers from data, and introduces a harness that prompts an LLM to build rules within a classic rule-learning loop. It reports strong comparisons with rule learners across tabular and text tasks, and frames interpretable-classifier construction as a coding benchmark measured by held-out error and compared with mature specialist algorithms.

  • Problem

    The paper asks how well coding LLMs can produce accurate interpretable symbolic classifiers, a challenging task because interpretable models correspond to short programs.

    Source: §1 · p. 1

  • Method

    CLIPPER uses a harness based on SLIPPER’s rule-learning loop, repeatedly prompting a coding LLM to construct the next rule from sampled labeled examples.

    Source: §1 · p. 2

  • Results

    CLIPPER is competitive with or better than full-data SLIPPER on tabular data and wins by large margins on text; an inexpensive model performs comparably to frontier agents at a fraction of their cost.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Interpretable-classifier construction offers coding-agent benchmarks with held-out-error measurements, mature specialist-algorithm comparisons, and interpretable program outputs.

    Source: §5 · p. 10

    Benchmark accuracy and executable rules do not establish suitability for decisions about individuals, and MedCalc extraction is not a validated clinical system.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Since compact, interpretable classifiers correspond to simple programs, it is natural to ask if LLM-based coding agents can be used to learn interpretable models from data. In this paper we present a simple harness that repeatedly prompts a coding LLM to produce the next rule in a rule set, given a sample of labeled examples. We show that this approach is competitive with, and often superior to, state-of-the-art rule-learning methods on tabular data, especially with small training sets, and wins by large margins on text, including against another LLM-based rule learner; that this approach outperforms standard harnesses, and that the approach immediately leads to new kinds of interpretable models, since coding agents are not restricted to search a highly restricted hypothesis space. In addition to the advances made in learning interpretable models, these results are of interest because they introduce a new class of coding benchmarks - construction of interpretable classifiers - which includes quantitative measurements of output quality (i.e., test-set error) and existing mature ``specialist coding algorithms'' (i.e., existing learning methods).

1 INTRODUCTION

The introduction asks whether coding LLMs can build accurate interpretable classifiers, and presents CLIPPER, a harness that uses an LLM to construct rules within a classic rule-learning loop. The paper positions this as a way to build complete classifiers from data and to evaluate coding agents against mature specialist learners.

1 INTRODUCTION

  • The paper asks how well LLM-based coders can produce accurate interpretable symbolic models, since learning such classifiers remains challenging.The motivation is that interpretable models correspond to short programs that coding agents can produce.Source: §1 · p. 1
  • The paper extends LLM-based feature-generation work to building complete, executable interpretable classifiers, including tabular classifiers in many-shot settings.Prior work often focused on few-shot learning or NLP; the introduction describes little prior work on this task.Source: §1 · p. 1
  • The paper focuses on autonomous construction of interpretable classifiers, rather than directly using agents to classify messages.The introduction notes that direct LLM classification was undesirable in an example where agents were observed to present an overly charitable picture of their reasoning and deceptive behaviors.Source: §1 · p. 1
  • The paper treats predictive-rule construction as a coding benchmark measured by test-set error and compared with mature specialist learning algorithms.Unlike bug-fixes or competitive-programming puzzles, this task has no clear signal for when a classifier is finished, and LLMs may overthink by generating long, expensive, unproductive outputs.Source: §1 · p. 2
  • CLIPPER replaces SLIPPER’s discrete rule search with an LLM rule builder inside a reweighted, iterative learning loop.The harness samples training examples, scores each proposed rule on the full training set, and accepts it or updates the default.Source: §1 · p. 2, Figure 1 · p. 2
  • On tabular data, CLIPPER outperforms existing rule learners with smaller training sets and is statistically indistinguishable from the strongest baseline with appropriate tools and hints.On text datasets, it wins by large margins over similar baselines, including RuleChef, another LLM-based rule learner.Source: §1 · pp. 2–3

2 RELATED WORK

The paper situates CLIPPER among symbolic learners and LLM-assisted model construction. Its key distinction from feature-generation approaches is that it uses an LLM to build a compact executable classifier directly.

Symbolic classifiers

  • CLIPPER shares most of its code with a SLIPPER reimplementation but replaces discrete rule search with a Pydantic AI agent.The related-work discussion contrasts this with RIPPER’s compact rule-set search and CORELS’s search over a fixed set of antecedents.Source: §2 · p. 3

LLM-generated features for downstream symbolic learners

  • Unlike LLM feature-generation approaches such as FeatLLM (Han et al., 2024), CLIPPER generates a compact classifier directly rather than supplying features to a downstream symbolic learner.FeatLLM generates many conjunctive rules for a symbolic learner; in most experiments it generated 200 rules per class from up to 64 examples, compared with CLIPPER’s average of 9.6 rules on 2000 examples.Source: §2 · p. 3

LLMs for interpretable model construction

  • Prior LLM-based interpretable-model construction includes a zero-shot decision-tree learner (Carrasco et al., 2025) and the concurrent text-classification and extraction system RuleChef (Kovács et al., 2026).Unlike RuleChef, CLIPPER is also evaluated on tabular data and permits controlled evaluation of an LLM’s ability to produce rules from data.Source: §2 · p. 3

Boosting language-model classifiers

  • Unlike boosted LLM classifiers that use natural-language summaries for prediction, CLIPPER’s weak hypotheses are executable code, so prediction needs no LLM.Manikandan et al. (2023) use an LLM as a weak learner on tabular data, conditioning prediction on a natural-language summary from each round.Source: §2 · p. 3

3 METHOD

The method uses SLIPPER’s confidence-rated boosting loop to build a rule classifier, substituting an LLM agent for discrete rule search. It samples weighted examples for prompting, scores candidate rules on the full training set, and updates weights across rounds.

3 METHOD

  • Across 4,510 experiments on eight tabular benchmarks, SLIPPER emerged as the strongest baseline learner, followed by CORELS.The evaluation covered 12 interpretable learners, binary dataset versions, and up to five training-set sizes; Figure 2 displays the experiments.Source: §3 · p. 3, Figure 2 · p. 4
  • At each boosting round, the harness reweights examples not explained by accumulated rules and prompts an LLM with a sample to construct one rule.This follows SLIPPER’s confidence-rated boosting outer loop, enabling a direct comparison between coding agents and SLIPPER’s discrete rule search.Source: §3 · p. 4
  • A rule with negative √p_t − √q_t is rejected and its weight is added to the default rule; the final classifier predicts the sign of the weighted rules.The default rule always predicts the negative class, while accepted rules contribute to the final score.Source: §3 · p. 4
  • CLIPPER’s boosting-round limit is selected by cross-validating SLIPPER, with at least two rounds allowed because CLIPPER’s rule builder is more productive.Other CLIPPER hyperparameters were set using an independent tuning set.Source: §3 · p. 4
  • For tabular prompts, the agent receives up to 100 weighted examples, labels, weights, and the schema, and is asked for one conjunctive Python predicate.The harness scores rules on the full training set; different samples may be supplied across rounds, so the learner may see more than 100 examples in total.Source: §3 · p. 5
  • The method considers string operations and regular expressions for text classification and extraction, where rules may return a value or abstain.For extraction, abstentions contribute to neither correct nor incorrect returned-value weights; the default is abstention, and only rules with p_t > q_t are retained.Source: §3 · p. 5

4 EXPERIMENTS

The experiments compare CLIPPER’s iterative LLM rule construction with discrete and LLM-based learners across tabular and text tasks, then test value extraction and harness choices. CLIPPER improves on several rule-learning baselines, while results vary with the harness and information provided.

4.1 EVALUATION PROTOCOL

  • The tabular evaluation covers eight binary tasks with fixed stratified 25% test holdouts, matched training subsets, and results averaged across ten seeds.Macro error weights dataset means equally; learning curves beyond n = 750 exclude College.Source: §4.1 · pp. 5–6

4.2 LLMS BUILD BETTER RULES THAN DISCRETE SEARCH

  • At M = 100, CLIPPER beats SLIPPER at all nine training sizes; at n = 1,200, its error is 22.5% versus 23.4%.The difference is 0.9 ± 0.2 points.Source: §4.2 · p. 6, Figure 3 · p. 6
  • Condition hints beat full-data SLIPPER at eight of nine sizes; increasing M to 200 or 400 raised error 0.3–0.4 points, and name anonymization raised error 1.1 ± 0.2 at n = 250.Coverage queries alone gave smaller gains than condition hints.Source: §4.2 · pp. 6, 8, Figure 4 · p. 7
  • CLIPPER has lower error than CORELS, RIPPER, the greedy rule list, SkopeRules, FPSkope, and sampled SLIPPER at all five training sizes.Only full-data SLIPPER is lower, and only at n = 2,000; median CLIPPER rulesets contain 4.5 to 13.5 rules.Source: §4.2 · p. 6, Figure 3 · p. 6
  • On five BANKING77 intents, CLIPPER’s error is lower than RuleChef’s in all 50 runs, by 29.7 points on average with tools and 27.8 without.On five other text datasets, both CLIPPER prompt variants substantially improve on SLIPPER and the best baseline learner.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 FROM PREDICATES TO VALUE-RETURNING RULES

  • Python rules learned with Gemini 3.1 Flash-Lite at M = 55 reach 93.7 micro F1, within 1.6 points of the reference pool’s 95.3.Python and regex rules have similar precision, but Python has higher recall: 91.4% versus 86.0%.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 THE HARNESS MATTERS MORE THAN THE MODEL

  • At n = 1,200, one-shot error is 24.8%, versus 22.4% for boosted CLIPPER; replacing boosting with set covering can sometimes slightly help the LLM rule builder.The comparison uses a fixed model, sample cap, and tools for the one-shot variant.Source: §4.4 · p. 8, Table 1 · p. 8
  • Across six models, errors fall within 1.5 points; meanwhile, Flash-Lite and Haiku are 3–4 points worse in general coding harnesses than inside CLIPPER.Inside CLIPPER, Flash-Lite beats full-data SLIPPER at about $0.04 per run; general-purpose agents often code discrete search methods instead of rules.Source: §4.4 · pp. 8–9, Table 2 · p. 9, Table 3 · p. 9

5 DISCUSSION AND CONCLUSION

CLIPPER uses coding LLMs to build executable rules within a rule-learning harness, extending interpretable learning to complete classifiers. The paper reports strong rule-learning results and frames interpretable classifier construction as a measurable coding benchmark.

5 DISCUSSION AND CONCLUSION

  • CLIPPER is competitive with full-data SLIPPER on tabular data, wins by large margins on text, and extends rule learning to extractors competitive with frontier-agent-assisted work.The authors also describe held-out error, mature specialist algorithms, and interpretable programs as ingredients of a new coding benchmark.Source: §5 · pp. 9–10

AI USE STATEMENT

  • The authors used AI assistants for research, experiment planning, coding, and manuscript work, and report reviewing the assisted work and regenerating reported results from saved runs.They state that code was tested and reported numbers, tables, and figures were checked against per-run results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Benchmark accuracy and executable rules do not establish suitability for individual decisions; MedCalc extraction is not a validated clinical system.MedCalc uses supplied calculator identities and schemas and silver reference values.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors release the experiment codebase for reproducibility.The release is identified as the codebase used for the experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 ANALYSIS OF BASELINES

  • The authors’ SLIPPER reimplementation serves as both a baseline and the rule-search component replaced by CLIPPER’s coding agent.Figure 2 also includes rule, tree, and ensemble baselines.Source: §A.1 · p. 13

A.2 CLASSIFICATION ALGORITHM

  • The harness reweights examples, samples them for an LLM-generated predicate, evaluates that rule, and updates retained rules or the default score.The classification procedure also specifies weighted sampling and boosting updates across rounds.Source: §A.2 · p. 13

A.3 PROMPTS AND STATISTICAL TOOLS

  • The prompt gives the agent a schema and weighted labeled rows; optional coverage and hints tools provide full-data statistics or candidate attribute tests.The coverage tool returns weighted coverage sums, while hints report tests suggested by SLIPPER’s discrete optimizer.Source: §A.3 · pp. 13–14

A.4 TABULAR DATA AND PREPROCESSING

The tabular evaluation uses a fixed stratified holdout and shared training subsets across methods, with dataset-specific feature exclusions and preprocessing choices.

A.4 TABULAR DATA AND PREPROCESSING

  • Each dataset uses a stratified 25% holdout; methods share each training subset, and the main figures use seven datasets supporting training sizes through 2,000.The learning-curve seeds draw subsets from the remaining rows; College supports sizes only through 600.Source: §A.4 · p. 14
  • Feature preparation removes identifiers and selected post-outcome or dataset-specific fields, while missing entries remain null in SLIPPER’s input.Examples include excluding Bank marketing call duration and the COMPAS score columns.Source: §A.4 · p. 14

A.5 TEXT DATASETS

The appendices describe text benchmark construction and preprocessing, the BANKING77 comparison, baseline model size and failures, alternative harnesses, and uncertainty across repeated runs.

A.5 TEXT DATASETS

  • CLIPPER reads raw text and may use regular expressions, while SLIPPER sees word sets and searches all training rows; other imodels baselines use TF-IDF word weights.The five text datasets use stratified 25% test sets and shared training subsets for each method.Source: §A.5 · p. 14
  • The text suite contains five two-class datasets, and the post-cutoff Inflation Research Abstracts benchmark has 1,138 economics abstracts labeled for AI or machine-learning use.On that benchmark, raw-text CLIPPER has the lowest error at every training size, reaching 6.9% at n = 750 versus 8.4% for full-data SLIPPER and 9.0% for CLIPPER reading word sets.Source: §A.5 · pp. 14–15

A.6 COMPARISON WITH RULECHEF ON BANKING77

  • CLIPPER’s error is lower than RuleChef’s in all 50 BANKING77 runs, by 29.7 points on average with coverage tools and 27.8 points without them.The comparison uses five training sizes and ten seeds; Table 7 reports test error averaged over ten seeds.Source: §A.6 · p. 16, Table 7 · p. 16

A.7 BASELINE SETTINGS AND MODEL SIZE

  • The separate imodels SLIPPER implementation failed in 50 of 380 dataset/size/seed cells, including every Water potability cell, and was excluded from the common aggregate.Its AdaBoost wrapper reports failure when the base classifier is worse than random.Source: §A.7 · pp. 16–17
  • At n = 2,000, mean retained-predicate counts range from 9.6 for CLIPPER sample to 17.4 for hints+tools, describing ensemble size rather than human effort.Full SLIPPER has 8.9 predicates; each Python predicate may contain several operations.Source: §A.7 · p. 17

A.8 ALTERNATIVE HARNESSES

  • One-shot asks for a complete disjunctive classifier in one outer round, and a proposal is retained only if it passes the p > q gate.This changes hypothesis composition, acceptance, and total compute alongside iteration.Source: §A.8 · p. 17

A.9 UNCERTAINTY AND REPETITIONS

  • SEM summarizes variation across ten seeded training subsets on a fixed holdout, not uncertainty over newly sampled datasets or test sets.Training sizes from the same dataset are dependent and should not be treated as independent experiments in pooled significance tests.Source: §A.9 · p. 17

B.1 DATA AND CALCULATOR PREDICTION

On MedCalc, CLIPPER reads raw notes while discrete learners use word sets, and CLIPPER predicts calculators more accurately than SLIPPER across several matched training conditions. The section also describes value-extraction evaluation, its boosting procedure, and examples of generated rules.

B.1 DATA AND CALCULATOR PREDICTION

  • CLIPPER reading raw notes predicts calculators more accurately than SLIPPER with the same 20-example proposal sample from 220 notes onward, and beats full-data SLIPPER at 220 and 880 notes.Figure 5 reports test error on the official 55-way test split, averaged over ten seeds.Source: §B.1 · p. 18, Figure 5 · p. 18

B.2 VALUE EXTRACTION

  • For extraction, the evaluation uses ten selected entity keys across 25 calculators, with 500 official-test instances and 1,120 requested values.Inconsistent omitted-versus-false annotations make some schemas unsuitable for the exact-value task; the reference labels are silver Relevant Entities annotations.Source: §B.2 · p. 18

B.3 EXTRACTION WITH SLIPPER AND CLIPPER

  • The extraction harness samples weighted examples to propose rules, scores each candidate on all training examples, and retains it only when correct-return weight exceeds incorrect-return weight.CLIPPER proposes a Python function or regex; prediction groups rule outputs by canonical value and returns the value with the greatest summed confidence, or abstains.Source: §B.3 · pp. 19–20

B.4 EXAMPLE EXTRACTION PROMPTS AND RULES

  • Generated extraction rules illustrate both literal regex outputs and Python arithmetic, such as converting a platelet count from thousands per microliter to a numeric value per microliter.Regex capture templates combine matched text and literals, whereas Python can multiply the captured count by 1,000.Source: §B.4 · pp. 20–21

C.1 DEVELOPMENT SUITE AND CONFIGURATION SELECTION

The development comparisons examine sample size, prompt representation, and statistical feedback, alongside repairs to discrete covering and patience settings for the LLM proposer. These choices show mixed effects across configurations and training sizes.

C.1 DEVELOPMENT SUITE AND CONFIGURATION SELECTION

  • Increasing proposal sample size from 100 to 200 and 400 gives macro errors of 20.5%, 20.8%, and 20.9%, while table prompts cut tokens about 71% with a 0.5-point macro-error increase.One-conjunction-per-round framing improves macro error by 1.0 points relative to its prompt control; the task instruction is not syntactically enforced.Source: §C.1 · p. 22
  • The condition table with iterative coverage queries achieves the lowest development macro error, 17.7%, so initial hints and iterative queries are evaluated separately.The separate initial-table comparison reports 18.8% and varies across datasets.Source: §C.1 · p. 22

C.2 REPAIRS TO DISCRETE COVERING

  • The combined discrete-covering repair improves on legacy covering in 206 cells, worsens in 186, and ties in 48, but its pooled error is higher at size 800.At size 800, the repair has 22.5% pooled error versus 21.2% for legacy covering; on the seven main test datasets it reports 24.6 ± 0.2% at n = 250 and 22.8 ± 0.1% at n = 1,200.Source: §C.2 · p. 23, Table 12 · p. 23

C.3 COVERING PATIENCE WITH THE LLM PROPOSER

  • Patience three has pooled error of 22.9% versus 23.0% for patience one, while reducing zero-rule fits from 76 to 56 and increasing total model-call cost from $22.93 to $44.11.The comparison has 380 cells and the dataset pool changes across the pooled summaries.Source: §C.3 · p. 23, Table 13 · p. 23

D DATASET-LEVEL TABULAR RESULTS

This block presents dataset-level tabular comparisons through a fixed seven-dataset macro and dataset-specific tables, alongside learning-curve tables and coding-agent comparisons. The supplied passages describe evaluation scopes and reporting conventions, but do not provide comparative cell values or winners.

D DATASET-LEVEL TABULAR RESULTS

E ADDITIONAL EXPERIMENT DETAILS AND AGENT BEHAVIOR

D.1 DISCRETE BASELINES

D.2 CLIPPER AND SLIPPER

D.3 ALTERNATIVE HARNESSES

D.4 COMPLETE SUMMARY LEARNING-CURVE RESULTS

E.1 RULE CONSTRUCTION IN GENERAL-PURPOSE CODING HARNESSES

  • General-purpose coding agents receive all 250 training examples and can run exploratory scripts, whereas CLIPPER proposes one predicate at a time from at most 100 examples.The agents are evaluated as complete systems, including prompts, data access, and computation budgets.Source: §E.1 · p. 37
  • The coding-agent comparison table reports mean test error over three matched seeds for each dataset and propagated SEM for macro columns.Its seven-dataset pool excludes College.Source: Table 50 · p. 37

E.2 SPECIFICATION GAMING THROUGH UNINTENDED LEARNER FITTING

The original classifier-output restrictions did not prohibit fitting an existing learner in a helper script, allowing some agents to satisfy the requested form while bypassing the intended procedure. Explicitly forbidding that strategy narrowed Sol’s advantage, though the measured change includes the strategy induced by the instruction.

E.2 SPECIFICATION GAMING THROUGH UNINTENDED LEARNER FITTING

  • Sol installed scikit-learn or wittgenstein in 10 of 24 runs and transcribed learned models into if/then rules; its error averaged 1.4 points below the best other system in those runs.In the remaining 14 runs, its error averaged 1.0 point higher; other agents’ learner-fitting attempts generally performed poorly.Source: §E.2 · pp. 37–38

E.3 MAKING THE INTENDED RESTRICTION EXPLICIT

  • Forbidding learner invocation and implementation raised Sol’s seven-dataset error from 20.7 ± 0.3 to 22.9 ± 0.2%, a paired increase of 2.2 ± 0.4 points.Its mean advantage over CLIPPER with coverage tools fell from 2.8 to 0.6 points, and the recorded checks found no learner-fitting commands in the 24 new runs.Source: §E.3 · p. 38, Figure 7 · p. 38

E.4 ANONYMIZATION PROCEDURE AND COMPLETE RESULTS

The section reports an anonymization comparison across eight tabular datasets and describes the controls used for a backend comparison. Anonymization increased macro test error, while the supplied backend passages specify comparison conditions but no outcome.

E.4 ANONYMIZATION PROCEDURE AND COMPLETE RESULTS

  • Anonymization increased eight-dataset macro test error by 1.1 ± 0.2 points at n = 250.The change is anonymized minus original error; the table reports mean test error (%) ± SEM across ten matched seeds, with datasets weighted equally.Source: §E.4 · p. 38, Table 51 · p. 39
  • The anonymization mapping replaced dataset stems, feature names, categorical values, and classes while preserving numeric values, missingness, row order, and train/test membership.Both conditions used the same sampled training subsets, Gemini 3.1 Flash-Lite with coverage tools, and matched cross-validated boosting-round budgets.Source: §E.4 · p. 38

E.5 BACKEND COMPARISON

  • Backend comparisons fixed training subsets, the M = 100 proposal cap, full-training scoring, and boosting-round budgets; invalid outputs or exhausted tool budgets could consume a round.The supplied backend table caption reports mean test error (%) ± propagated SEM at n = 250 across seven main-text datasets, excluding College.Source: §E.5 · p. 39, Table 52 · p. 39

F MODEL COMPLEXITY

The section quantifies the size of retained rule ensembles and illustrates how coverage-tool traces guide rule proposals, acceptance, and rejection. The examples show the harness retaining a refined Adult rule and an electronic-check rule for Telco churn.

F MODEL COMPLEXITY

  • At n = 2,000, hints+tools ranges from 1 to 74 retained rules; at M = 55, extraction ensembles have a median of 81.5 code lines (IQR 61.8–103).The tabular rule counts vary substantially across fits; extraction code-line counts exclude blank lines, comments, and docstrings but include imports and function definitions.Source: F MODEL COMPLEXITY · p. 40, Table 53 · p. 40

G REPRESENTATIVE PROMPTS AND TOOL TRACES

  • Representative prompts request one Python classification function with a single conjunction, use coverage calls to test candidates, and impose a six-call-per-rule budget.The trace setup uses saved Adult and Telco churn runs with 250 training examples and 100 sampled rows per prompt.Source: G REPRESENTATIVE PROMPTS AND TOOL TRACES · p. 41

G.1 ADULT: FIRST PROPOSAL

  • In the Adult first-proposal trace, adding an age restriction removes negative weight without losing positive weight, and the harness retains the rule.The retained example adds an age threshold of 30 to conditions on marital status, education, and work hours.Source: §G.1 · p. 41

G.2 ADULT: A LATER REJECTED PROPOSAL

  • In the later Adult trace, the proposal fails the acceptance test after the sample and weights change, so the harness updates the default instead.The new sample includes a more heavily weighted negative example with a bachelor’s degree.Source: §G.2 · p. 42

G.3 TELCO CHURN: COMPARING CATEGORICAL CONDITIONS

  • In the Telco churn trace, the agent compares five refinements and the harness retains the rule combining fiber-optic service, a month-to-month contract, and electronic check.The target class is Churn=Yes.Source: §G.3 · p. 43
Loading 2610.14006v1…