PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Foundation Models for Agentic What-If Reasoning:Cost, Latency, and Governance in a Hybrid LLM+SLM Architecture

Sourish Dey, Aditya Kumar

arXiv:2609.16091v1cs.LG

TL;DR

Tabular foundation models offer near-zero-training prediction, but their inference latency creates a bottleneck in interactive agentic loops. This paper distills a TabPFN teacher into a compact student and evaluates it in a hybrid pipeline; the student retains reported benchmark performance while the hybrid pipeline reduces end-to-end latency and cloud token cost.

  • Problem

    Tabular foundation models provide near-zero-training prediction, but their inference latency creates a bottleneck in interactive, multi-step agentic loops.

    Source: §1 · pp. 1–2

  • Method

    The paper distills a TabPFN teacher into a compact feed-forward student and uses a hybrid pipeline with cloud tool selection and parsing, student prediction, and local response generation.

    Source: §4 · p. 4, §8 · p. 10

  • Results

    The student retains 95.4–100.5% accuracy and 96.8–100.0% AUC; the hybrid pipeline reduces end-to-end latency by 3.8–4.8× and cloud token cost by 2.1×.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Distilling into a feed-forward student eliminates serving-time attention overhead and provides a ≥2,200× per-query inference speedup after an offline training step of under two minutes.

    Source: §7 · p. 9

    The orchestrator uses gpt-4o-mini as a proxy for a cheaper gpt-4o-nano production tier, so absolute token prices and orchestrator latency will not transfer.

    Source: §7.1 · p. 10

Abstract

from arXiv · show

Tabular foundation models deliver strong zero-training predictive performance via in-context learning, but their high inference latency makes them impractical as hot-path decision backends in interactive agentic loops. We distill a TabPFN teacher into a compact feed-forward student across a business-decision simulation on UCI Adult and five OpenML benchmarks: the classification head compresses 53.2M parameters to 8,546 (6,220x); the deployed two-head loan pipeline compresses 111.4M parameters to 17,059 (6,532x). The student retains 95.4-100.5% accuracy and 96.8-100.0% AUC, with the lowest accuracy retention on credit-g at 95.4%; an alpha = 0 hard-label control shows that the teacher's soft targets provide a 2.1-7.0 AUC point gain.

1 Introduction

Tabular foundation models offer near-zero-training prediction but their inference latency burdens interactive agentic loops, while cloud-based orchestration can expose sensitive attributes. The paper evaluates distillation and a hybrid deployment split for preserving task quality while reducing latency, cost, and data exposure.

Motivation

  • Tabular foundation models such as TabPFN [14] use in-context learning by ingesting the training set at inference, without gradient-based fitting on the target dataset.Their inference cost scales with context size.Source: §1 · p. 1

Motivation and governance

  • A full-LLM decision pattern repeatedly sends sensitive attributes such as age, income, and marital status to external APIs, adding token latency and data-governance risks.The paper also audits data minimization under GDPR and subgroup fairness, finding that aggregate retention can hide widened race equalized-odds gaps under compression.Source: §1 · p. 2

Research question

  • The paper evaluates whether distillation and an asymmetric hybrid split can retain task quality while reducing latency, operational cost, and sensitive-data exposure.The cloud LLM is reserved for tool selection and parsing while prediction and response generation are hosted on-premises.Source: §1 · p. 2

Contributions

  • The 53.2M-parameter TabPFN classifier is distilled to an 8,546-parameter student (6,220×); the two-head loan pipeline compresses from 111.4M to 17,059 parameters (6,532×).Hard-label-only controls show a 2.1–7.0 AUC-point gain from teacher soft targets.Source: §1 · p. 2
  • 3.8–4.8× lower end-to-end latency, tool-call accuracy rising from 0.90 to 0.98, and 2.1× lower cloud token cost are reported for the hybrid pipeline on 50 what-if queries.The reported latency reduction is driven by removing TabPFN from the hot path.Source: §1 · p. 2

2 Related Work

The related work situates the paper at the intersection of knowledge distillation, tabular foundation models, and agentic workflow evaluation. It distinguishes compressing a fitted TabPFN teacher into an independent MLP from adaptation approaches that preserve the original architecture.

Knowledge distillation

  • The paper compresses a fitted TabPFN teacher into an independent lightweight MLP, comparing it with plain MLP baselines [9] and evaluating subgroup compression bias [16].Prior work includes distillation approaches for vision and language encoders and efforts to condense TabPFN context representations [24] [7].Source: §2 · p. 2

Tabular foundation models and adaptation

  • Distillation changes the operational footprint, unlike fine-tuning [17] [28] or parameter-efficient adapters such as LoRA [18], which preserve the model architecture and inference compute.TabPFN [14] treats tabular learning as in-context transformer inference.Source: §2 · p. 2

Agentic workflows and evaluation

  • The evaluation uses LangGraph for multi-step orchestration and an LLM judge with the RAGAS faithfulness/relevance triad [5].It also assesses small-model parsing capabilities zero-shot.Source: §2 · p. 2

3 Background: Distillation versus Fine-Tuning

This background contrasts fine-tuning, which adapts a fixed-capacity architecture, with distillation, which trains a separate compact student from teacher outputs and labels. The paper applies soft-target classification and a corresponding teacher-and-label regression loss.

3.1 Fine-tuning

  • Fine-tuning continues optimization of a pretrained fixed architecture on target-task data, retaining the checkpoint architecture and its inference-compute and memory requirements.LoRA [18] parameterizes updates with a low-rank decomposition while leaving original weights frozen.Source: §3.1 · p. 3

3.2 Distillation

  • Distillation trains a separate compact student to reproduce a fixed teacher’s output distribution alongside ground-truth labels, allowing a ∼53–111M-parameter transformer to be replaced by an 8.5K-parameter feed-forward network.Unlike fine-tuning, student and teacher need not share architectural families or internal representations.Source: §3.2 · p. 3
  • Classification uses softened teacher probabilities and hard labels, with temperature T = 2.0 and soft/hard mixing weight α = 0.7.For regression, the corresponding loss mixes mean-squared error against the teacher point prediction and the ground-truth label at α = 0.7.Source: §3.2 · p. 3

4 System Architecture

The hybrid pipeline reserves cloud orchestration for tool selection and scenario parsing, while using a distilled student for prediction and a local SLM for response generation. The pipelines are evaluated with matched schemas, prompt intent, and query sets; tracing instruments their operational metrics.

4 System Architecture

  • The hybrid architecture uses one cloud call for tool selection and structured parsing, a distilled student MLP for prediction, and a local Qwen2.5-3B-Instruct SLM for the final answer.The full-LLM baseline uses the cloud model for orchestration and final response generation and may require up to three calls per query.Source: §4 · p. 4
  • The pipelines share tool schema, system-prompt intent, and evaluation queries; the manipulated variables are the prediction model and whether response generation is cloud-based or local.The split keeps cloud handling for structured extraction, where zero-shot local SLM parsing has 58% tool-call accuracy versus 98% with cloud schema-constrained function calling.Source: §4 · p. 4

4.1 Observability

  • LangSmith tracing captures token use, prediction latencies, and local SLM generation, and writes offline evaluation metrics onto live traces for filtering and audit.Operators can inspect end-to-end trace spans for a query; the tracing layer is described as adding zero overhead when disabled.Source: §4.1 · pp. 4–5

5 Distillation Benchmarking on Structured Data

The section evaluates TabPFN distillation on an Adult-derived business simulation and five OpenML classification benchmarks. It reports compression and retained classification quality, tests the contribution of soft targets, and evaluates the loan regression head.

5.1 Datasets

  • The evaluation combines an Adult-derived business simulation with five OpenML classification benchmarks, including high-dimensional or imbalanced datasets.The simulated max_loan regression target is assigned only to rows with a positive income outcome and has no external ground truth.Source: §5.1 · p. 5

5.2 Training protocol

  • Classification runs use the scripts’ defaults, with no hint or embedding loss.The training protocol passage specifies the defaults of the distillation and classification-benchmark scripts unless a caption names an exception.Source: §5.2 · p. 5

5.3 Teacher and student architectures

  • A shared compact feed-forward student varies only in input width; the classifier uses a sigmoid output and the regressor a ReLU output.The student is evaluated against teacher-produced soft logits precomputed once per training run; the listed hidden-layer activations are ReLU.Source: §5.3 · pp. 5–6

5.5 Accuracy and AUC retention

  • 95.4–100.5% accuracy retention and 96.8–100.0% AUC retention span six classification datasets at 6,220×–11,611× parameter reduction.Retention is student metric divided by teacher metric; it is uncapped because small held-out test sets can yield nominal student scores above the teacher’s.Source: Table 3 · p. 6, §5.5 · p. 6

5.4 Parameter-count accounting

  • 6,532× compression reduces the primary loan predictor from 111,428,088 teacher parameters to 17,059 student parameters.The student has 8,546 classification parameters and 8,513 regression parameters; smaller-input datasets show higher compression because the student’s first layer scales with input width while teacher parameter count stays fixed.Source: §5.4 · p. 6

5.6 Hard-Label-Only Control Ablation

  • 2.1–7.0 AUC-point gains over α = 0 hard labels occur at canonical α = 0.7 across the six datasets.The hard-label baseline has the lowest ROC-AUC across all six datasets, with ROC-AUC increasing as α rises toward 0.7.Source: §5.6 · p. 6

5.7 Regression Head Evaluation and Soft-Target Control

  • In approval-conditioned evaluation, the student regressor has RMSE $88,115 and WMAPE 0.873; its unconditional WMAPE is 1.213.The approval-conditioned setting evaluates the student on its own predicted approvals; the teacher’s test-set RMSE is $75,737 and WMAPE is 0.712.Source: §5.7 · pp. 6–7
  • Unconditional test-set WMAPE falls from 1.266 at α = 0 to 1.107 at α = 1.0, with intermediate values 1.226 at α = 0.3 and 1.160 at α = 0.7.The authors report this as consistent with the classification findings and state that teacher supervision benefits both task heads.Source: §5.7 · p. 7

6 Hybrid Architecture: Token Cost, Latency, and Quality

Across the 50-query benchmark, the hybrid pipeline reduced latency and cloud cost while improving tool-call accuracy; its answers scored lower on faithfulness, context relevance, and answer relevance.

6.1 Tool-call and parsing accuracy

  • Both pipelines answered all 50 queries without an unhandled exception.Tool-call accuracy, parsing quality, and LLM-judge scores were hardware-independent.Source: §6.1 · p. 7
  • Tool-call accuracy reached 0.98 versus 0.90 for the full-LLM baseline using schema-constrained function calling; under a generic schema, exact-match accuracy was 0.58 for Qwen and 0.38 for gpt-4o-mini.The generic-schema parser ablation is distinct from the deployed interface with named parameter declarations.Source: §6.1 · p. 7

6.2 LLM-judge answer quality

  • Faithfulness favored the full-LLM pipeline, 0.83 versus 0.75; the hybrid formatter sometimes exposed raw feature keys or malformed unicode.The reported answer-quality scores use an LLM judge on faithfulness, context relevance, and answer relevance.Source: §6.2 · pp. 7–8

6.3 Cost and latency

  • 3.84×, 3.83×, and 4.82× faster across the three profiles, while total cost fell 2.11× on the shared-host benchmark.The latency comparisons are within each hardware profile; the two CPU hosts are different machines.Source: Table 6 · p. 8, §6.3 · p. 8
  • The 2.11× cost saving equals $0.106 per 1,000 queries; recovering a dedicated $500 GPU's cost requires approximately 4.71M queries, excluding power and maintenance.The authors state that marginal savings apply at enterprise scale or on existing infrastructure, while latency and governance benefits apply from query one.Source: §6.3 · p. 8
  • TabPFN plus the unaccounted residual made up 90–92% of full-LLM wall-clock time, while the hybrid orchestrator and formatter took longer on every profile.The latency decomposition uses measured spans additively; a crossed 2×2 experiment was not run.Source: §6.3 · p. 8, Table 7 · p. 9

7 Discussion: Production Hardening and Data Governance

The discussion weighs production benefits of distillation and local processing against formatting, subgroup-fairness, and data-governance concerns. It also bounds generalization by benchmark, hardware, and audit scope.

7 Discussion: Production Hardening and Data Governance

  • Distilling away in-context attention yields at least 2,200× per-query speedup after under two minutes of offline training, while matching teacher accuracy and improving on hard-label training.The reported soft-target advantage is 2.1–7.0 AUC points.Source: §7 · p. 9
  • Hybrid context- and answer-relevance scores are 0.78/0.94 versus 1.00/1.00, although both pipelines had 100% execution reliability across 50 queries.The local Qwen2.5-3B formatter is not task-fine-tuned and can expose raw feature keys; schema constraints or parameter-efficient fine-tuning are proposed remedies.Source: §7 · p. 9
  • Race equalized-odds difference widens from 0.431 to 0.490 (+14%) under compression, even as sex disparities remain stable and individual racial-subgroup true-positive rates improve.The authors warn that aggregate retention can mask subgroup effects and call for fairness audits in regulated deployments.Source: §7 · p. 9
  • The hybrid system keeps raw 14-column feature rows and predictions local, but free-text queries and customer_id still cross the boundary, so data minimization does not eliminate GDPR exposure.The text identifies both remaining items as personal data when identifiable or linkable to a person.Source: §7 · p. 9
  • LangSmith trace trees expose tool parameters, teacher–student prediction differences, and formatter output for operational review without rerunning offline evaluations.Evaluative scores are attached to production spans for auditor inspection.Source: §7 · p. 9

7.1 Limitations

  • Generalization is bounded because gpt-4o-mini proxies a cheaper tier whose prices and latency will not transfer, 50 queries underpower subtle language or judge differences, and max_loan is synthetic.Schema adherence is expected to transfer; the classification labels, by contrast, are the real Adult income indicator.Source: §7.1 · p. 10
  • Latency measurements on shared CPU and GPU development machines are contention-sensitive; additive span attribution was used, and the factorial student/teacher-by-formatter comparison was not run.The large full-LLM residual shrank from 67.88s on the shared CPU to 1.97s on the dedicated M5 Pro, consistent with contention.Source: §7.1 · p. 10
  • Distillation results use one seed, and fairness audits cover only sex and race in Adult groups with at least 20 test rows.Multi-seed weight-initialization sweeps are future work.Source: §7.1 · p. 10

8 Conclusion

The conclusion reports a compact student that retains most teacher performance and improves over hard-label controls, then places it in a hybrid agentic pipeline. Latency, cost, governance, and subgroup fairness results qualify the deployment picture.

8 Conclusion

  • The student retains 95.4–100.5% accuracy and 96.8–100.0% AUC, with 6,220× compression for the classifier and 6,532× for the two-head loan pipeline.Hard-label controls show a 2.1–7.0 AUC-point gain from teacher soft targets alongside monotonic regression improvements.Source: §8 · p. 10
  • The hybrid LangGraph pipeline reduces end-to-end latency by 3.8–4.8× and cloud token cost by 2.1×, while tool-call accuracy rises from 0.90 to 0.98.The latency reduction is attributed almost entirely to removing TabPFN from the hot path; fewer cloud calls reduce token cost rather than wall-clock time.Source: §8 · p. 10
  • Raw feature rows stay on-premises, but free-text queries and customer_id still leave the local boundary, and amortizing a dedicated GPU requires multi-million-query scale.The conclusion also reports a 58% zero-shot parsing accuracy for local SLMs under a generic schema.Source: §8 · p. 10
  • Race equalized-odds differences widen by 14% under compression despite individual true-positive-rate gains, motivating subgroup evaluation in distillation workflows.Sex disparities remain stable, and future work includes formatter fine-tuning, unlabelled transfer sets, and span-level redaction.Source: §8 · p. 10

A Extended Classification Benchmark Sweeps

The extended classification sweep broadens evaluation beyond the datasets in Table 3 and reports additional classification metrics on the same checkpoints.

A Extended Classification Benchmark Sweeps

B Extended Regression Benchmark Sweeps

The appendices extend regression evaluation and document the associated prompts, instrumentation, model accounting, artefacts, dependencies, and hardware. The regression results show dataset-dependent retention, with teacher supervision improving retention over hard-label controls across all five datasets.

B Extended Regression Benchmark Sweeps

  • The five-dataset sweep uses α = 0.9, T = 1.0, 200 epochs, and seed 42; all datasets have at most 10,000 rows, with topo-2-1 and coil-20 row-capped for tractable CPU inference.The datasets range from 11 to 1,024 raw features, and the two largest have 266 and 1,024 features.Source: B Extended Regression Benchmark Sweeps · p. 14
  • Table 10 accounts for student parameters using encoded input width, with classifier parameter count given as 64din + 2,210; the teacher’s stored parameter count stays fixed across datasets.The input width differs from raw column count because numeric features are standardized and categorical features are one-hot encoded.Source: Table 10 · p. 15
  • Teacher supervision improves WMAPE retention over hard-label controls on all five further OpenML regression datasets, from +2.3pp on cpmp-runtime to +21.9pp on coil-20.topo-2-1 reaches parity with the teacher; coil-20 has the weakest WMAPE retention at 24.4%.Source: Table 12 · p. 16, B Extended Regression Benchmark Sweeps · p. 17

C.1 Prompt Templates and Function-Calling Schemas

  • The scenario set contains 50 natural-language what-if questions generated from customer-attribute changes, while the local formatter turns tool output and the query into JSON and the cloud orchestrator selects tools and parses scenarios.The function schema specifies customer_id and structured feature modifications as parameters.Source: §C.1 · pp. 17–18

C.2 Observability Instrumentation and Trace Schema

  • Table 13 describes which LangSmith spans capture pipeline information and whether capture is automatic or requires an explicit annotation.Source: Table 13 · p. 19

C.3 Detailed PyTorch Layer-by-Layer Parameter Accounting

  • The SmallClassifier and SmallRegressor use the same 64- and 32-unit hidden layers with dropout p = 0.2, differing in output dimensions and activations; Table 14 gives layer-level parameter counts.Both architectures use din = 99 one-hot encoded features on the primary loan dataset.Source: §C.3 · p. 19, Table 14 · p. 20

D.1 Code, Data, and Artefacts

  • Code and evaluation artefacts are linked in the repository, but the extended benchmark raw results and source CSVs are not checked in.The classification, regression, and OpenML download scripts are identified as reproduction routes.Source: §D.1 · pp. 19–20

D.2 Environment and Dependencies

  • The environment uses uv with Python ≥3.11 and lists LangSmith and Outlines among its dependencies.LangSmith supports observability tracing and evaluation feedback; Outlines supports constrained JSON generation for the SLM ablation.Source: §D.2 · p. 20

D.4 Standalone SLM-as-Parser Ablation

The zero-shot standalone SLM parser reaches 0.58 exact-match accuracy, but remains below the hybrid pipeline’s 98%. On a dedicated Apple M5 Pro, the hybrid retains similar latency and cost ratios to the other reported comparisons.

D.4 Standalone SLM-as-Parser Ablation

  • The standalone ablation tested gpt-4o-mini against zero-shot Qwen2.5-3B-Instruct with constrained JSON decoding via Outlines on 50 examples across two runs.The evaluation asked whether a local SLM could parse scenarios and close the loop entirely on-premises.Source: §D.4 · pp. 20, 22
  • On the dedicated M5 Pro, hybrid-vs-full-LLM ratios remain 4.82× for latency and 2.15× for cost.The passage attributes the small cost difference from Table 6 to token-count noise, not hardware.Source: Table 15 · p. 21, Figure 6 · p. 21, Figure 7 · p. 21
  • 0.58 exact-match accuracy in both runs exceeds gpt-4o-mini’s 0.38 but remains below the hybrid pipeline’s 98%.Table 16 reports exact-match tool-call accuracy on extracted modification sets; the hybrid result is from Table 4.Source: §D.4 · p. 22, Table 16 · p. 22
  • The SLM often copied unmentioned applicant fields into modification JSON, supporting retention of cloud parsing while prediction and response formatting stay local.The passage identifies failure to isolate user-specified diffs as the parsing issue.Source: §D.4 · p. 22

E.1 Detailed Subgroup Fairness Protocol

The Adult fairness audit finds that subgroup gaps narrowed for sex but widened for race under student predictions. The accompanying validity accounting limits the audit to well-represented sex and race groups and notes other study boundaries.

E.1 Detailed Subgroup Fairness Protocol

  • Sex disparities narrowed (DPD 0.180 teacher vs. 0.176 student; EOD 0.093 teacher vs. 0.080 student), while race disparities widened (DPD 0.240 vs. 0.256; EOD 0.431 vs. 0.490, +14% relative).The sex groups comprised 679 female and 1,321 male applicants; the race audit included four groups with at least 20 test rows.Source: §E.1 · p. 22
  • Although true-positive rates increased across all racial groups, non-uniform changes widened the disparate-impact gap, so aggregate accuracy retention alone cannot validate compression safety in sensitive domains.The audit used demographic parity difference and equalized-odds difference across protected groups on the UCI Adult test set.Source: §E.1 · pp. 22–23

E.2 Detailed Accounting of Threats to Validity

  • The fairness audit covers Adult sex and race only; attributes with small samples were excluded to avoid deceptive small-sample statistics.The evaluation used fixed seed 42, with multi-seed weight-initialization sweeps left for future work.Source: §E.2 · p. 23
  • The what-if evaluation uses 50 queries, limiting statistical power for subtle natural-language variations across LLM judges; the synthetic max_loan target is an experimental simulation target.The orchestration proxy was gpt-4o-mini rather than the target gpt-4o-nano production tier, whose absolute pricing and latency differ.Source: §E.2 · p. 23
Loading 2609.16091v1…