PaperlayerView exact revision on arXiv

Source-linked AI summary

Typed Decision Models on Tabular Data

Baowen Zhang, Hangting Ye, Hongyuan Zha

arXiv:2610.19536v1cs.CLcs.AIcs.LG

TL;DR

Typed decision models return probabilities for fixed answers, but their robustness on tabular data has not been established. Across three tasks and content-preserving rewrites, they perform as well as LLMs at lower cost, while table organization remains a source of changed decisions.

  • Problem

    Although LLM decisions on tables are known to depend on serialization and row or column order, decision models had not been studied as a tabular method.

    Source: §1 · p. 2

  • Method

    The study compares decision models and LLMs on three tabular tasks by sending requests in their original form and under content-preserving rewrites.

    Source: §1 · p. 2, §9 · p. 8

  • Results

    Decision models perform as well as LLMs at lower cost; table organization affects both, and about a third of row decisions change under some rewrite.

    Source: §1 · p. 2, §9 · p. 8

  • Takeaways & Limitations

    Decision models are practical for fixed-answer tabular decisions, especially when many rows need decisions, question wording varies, or uncertain cases need identifying.

    Source: §8 · p. 8

    No tested model is robust to table organization, the main limitation of decision models on tables identified by the authors.

    Source: §8 · p. 8

Abstract

from arXiv · show

Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.

1 Introduction

Prior work finds that LLM decisions on tables can vary with serialization and question wording, while typed decision models avoid text generation but their robustness on tabular data is unstudied. This paper compares decision models and LLMs across tabular tasks and content-preserving rewrites.

1 Introduction

  • Prior work reports that LLM decisions can vary with table serialization, row and column order, and question wording.LLMs serialize rows or tables into text and then generate an answer that must be mapped to an option.Source: §1 · pp. 1–2
  • Typed decision models return a probability for each fixed answer in one request instead of generating text, but their robustness to table and question wording is open.The paper describes this model family as avoiding generation and decoding for each decision.Source: §1 · pp. 1–2
  • The study evaluates four decision models and three LLMs across row classification, fact verification, anomaly detection, and 21 rewrite types.Rewrites preserve the information needed for a decision, including changes such as renaming options or using another table format.Source: §1 · p. 2
  • Decision models perform as well as LLMs at lower cost, but table organization changes many decisions and low-probability decisions change most often.The introduction reports that the table can be written differently even when accuracy stays almost the same.Source: §1 · p. 2

2 Experimental setup

The study compares typed decision models, which return probabilities over fixed answers, with LLMs that generate text for the same tabular decisions. It evaluates three task types and tests content-preserving rewrites of requests.

2.1 Decision interfaces

  • A decision model receives a state and fixed-answer questions, returning probabilities for every option; an LLM generates text that is mapped to an option.The request may include several items, and the model output for one item is a decision.Source: §2.1 · p. 2

2.2 Tasks and models

  • The tasks are row classification, fact verification on TabFact, and anomaly detection on 20 ReTabAD sources.Row classification uses ten datasets from TabLLM and FeatLLM.Source: §2.2 · pp. 2–3, Table 2 · p. 3
  • The comparison covers four decision models and three LLMs, with row-classification requests grouped as 16 typed questions and LLM prompts containing one row.For fact verification and anomaly detection, every model receives the same claim or batch of 15 samples.Source: §2.2 · p. 3

2.3 Evaluation

  • Performance is measured by Macro-F1, accuracy, and AUROC, while latency and price are also reported.Rewrites preserve decision-relevant information; the evaluation compares flips under rewrites with each model’s re-send rate for identical requests.Source: §2.3 · p. 3

3 RQ1: Are decision models practical for tabular decisions?

Decision models perform as well as LLMs or better on the three tabular tasks, with generally lower latency and price per decision. In-context examples help, but models trained on sufficient labelled rows can match or outperform in-context approaches.

3 RQ1: Are decision models practical for tabular decisions?

  • Decision models perform as well as LLMs or better on two of three tasks; Jev leads on TabFact, Kev-27B on anomaly detection, and GPT-6 Luna on row classification.GPT-6 Luna Decisions comes within 1.3 points of GPT-6 Luna on row classification and TabFact, and is 3.2 points above it on anomaly detection.Source: §3 · pp. 3–4, Table 3 · p. 4
  • With as many rows as fit in context, Clef and Kev-27B beat every LLM on row classification and every LLM and same-sample detector on anomaly detection.From about 64 labelled rows, TabPFN-3.5 and methods trained on the same rows match or beat every in-context model on row classification.Source: §3 · p. 4, Figure 2 · p. 4
  • GPT-6 Luna costs 1.6 to 8 times as much per decision as Jev; the Decisions API version costs less on row classification and anomaly detection.The same API interfaces cost the same per decision on TabFact.Source: §3 · p. 4
  • Decision models are practical for tabular decisions, but a tabular foundation model or trained model is better when around 64 or more labelled rows are available.The paper characterizes decision models as generally faster and cheaper per decision than LLMs.Source: §3 · p. 4

4 RQ2: Are decision models robust to the formulation of the question?

Decision models are generally robust to option names and order, though GPT-6 Luna Decisions is a notable exception. Paraphrasing instructions affects both model types, while yes/no reformulation can disrupt some LLM outputs.

4 RQ2: Are decision models robust to the formulation of the question?

  • Renaming or reordering options changes 2–8% of Jev, Clef and Kev-27B classifications, versus 14–30% for open-weight LLMs; Macro-F1 changes by at most 1.3 points for the decision models.The decision models’ classification changes are at most 5.5 percentage points above their re-send rates.Source: §4 · p. 4
  • Yes/no reformulation changes the three decision models’ results by at most 2 points but costs Qwen3.8-27B 14 points; half of Llama-3.3-70B’s outputs are unusable.Clef sorts options, accounting for its apparent robustness to option order.Source: §4 · pp. 4–5
  • Paraphrasing instructions affects both model types; on TabFact, these rewrites change under 5% of decision-model verdicts and under 7% of open-weight LLM verdicts.Clef and Kev-27B change fewer decisions under these rewrites than Qwen3.8-27B, which shares their backbone.Source: §4 · p. 5
  • Reversing options changes 23% of GPT-6 Luna Decisions classifications, 12% of its TabFact verdicts and 22% of its anomaly decisions; 92–100% of flips move to the newly first option.Even among classifications with probability of at least 0.8, 14% flip.Source: §4 · p. 5

5 RQ3: Are decision models robust to how the table is organized?

Changing table format or organization alters decisions across models, including decision models and LLMs alike. These changes can affect many individual decisions while leaving aggregate accuracy nearly unchanged.

5 RQ3: Are decision models robust to how the table is organized?

  • Writing each row as a list, sentences, a JSON object, or a Markdown table instead of an array of values changes 12–21% of every model’s row classifications.This is about ten times the re-send rate of the open-weight models.Source: §5 · p. 5
  • On TabFact, table formats and row or column orders change 3–13% of decision-model verdicts and 4–12% of open-weight LLM verdicts, with no difference between model types.Clef and Kev-27B change about as many decisions as Qwen3.8-27B, which shares their backbone.Source: §5 · p. 6
  • Presenting the same in-context examples as interleaved classes or text lines changes 7–12% of decision-model classifications, 7–8% of Qwen3.8-27B’s and 15% of Llama-3.3-70B’s.Macro-F1 changes by at most 3 points; GPT-6 Luna’s 14% change is close to its re-send rate.Source: §5 · p. 6
  • Some rewrite changes about a third of every model’s row classifications and a fifth or more of TabFact claims and anomaly samples, while Macro-F1 changes by less than a point for some models and up to 7 for others.The flips fall on borderline items that are right about as often as wrong, so accuracy largely hides the changes.Source: §5 · p. 6

6 RQ4: Does their confidence indicate which decisions are sensitive to rewrites?

Decision-model confidence ranks decisions likely to change under table rewrites, and re-querying only the least-confident rows can recover much of the gain from re-querying all rows. LLM confidence is a weaker indicator, while stable errors remain difficult to identify from confidence.

6 RQ4: Does their confidence indicate which decisions are sensitive to rewrites?

  • Decision-model confidence is the chosen option’s probability; LLM confidence is estimated from ten sampled answers agreeing with the original decision, with stated confidence and first-token probability also compared.Flip AUROC evaluates how well low confidence ranks flipped decisions; 0.5 is chance.Source: §6 · p. 6
  • 0.82–0.87 (row classification), 0.81–0.90 (TabFact), and 0.84–0.93 (anomaly detection) are the decision models’ flip AUROC ranges.GPT-6 Luna Decisions has the lowest flip AUROC on every task.Source: §6 · pp. 6–7
  • Only 0.1–5% of the highest-confidence decision-model items change under some table rewrite, versus about a third of all items; GPT-6 Luna Decisions reaches 15.5% on TabFact.Source: §6 · p. 6
  • Re-querying the least-confident rows undoes most of a weak format’s effect: the array is weakest for six of seven models, while the best other format alone is within 0.2 Macro-F1 points of or exceeds the combination.Figure 4 reports the share of the Macro-F1 gain obtained when every row is re-queried.Source: §6 · p. 7, Figure 4 · p. 7
  • LLM confidence ranks flips less well than Jev, Clef, and Kev-27B on every task, and its stated confidence is the least informative readout.The best LLM readout comes within 0.02 of the weakest of those decision models on row classification and TabFact, but remains 0.05 behind on anomaly detection and requires ten decoding passes or token probabilities.Source: §6 · p. 7
  • Stable but wrong decisions retain almost as much confidence as stable correct decisions on row classification and anomaly detection, so confidence tracks stability better than correctness.Stable errors account for 17–23% of decisions on those tasks and 4–5% on TabFact; Llama-3.3-70B confidence is high in all four groups.Source: §6 · p. 7
  • A rewrite flips a decision when its margin falls below zero; small, nearly item-independent margin shifts predict observed decision-model flip rates decile by decile.The account explains why borderline decisions flip while accuracy changes can cancel, and why confidence predicts flips.Source: §6 · p. 7

7 Related work

Prior work studies LLMs on individual table tasks and their sensitivity to prompt and table changes, while emerging decision-model studies leave tabular comparisons across tasks and individual decisions largely unexamined.

7 Related work

  • LLMs have been applied to row classification, anomaly detection, and TabFact verification, but prior task-specific studies report accuracy rather than comparing models across tasks and individual decisions.Source: §7 · pp. 7–8
  • LLM outputs vary with option and prompt wording, table serialization, and row or column ordering; prior single-prediction studies also examine uncertainty and inconsistency.The present work measures which individual table decisions rewrites change for both LLMs and probability-returning models.Source: §7 · p. 8
  • Early decision-model studies examine text accuracy, calibration, and consistency, while other work covers single tabular applications and audits missing tests.This paper compares decision models, including one built on an LLM, with LLMs across tabular tasks.Source: §7 · p. 8

8 Discussion

Decision models are practical for fixed-answer tabular tasks, especially when many rows need decisions or question wording varies. They are less suited to settings with many labelled rows or answers requiring free text.

When to use decision models?

  • Decision models suit fixed-answer tabular decisions, especially when many rows need decisions, question wording may vary, or uncertain cases need identification.They return confidence with each decision and are described as faster and cheaper per decision than LLMs.Source: §8 · p. 8
  • With many labelled rows, tabular foundation models and trained models perform better, while decision models’ shorter context windows limit how many labelled rows they can read.Source: §8 · p. 8

When to use LLMs?

  • LLMs suit table tasks requiring token-generated answers, such as explanations or open questions, and can be a reasonable starting point when almost no labelled rows exist.Source: §8 · p. 8

Future directions

  • Training decision models on table rewrites and combining them with LLMs to explain or review low-confidence decisions are proposed future directions.Source: §8 · p. 8

9 Conclusion

The study finds typed decision models perform as well as LLMs on three tabular tasks at a fraction of the cost. Their robustness to table organization remains unresolved.

9 Conclusion

  • Decision models perform as well as LLMs on row classification, fact verification, and anomaly detection at a fraction of the cost.Source: §9 · p. 8
  • Except for GPT-6 Luna Decisions, renaming or reordering options barely changes decision-model outputs.Source: §9 · p. 8
  • About a third of row decisions change under some table rewrite, and low-probability decisions are most likely to change.Source: §9 · p. 8

Limitations

The evaluation is limited to single versions of seven models and subsets of three tabular benchmarks. Its 21 rewrites do not cover all content-preserving changes.

Limitations

  • Each of the seven evaluated models is tested in one version, so other models and later versions may behave differently.The two open-weight decision models reproduce the findings relied on in the paper.Source: Limitations · p. 9
  • The evaluation uses subsets of three benchmarks, and its 21 rewrites omit content-preserving changes such as renaming columns or converting units.Source: Limitations · p. 9

A Models

B Compute

Runs used different hardware by model type: open-weight models ran on GPUs, while trained methods and TabPFN-3.5 ran on CPUs; closed models were accessed through provider APIs.

B Compute

C Further results

Further results examine whether confidence predicts decision changes and errors, and describe sensitivity to rewrites of questions and tables. They also report raw results for rewrite conditions and in-context examples.

C Further results

  • Figures 5–9 describe analyses of rewrite sensitivity by margin, rewrite type, dataset, and confidence, including observed versus noise-predicted flip rates.The captions specify flip AUROC, correctness changes, margin variation, confidence bands, and cluster-bootstrap intervals, but provide no plotted outcomes.Source: Figure 5 · p. 12, Figure 6 · p. 16, Figure 7 · p. 19
  • Table 9 evaluates whether confidence predicts both a changed decision under information-preserving rewrites and an incorrect original decision, using flip AUROC and error AUROC.The caption identifies 0.5 as chance and reports a strict-parse caveat for one Llama row-classification result.Source: Table 9 · p. 13
  • Supplementary tables cover LLM confidence readouts, information-preserving table formats, raw question, ordering, and format rewrites, and in-context-example results.Their captions define the reported metrics and setups; they also identify a set of decision-model papers whose main input is a table or table rows.Source: Table 10 · p. 13, Table 11 · p. 13, Table 12 · p. 14
  • Table 18 measures changes under information-preserving table rewrites and question rewrites, including shares across confidence-ranked groups and a re-send-based noise bound.Invalid outputs count as changes; the table also specifies confidence measures and bootstrap intervals.Source: Table 18 · p. 20
  • Table 19 compares confidence with correctness and fragility, defining fragility as a decision changed by at least one table rewrite.It reports the share of right decisions among the highest- and lowest-confidence quartiles and group-level shares and mean confidence.Source: Table 19 · p. 22

Related papers

Loading 2610.19536v1…