PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost-Effective Numerical QA over Semi-Structured Table: Structuring, Resolution, Planning

Feng Luo, Hui Luo, Zhifeng Bao, J. Shane Culpepper, Xiaoli Wang, Shazia Sadiq

arXiv:2610.07749v1cs.DB

TL;DR

Numerical QA over semi-structured tables must account for structural layout and underspecified phrases, while existing methods have limitations in handling question uncertainty. SemiBonsai combines table structuring, context-aware resolution, plan-guided reasoning, and budget-aware LLM routing; across three benchmarks it improves answer accuracy and cost-effectiveness.

  • Problem

    Semi-structured tables encode rich layout semantics, and existing approaches can struggle to resolve underspecified phrases in numerical QA.

    Source: §I · p. 1, §IV · p. 6

  • Method

    SemiBonsai combines a multiway layered table structurer, context-aware uncertainty resolver, plan-guided reasoner, and budget-aware bandit router.

    Source: §I · p. 2, A. Query Planning · p. 7, §VIII · p. 12

  • Results

    Across three benchmarks, SemiBonsai achieves an 18% average improvement over the best baseline and consistently improves answer accuracy with strong cost-effectiveness.

    Source: 1) Main results · p. 10, §VIII · p. 12

  • Takeaways & Limitations

    The results support jointly modeling table structure, question uncertainty, and complex intent for numerical QA across three benchmarks.

    Source: 1) Main results · p. 10, §VIII · p. 12

Abstract

from arXiv · show

Semi-structured tables encode rich semantic information through diverse layout elements, such as hierarchical row and column headers. Answering numerical questions over such data is challenging because it requires a joint effort of accurate structural understanding of tables, question uncertainty resolution, and complex query intents interpretation. Existing methods are rarely effective in handling the above multifaceted challenges in a holistic manner; moreover, they rely on LLMs without considering financial implications. We propose SemiBonsai, a cost-effective, LLM-powered framework featuring: (1) a Multiway Layered Table Structurer that converts a table into a multiway layered tree organized by structural elements; (2) a Context-Aware Uncertainty Resolver that grounds underspecified phrases to context-coherent structural elements; (3) a Plan-Guided Reasoner that decomposes complex query intents into a query plan for reasoning; (4) a Budget-Aware Bandit Router that learns per-instance LLM utilities and optimizes LLM selection under fixed monetary constraints. Experiments on three benchmarks show that SemiBonsai improves both QA effectiveness and cost-effectiveness. Notably, the Table Structurer improves answer accuracy by 46% over alternative tree construction methods, and Router achieves a 5% average relative gain in answer accuracy compared with existing routing methods across budgets.

I. INTRODUCTION

Numerical QA over semi-structured tables must jointly handle rich table layouts, underspecified questions, and multi-step query intent. SemiBonsai combines components for these challenges with budget-aware LLM routing, and reports gains in accuracy and cost-effectiveness.

Introduction

  • Semi-structured tables encode structural semantics through subtables and hierarchical row and column headers, while nearly 70% of their data is numerical.These properties motivate natural-language interfaces for numerical QA.Source: §I · p. 1
  • The task jointly requires accurate structure understanding, resolution of question uncertainty, and interpretation of complex query intent.For example, “last two years” needs grounding, while finding annualized uplift for a minimum-cost year involves multiple retrieval, ranking, aggregation, and calculation steps.Source: §I · p. 1
  • Existing prompt-based, code-driven, and representation-driven approaches rarely address these challenges holistically, with limitations including simplified layouts and weak uncertainty or intent handling.The cited ST-Raptor example omits hierarchical row headers and does not explicitly resolve underspecified phrases.Source: §I · p. 1

Our Contributions

  • SemiBonsai combines a multiway layered tree, context-aware phrase grounding, query-plan-guided reasoning, and budget-aware LLM routing.The Router learns per-instance utility scores and adaptively updates a cost penalty while enforcing the global budget constraint.Source: §I · p. 2

Experimental Evaluation

  • The Table Structurer improves answer accuracy by 46% over alternative tree construction methods, while the Router gains 5% average relative answer accuracy over existing routing approaches across monetary constraints.Experiments cover three benchmarks.Source: §I · p. 2

II. PROBLEM FORMULATION AND LITERATURE REVIEW

The formulation treats a semi-structured table as subtables within a two-dimensional grid, whose values are interpreted through structural metadata. Numerical QA computes an answer to a numeric question from that table.

A. Problem Definition

  • A semi-structured table is a two-dimensional grid whose layout cues, such as indentation and merged cells, indicate when one header is nested under another.The paper denotes this nesting relation as h1▷h2 when h2 is nested under h1.Source: A. Problem Definition · p. 3
  • Hierarchical row and column headers are maximal nested header sequences that identify rows and columns, respectively.When no nesting exists, header depth is treated as one.Source: A. Problem Definition · p. 3
  • A subtable is a rectangular region delimited by contiguous groups of hierarchical row and column headers, and may have an optional merged title cell.The title typically spans all columns or all rows in the subtable.Source: A. Problem Definition · p. 3
  • A subtable’s content is the set of cell values within the region delimited by its hierarchical row headers.The formulation illustrates contiguous header groups delimiting a subtable region.Source: A. Problem Definition · p. 3
  • The table’s structural elements comprise the set of subtables and their metadata.The example distinguishes subtables by their titles.Source: A. Problem Definition · p. 3
  • The formulation assumes layout cues reliably indicate structural elements and focuses on rectangular subtables, excluding misleading cues and nested or interleaved subtables.Source: A. Problem Definition · p. 3
  • Numerical semi-structured table QA computes the correct numeric answer to a question using arithmetic computation or aggregation over a semi-structured table.The task is expressed as learning a function from the question and table to the answer.Source: A. Problem Definition · p. 3

1) Semi-structured table QA:

Semi-structured table QA methods have gaps in structural fidelity and explicit handling of question uncertainty and complex intent, while existing routing approaches face data and feedback constraints. SemiBonsai’s structurer addresses structural representation with a layered tree and separate value index.

1) Semi-structured table QA:

  • Prior QA methods can lose table structure or leave uncertainty and complex intents implicit, while budget-constrained routers face training-data and feedback constraints.Normalization may discard subtable structure, and representation-driven approaches may omit row headers or rely on implicit LLM handling.Source: 1) Semi-structured table QA · pp. 3–4, 2) LLM Routing · p. 4

2) LLM Routing:

  • Budget-constrained routing methods require unavailable large training data or online answer-quality feedback, limiting their applicability in this setting.Prediction errors can also degrade routing decisions when per-instance costs and accuracy vary.Source: 2) LLM Routing · p. 4

III. MULTIWAY LAYERED TABLE STRUCTURER

  • The Multiway Layered Table Structurer combines a logical tree for structural elements with a physical value index for storage-efficient lookup.It uses a rule-guided procedure to construct the tree, then indexes content by de-duplicating cell values.Source: §III · p. 4

A. A Logical–Physical Structural Model

  • The layered tree organizes subtables and hierarchical row and column headers, while a separate value index stores distinct cell values and maps header paths to them.A path from a subtable to a leaf header identifies a hierarchical row or column header.Source: A. A Logical–Physical Structural Model · pp. 4–5

B. Constructing the Layered Tree

The layered tree is built by combining VLM-proposed nodes with layout-driven rules for header hierarchy and subtable containment. Value indexing, context-aware phrase resolution, and pruning then support efficient and coherent grounding.

B. Constructing the Layered Tree

  • VLMs propose candidate titles and headers, then layout-driven rules match cells, infer header nesting, group contiguous header subtrees, and identify subtables by span alignment.Nesting uses indentation and merged-cell row or column spans; title spans aligned with corner-adjacent header groups establish subtables.Source: B. Constructing the Layered Tree · pp. 5–6

C. Storage-efficient Value Indexing

  • The Value Index Structure stores each distinct cell value once and maps row- and column-header paths to that value.Root-to-leaf paths in the layered tree identify the relevant headers.Source: C. Storage-efficient Value Indexing · p. 6

IV. CONTEXT-AWARE UNCERTAINTY RESOLVER

  • The Context-Aware Uncertainty Resolver grounds multiple underspecified phrases jointly, avoiding irrelevant mappings that can arise from independent resolution.Its three stages identify uncertainty, resolve phrases, and rewrite the question with grounded elements.Source: §IV · p. 6

A. Context-Aware Pruning

  • Context-aware pruning selects a minimum-size collection of connected tree components covering all phrases, then lifts paths to subtable-to-leaf header paths.The method removes candidate groundings unsupported by the selected local and global contexts.Source: A. Context-Aware Pruning · p. 6

B. From an Uncertain Question to a Clear One

SemiBonsai grounds underspecified phrases in table structure and decomposes complex numerical questions into dependent subintents. The plan guides evidence retrieval and program generation, while the pipeline incorporates budget-aware LLM routing.

B. From an Uncertain Question to a Clear One

  • The resolver identifies phrases that need table grounding, aligns candidate structural nodes, prunes irrelevant elements, and rewrites the question with selected groundings.For large tables, hybrid retrieval can form a smaller candidate pool before identification.Source: B. From an Uncertain Question to a… · pp. 6–7

V. PLAN–GUIDED REASONER

  • The Plan–Guided Reasoner uses planning, grounding, and coding agents to turn complex query intents into an executable reasoning program.The approach addresses the fragility of synthesizing a full program in one step when multiple operands and operators are involved.Source: §V · p. 7

A. Query Planning

  • Query planning decomposes a question into minimal dependent subintents that specify computations, required evidence, and operand or operator relationships.Operand-based decomposition separates distinct operands; operator-based decomposition splits a multi-operator chain at its first operator.Source: A. Query Planning · p. 7

B. From Plan to Execution

  • The system retrieves evidence for independent subquestions and generates Python subprograms bottom-up, combining child outputs according to plan dependencies.Failed code execution can trigger up to three repair attempts.Source: B. From Plan to Execution · p. 8

VI. BUDGET-AWARE BANDIT ROUTER

  • The pipeline adds smaller LLMs and budget-aware routing because relying on high-cost models makes it impractical under budget constraints.The passage introduces a two-stage routing strategy but does not specify its mechanism here.Source: §VI · p. 8

A. Problem Setup

The section defines budget-aware routing as assigning one candidate LLM to each question-table instance to maximize average answer accuracy under a total budget. It then describes a two-stage router and the evaluation dimensions used to assess SemiBonsai.

A. Problem Setup

  • Given question-table instances, candidate LLMs, and a total budget, routing selects one LLM per instance to maximize average answer accuracy within that budget.Each instance pairs a question with a semi-structured table, and each selected LLM incurs a monetary cost.Source: A. Problem Setup · p. 8

B. Two-Stage Routing: Contextual Bandit Scoring and Budget-Constrained Control

  • The router learns per-instance LLM utility scores with a contextual bandit, then adaptively updates a cost penalty during execution to enforce the global budget.The two-stage approach replaces precise per-instance accuracy and cost prediction with utility ranking, motivated by limited training data.Source: B. Two-Stage Routing: Contextual… · pp. 8–9

VII. EXPERIMENTAL EVALUATION

  • The evaluation asks about overall QA effectiveness, robustness to table- and question-side complexity, runtime and token efficiency, component effects, and budget-constrained cost-effectiveness.The budget-aware router is assessed on whether it keeps total monetary cost within budget while maximizing answer accuracy.Source: §VII · p. 9

A. Experimental Setup

The experiments use three semi-structured table QA benchmarks and numerical-question subsets, with additional complexity-based subsets. They compare representative methods using arithmetic exact match and report the evaluation setup.

1) Datasets:

  • HiTab, MultiHiertt, and RealHiTBench are selected for their higher numerical-value ratios; numerical-only subsets are derived from specified splits.The reported ratios are 0.72, 0.68, and 0.79, respectively.Source: 1) Datasets · p. 9
  • The study creates COMPLEX-Q, COMPLEX-T, and COMPLEX-QT subsets using feature-based classifiers to distinguish simple from complex questions and tables.The classifiers use manually labeled pools and table- and question-side complexity features rather than a fixed heuristic threshold.Source: 1) Datasets · p. 9

3) Evaluation metrics:

  • Answer accuracy is measured with Arithmetic Exact Match, with an LLM-based value-equivalence check for numerically equivalent formats.The equivalence check matched human judgments in 95% of 40 sampled prediction–gold answer pairs.Source: 3) Evaluation metrics · p. 9

4) Implementation:

  • GPT-4.1 serves as the primary backend for baselines and SemiBonsai and as the table-structurer VLM; tables are supplied in HTML or Excel according to method format.The matching threshold is 0.9.Source: 4) Implementation · p. 10

B. QA Effectiveness (Q1)

1) Main results:

SemiBonsai improves answer accuracy over baselines overall and on complex subsets, and remains robust as table- and question-side complexity increases. Its runtime is modest, with the best accuracy–latency trade-off across the benchmarks.

1) Main results:

  • 18% average improvement in answer accuracy over the best baseline is reported across the three benchmarks.The authors note that no single baseline is consistently strongest, with representation-driven and code-driven methods excelling at different capabilities.Source: 1) Main results · p. 10

2) Results on complex questions and tables:

  • 50% improvement over the best baseline on COMPLEX-Q is reported, and SemiBonsai attains the highest accuracy across all complex subsets.The authors also report clear gains on COMPLEX-T.Source: 2) Results on complex questions and… · p. 10

1) Impact of table-side factors:

  • SemiBonsai is the most robust across table-side splits, and its accuracy remains stable in augmented COMPLEX-Q variants with 3, 5, or 7 subtables.Table-side factors include single versus multiple subtables and hierarchical headers with mean nesting depth above 1.Source: Figure 6 · p. 10, 1) Impact of table-side factors · p. 10

1) Time efficiency:

  • SemiBonsai incurs modest runtime and offers the best accuracy–latency trade-off across all benchmarks.Prompt-based methods are fastest, while GraphOTTER is slowest because it repeatedly invokes LLMs to retrieve values before answering.Source: Figure 8 · p. 11, 1) Time efficiency · p. 11

2) Impact of question-side factors:

2) Token efficiency:

SemiBonsai performs best across the reported routing budgets and benchmarks, while the appendices examine component effectiveness, token use, and the framework’s overall design.

2) Token efficiency:

  • Prompt-based methods use the fewest input tokens, while prompt-based methods and TreeThinker use the fewest output tokens; SemiBonsai remains moderate in usage.Repeated LLM-based intermediate reasoning increases token consumption for the other methods.Source: 2) Token efficiency · p. 11

E. Ablation and Component Analysis (Q4)

1) Experimental setup:

  • Evaluation used three benchmarks with 20% of instances for training, seven LLMs, and ten budgets, averaging five permutations under a budget-stop protocol.Source: 1) Experimental setup · p. 12

2) Main Results:

  • Across all three benchmarks and budgets, SemiBonsai attains the highest accuracy among routing baselines and steadily increases accuracy as budgets loosen.It also surpasses NO ROUTING on two benchmarks, where some smaller LLMs outperform the large model on individual instances.Source: Figure 10 · p. 12, 2) Main Results · p. 12

VIII. CONCLUSION

  • SemiBonsai combines tree structuring, context-grounded question rewriting, plan-guided executable reasoning, and utility-based routing under a fixed budget.Source: §VIII · p. 12

APPENDIX A ILLUSTRATIVE STRUCTURAL UNDERSTANDING ERRORS

The appendices illustrate structural interpretation errors, describe data-driven complexity subsets, and derive instance-wise LLM routing from a budget-constrained objective.

APPENDIX B COMPLEXITY OF QUESTIONS AND TABLES

  • Complexity subsets are formed by classifying questions and tables as simple or complex using table-side and question-side features.The authors use learned decision boundaries because a fixed heuristic threshold can misclassify borderline cases when multiple factors interact.Source: Appendix B · p. 15

A. Feature Construction

  • Table features count subtables and hierarchical headers and measure nesting depth; question features include underspecification, operators, operands, and involved structures.Source: A. Feature Construction · p. 15

B. Complexity Subset Construction

APPENDIX C LAGRANGIAN RELAXATION FOR ROUTING

  • A nonnegative cost multiplier relaxes the budget constraint; with fixed λ, the objective decomposes into per-instance LLM choices scored by utility.Source: Appendix C · pp. 15–16

APPENDIX A ILLUSTRATIVE STRUCTURAL UNDERSTANDING ERRORS

APPENDIX D EXPERIMENT SUPPLEMENTS

Supplementary experiments compare SemiBonsai with GPT-5.5, test robustness under increased table complexity, and break down component efficiency.

A. Comparison with GPT-5.5

  • SemiBonsai exceeds GPT-5.5 in answer accuracy on COMPLEX-Q and COMPLEX-QT, achieves comparable EM on COMPLEX-T, and costs less or comparably across all three subsets.Source: A. Comparison with GPT-5.5 · p. 16

B. Robustness under Complexity-augmented Tables

  • On variants that preserve values but raise subtable counts or header depth, SemiBonsai remains stable as table-side complexity increases.The augmented collections increase subtables or row/column header depth within the observed benchmark upper bounds.Source: Figure 12 · p. 16, B. Robustness under… · p. 16

C. Component-level Efficiency

D. The Impact of Operand-based Decomposition

The operand-based decomposition variant is evaluated against operator-only decomposition across datasets.

D. The Impact of Operand-based Decomposition

  • SemiBonsai consistently improves EM across all datasets compared with operator-based decomposition alone.The authors suggest that explicitly grounding operands during planning helps specify complex query intents.Source: D. The Impact of Operand-based… · p. 17

APPENDIX E PROMPTS

The appendix documents SemiBonsai prompts for extracting table structure, resolving ambiguous phrases, rewriting questions, planning numerical reasoning, and generating answer code.

Candidate Node Identification Prompt

  • The table-image prompt represents row and column header structure as flat or multi-level groups, preserving leaf-header order and handling spanning headers without mixing visual blocks.It directs the model to use image cell text verbatim and inspect the left-most column below the header band for row headers.Source: Appendix E · p. 18

Plan-Guided Reasoner Prompts

  • The planning prompt selects an action for an unresolved subquestion: execute direct retrieval, infer a calculation formula with operand subquestions, or decompose multi-hop reasoning into up to three ordered single-hop subquestions.It records subquestion dependencies and uses action history; the first two action types take the target subquestion ID as their first argument.Source: Appendix E · p. 20

Uncertainty Identification and Resolution Prompt

  • The grounding prompt identifies underspecified phrases and ranks relevant metadata labels across subtable titles, column headers, and row headers, using only provided labels verbatim.It forbids cross-phrase pruning at this stage; the rewriting prompt then makes grounded labels explicit while preserving intent and adding no new facts.Source: Appendix E · pp. 19–21

Plan-Guided Reasoner Prompts

  • The code-generation prompt requires one executable Python snippet that computes atomic-subquestion intermediates from relevant subtable data and assigns the answer to final_answer.It specifies a minimal pandas DataFrame, numeric-string normalization, and filtering summary rows when they affect aggregation.Source: Appendix E · p. 21

Related papers

Loading 2610.07749v1…