PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?

Daniel Gomm, Maarten de Rijke, Madelon Hulsebos

arXiv:2609.07629v1cs.IRcs.AIcs.CLcs.DB

TL;DR

Knowledge users need from tables must often be derived, while research is fragmented across task-specific communities. This paper formalizes OpenTI and reviews systems and benchmarks, finding that current approaches and evaluations do not cover its open, end-to-end scope.

  • Problem

    Knowledge valuable for users often must be derived from tables, while research remains organized around separate task labels such as table question answering, text-to-SQL, and data science agents.

    Source: §1 · pp. 3, 5

  • Method

    The paper formalizes OpenTI around insight needs, realizations, and result quality, then uses the framework to position systems and benchmarks across communities.

    Source: §1 · pp. 3, 5

  • Results

    Current systems focus on limited analyses over available data, 41 of 58 lack table retrieval, and benchmarks are largely unfit for open-setting evaluation.

    Source: §1 · p. 5

  • Takeaways & Limitations

    OpenTI frames the field around users’ insight needs and the end-to-end derivation and presentation of analytical knowledge, rather than surface-level answers.

    Source: §7 · p. 55

    Interpretation mechanisms are predominantly applied before analysis, while explicit revision as new observations emerge remains limited outside iterative retrieval approaches.

    Source: §3.3.1 · p. 23

Abstract

from arXiv · show

Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person's insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at https://open-tabular-insight-extraction.github.io.

1 Introduction

Knowledge from tables must be located and analyzed before it can inform users, while efforts addressing these stages remain dispersed across research communities. The paper frames this end-to-end task as OpenTI and reviews how systems and benchmarks cover it.

1 Introduction

  • Useful knowledge is implicit in tables and must be located and analyzed, tasks that demand skills and familiarity with data that people seeking the knowledge may lack.In open organizational settings, people may not know the relevant data’s location, contents, structure, or coverage in advance.Source: §1 · p. 3
  • OpenTI is the task of deriving and presenting analytical knowledge that serves a sufficiently directed insight need over an open corpus of tabular data.The corpus’s contents, schema, and coverage are unknown when the need is expressed.Source: §1 · p. 3
  • From 2021 to 2025, relevant works grew roughly eightfold, while benchmarks broadened beyond lookup and aggregation only recently.The paper links this broadening scope to maturing retrieval, learned table representations, and language-model-based agent capabilities.Source: §1 · p. 4
  • Research remains fragmented across labels such as table question answering, text-to-SQL, and data science agents, with works citing within the same label six times as often as across labels.The paper argues these labels describe facets of one problem and proposes shared vocabulary for comparing work across communities.Source: §1 · p. 5
  • The conceptual framework formalizes insight needs, realizations that derive knowledge, and whether a valid realization actually serves the need.The authors report that current benchmarks approximate validity but largely leave how well results serve the need unmeasured.Source: §1 · p. 5
  • The systematic review positions systems by their functional capabilities and orchestration, and assesses benchmarks against criteria for robust evaluation.The analysis finds that systems and benchmarks only partially cover OpenTI’s scope and requirements.Source: §1 · p. 5
  • The paper consolidates human-side work under cooperative interaction and derives a research agenda from gaps in pursuit of reliable, accessible insight extraction.It examines interpretability, personalization, and contextualization as means of aligning system realizations with latent insight needs.Source: §1 · p. 5
  • Systems concentrate on limited analysis over data already at hand, while 41 of 58 reviewed systems lack table retrieval and benchmarks are largely unfit for open-setting evaluation.Benchmark inputs often reference tables or values users would not know in advance, and validation commonly relies on a single reference.Source: §1 · p. 5

2 Open Tabular Insight Extraction

This section establishes OpenTI’s conceptual framework by formalizing the insight a person seeks, how systems derive it from tables, and what it takes to satisfy that need. It connects data sufficiency to retrieval over candidate sets and organizes analytical insights by the assumptions that justify them, making visible how task labels divide research on overlapping capabilities.

2.1.1 Insights

  • An insight links analytical knowledge derived from data with a person’s pre-existing contextual understanding, rather than being the analytical output alone (Battle & Ottley, 2024).Contextual understanding can range from minimal to substantial, and different presentations may invoke the same insight; insights can also build on prior insights (Battle & Ottley, 2024).Source: §2.1.1 · pp. 6–7

2.1.2 Insight Needs

  • An insight need is a requirement for analytically derived knowledge that becomes an insight when interpreted through a person’s contextual understanding.Needs can vary in directedness, remain partly latent in their expression, and decompose into recursively composed sub-needs; they seek analytical conclusions rather than artifacts as ends in themselves.Source: §2.1.2 · p. 7

2.1.3 Open Tabular Insight Extraction

  • OpenTI seeks to satisfy sufficiently directed needs by deriving and presenting analytical knowledge that serves them from an open corpus of tables whose contents may be unknown.Needs separate data, methodological, and output factors: what data matters, what analysis to perform, and how results should be presented.Source: §2.1.3 · p. 8

2.1.4 Realizations

  • A realization is an executable, table-bound DAG of operations that derives analytical knowledge, while its utility grades how well it serves a need.A realization declaratively specifies the derivation, not how a system found it; utility can reflect data quality, analytical rigor, and presentation, but not computational effort or interaction burden.Source: §2.1.4 · pp. 9–10

2.1.5 Problem Formulation

  • The ideal realization is the highest-utility candidate that is both valid for the need and feasible in the corpus; if none exists, the need is unresolved.Validity concerns whether the tables, operations, and output address the need; feasibility concerns whether the required tables exist in the corpus. Systems observe only the expressed need and must approximate the ideal across a combinatorially vast realization space.Source: §2.1.5 · pp. 10–11

2.2.1 Sufficient Specification

  • A need is sufficiently specified when its expression lets a system distinguish realizations that satisfy it from those that do not.Data, methodological, and output components must be specified or inferable; reasonable conventions can resolve implicit details, but an underspecified comparison may leave valid and invalid realizations indistinguishable.Source: §2.2.1 · p. 11

2.2.2 Data Sufficiency

  • Corpus sufficiency requires at least one valid realization, with tables covering the need’s semantic concepts and being structurally combinable; both forms of alignment are necessary.Semantic coverage and structural compatibility are independent: a set can satisfy one while failing the other, leaving the need unresolved.Source: §2.2.2 · p. 11
  • Data sufficiency can require aligning conditions, such as French and English contents; retrieval must surface candidate sets and navigate the search they induce.The passage gives structural alignment of a middle column as an example and locates this search problem under data retrieval.Source: §2.2.2 · p. 12

2.3 Taxonomy of Insight Types

  • OpenTI groups insight types into descriptive, inferential, and causal tiers: describing observed data, generalizing to its generating process, and reasoning about interventions.Descriptive examples include lookup, aggregation, characterization, association, and grouping; inferential examples include statistical inference and prediction; causal examples include interventional and counterfactual insights.Source: §2.3 · pp. 12–13
  • Higher-tier knowledge entails lower-tier regularities, but observed data or associations alone cannot determine the generating process or its response to intervention.A causal model entails the generated distribution and intervention effects, while the same distribution can fit distinct causal models with different effects.Source: §2.3 · p. 14
  • The tiers span analytical claims about observed data, its generating process, or interventions; listed types are representative, and each insight type is atomic.Compositions such as comparing means or analyzing trends inherit the relevant tier, while an expression can imply a different type from the user's underlying need.Source: §2.3 · p. 14
  • Works cite 10.6% of chronologically possible works sharing their task label, versus 1.8% across labels, while labels chiefly reflect conventions rather than analytical scope.Systems with shared capabilities are more likely to cite one another when they share a label, but barely more likely when their labels differ.Source: §2.3 · p. 15

3 Anatomy of OpenTI Systems

This section maps the anatomy of OpenTI systems, including how they navigate analytical processes through orchestration and capabilities spanning interpretation, retrieval, analysis, communication, and governance. It highlights gaps in capability coverage, open-corpus retrieval, data preparation, contextualized output, and mechanisms for validation, adaptation, and error recovery.

3.1 OpenTI Systems

  • OpenTI systems are agents that observe their environment, take actions, and follow a policy to construct a realization maximizing utility for an expressed insight need.The process entails interdependent decisions under uncertainty and partial observability, including uncertainty about the latent need and corpus contents.Source: §3.1 · p. 15

3.2 Orchestration Strategies

  • Of 58 reviewed systems, 54 orchestrate multiple capabilities; most use fixed orchestration, fewer use reactive or hybrid strategies, and only two are plan-based.The remaining four systems use a single capability and therefore require no orchestration.Source: §3.2 · p. 17, Figure 7 · p. 17

3.2.1 Fixed Orchestration

  • Fixed orchestration sets capability order at design time, enabling predictable, traceable workflows and independent capability optimization but limiting adaptation to unexpected needs, data, or errors.A fixed retrieve-then-analyze sequence, for example, cannot retrieve tables that become relevant only after initial analysis.Source: §3.2.1 · p. 18

3.2.2 Plan-Based Orchestration

  • Plan-based orchestration creates an insight-specific capability topology before execution, but its anticipated dependencies and recovery paths may fail when execution reveals unexpected data, methods, or needs.Plans can provide an end-to-end view for checking coherence and dependencies before execution, but they are generated without execution feedback.Source: §3.2.2 · pp. 18–19

3.2.3 Reactive Orchestration

  • Reactive orchestration selects actions from current observations, enabling adaptation and recovery but risking locally sensible actions that are globally suboptimal without sufficient anticipation.Agents can respond to observed errors by debugging and retrying operations.Source: §3.2.3 · p. 19

3.2.4 Hybrid Orchestration

  • Hybrid orchestration combines strategies across process phases or levels, using fixed structure with local reactive adaptation, planned workflows with reactive fallback, or revisable plans within reactive loops.These combinations aim to retain strengths such as global coherence while counteracting rigidity or limited adaptability.Source: §3.2.4 · pp. 19–20

3.3 Functional Capabilities

  • OpenTI’s object-level capabilities are interpretation, data retrieval, analytical composition and execution, and output synthesis; process governance supplies meta-level navigation toward high-utility realizations.Process governance includes assessing progress, validating partial results, handling errors, and deciding when to backtrack or pursue alternatives.Source: §3.3 · p. 20
  • 41 of 58 systems lack retrieval; 18 of 58 interpret implicitly, while outputs are limited to text or code results in 46 of 58 systems.Only 15 of 58 systems produce tables and 6 of 58 produce visualizations.Source: §3.3 · pp. 20–21

3.3.1 Interpretation

  • Scoping selects relevant rows, columns, or cells, but its effectiveness is disputed, with one study finding reduced accuracy.Methods range from semantic-similarity selection to retrieving schema elements and cell values or programmatically filtering rows.Source: §3.3.1 · p. 23
  • Interpretation is usually front-loaded; systems may solicit clarification, but explicit reinterpretation during processing remains limited and depends on orchestration.Reactive orchestration can condition decisions on observations, but counts as reinterpretation only when the agent revises its representation of the insight need.Source: §3.3.1 · p. 23

3.3.2 Data Retrieval

  • Only four reviewed systems target open-corpus retrieval, and multi-table needs require a sufficient set aligned both semantically to the need and structurally for integration.Retrieval may also need to satisfy methodological constraints, such as a minimum sample size.Source: §3.3.2 · p. 23
  • Text linearization remains dominant despite structural and datatype limitations, while table-, column-, and row-level indexing trade contextual breadth against finer-grained matching.Table-level indexing preserves cross-column semantics; column-level indexing can lose inter-column context and requires a larger index, while row-level indexing captures specific records.Source: §3.3.2 · p. 24
  • Multi-table retrieval must balance semantic coverage with structural compatibility, yet current approaches generally do not detect when the corpus contains no sufficient set.Structural compatibility assessments should reflect the agent’s integration capabilities, which may include transformations such as schema matching or feature engineering.Source: §3.3.2 · pp. 24–25
  • Generative retrieval jointly captures semantic and structural properties but risks synthetic-data errors and corpus-change retraining, while state-informed dynamic retrieval remains nascent.One approach makes retrieval invocable during execution but does not systematically use intermediate analytical state to guide it.Source: §3.3.2 · pp. 25–26

3.3.3 Analytical Composition and Execution

  • SQL offers formal, constrained execution but limited scope for complex analysis; Python is more expressive yet error-prone, while direct inference is less inspectable and unreliable for precise computation.Most Python systems demonstrate descriptive needs, and the practical effectiveness of Python for more complex analytical intents remains largely undemonstrated.Source: §3.3.3 · pp. 26–27
  • Incremental composition lets agents use intermediate results to adjust analysis and has reported accuracy gains over one-shot generation, contingent on using those results effectively.Execution mechanisms and composition patterns are coupled but not rigidly: tool calling is incremental, while general-purpose code supports both one-shot and incremental approaches.Source: §3.3.3 · pp. 27–28

3.3.4 Output Synthesis

  • Output synthesis ranges from returning raw results to composing contextualized, multimodal responses, but output needs are often absent from inputs and presentation is left to systems.The choice affects how accessible the analytical knowledge is to users.Source: §3.3.4 · p. 28
  • Most reviewed systems output text, while visualizations remain less integrated; modality selection is usually task-fixed, with only a few systems choosing modalities dynamically or by heuristic.Text suits retrieved facts and summary statistics, while visualizations may better communicate trends and distributions.Source: §3.3.4 · pp. 28–29
  • Systems can use LLMs to integrate parallel sub-need results, but contextualization is largely absent and evaluation often treats explanatory verbalization as noise rather than value.A few systems target contextualization: DAgent generates interpretive reports, DACO translates findings into domain-specific recommendations, and AOP attaches generated reasoning.Source: §3.3.4 · p. 29

3.3.5 Process Governance

  • Process governance operates at the meta-level to shape the analytical realization; error detection commonly uses execution checks, while recovery ranges from fixed fallbacks to LLM-driven retries.Adaptive retries are often bounded by retry or step limits.Source: §3.3.5 · pp. 29–30
  • Result validation goes beyond successful execution, ranging from checks for non-empty or compliant outputs to relevance assessment and process-level checks that all insight needs are addressed.AOP searches external knowledge for supporting or contradicting evidence, while AutoTQA compares results with the original need and can trigger replanning.Source: §3.3.5 · p. 30
  • Adaptive steering ranges from implicit reflection in ReAct steps to replanning and branch pruning, while Table-Critic uses a judge–critic–refiner loop to correct reasoning.AOP evaluates intermediate results after each execution layer and adjusts its plan forward.Source: §3.3.5 · p. 30
  • Ensembling aggregates multiple trajectories through voting or selectors, and can operate over full processes, candidate operations, or individual steps.Zhang et al. (2024b) also group computationally equivalent intermediate steps and select using aggregated log-probability.Source: §3.3.5 · pp. 30–31
  • Most systems use the analytical language model for governance, while others separate roles through prompts, distinct models, multiple agents, or deterministic programmatic checks.Separation matters because LLM self-correction can be unreliable without external verification.Source: §3.3.5 · p. 31

4 Interaction and Interpretability in OpenTI

OpenTI interaction must bridge the gap between users’ latent insight needs and what they can express. This section examines how interaction and interpretability can help systems align with those needs and let users assess the resulting analysis.

4 Interaction and Interpretability in OpenTI

  • Systems can resolve underspecification through conventions, disclosed choices among valid alternatives, or user engagement when neither approach suffices.User engagement is needed when information is missing or users cannot yet specify what they need without learning about the system’s data or capabilities.Source: §4 · p. 32

4.1 Interaction Modes

  • Interaction modes vary by when input occurs and who takes the initiative, and should fit user expertise and the analytical task.Modes form a continuum; less-directed needs tend to require sustained interaction, while more concrete needs place lower demands on it.Source: §4.1 · pp. 33–35
  • Upfront-only interaction minimizes involvement but cannot respond to data-specific ambiguities; continued input enables steering but increases interaction complexity.Output-driven refinement can require many iterations, while step-wise control shifts orchestration to users and requires greater analytical expertise.Source: §4.1 · pp. 33–34

4.2 Interaction Means

  • Specification interfaces guide users through data, methodological, or output choices, while direct intervention lets them inspect and edit emerging realizations.Intervention can expose code, visual graphs, or editable plans; users can also adjust operations or data bindings.Source: §4.2 · pp. 35–36
  • Natural language accommodates open-ended needs but can leave users unsure what to specify; combining it with direct manipulation remains largely unexplored in reviewed systems.Natural language supports open-ended intent, while direct manipulation can support precise, reference-dependent operations.Source: §4.2 · pp. 35–36

4.3 Interpretability

  • End-to-end verification requires users to assess the interpreted need, retrieved tables, and analytical operations; provenance can expose how data and results were derived.Data provenance concerns data binding and data flow, while analytical provenance makes the operations and their composition explicit.Source: §4.3 · p. 37
  • Interpretation format trades fidelity against accessibility, and excessive explanation risks cognitive overload; progressive disclosure offers a possible design approach.The appropriate default abstraction and drill-down depend on user expertise and task complexity, which reviewed works do not study systematically.Source: §4.3 · pp. 37–38
  • Concurrent interpretability supports output-driven, step-wise, and collaborative interaction, whereas post-hoc interpretation lets users inspect completed realizations.The section also cautions that exposing information alone can project transparency without fostering warranted trust.Source: §4.3 · p. 37

5 Evaluations in OpenTI

This section examines how OpenTI systems are evaluated, focusing on benchmarks that pair table corpora and user inputs with validation mechanisms, and on measuring the gap between theoretical capability and practical utility. It describes benchmark limitations in open settings and validation methods, their tradeoffs, and ways to combine signals, noting that evaluations often emphasize outputs over analytical processes and leave user utility unmeasured.

5 Evaluations in OpenTI

  • A benchmark pairs a corpus of tables with evaluation instances, each comprising an input expressing an insight need and a mechanism that scores the system’s realization.Benchmarks range from single-table question answering over web tables to multi-step analytical workloads over relational and file-based corpora.Source: §5 · p. 38

5.1.2 Analysis of Corpora

  • Benchmarks trade corpus scale against realism: Wikipedia corpora contain tens of thousands of mostly small, clean tables, while analytical corpora have more realistic structures but usually only hundreds.KramaBench combines raw realistic data with a multi-format data-lake organization, but still has a modest number of tables.Source: §5.1.2 · p. 40, Figure 9 · p. 41
  • Corpus realism, messiness, and domain coverage largely reflect source choices rather than deliberate design, leaving realism and scale in tension.TACO has 13,000 tables from open data portals but parses and cleans them into normalized relational databases; only a few benchmarks deliberately retain dirty data.Source: §5.1.2 · pp. 40–41

5.2.1 Desiderata

  • OpenTI inputs should be data-independent and sufficiently specified for their validation method, while representing multi-step analysis and integration across multiple tables.A single-reference validator requires inputs to determine the needed data and method; multiple-valid-realization validation can accommodate ambiguity, and mismatches can produce unreliable evaluation signals.Source: §5.2.1 · pp. 41–42

5.2.2 Input Protocols

  • Most benchmarks use static single-utterance inputs; user simulation is rare, and fixed multi-turn sequences trade contingent interaction realism for replicability.ConDABench, BIRD-INTERACT, and COTA (Li et al., 2025b) use LLM user simulators, which can risk goal misalignment or unrealistic responses.Source: §5.2.3 · pp. 42–43, §5.2.2 · p. 42

5.2.3 Analysis of Inputs

  • Insight coverage centers on descriptive lookups and aggregations, with inferential and causal insights sparse; 9 of 23 benchmarks provide relevant tables, while only 6 require open-setting retrieval.Most benchmarks that cover more complex aggregation, characterization, and association come from data-analysis workflows; genuine interventional analysis is targeted only by CausalReasoningBenchmark (Sawarni et al., 2026).Source: §5.2.3 · pp. 43–44
  • Inputs are often data-privileged, and all but NQ-Tables (31.6%) and FeTaQA (47.4%) have more than 50% insufficiently specified inputs.Data privilege is high in InfiAgent-DABench (70.0%), DABstep (78.4%), and TOPBench (85.0%); exact-match validation can conflate execution accuracy with guessing the intended interpretation.Source: §5.2.3 · pp. 43–44, Figure 10 · p. 44

5.3 Validation Mechanisms

  • Existing validation primarily approximates whether a realization is valid, leaving the quality of its outputs and how well it serves the user largely unjudged.OpenTI-suitable validation should be scalable and reproducible, cover relevant output modalities, and robustly measure analytical quality; traditional text-similarity metrics may correlate poorly with response quality.Source: §5.3 · p. 44, §5.3.1 · pp. 45–46

5.3.1 Desiderata

  • Validation should accept multiple valid realizations of an insight need rather than penalizing outputs that differ from a single gold standard.This can require sets of valid reference outputs or mechanisms that assess correctness beyond exact matching.Source: §5.3.1 · p. 46

5.3.2 Output-Level Validation

  • Output-level validation ranges from exact and similarity matching to LLM judges, programmatic tests, and human ratings, but treats the analytical process as a black box.Exact matching is the most common benchmark mechanism; free-form text similarity can correlate poorly with human judgments, while LLM judges introduce bias, cost, and reproducibility concerns. For complex insight needs, one study found 67% consistency between human experts and LLM judges.Source: §5.3.2 · pp. 46–47

5.3.3 Realization-Level Validation

  • Realization-level validation can distinguish correct from flawed analysis and award partial credit, using program comparison, structural matching, or component checks.Program comparison depends on code artifacts and reference realizations; structural and component methods can accommodate more variation, but may rely on error-prone extraction or costly per-instance curation.Source: §5.3.3 · pp. 48–49

5.3.4 Combining Validation Signals

  • Benchmarks can combine validation signals into one composite score or report separate criteria, trading diagnostic detail against simpler comparisons.Output- and realization-level signals can capture complementary failure modes; separate reporting helps attribute performance but complicates cross-system comparisons.Source: §5.3.4 · p. 49
  • Validation remains concentrated on outputs, with narrow modality coverage and little assessment of the utility of different valid realizations.Only DAComp evaluates multimodal text-and-chart outputs; genuine assessment of open-ended analytical processes appears in few works, and rubric-based approaches only partially surface utility.Source: §5.3.4 · p. 49

6 A Research Agenda for OpenTI

The agenda identifies gaps in retrieval, analysis, trust, cooperation, orchestration, model architectures, and evaluation that stand between current systems and end-to-end OpenTI. It proposes research directions spanning these connected challenges.

6.1.1 Retrieving Compatible and Relevant Data at Scale

  • Retrieval should measure task-specific utility and data quality, not only semantic similarity and structural compatibility, and indexes should account for relationships, quality, and analytical use.Real-world corpora also demand methods robust to heterogeneous schemas, inconsistent names, varying quality, and overlapping content, challenges current benchmarks leave largely untested.Source: §6.1.1 · p. 50, §6.1 · p. 50

6.1.2 Addressing Inferential and Causal Insights

  • Systems should extend beyond descriptive analysis by verifying assumptions for inferential and causal conclusions and jointly selecting methods and data that support valid analysis.The assumptions vary by tier; the appropriate methodology also depends on which features or variables retrieval makes available.Source: §6.1.2 · p. 51

6.2.1 Ensuring Robustness of Results and Realizations

  • Systems should detect unanswerable needs and quantify uncertainty across realizations, communicating when missing data, limited capabilities, or invalid assumptions undermine reliable results.Errors can compound across operations, and current surveyed systems lack systematic uncertainty quantification.Source: §6.2.1 · pp. 51–52
  • Systems should recognize when a technically sound answer misses the user's underlying need, then communicate the mismatch and propose a better-suited analysis.Users may request a proxy such as correlation when they need an analysis of effect direction and significance that accounts for confounders.Source: §6.2.1 · p. 52

6.2.2 Enabling Cooperative Human-System Interaction

  • Cooperative systems should share responsibility for alignment through richer interaction, interpretable and actionable realizations, maintained contextual knowledge, and output presentations suited to users and results.The agenda calls for user studies across task complexity and expertise, and for balancing comprehensive provenance with users' cognitive effort.Source: §6.2.2 · pp. 52–53

6.3.1 Dynamically Orchestrating Analytical Workloads

  • Complex workloads require orchestration that can revise operations, data bindings, and strategy as results emerge, retrieve newly needed data, incorporate user input, and route tasks appropriately.Current reactive systems offer initial adaptivity, but largely select predefined operations with data bindings fixed upfront and limited backtracking.Source: §6.3.1 · pp. 53–54

6.3.2 Superseding and Complementing Language Models

  • Language models' linear text representations fit tabular structure and evolving analytical workflows poorly, motivating architectures suited to both, while the value of single models versus specialized components remains open.Table-native models have shown limited downstream improvements over general-purpose language models; tabular foundation models have mainly entered insight extraction as tools for language-model agents.Source: §6.3.2 · p. 54

6.3.3 Designing Realistic and Faithful Evaluations

  • Evaluation should move beyond output-only exact matching toward realistic corpora, realization-level checks, interactive inputs, and joint validation of text, tables, and visualizations.Output-only scoring can reward guessed answers or penalize legitimate alternative realizations; interactive user simulations can also misalign with real user behavior.Source: §6.3.3 · pp. 54–55

7 Conclusion

The paper frames OpenTI as an end-to-end, insight-centered task and uses it to synthesize fragmented work across research communities. Its review finds that relevant capabilities exist but remain fragmented and that retrieval, contextualization, cooperation, inferential and causal analysis, and evaluation remain underdeveloped.

7 Conclusion

  • OpenTI centers on deriving and presenting analytical knowledge that serves a person's insight need, formalizing the process through needs, realizations, and utility beyond literal answer accuracy.The paper frames systems as agents navigating a large realization space under partial observability and treats human interaction and evaluation as part of the task.Source: §7 · p. 55
  • The review finds that existing capabilities remain fragmented and mostly answer-centered, with thin coverage of retrieval, contextualization, cooperation, inferential and causal analysis, and evaluation.The paper argues that systems should be designed and evaluated around insights, producing robust, contextualized results that leave users better informed.Source: §7 · p. 55

A Overview of Notation

  • The notation distinguishes insight, data, methodological, and output needs, and represents a realization as operations connected by dataflow and bound to corpus tables.The appendix also defines realization utility and candidate realizations as those both valid for the need and feasible in the corpus.Source: A Overview of Notation · p. 85

B Comparison with Related Work

The paper distinguishes OpenTI from related surveys by treating tabular insight extraction as one open, end-to-end problem rather than a collection of community-specific tasks. The comparison foregrounds user-facing concerns alongside retrieval, analysis, and presentation.

B Comparison with Related Work

  • Table 5 compares surveys across end-to-end scope, openness, insight-centered framing, formalization, and user-facing concerns.Its scope column tracks explicit treatment of user interaction, retrieval, data integration, analysis, and output presentation; symbols distinguish unaddressed, partial, and foregrounded aspects.Source: Table 5 · p. 86, B Comparison with Related Work · p. 86
  • Related surveys address only parts of the path from expressed need to presented insight, often focusing on analysis while assuming relevant tables are already supplied.The paper contrasts this closed setting with OpenTI's premise that corpus contents are unknown when a need is expressed.Source: B Comparison with Related Work · p. 86
  • The paper says related work generally targets surface-answer correctness rather than the insights that serve users, and formalizes isolated subtasks rather than the whole problem.Interaction, personalization, and interpretability are more often raised as open challenges than developed as central concerns.Source: B Comparison with Related Work · p. 86

C Literature Review Protocol

The review uses a structured protocol to identify, screen, and select the works informing its analysis of systems and benchmarks.

C Literature Review Protocol

  • A broad keyword search across 18 venues identified 399 distinct works after deduplication.The protocol's identification stage was followed by manual screening against inclusion criteria.Source: Figure 11 · p. 87, C Literature Review Protocol · p. 87
  • Relevance screening reduced the search results to a core set of 160 works.The protocol was organized into identification, screening, and selection stages and was inspired by PRISMA guidelines.Source: Figure 11 · p. 87, C Literature Review Protocol · p. 87
  • Backward and forward snowballing extended the core set, from which the review selected 29 systems and 23 benchmarks for its overview tables.The broader relevant-work set informs the paper, while the selected systems and benchmarks are analyzed in Sections 3 and 5.Source: Figure 11 · p. 87, C Literature Review Protocol · p. 87

C.1 Identification

The review searches leading venues across OpenTI-related fields from 2021 to May 2026, then screens keyword-matched works for relevance to its end-to-end scope. Snowballing extends the system and benchmark sets used for detailed surveys.

C.1 Identification

  • The search covers 18 targeted venues across machine learning, information retrieval, human-computer interaction, database research, natural language processing, and tabular-data workshops.The review begins in 2021, when relevant work accelerates markedly, and adds TRL workshops as a direct intersection of tabular data and machine learning.Source: §C.1 · pp. 87–88
  • Seven diverse keyword searches yield 399 distinct works for screening, targeting labels such as data analysis agents, table question answering, and text-to-SQL.The queries mirror fragmented vocabulary across communities; Table 6 reports matches and relevant works for each query.Source: §C.1 · p. 88, Table 6 · p. 88

C.2 Screening and Selection

  • Screening retains 160 core works that inform OpenTI’s conceptual treatment, system components, benchmarks, or relevant user-system interaction.The inclusion criteria cover end-to-end systems or assessments, benchmarks for directed tabular insight needs, and interaction; relevant conceptual works are also retained.Source: §C.2 · p. 88
  • Snowballing adds 8 system and 13 benchmark works, yielding survey sets of 58 methods and 42 benchmarks.Backward and forward snowballing targets substantial contributions outside the specified venues, date range, or queries.Source: §C.2 · p. 88

C.3 Selection for the Overview Tables

The overview tables select systems with substantial OpenTI capability coverage and benchmarks that cover substantial parts of insight needs, while retaining several widely used reference datasets.

C.3 Selection for the Overview Tables

  • 29 systems are selected for substantial coverage of OpenTI capabilities and an output beyond raw results or code.A system must cover at least three capabilities fully or four partially; Table 1 presents the resulting set.Source: §C.3 · p. 89
  • 23 benchmarks are presented: 18 meet the coverage threshold, alongside five widely used, highly cited reference datasets.Non-English datasets are excluded; the rubric scores data, methodological, and output needs from 0 to 2, with a total of at least 3 for inclusion among the 18 qualifying datasets.Source: §C.3 · p. 89, Table 7 · p. 89

C.4 Limitations

The review’s claims are bounded by its search and selection protocol, which does not provide exhaustive coverage of the fast-moving OpenTI space.

C.4 Limitations

  • Works found only on preprint servers or outside the 18-venue search enter only through snowballing, potentially underrepresenting recent developments.The authors frame findings as claims about surveyed works, whose scope is limited by the review methodology.Source: §C.4 · p. 89
  • Commercial and organizational systems are excluded because their undocumented internals cannot be assessed against the framework.The authors note that several deployed products plausibly attempt parts of OpenTI.Source: §C.4 · p. 89
  • The protocol is not fully PRISMA-compliant, and its search and inclusion criteria were not registered in advance.Criteria were refined alongside the developing understanding of OpenTI rather than fixed beforehand.Source: §C.4 · p. 89

D Input Classification Methodology

The authors apply LLM judges to classify data privilege and specification across selected benchmarks, then validate the judgments against expert annotations or corrections. Agreement is substantial for both aggregated classifications.

D Input Classification Methodology

  • The classification covers 18 benchmarks with self-contained natural-language inputs, sampling 500 input definitions per benchmark or all inputs when fewer are available.Five benchmarks are excluded because inputs are unavailable or incompatible with the LLM-judge setup.Source: D Input Classification Methodology · pp. 89–90
  • Data privilege flags references to schema, non-public values, or data containers, with five classifications aggregated by majority vote.An input is counted as privileged if at least one dimension is flagged; the final label requires at least three identical votes.Source: D Input Classification Methodology · p. 90
  • Sufficient specification requires all five data and procedural specification flags to hold; otherwise an input is insufficiently specified.The dimensions are entity, temporal, domain, intent, and methodological specification.Source: D Input Classification Methodology · p. 90
  • Validation uses a stratified sample of 168 annotations across the 18 classified benchmarks, comparing data-privilege labels with expert annotations and specification labels with expert corrections.The paper reports raw agreement and Cohen’s κ for these checks in Tables 8 and 9.Source: D Input Classification Methodology · p. 90, Table 8 · p. 90, Table 9 · p. 90
  • κ = 0.747 for aggregated data-privilege judgments and κ = 0.869 for sufficient-specification judgments; raw agreement is in line with Gomm et al. (2025).The paper characterizes these as substantial and almost perfect agreement, respectively, following Landis & Koch (1977).Source: D Input Classification Methodology · p. 90, Table 8 · p. 90, Table 9 · p. 90

E Citation Analysis

The citation analysis builds a graph among surveyed works, assigns each work its self-framing task label, and measures whether citations cluster by label.

E Citation Analysis

  • The analysis covers 181 works surfaced in the literature review, including works found through forward and backward snowballing.Source: E Citation Analysis · p. 91

E.1 Citation Graph Extraction

  • Reference extraction found 8,473 references across 180 works; title matching identified 959 surveyed-paper citation edges, 96% through exact title matches.Source: §E.1 · p. 91
  • Combining the extracted edges with 40 additional edges yielded a final citation graph of 999 edges.Semantic Scholar and OpenAlex contributed comparison and complementary edge sets.Source: §E.1 · p. 91

E.2 Assigning Task Labels

The analysis assigns surveyed works to the task labels they use to frame their contributions, with manual review resolving uncertain assignments.

E.2 Assigning Task Labels

  • The three labels are text-to-SQL, table question answering, and data analysis agents; retrieval works inherit the label of the downstream task they serve.Source: §E.2 · p. 91
  • Automatic title-and-abstract label counts and an independent LLM classifier agreed on 86.2% of works, with κ = 0.799.Source: §E.2 · p. 91
  • Manual review resolved 37 uncertain assignments, yielding 85 table question answering, 54 text-to-SQL, 33 data analysis agent, and 9 unlabeled works.Source: §E.2 · p. 91

E.3 Analysis

The analysis compares citation rates within and across task labels among pairs for which a citation was feasible.

E.3 Analysis

  • A feasible pair consists of works where the cited work was published before or in the same year as the citing work.Source: §E.3 · p. 91
  • Restricting to feasible pairs controls for differences in label-group sizes and works’ age distributions, leaving 18,243 pairs.Source: §E.3 · p. 91
  • The analysis compares within-label and cross-label citation rates and tests their ratio using 10,000 permutations of labels across works.Source: §E.3 · p. 91

E.3.1 Results

Citation rates are substantially higher within task labels than across them, while shared system capabilities correspond to increased within-label citation but not cross-label citation.

E.3.1 Results

  • 10.6% of feasible pairs cite within labels versus 1.8% across labels, a 6.02 ratio significant at p < 0.001.Source: §E.3.1 · p. 91
  • Across labels, citation rates stay at or below 3.0%, meaning systems converging on similar designs are no more likely to cite one another across a label boundary than systems sharing no capability at all.Source: §E.3.1 · p. 92
  • Citation absence indicates non-engagement, not proof of unawareness; around a quarter of data analysis agent works cite none of the surveyed works.That no-citation share also affects how to interpret the corresponding figure row and reflects the review’s reach.Source: §E.3.1 · p. 92

Related papers

Loading 2609.07629v1…