PaperlayerView exact revision on arXiv

Source-linked AI summary

A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain

Yiou Wu, Zezhi Tang, Ningwei Bai, Liuhaichen Yang

arXiv:2610.06914v2cs.AIcs.DB

TL;DR

Enterprise dashboard creation requires linking a business question to the right data, query, and visual artifact while controlling execution. This paper implements a governed natural-language dashboard agent for DataBrain and evaluates it on frozen tasks and controlled faults. Results include successful metadata, SQL, dashboard, and fault tests, alongside exact-column and convergence failures.

  • Problem

    Turning business questions into dashboards requires selecting entities and tables, inspecting schemas, writing SQL, choosing visuals, and checking the rendered result, often without users knowing the database structure.

    Source: A. Problem and motivation · p. 1

  • Method

    The system combines schema-constrained model decisions with typed tools, persistent state, lifecycle Hooks, and staged validation while deterministic software retains execution authority.

    Source: C. System approach · p. 2, §VII · p. 13

  • Results

    6/8 strict metadata and SQL tasks passed; metadata passed 4/4, SQL semantics 4/4 but exact output columns 2/4, and the final release passed 4/4 single-panel dashboard tasks.

    Source: §VII · p. 13

  • Takeaways & Limitations

    Artifact-producing agents require evaluation of machine-readable interface compatibility as well as query semantics, because semantically correct SQL can fail downstream panel contracts.

    Source: §VII · p. 13

    The evaluated prototype is DataBrain-specific, uses a shallow schema validator and lexical SQL policy, and does not expose standalone online publication or browser chat.

    Source: 7) Implementation limitations · p. 10, B. Threats to validity · p. 12

Abstract

from arXiv · show

We present a DataBrain-specific prototype that turns natural-language analytic requests into inspectable dashboards. An installable Codex plugin and standalone Agent Runtime combine schema-constrained model decisions with typed tools, persistent state, and deterministic Hooks for approval, audit, checkpointing, recovery, and failure handling. The pipeline resolves entities, discovers metadata, enforces read-only SQL, composes dashboards, and applies static checks, dynamic preflight, and browser inspection. The model proposes actions while deterministic software controls execution and records state transitions. We evaluate the workflow on frozen real-DataBrain tasks and controlled Hook faults. Strict success was 6/8 on metadata and SQL tasks: metadata selection passed 4/4, all four SQL tasks met semantic criteria, and 2/4 met the exact output-column contract. The final release passed 4/4 single-panel dashboard tasks, one two-panel task, and one existing-dashboard refinement; a parameterised task exceeded its step limit. All ten fault scenarios met their specified outcomes without unapproved external side effects. Model inference accounted for over 97\% of observed runtime in every reported group. These small, DataBrain-specific results do not establish production readiness, general text-to-SQL accuracy, or an efficiency advantage over manual dashboard construction.

I. INTRODUCTION

The paper addresses the linked work of turning business questions into valid, inspectable dashboards and aims to do so within DataBrain. Its approach keeps model-proposed actions under deterministic Runtime control and evaluates the resulting workflow.

A. Problem and motivation

  • Dashboard creation requires identifying entities and tables, inspecting schemas, writing SQL, choosing visuals and filters, and checking the rendered result.Users may know the decision they want to make without knowing the organisation’s database structure.Source: A. Problem and motivation · p. 1

B. Project aim and boundary

  • The aim is an inspectable, dynamically verified dashboard over DataBrain; the implemented endpoint is a validated local preview, not standalone online publication or web chat.The system is designed around DataBrain identifiers, metadata, data domains, SQL, and dashboard artifacts.Source: B. Project aim and boundary · p. 1

C. System approach

  • The workflow resolves an entity, discovers and inspects tables, generates bounded read-only SQL, builds panel artifacts, and applies static validation, approved preflight, and browser inspection.Lifecycle Hooks support approval, duplicate-call control, audit logging, checkpointing, recovery, and bounded handling of external failures.Source: C. System approach · p. 2

D. Research objectives

  • Four objectives operationalise the study: separate model decisions from authority and state, measure metadata and SQL correctness, test lifecycle Hooks, and assess dashboard completion, refinement, cost, and convergence.The evaluation concerns the company’s DataBrain environment.Source: D. Research objectives · p. 2

E. Contributions and evidence

The paper contributes a governed execution architecture and an evaluation of its DataBrain prototype. Results show successes across task groups alongside limits in exact SQL contracts and task convergence.

E. Contributions and evidence

  • The technical contribution is an architecture where the model proposes schema-constrained actions while deterministic software retains authority through typed tools, Hooks, and durable state.The evaluation preserves frozen criteria, failed runs, staged validation, timing records, and controlled fault scenarios.Source: E. Contributions and evidence · p. 2
  • 6/8 frozen metadata and SQL tasks met every strict criterion; metadata passed 4/4, all four SQL tasks met semantic criteria, and 2/4 met the exact output-column contract.The final release completed 4/4 single-panel tasks; ten controlled fault scenarios met their specified outcomes without unapproved external side effects, while an extension task exceeded its step budget.Source: E. Contributions and evidence · p. 2

A. Tool-using language-model agents

  • Prior agent benchmarks motivate trajectory-level evidence, but do not establish that this enterprise workflow enforces its own control boundaries.The evaluation records tool order, terminal state, approvals, latency, and prohibited actions alongside generated artifacts.Source: A. Tool-using language-model agents · p. 2

B. From language to data and visualisations

  • Text-to-SQL work addresses schema grounding and constrained generation, but the cited approaches do not cover this workflow’s exact result-column contract, approval state, and persistent visual artifact.The prototype operates within DataBrain and does not claim generalisation across unrelated databases or a new state-of-the-art text-to-SQL model.Source: B. From language to data and… · p. 3
  • Visualisation research covers analytic intent and chart specification, while this workflow also requires live-data safety and valid interactive reports across parameters and state transitions.Its panels retain SQL templates, parameters, chart configuration, and state; static validation, preflight, and browser inspection are separate gates.Source: B. From language to data and… · p. 3

C. Human control and lifecycle governance

  • Approval is bound to the exact tool name and canonical arguments, while checkpoint, approval, and policy failures stop execution; these mechanisms do not establish full organisational governance compliance.The sources motivate human control and inspectability but do not prescribe this workflow’s execution semantics.Source: C. Human control and lifecycle… · p. 3

D. Position relative to orchestration frameworks and retrievalaugmented generation (RAG)

The paper positions its small custom Runtime as an inspectable DataBrain-specific implementation, not a demonstrated improvement over existing orchestration frameworks. It frames the engineering problem around retaining deterministic authority while supporting natural-language dashboard work.

D. Position relative to orchestration frameworks and retrievalaugmented generation (RAG)

  • The custom Runtime makes event order, state representation, and approval authority inspectable, but no controlled comparison establishes it as better than LangGraph.The standalone Agent does not implement retrieval-augmented generation; it injects a source-labelled routing summary while verifying physical tables and columns through live metadata tools.Source: D. Position relative to orchestration… · p. 3

E. Engineering problem addressed

  • The engineering problem is to integrate metadata and dashboard actions while deterministic software retains authority over permissions, approval, persistence, and validation.The paper limits this claim to the DataBrain workflow rather than a general gap across databases or orchestration frameworks.Source: E. Engineering problem addressed · p. 3

1) Operational context:

  • The target workflow starts with a natural-language request and includes metadata selection, approval of real-data execution, artifact validation, and local browser inspection.It also supports constrained revision of an existing dashboard while preserving unrelated panels.Source: 1) Operational context · pp. 3–4

2) System requirements:

  • Table II organises functional requirements for the user-visible workflow and non-functional requirements that constrain execution.Approval attaches to a fingerprint of one tool name and canonical argument object; changing SQL, date range, or preflight request requires new approval.Source: 2) System requirements · p. 4

3) Domain boundary:

The study is bounded to a DataBrain prototype and a controlled evaluation that separates evidence by task and validation stage. Its data-handling procedure limits dissertation disclosure but does not replace company governance processes.

3) Domain boundary:

  • The project defines rules for DataBrain entities, public and business data domains, DataLab metadata, BigQuery-oriented SQL, permissions, and dashboard format; it does not claim general text-to-SQL accuracy.The evaluated standalone boundary ends at an approved, dynamically preflighted local preview; online publication and a user study are outside the completed evaluation.Source: 3) Domain boundary · p. 4

4) Privacy, ethics, and company data handling:

  • Real DataBrain observations may enter later model context, so each real run and cross-process resume requires explicit acknowledgement; raw evidence stays local and only manually reviewed, redacted derivatives enter the dissertation.The evaluation recruited no participants or collected new personal data, and its disclosure procedure does not replace company access-control and information-governance processes.Source: 4) Privacy, ethics, and company data… · p. 4

5) Evaluation design:

  • The evaluation uses frozen criteria and retained failures across metadata/SQL tasks, dashboard tasks, extension tasks, and ten controlled fault scenarios.Only tasks with frozen criteria and valid execution records enter reported denominators; infrastructure and benchmark-construction failures remain in local records but are not Agent outcomes.Source: 5) Evaluation design · p. 4

6) Outcome measures and analysis:

  • Strict dashboard success requires correct scope and fields, valid read-only SQL and panel, successful validation and preflight, usable rendering, and no prohibited or unapproved action.Run records, artifacts, live preflight, and browser inspection provide separate evidence; small-sample success proportions use Wilson 95% intervals, with no significance test or manual-workflow speed comparison.Source: 6) Outcome measures and analysis · p. 5, 5) Evaluation design · p. 5

B. System Architecture and Design

The architecture separates model proposals from deterministic enforcement and state recording. Plugin and standalone entry points share DataBrain and Dashboard capabilities, while a structured decision loop keeps proposed actions inspectable.

1) Design principles:

  • The model proposes actions, but deterministic code decides whether they may run and records their results.This separation prevents a fluent response from counting as approval, valid SQL, or a verified Dashboard.Source: 1) Design principles · p. 5

2) Dual-entry architecture:

  • The Codex plugin and standalone package are two orchestration surfaces over one shared DataBrain and Dashboard capability layer.The standalone package adds its own manager loop, typed tools, state, Hooks, resume command, and experiment runner.Source: 2) Dual-entry architecture · p. 6

3) Structured decision loop:

  • Each bounded iteration requests one schema-constrained decision, records it, and routes tool calls through validation and lifecycle Hooks.The provider interface allows a tool call, a user question, or completion; the Runtime remains authoritative for trace, approvals, stopping, and persisted state.Source: 3) Structured decision loop · p. 6

4) Hook governance layer:

Hooks place policy at stable lifecycle boundaries, while separate persistent state supports resumption and Dashboard continuation. The artifact pipeline moves from typed data access and panel construction through validation and preflight to a local preview.

4) Hook governance layer:

  • Hooks can continue, pause, block external dependencies, or block, with failure policy determined by the guarantee each Hook provides.Audit is fail-open; approval, state loading, circuit breaking, loop protection, and checkpoint persistence are fail-closed.Source: 4) Hook governance layer · p. 6

5) Two-layer state and recovery:

  • AgentState and Dashboard artifacts persist separately, allowing a later process or reviewer to resume work without replaying the original conversation.Corrupt Dashboard state fails closed, and real company-data runs require acknowledgement again after cross-process resume.Source: 5) Two-layer state and recovery · p. 6

6) Data and artifact pipelines:

  • The DataBrain boundary provides typed metadata and SQL tools, while separating entity identity from data scope and distinguishing authentication from permission failures.The common metadata path resolves entities, searches bounded keywords, selects candidates, and retrieves schemas.Source: 6) Data and artifact pipelines · p. 7
  • Panel artifacts are checked before verification; report validation is static, while approved preflight tests bounded queries against live DataBrain results.Preview requires a successful receipt bound to the exact HTML hash, and regenerating the report invalidates that receipt.Source: 6) Data and artifact pipelines · pp. 7–8

7) Security trade-offs:

The prototype combines a bounded Runtime, provider adapter, governance Hooks, and DataBrain and Dashboard tools. Its security measures establish a prototype boundary, not a formal security proof.

7) Security trade-offs:

  • Security controls include a sanitised model environment, a read-only sandbox, and workspace-constrained panel identifiers and paths.An operating-system token may still be inherited by general Dashboard child processes.Source: 7) Security trade-offs · p. 8
  • The prototype’s SQL checks are lexical rather than parser-based, and secret redaction relies on recognised patterns.The localhost preview proxy remains a credential-bearing process while active; the authors call for further security assessment before wider deployment.Source: 7) Security trade-offs · p. 8

C. Implementation

  • The standalone implementation is a Python package with a bounded Runtime, provider adapter, governance Hooks, typed tools, and evaluation instrumentation.It uses Python data classes, protocols, subprocess management, JSON persistence, and a typed-tool layer.Source: C. Implementation · p. 8

1) Runtime implementation:

  • The Runtime accepts one tool request, user question, or final response per iteration and records each transition in a structured trace.The model cannot register tools, bypass argument checks, grant approval, alter checkpoints, or assign terminal phases directly.Source: 1) Runtime implementation · p. 8

2) Model-provider integration:

  • The provider adapter uses the official Codex CLI and a required decision schema; malformed outputs become provider failures rather than executable actions.Before launching the model subprocess, it removes billing and credential-like environment variables, including DataBrain credentials.Source: 2) Model-provider integration · p. 8

3) Governance and persistence:

Governance Hooks and separate persisted state constrain approvals, recovery, and failure handling. Typed DataBrain tools enforce scope and read-only SQL rules, while the dashboard lifecycle distinguishes local structural checks from live preflight.

3) Governance and persistence:

  • Deterministic Hooks pause or block runs at lifecycle boundaries, and safety or recovery Hooks fail closed when their guarantees fail.Audit is fail-open; approvals are bound to exact tool arguments, consumed after execution, and renewed when arguments change.Source: 3) Governance and persistence · pp. 8–9

4) DataBrain integration:

  • Separate Agent and Dashboard state lets later processes continue artifacts without replaying model history, while corrupt Dashboard state blocks resume.The DataBrain adapter enforces scope routing, bounds metadata retries, classifies 401 and 403 separately, and does not automatically retry real SQL execution.Source: 3) Governance and persistence · p. 9, 4) DataBrain integration · p. 9

5) Dashboard artifact lifecycle:

  • The artifact lifecycle verifies panels before composition, then separates static report checks from approved live preflight.Preflight receipts bind success to the exact report hash, and preview is allowed only while that hash still matches.Source: 5) Dashboard artifact lifecycle · p. 9

6) Evaluation instrumentation:

The evaluation uses frozen task definitions, retained failures, and separate evidence sources to assess a governed DataBrain workflow. Metadata selection passed, while SQL strict success was limited by exact output-column mismatches; the implementation also has defined technical boundaries.

6) Evaluation instrumentation:

  • Each run records task configuration and policy, while append-only run and browser records support reproducible outcome derivation and regenerated summaries.Timing separates model, tool, and wall time; browser inspection can reject structural success but cannot promote a structured failure.Source: 6) Evaluation instrumentation · pp. 9–10

7) Implementation limitations:

  • The evaluated prototype uses a shallow schema validator and lexical SQL policy, with synchronous tools and pattern-based secret redaction.An operating-system DataBrain token may be inherited by operational Dashboard subprocesses; standalone Runtime also omits online publication and browser chat.Source: 7) Implementation limitations · p. 10

A. Evidence and adjudication

  • Task groups remain separate because they cover different task families and step budgets; small binomial groups use Wilson 95% confidence intervals.No significance test, ablation claim, or manual-workflow speed comparison is made.Source: A. Evidence and adjudication · p. 10

B. Metadata and read-only SQL

  • 6/8 core tasks met every strict criterion; metadata passed 4/4, SQL semantics passed 4/4, but exact output-column compliance was 2/4.The two strict SQL failures returned aliases different from the frozen contract; one query was repaired after an identifier error and renewed approval.Source: B. Metadata and read-only SQL · p. 10

C. Final single-panel Dashboard correctness

The final release completed all four frozen single-panel tasks, with one successful two-panel task and one state-preserving Dashboard refinement. A parameterised task failed to terminate within its step budget, while observed runtime was dominated by model decisions.

C. Final single-panel Dashboard correctness

  • 4/4 frozen single-panel tasks passed strict criteria, and browser inspection found the requested visible results without page-console warnings or errors.Each task also had one rejected panel-build call using sql instead of sql_template, which the Agent repaired on its next decision.Source: C. Final single-panel Dashboard… · p. 10

D. Extension, parameters, and refinement

  • D06 passed structured checks, preflight, and browser inspection with exactly two non-empty panels, establishing success only for this evaluated multi-panel task.The joint D05/D07 parameter task reached correct intermediate parameter and preflight states but failed to return a final decision within 30 steps.Source: D. Extension, parameters, and refinement · p. 10
  • R03 preserved the copied Dashboard’s panel and chart configuration while changing its time window from 30 days and 31 rows to 14 days and 15 rows.Preflight and browser inspection passed, and no initialise, add-panel, preview, or publication action was attempted.Source: D. Extension, parameters, and refinement · pp. 10–11

E. Efficiency

  • Model time averaged 438.834 seconds per core task versus 9.141 seconds of mean tool time; final single-panel model decisions averaged 28.65 minutes versus 16.31 seconds for tools.The measurements do not establish a speed advantage over human dashboard construction because no manual baseline was collected.Source: E. Efficiency · p. 11
  • Repeated sequential model decisions were the main observed efficiency constraint, with each repair, validation, or completion decision adding another model round trip.The D05/D07 run also failed to terminate within its step budget.Source: D. Extension, parameters, and refinement · p. 10, E. Efficiency · pp. 11–12

F. Controlled Hook fault evaluation

The controlled Hook tests met all ten specified outcomes without unapproved external side effects, while the broader evaluation found uneven task completion and important limits on what the evidence establishes.

F. Controlled Hook fault evaluation

  • All ten controlled Hook scenarios produced their specified outcomes without an unapproved external side effect.Table VII groups the observed outcomes by governance function.Source: F. Controlled Hook fault evaluation · p. 12, Table VII · p. 13
  • The fault matrix supports controls only for registered tools and injected fault classes, not arbitrary prompt injection, lexical-policy bypass, or reliability under live outages.It provides evidence for deterministic approval, bounded metadata failure, observable fail-open telemetry, and fail-closed recovery.Source: F. Controlled Hook fault evaluation · p. 12

A. Interpretation of findings

  • The architecture separates probabilistic model proposals from deterministic execution authority and persists Agent and Dashboard state separately for recovery.The plugin and standalone entry points reuse DataBrain and artifact capabilities but retain distinct orchestration mechanisms.Source: A. Interpretation of findings · p. 12
  • Strict success was 6/8: metadata selection passed 4/4, while SQL passed 2/4 strictly despite all four tasks meeting semantic criteria.Two semantically valid queries returned aliases that violated the frozen downstream panel contract.Source: A. Interpretation of findings · p. 12
  • The controlled fault matrix covered ten declared scenarios, including changed-argument reapproval, bounded retries, loop interruption, checkpoint failure, and fail-open audit behaviour.No unapproved external side effects were recorded; the finding is limited to registered tools and injected fault classes.Source: A. Interpretation of findings · p. 12
  • The final dashboard release passed four single-panel tasks, one two-panel task, and one refinement, but the parameter task exceeded 30 steps; model decisions took over 97% of wall time.Successful intermediate preflights did not prevent the manager loop from continuing to rebuild the artifact.Source: A. Interpretation of findings · p. 12

B. Threats to validity

  • The evidence supports the specified prototype tasks, but not production readiness, a general reliability rate, or unattended daily reporting.Small denominators, one confirmatory run per task, evaluator and experimenter bias, construct choices, and restricted domains constrain interpretation.Source: A. Interpretation of findings · p. 12, B. Threats to validity · pp. 12–13

VI. FUTURE WORK

The conclusion reports successful outcomes on several frozen DataBrain tasks and controlled faults, alongside failures in exact output contracts and parameterised-task termination. Future work targets contract adherence, convergence, broader evaluation, and portability beyond DataBrain.

VI. FUTURE WORK

  • Future engineering should improve output-contract adherence and deterministic convergence, including avoiding unnecessary rebuilds after successful preflight.The proposed work also targets fewer sequential model calls, parser-backed SQL validation, credential inheritance, and preview-process cleanup.Source: §VI · p. 13
  • Future evaluation should repeat representative tasks, add a governance ablation, broaden entity and schema coverage, use independent visual reviewers, and include a manual baseline for efficiency comparisons.The proposed ablation would compare the full Runtime with variants that remove approval or loop-control mechanisms.Source: §VI · p. 13
  • Portability remains unestablished because the prototype relies on DataBrain-specific metadata, permissions, artifact formats, and validation rules.Future work could separate reusable governance mechanisms from DataBrain-specific adapters and evaluate them in other enterprise analytics environments.Source: §VI · p. 13

VII. CONCLUSION

  • The system combines schema-constrained model proposals with typed tools, lifecycle Hooks, durable state, staged validation, and controlled recovery.The plugin and standalone Runtime share DataBrain and dashboard capabilities.Source: §VII · p. 13
  • The final release passed 4/4 single-panel tasks, one two-panel task, and one state-preserving refinement; exact SQL output contracts passed 2/4, and the parameterised task exceeded its step budget.The ten controlled fault scenarios met their specified outcomes without unapproved external side effects; these are declared-task observations, not general rates.Source: §VII · p. 13
  • Exact field compatibility matters alongside query semantics because a semantically correct query can still violate the downstream panel contract.The evaluation found two SQL tasks with alias mismatches despite all four satisfying the semantic rubric.Source: §VII · p. 13
  • Successful intermediate tool execution did not ensure agent-level convergence: the parameterised task produced valid panels and preflights before reaching its step limit.The conclusion recommends treating completion and termination as explicit system properties rather than leaving them entirely to model judgement.Source: §VII · p. 14

Related papers

Loading 2610.06914v2…