Source-linked AI summary
MasterControl Seventeen Every Time
MasterControl AI Lab
TL;DR
Enterprise analytics must preserve not only an answer but also its meaning and evidence. This paper lets a language model interpret requests while deterministic policy selects a pre-approved analytical program, showing exact expressiveness within a stated class and a configuration-specific advantage over runtime planning. The approach is bounded by model interpretation, data completeness, policy correctness, and the tested setup.
Problem
Enterprise answers can share a number while measuring different populations, so governed analysis needs stable meaning, value, and supporting evidence.
Method
A language model interprets accepted meaning, while deterministic policy selects and executes pre-written analytical programs covering relational operations and explicit analytical kernels.
Results
In the tested 440-episode setup, runtime-planning agents produced no exact full-suite analytical contracts, while policy execution matched 110/110.
Takeaways & Limitations
For supported factual analyses whose approved method is known, governing method ownership can provide replayable results and evidence without requiring runtime method composition.
Takeaways & Limitations
The benchmark is configuration-specific, and deterministic execution does not correct wrong meanings, incomplete data, or substantively wrong policies.
Abstract
from arXiv · showhide
We study a governed approach to enterprise analytics: a language model interprets the question, while deterministic policy selects and runs a pre-approved analytical program that returns both results and evidence. We show that this restriction can remain expressive within a defined analytical class, using relational operations plus aggregation, comparison, windows, ranking, and similarity. Fixed meaning, policy, data, and execution rules also make results replayable. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B interpreted intent only and policy executed the approved program. None of 330 runtime-planning episodes matched the full answer-and-evidence contract across all test datasets; the policy-executed analyzer matched 110 of 110. This is a configuration-specific result, not evidence that runtime agents cannot succeed under other designs.
1 The problem: 17 is not just a number
The paper frames enterprise answers as governed analytical contracts rather than bare numbers: meaning, value, and evidence must remain aligned. Language-model flexibility is retained for interpretation, while policy fixes the analytical method.
- The problem: 17 is not just a number: “Last week,” category definitions, joins, and voided-record rules can change what an identical answer measures.Two systems may print 17 while measuring different populations.
- The problem: 17 is not just a number: A governed answer includes accepted meaning, analytical value or table, and role-labeled supporting evidence.An execution certificate also records snapshot, policy, program, kernel, and serialization versions.
- The problem: 17 is not just a number: The model may interpret the question, but it does not choose the analytical method.This boundary separates flexible language interaction from fixed policy and versioned code.
- The problem: 17 is not just a number: Runtime choice of tools, joins, filters, sequence, and stopping rule makes the agent a designer of the analytical procedure.That differs from merely calculating a requested answer.
2 A small analytical language
The proposed execution language is a small, typed set of analytical families whose contracts govern inputs, outputs, parameters, errors, and evidence. It is intentionally restrictive, yet broad enough for the paper’s stated class of database analyses.
- A small analytical language: Each operation has typed inputs, typed outputs, declared parameters, error behavior, and evidence semantics.The contracts define how approved operations behave rather than leaving semantics to runtime composition.
- A small analytical language: A running count can be implemented as a reviewed program chaining restriction, a calendar-week window, a temperature definition, aggregation, and shaped evidence.The example counts distinct deviation_id values for the previous calendar week.
- A small analytical language: Approved programs are authored and tested before deployment, or deterministically expanded from approved policy rules, rather than generated by the model.Different supported questions map to different approved programs.
- A small analytical language: “Complete” means complete relative to a stated query class, not universal computation.The paper uses relational algebra’s restricted-but-complete role for an important database-query class as the analogy.
3 From relational logic to analytical programs
The paper constructs analytical programs from finite-domain relational logic and extends that foundation with explicit kernels for common analytical operations. It proves exact implementation for the resulting stated class, while marking boundaries around counting and catalog scope.
- From relational logic to analytical programs: A relational question is defined by a first-order formula over finite candidate domains, independently of how it is executed.The formula specifies which tuples satisfy the question.
- From relational logic to analytical programs: Selection, projection, renaming, product, union, and difference translate logical predicates, quantification, disjunction, and negation into relational operations.Universal quantification is handled by removing assignments with counterexamples.
- From relational logic to analytical programs: For every finite-domain first-order specification, a finite relational-core program produces the same result on every admissible database, and the converse also holds.This is the constructive relational-expressiveness theorem.
- From relational logic to analytical programs: Pure relational operations do not produce variable-size counts as new scalars, so counting requires an explicit analytical extension.Difference or an equivalent non-monotone operation is also needed for absence questions.
- From relational logic to analytical programs: The registry adds scalar derivation, aggregation, windows, comparisons, ranking, and fixed proximity functions to the relational stages.These kernels have mathematical definitions independent of the family names.
- From relational logic to analytical programs: Every finite acyclic specification in the declared analytical class has a finite well-typed program returning the same value, evidence, or declared domain error.The construction processes stages in dependency order and applies the corresponding registered kernels.
4 Policy owns the method
Policy separates analytical expressiveness from deployment approval: approved templates are authored, reviewed, tested, and versioned before requests arrive. At runtime, deterministic policy binds accepted meaning to an approved program without model-selected composition.
- Policy owns the method: Supported analytical requests are represented as templates with business parameters such as sites, products, intervals, or thresholds, not code or primitive lists.The supported family is defined independently of implementation.
- Policy owns the method: Engineers and domain experts author, review, test, and version a program for each supported template at design time.Request-time execution binds only the allowed typed business parameters.
- Policy owns the method: The language model does not emit SQL, primitive names, primitive order, or a program identifier.This prevents the model from composing the analytical method at request time.
- Policy owns the method: If templates belong to the stated analytical class, pre-written programs and deterministic policy execute correctly bound requests exactly without model-selected composition.The guarantee follows by compiling each symbolic template once and binding typed parameters at runtime.
- Policy owns the method: The algebra answers whether an analysis can be expressed, while policy answers whether it is approved and supported in the deployment.An uncovered request becomes a coverage gap rather than an invented method.
5 What is guaranteed - and what is not
Replay is guaranteed when the accepted request and complete governed state are fixed and policy resolution, kernels, and execution conventions are deterministic. This guarantee makes the selected analysis reproducible, not necessarily correct.
- The governed state includes data, authorization, meaning definitions, policy, program catalog, kernels, numerical conventions, and output contract.
- Repeated successful executions with the same canonical request and governed state return the same analytical contract and execution certificate.
- Replay follows by induction over the program dependency graph and canonical serialization; latency and run identifiers are excluded from analytical equality.
- Determinism begins after meaning acceptance and does not correct wrong meanings, incomplete data, or substantively wrong policies.
- Approximate kernels preserve threshold or top-k membership only when scores have sufficient margins from the threshold or ranking boundary.
- Deterministic tools do not ensure runtime agents select the same analytical function without a specified search process, progress condition, and sound acceptance rule.
6 Empirical demonstration: does runtime planning help?
The experiment asks whether creating analytical procedures at runtime helps when policy already supports the analyses. It compares runtime procedure construction with deterministic execution across eleven tasks and controlled panels.
- The benchmark used one synthetic governed quality/manufacturing dataset and eleven tasks spanning counts, grouping, absence, ranking, rates, joins, means, thresholds, and regional aggregation.
- Four setups were compared on the same hardware, including three runtime-planning agents and a policy-executed analyzer.
- Runtime agents inspected data, wrote and executed SQL, revised procedures, and selected the final method, whereas policy execution ran a pre-written program.
- The same Qwen3-8B model appears in both conditions, isolating whether the model owns analytical method selection rather than model capability.
- The natural-language panel required runtime agents to interpret questions and construct procedures, while policy execution interpreted intent and then ran the approved method.
- The fixed-request panel supplied accepted meaning to every system, removing semantic interpretation as an explanation for analytical failure.
7 Results
Across the completed benchmark, runtime planning failed to satisfy the full analytical contract, while policy execution succeeded exactly in the fixed-request comparison. Correct numbers alone did not guarantee correct governed analysis.
- 440 episodes completed the benchmark, but “Final” indicated only that a procedure or result was returned, not that it was correct.
- Of 330 runtime-planning episodes, 220 were rejected, 55 exhausted the tool budget, and 55 returned final programs; none was exact across five snapshots.
- Five episodes produced the correct visible-snapshot number, but wrong evidence roles and empty-population behavior prevented exact governed analysis.
- In the fixed-request panel, runtime agents achieved 0 exact contracts in 165 episodes, while policy execution achieved 55/55 exactly with no model inference.
- In the natural-language panel, policy used about 4.8 times fewer tokens and 29 times less online time than Qwen runtime planning while moving from 0/55 to 55/55 exact contracts.
- Repeatable procedures were sometimes wrong, so low retries and temperature-zero repeatability should not be interpreted as efficient correctness.
8 Discussion
The paper separates what its proofs establish from what its benchmark demonstrates. Its narrower conclusion concerns supported factual analyses whose approved methods are already known, while deployment requires deliberate coverage work.
- The proofs establish representability within the declared analytical class and replay under fixed governed state, whereas the benchmark illustrates the operational importance of method ownership.
- The benchmark does not show that runtime-planning agents are generally incapable of analysis, because other models, prompts, budgets, verification, or search procedures could perform better.
- Approved programs must be authored, reviewed, tested, versioned, and maintained, so coverage expands deliberately rather than invisibly.
- Product evaluation therefore requires measuring how many real user questions map to reviewed programs and the cost of extending coverage.
9 Conclusion
A governed analyzer separates flexible language interpretation from stable factual measurement: policy selects an approved analytical program, which returns results with evidence. In the tested configuration, this design illustrates a practical trade-off between runtime planning and deterministic execution.
- Human language may benefit from model-based interpretation, while known analytical tasks should use governed methods for stable factual measurement.
- A compact relational core with explicit analytical kernels can exactly implement a broad stated class of finite data analyses.
- Across 440 episodes, runtime-planning agents produced no exact full-suite analytical contracts, while semantic interpretation followed by deterministic policy execution produced 110/110.The result is explicitly limited to the tested configuration.
- The recommended architecture lets the model determine user meaning while policy determines how an approved analysis is performed and returns the evidence with the answer.
A.1 Proof of Theorem 1
The proof translates finite first-order specifications into relational expressions and establishes that tuple membership matches formula satisfaction. Structural induction covers logical operators and quantifiers, while a separate induction argument establishes termination and replay under fixed execution state.
- For each finite variable set, the construction assigns a relation with that schema to a formula after bound variables are alpha-renamed.
- The structural-induction proof establishes that a tuple belongs to the constructed relation exactly when it satisfies the corresponding formula.
- Relational difference implements negation and conjunction, union implements disjunction, projection implements existential witnesses, and nested difference handles universal quantification.
- The reverse induction maps relational operations to logical constructs, preserving tuple membership at every step.
- A finite acyclic program of terminating typed kernels terminates by topological induction, and fixed state makes node outputs and sink results replayable.