Source-linked AI summary
Output Discipline Sets the Floor: A Falsification-Guided, Model-Agnostic Agent Protocol for Auditable Hypothesis-Driven Decision Making
Jing Liu, Yongxing Qi, Ping Yao, Ting Wu
TL;DR
On auditable, hypothesis-driven decisions, LLMs may lack reliable output discipline and candidate-level verification. This paper evaluates a model-agnostic falsification protocol and finds that it rescues weak-discipline configurations and stabilizes a high-variance model.
Problem
For auditable decisions, LLMs may reach conclusions without proposing candidate hypotheses and checking each against evidence, leaving the reasoning difficult to verify or audit.
Source: pp. 1–2Method
The paper uses a model-agnostic agent framework that enforces candidate-by-evidence falsification, requiring at least half the candidates to be falsified before licensing a conclusion.
Source: p. 2Results
Across TCMEval-SDT, the protocol rescues weak-discipline configurations (+0.267/+0.271 on DeepSeek-V4-Flash) and stabilizes high-variance Kimi K3 by an order of magnitude.
Source: future work · p. 18Takeaways & Limitations
On TCMEval-SDT, the findings identify output discipline and reasoning structure, not model capability alone, as binding constraints on performance.
Source: future work · p. 18Evidence is single-domain at benchmark scale; a second full-scale domain is needed before the boundary condition can be treated as domain-general.
Source: future work · p. 18
Abstract
from arXiv · showhide
What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the public TCMEval-SDT benchmark for Traditional Chinese Medicine syndrome differentiation (200 cases), the binding constraint is output discipline and reasoning structure, not model capability alone: two years of model scaling move zero-shot scores by only -0.03 to +0.08, while output-discipline failures (non-terminating reasoning, format non-compliance, large run-to-run spread) persist even in frontier models. We test this with a model-agnostic agent framework formalizing decision making as a five-stage closed loop, enforced by a candidate x evidence matrix falsification protocol: every candidate is judged against every evidence item, at least half of the candidates must be falsified before a conclusion is licensed, and survivors are emitted in a machine-verifiable single-line format. Across five base models, the effect is sign-conditional on zero-shot output discipline: it rescues models whose discipline fails (DeepSeek-V4-Flash: +0.267/+0.271, entirely termination rescue), reduces a high-variance model's run-to-run standard deviation from 0.218 to 0.007 (Kimi K3), and significantly costs already-disciplined models (-0.087 to -0.120 on budget-matched Qwen3-4B/32B). Ablations attribute the cost to the candidate-bounded format layer and show the effect is independent of injected knowledge content (protocol-knowledge decoupling). The protocol also transfers to misinformation-robust mathematical reasoning, detecting injected false premises at 89.8% recall with a 6.8% false-positive rate (MCC 0.54; a flag-everything baseline scores 0) and restoring accuracy to the no-misinformation ceiling. Where zero-shot output discipline is weak, the reasoning protocol sets the floor; base-model capability always sets the ceiling.
future work.
Future work is framed around testing whether the findings generalize beyond one benchmark-scale domain, improving measurement and mechanism evidence, and strengthening the framework’s questioning and formal-validation methods.
future work
- A second full-scale domain is needed to establish whether the observed boundary condition generalizes, and an instrumented rerun should archive reasoning traces and per-case token usage.Current budget certification relies on wall-clock time, which includes network and retry overhead; empty visible outputs also prevent post-hoc classification of non-termination.Source: future work · p. 18
- Further mechanism tests should add budget-matched DeepSeek zero-shot, same-model CoT and self-consistency baselines, and a matrix-only ablation to isolate the matrix’s contribution.The budget-matched DeepSeek control was substituted by post-hoc elapsed-time certification because the endpoint no longer served the original Flash weights.Source: future work · p. 18
- The deployed questioning policy scores coverage rather than expected information gain; a response model could enable an information-gain-based upgrade.The current rule is presented as a coarse proxy for greedy information-gain selection, not as an implemented information-gain system.Source: future work · p. 19
- Expert-annotated cell audits are future work for estimating the unmeasured error rate, and a full multi-label bound under the official metric remains undeveloped.The bound also relies on idealized conditional independence and does not control candidate-pool coverage; at m=10 and n≈10 it is vacuous for ε=0.20.Source: future work · p. 21
■Data and Code Availability
The benchmark, evaluation materials, and case-level logs are publicly available, and the protocol is specified for independent reimplementation. Agent code and private medical datasets are not publicly released.
Data and Code Availability
- TCMEval-SDT is public under CC-BY 4.0, and the repository provides evaluation scripts, the reconstructed 243-question misinformation set, and per-case logs.The misinformation set was rebuilt from the retained evaluation aggregate of [6] and has documented selection bias.Source: ■Data and Code Availability · p. 22
- The protocol is specified in Section 3 for independent reimplementation, but agent source code and prompt templates are available only from the corresponding author while a patent application is pending.The specification covers matrix construction, cell judgments, aggregation and sufficiency rules, and output format.Source: ■Data and Code Availability · p. 22
- The 8,147-record case database, 539-case specialty dataset, and 1,103-pair distilled dataset cannot be released for privacy and licensing reasons.Source: ■Data and Code Availability · p. 22
■Acknowledgements
The authors acknowledge the TCMEval-SDT benchmark creators and the developers of the DeepSeek, Kimi, and Qwen model families.
■Acknowledgements
- The authors thank the TCMEval-SDT creators for making their data public and the developers of the DeepSeek, Kimi, and Qwen model families.Source: ■Acknowledgements · p. 22
■Competing interests
The authors disclose a pending patent application covering the agent mechanism and report no other competing interests or financial conflicts.
Competing interests
- A patent application covering the agent mechanism is pending, and the authors declare no other competing interests or financial conflicts.Source: ■Competing interests · p. 22