Source-linked AI summary
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Hanlin Tian, Minhao Li, Yu Mi, Sihan Zhu, Zhao Yang, Yuxiang Wang, Hongquan Zhu, Qiufei Hu
TL;DR
Professional agent tasks may depend on conventions absent from public corpora, yet benchmarks often do not control access to those conventions. The paper introduces an audited, knowledge-gated construction and verification protocol, finding a 68.0% versus 0% pass-rate contrast across fifteen calibration tasks for one frontier configuration. The experiments validate protocol behavior, not whether retained tasks improve post-training.
Problem
Professional agent tasks can require specialised conventions unavailable from public corpora, while benchmarks rarely control whether agents can access them.
Method
The protocol separates byte-identical task instructions from compact human-curated artefacts, audits leakage, uses executable witnesses or named rubrics, and calibrates candidates under repeated ablations.
Results
Across fifteen tasks, the tested frontier configuration achieves a 68.0% pass rate with the artefact and 0% without it; the screen retains seven tasks.
Takeaways & Limitations
The protocol makes dependence on supplied knowledge explicit and testable, producing an operationally calibrated release candidate for knowledge-gated task evaluation.
Takeaways & Limitations
The study does not establish that retained tasks improve post-training, and its strongest verification guarantee applies to deterministic, checkable deliverables.
Abstract
from arXiv · showhide
Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.
1 Introduction
The paper frames specialised-knowledge dependence as a task-construction problem: curators must make private conventions necessary while preserving verifiable correctness. It proposes a protocol that separates compact artefacts from byte-identical instructions, audits leakage, and calibrates empirical dependence without claiming downstream training value.
- Professional tasks can be under-specified when relevant rules are unavailable, but add little beyond generator capability when the generator already knows them.
- The protocol uses construction-time asymmetry: curators plant non-derivable choices while retaining cheap verification of structured outputs.
- Each task pairs byte-independent instructions with a human-curated kilobyte artefact, while leak audits test whether gated content appears in the instruction or environment.
- Across fifteen tasks, Opus moves from a 0% pass rate without the artefact to 68.0% with it under byte-identical instructions.
- Deterministic witnesses verify structured outputs exactly, named rubric criteria localise open-form failures, and provenance limits each task’s evidential claims.
- The experiments characterise protocol behavior rather than demonstrating that the retained tasks improve post-training.
2 Related Work
The paper distinguishes knowledge-gated task construction from generator-limited synthesis, optimisation-based dataset distillation, and ordinary procedural grounding. Its contribution is an audited enforcement protocol, while calibration is used only to identify an intended operational profile rather than to establish training value.
- Prior synthetic-instruction methods can plateau at generator capability, while dataset distillation optimises synthetic training data rather than runtime context.
- This work hand-authors inference-time artefacts and adds instruction–artefact separation plus leak audits to isolate their contribution.
- Knowledge-grounded benchmarks commonly provide procedural or domain information, but this protocol plants gated content that is not derivable from the instruction or environment.
- Success measures whether an agent holds and applies a supplied key, which overlaps with but is not identical to professional competence.
- The authors use related verifiable-reward ideas only to motivate offline calibration, not to estimate policy-optimisation value or prove downstream training benefit.
3 Methodology
The methodology pairs synthetic tasks with compact knowledge artefacts, audits their separation, verifies outputs through executable or rubric-based procedures, and calibrates retention using repeated trials. The resulting screen is configuration-relative and deliberately treated as a coarse quality-control rule.
- Each curation unit combines a task specification with a reusable artefact containing terminology, conventions, formulas, tables, and utility operators rather than an end-to-end solution.
- The study uses synthetic records and combines study-specific tasks with adapted or vendor-sample material, rather than claiming fifteen wholly new tasks.
- Convention gates plant private non-derivable choices, whereas operator gates place difficult algorithms in the artefact.
- Instructions remain byte-identical across artefact conditions, and static audits check for explicit gated constants or artefact mentions while acknowledging semantic leakage may remain.
- Each candidate receives five trials per calibration cell, with empirical pass rate defined as kt,A,c/5 across agent configurations and artefact conditions.
- Retention is configuration-relative and encodes separation between conditions, not variance reduction, statistical testing, or intrinsic model ranking.
- Structured outputs use independent deterministic implementations and strict tolerances, while free-form rewards require every named rubric criterion to pass.
4 Experiments
The experiments evaluate whether knowledge-gated tasks behave as intended, using paired artefact conditions, task-level calibration, executable verification, and a recitation probe. The artefact strongly changes Opus’s pooled pass rate, while additional controls show configuration- and harness-dependent limits on interpretation.
- 4.2 The artefact induces the intended task dependency: 68.0% pooled pass rate with the artefact versus 0% without it was observed for AF across fifteen calibration tasks.Amid,+A achieved 22.7%; each reported cell contained exactly five trials.
- 4.2 The artefact induces the intended task dependency: Seven of fifteen candidates were retained by the configuration-relative screen, while eight were marked for revision or removal.The retained tasks were legal-clause, paper-index, bom-costing, airline-booking, clinical-variant, mortgage-escrow, and dependency-audit.
- 4.2 The artefact induces the intended task dependency: A perturbed bom-costing artefact with plausible but incorrect conventions performed no better than no artefact in the reported control.This was a single-task, single-model control.
- 4.2 The artefact induces the intended task dependency: Every calibration task used a deterministic executable verifier, and withholding the artefact yielded no passes on any task.The table reports per-task passes/runs with Wilson 95% intervals; no task relied on LLM-judged rubric scoring.
- 4.3 A recitation probe: locating a configuration failure: Qwen recited every gated convention correctly and achieved a 100% pass rate in 20 reduced conversations, yet scored 0% on both tasks under the full agentic harness.The probe therefore points toward tool use, file discovery, or output-format compliance rather than missing conventions or computation inability, while remaining a two-task, single-model case study.
- 4.5 Descriptive behavior of the calibration screen: 42.9% of retained-task +A cells were mixed versus 40.0% across all candidates, but the comparison is descriptive rather than independent confirmation.The retained set was selected using the same configuration outcomes, and connecting variance to learning utility would require a held-out learner or post-training experiment.
5 Discussion and Future Work
The paper frames its contribution as a construction and validation protocol, not evidence that retained tasks improve post-training. Future work should strengthen auditing, repeated-trial calibration, and cross-configuration validation before testing training benefits.
- The retained tasks demonstrate controlled dependence on supplied artefacts under tested configurations, but have not been shown to produce superior post-training.
- The authors conjecture that private-convention gating may resist self-correction because planted thresholds and normalisation tables are absent from the instruction and environment.
- The calibration pool has fifteen tasks with five runs per cell, automated leak audits cover nine tasks, and the two configurations change both model and harness.These factors make individual retention decisions sensitive to single outcomes and complicate interpretation of low full-harness scores.
- Seven of fifteen calibration tasks satisfy the retention rule, while five representative retained tasks are publicly released.The release includes contract clause risk scoring, publication indexing, BOM cost rollup, airline booking audit, and mortgage escrow analysis.
- The next step is validation with complete audits, more trials near decision boundaries, a common harness, and held-out agent configurations before post-training comparisons.
6 Conclusion
The paper presents controlled task construction as a way to make knowledge access explicit and auditable. Its calibration identifies configuration-relative artefact dependence, while leaving post-training effectiveness for future study.
- The protocol makes knowledge access an explicit, auditable variable rather than an incidental property of a prompt.
- Across fifteen calibration tasks, the compact artefact changes the tested frontier configuration from total failure to majority success under byte-identical instructions.
- Deterministic witnesses, leak audits, and named rubric criteria make the intended dependency inspectable across structured and non-executable evaluations.
- The contribution is a method for constructing and validating knowledge-gated agent tasks, not evidence that retained tasks are better training data.
7 Appendix: Signal Retention under Procedural Compression
A single paper-index case study tests whether the artefact’s task-solving signal survives procedural compression. The minimal 1078 B artefact matches the full document, while the unprovided condition fails, but the result is not a general compression curve.
- 100% pass rates occur with both the 1078 B minimal artefact and the 4379 B full document, versus 0% with no artefact.Each condition uses five trials, and the task instruction, verifier, and harness remain byte-identical.
- The operative content is a small set of conventions: an inclusion rule, precedence order, two alias tables, and one operator convention.
- The experiment locates no degradation point because both tested non-empty artefacts achieve 100%, leaving the breaking size below 1078 B untested.
- A general compression–performance curve would require intermediate sizes, multiple tasks and domains, and per-rule ablations.The authors present this study as a single existence proof rather than a general compression result.
- The minimal artefact preserves the paper-index task-solving signal through 4× compression.It retains the gated conventions while removing explanations and worked examples.