Source-linked AI summary
Structure for Reading, Prose for Writing: Asymmetric Structural Conditioning in Multi-Agent Document Authoring
Cheng Yu, Nikhil Mathew, Zhengjie Wang
TL;DR
Formal document authoring requires systems to read heterogeneous requester files and write answers within their layouts under sovereignty constraints. This paper reports a deployed multi-agent tender-response system and finds that structural markup improves reading, while prose and self-applied tests are better suited to writing.
Problem
Formal document authoring requires reliable extraction, multi-step analysis, validated output, and preservation of requester wording under data-sovereignty constraints.
Method
The paper evaluates a deployed multi-agent tender-response system against human-submitted bids and tests structural markup, forbidden-construction instructions, and annotation-windowing choices.
Results
The system matched or exceeded human answers on 40 of 55 sections, while structural markup improved reading tasks but reduced paired conditioning quality from 74% to 48%.
Takeaways & Limitations
Ground-truth gaps often reflect unavailable organisational knowledge rather than writing quality, and structure should support reading while output tests support writing.
Takeaways & Limitations
The ground-truth comparison used one procurement and one same-family LLM judge without blinded human scoring, while gap classification retained model-family circularity.
Abstract
from arXiv · showhide
Multi-agent pipelines that author formal documents must both read a requester's forms and write against them. We report a deployed tender-response system, running an open-weights model under sovereignty constraints, and evaluate it against human-written bids the same organisation actually submitted. On a blind comparison where the system had no worked example available, an LLM judge rated its answers at least as good as the human-submitted answer on $40$ of $55$ ground-truth sections, better on $4$, missing on none, and flagged one unsupported claim in total. Classifying every gap the judge identified shows that $68\%$ were content absent from the system's own sources -- knowledge the human author held and the pipeline was never given -- so only $6$ of the $15$ adverse verdicts involve a deficiency the system could have avoided. A divergence from ground truth is more often an information-availability result than a writing-quality one, and evaluations that do not separate the two understate such systems. Against this backdrop we report a conditioning asymmetry. It is well established that rendering documents as structural markup rather than flat prose improves extraction, and we reproduce that on three reading tasks. The benefit does not transfer to conditioning: converting a bid's \emph{instruction} material from prose to nested XML dropped answer quality from $74\%$ to $48\%$ under a paired comparison. We further find that naming a forbidden construction concentrates rather than removes it -- $96\%$ of surviving defects fall in the two forms the prompt explicitly names -- and that coupling a stochastic annotation to a deterministic windowing function moves the extracted requirement count from $68$ to $51$ on a byte-identical file. Structure belongs where the model reads; prose and self-applied tests belong where it writes.
1 Introduction
Formal document authoring requires systems to read heterogeneous, structured requests and write answers that preserve the requester’s layout and wording. This paper tests whether structural conditioning benefits writing as it benefits reading, and reports that it does not.
- Motivation: Formal document authoring must answer every obligation in scattered files while preserving the requester’s layout and wording.Under sovereignty constraints, the model also cannot rely on a frontier API to absorb document complexity.
- Related work: 31% higher F1, up to 8.8 additional accuracy points, and compilability from 45.0% to 85.2% summarize prior benefits of structural representations on reading tasks.These results concern extraction, questionnaire understanding, and decompiler-output comprehension rather than writing from structured instructions.
- Research gap: The paper identifies an untested boundary: structural benefits measured when models locate information may not hold when models internalise instructions and write from them.The authors report that the benefit does not transfer to this conditioning setting.
- Contributions: The paper evaluates a deployed multi-agent system against human bids, including a blind comparison without a worked example and a second procurement involving human post-editing.The contribution also measures how information availability changes interpretation of gaps.
- Contributions: The evaluation separates information-availability failures from writing failures, moving the reported score from 73% to 89% after excluding gaps caused by unavailable substance.This distinction is presented as a way to interpret comparisons against human work more accurately.
- Contributions: The paper reports two additional findings: naming forbidden forms concentrates residual defects in those forms, while stochastic annotations can amplify variance through deterministic windowing.These findings extend the paper’s analysis beyond the structural-conditioning comparison.
2 System
The system uses a staged directed graph of specialised agents, structured document representations, code-enforced locators, and staged drafting sessions. Its design persists inspectable intermediate outputs and constrains how requirements and answer slots reach the writer.
- Architecture: 43 single-shot agent roles and one multi-turn drafter operate in six stages across three human edit gates.Intermediate semantic work is performed once, persisted as inspectable JSON, and can be corrected before the next stage.
- Markup and locators: Each input becomes an ordered stream of paragraphs, table cells, spreadsheet cells, or slide shapes with identifiers, source locations, rendered text, and format flags.The stream is rendered as nested markup to preserve document containment and adjacency.
- Markup and locators: Locators address elements but never classify them, while code splices requirement text from the element stream instead of having the model retype it.These rules prevent hallucinated coordinates and paraphrased requirements from reaching the file writer or evaluator.
- Answer-slot extraction: The model identifies answer slots by echoing a markup window and replacing respondent-entered cells with sentinel tokens, after which code diffs the echo against the input.This uses one transformation instead of a twelve-field classification and prevents unseen identifiers from appearing in output.
- Windowing: An 8k-character window found 32 of 32 slots and echoed all 407 elements, whereas a 23.5k window found 30 of 32 and silently dropped marks mid-window.The shorter window was 5.7× faster; chunk size is therefore task-specific rather than a system constant.
- Drafting sessions: Drafting uses one chat session per section, serial questions, continuous first-completion scheduling, and four turns covering planning, drafting, compliance, and quality.Past exemplars are matched by incoming requirement to past question, excluded when from the same tender, and may be absent without affecting the path.
3 Experiments
The deployed system was evaluated against submitted human work using blind section-level judgments and a cause-based analysis of adverse verdicts. The results distinguish writing deficiencies from information unavailable to the pipeline.
- 3.2 Against human ground truth: 0 of 120 T3 prompts carried an exemplar block, so the blind comparison tested generation without a worked example.The system wrote against the same blank forms for which the organisation had submitted the human ground truth.
- 3.2 Against human ground truth: 40 of 55 sections (73%) were judged at least as good as the human answer, with 4 better and none missing.The judge also flagged one unsupported claim across all sections.
- 3.2 Against human ground truth: 68% of 70 identified gaps reflected content absent from the pipeline’s sources, while 21% was available but unused and therefore avoidable.The classification separated unavailable substance, unused available content, policy divergence, and unnecessary detail.
- 3.2 Against human ground truth: Information-controlled rescoring reached 49 of 55 sections (89%), compared with 40 of 55 (73%) under the original judgments.Nine adverse verdicts involved only unavailable information or optional detail, leaving six with any avoidable gap.
- 3.2 Against human ground truth: The evaluation does not claim the system writes better bids, but reports competitive judged performance and one flagged inference across 55 sections.The paired scores bracket writing quality because the lower score charges the system for missing knowledge while the higher score discounts submission deficiencies caused by unavailable information.
3.3 Human post-editing
The experiments show an asymmetric effect of structural markup: it improved document-reading tasks but reduced quality when instruction material was converted from prose to nested XML. Human post-editing further indicates that the generated draft served mainly as scaffolding.
- 3.3 Human post-editing: In the post-editing study, the generated draft supplied a median 1.2% and mean 8.7% of retained answer text, with no answer exceeding 50%.A fifth of the final human text originated in the draft overall, concentrated in technical description; every pricing answer had 0% retention.
- 3.4 Markup on reading tasks: Every instrumented reading task improved with structural markup, including annotation routing, workbook slot extraction, and past-bid segmentation.Markup raised annotation coverage, stabilised slot counts, and made segmentation reproducible across runs.
- 3.4 Markup on reading tasks: Annotation routing coverage rose to 97.8%, while past-bid markup segmentation produced identical slice counts across six runs.The flat-text past-bid extraction yielded between 0 and 28 pairs across runs, whereas markup raised the yield floor from 3 to 47.
- 3.5 Markup on a conditioning task: Answering verdicts fell from 23/31 to 15/31 when the same instruction material changed from prose to nested XML.Mean restated share rose from 17.6% to 25.0%, and independently judged prompt fidelity fell from 19/24 to 13/24.
- 3.5 Markup on a conditioning task: Nested XML more than doubled requester-as-subject defects by foregrounding the requester as a tagged entity in the highest-precedence source.The reported mechanism is a salience shift: structure changed what was most available as the grammatical subject.
3.6 Naming a construction concentrates it
Naming forbidden constructions concentrated surviving defects in those named forms, whereas a self-applied test eliminated verbatim copying without naming a prohibition.
- 35/144 and 37/144 offending openings fell to 2/44 after a self-applied deletion-and-rewrite test.The test removed requirement-derived material from the first sentence rather than listing banned words.
- 96% of surviving defects were in the two surface forms explicitly named by the prompt.A fresh run reproduced this concentration at 5/5.
- 13 → 0 verbatim copies resulted when illustrative content was marked as an example shape rather than text to copy.The reduction held at zero in subsequent attempts.
- Rewording a prohibition failed, while changing the requested behavior worked immediately.The authors relate this to constraint design dominating constraint formatting for compliance.
3.7 Variance amplification through windowing
A byte-identical rerun produced different annotations, and deterministic length-based windowing propagated that upstream variation into different extraction results; compaction reduced cost but discarded question-local context.
- 54 versus 56 slots were marked across two runs on the byte-identical input file.Element parsing, segmentation, diagram conversion, exemplar selection, and reference assembly remained exactly reproducible.
- A two-slot difference shifted window boundaries by three elements because windows were computed over the marked stream.Adding answer_slot attributes changed rendered character lengths and therefore what each extraction call received.
- No hallucination or false-claim propagation occurred; variance arose because deterministic windowing consumed stochastic annotations.The proposed semantic-cascade mitigations do not address this mechanism because there was no false content to detect.
- Windowing over the unmarked stream and applying annotations afterward is the proposed remedy.The property affects pipelines that window, chunk, or paginate model-annotated text unless boundaries are logged.
- 10.2M to 2.1M characters posted across eleven sessions, a 79% reduction, followed question compaction.On one six-question session, the final call contained 355,024 characters, 73% concerning other questions.
- Zero of 22 checkpointed sessions retained the past-bid exemplar or capability-facts block after compaction.Only the explicitly re-attached reference block survived, so persistent material must use the channel compaction preserves.
4 Discussion
The discussion separates writing quality from information availability and places structural conditioning asymmetrically: structure supports reading, while writing benefits from prose and self-applied tests.
- Competitive answers depended on requester wording, readable structure, and tests applied to the system’s own output, not model capability alone.The system used an open-weights non-frontier model without a worked example on that tender.
- Only a fifth of the shortfall against human work was a generation problem, while two thirds was knowledge never supplied to the pipeline.The authors therefore argue for evidence capture rather than further prompt engineering.
- Structural markup improved three reading tasks but reduced answer quality by 26 points when applied to instruction material.The authors attribute the reduction to changing which entity was most salient at the prompt’s top privileged source.
- An instruction describing a failure supplies tokens for it, whereas a self-applied output test does not.This is presented as the same asymmetry as the structural-markup result in a different form.
- The paper’s engineering gains are expected to depreciate as models improve, according to the discussion’s comparison with Khan (2025).The authors identify this as a scope boundary and an untested prediction about replacing the deployed model with a frontier one.
5 Conclusion
The deployed tender-response system performed competitively with submitted human bids, while the broader evaluation found distinct boundaries for structural reading aids, prohibition prompts, and annotation-driven windowing.
- 40 of 55 ground-truth sections were rated at least as good as the human answer, with 4 better, none missing, and one unsupported claim flagged.The blind comparison gave the system no worked example.
- 68% of identified gaps reflected information absent from system sources, yielding 49 of 55 after excluding those gaps.The conclusion recommends reporting both figures to distinguish briefing from writing.
- 20.5% of final human text originated in the system draft when a human post-edited it.This was measured on a separate procurement.
- Structural markup improved all three instrumented reading tasks but reduced conditioning answer quality from 74% to 48%.The reading gains included 61% to 97.8% annotation coverage, a 124-cell swing to 1–2 cells, and unstable to bit-identical segmentation.
- Naming forbidden constructions concentrated 96% of surviving defects in the named forms, while stochastic annotations changed deterministic windowed extraction.The conclusion assigns structure to reading and self-applied tests to writing.
6 Limitations
The evaluation has important scope and validity limits: several comparisons are unreplicated or confounded, and the gap-adjusted scores are brackets rather than point estimates.
- Evaluation validity: The ground-truth comparison uses one procurement and one LLM judge from the drafting model family, without blinded human scoring.The judge’s documented markdown preference is an unmeasured confound.
- Score interpretation: 73% and 89% bracket the gap-adjusted result, and neither is a point estimate.The classifier’s model-family circularity and failure to detect a known policy-divergence case limit interpretability.
- Evaluation validity: The conditioning comparison has n=31 per arm on one procurement and has not been replicated, making its 26-point difference suggestive rather than conclusive.Runs occurred sequentially during active development, so several comparisons vary more than one factor.
- Evaluation validity: The post-editing study measures retained machine-draft content, not time saved, using one editor on one procurement.Its evidence therefore does not establish editing efficiency.
A The Four-Turn Chain
The four-turn chain separates planning, drafting, compliance, and quality control, using a self-ask plan and safeguards against destructive editing.
- Implementation: A stateful turn boundary serves as the reasoning envelope because the open-weights model has no thinking tokens.The plan reply conditions drafting, while code does not treat it as an answer.
- Planning: The plan turn asks seven targeted questions because counter-conventional tender instructions can override trained habits unless noticed first.Observed monologue diversity collapsed as answers lengthened, motivating the structured self-ask.
- Turn structure: Each question is handled through a self-ask plan, exemplar-conditioned draft, verbatim compliance pass, and quality pass.The quality pass may add, reframe, or flag text rather than delete it.
- Quality control: The quality turn independently reports answers shortened by more than 15%, after an earlier version removed 40–75% of strong evidence.Its permitted operations are now adding, reframing, flagging, or marking text for human decision.
B Ground-Truth Protocol
The ground-truth protocol reconstructs a human bid independently, verifies no exemplar leakage, and classifies gaps using the system’s actual source corpus.
- Independence verification: The human T3 submission was recovered as 55 question–answer sections and verified independent of the system before serving as ground truth.Verification used provenance, self-exclusion, and textual-overlap checks.
- Independence verification: 0 of 120 generated T3 prompts contained an exemplar block, and the system and human answer texts shared 0.0% overlap after excluding form 10-grams.The eight shared 10-grams were independently drawn source facts from common internal collateral.
- Ground-truth selection: Document-level overlap disqualified T4 as ground truth because 26% of the human document was blank-form text, leaving 20.5% reuse after boilerplate exclusion.The apparent 38.8% overlap was mostly requester questions repeated in both documents.
- Gap classification: Gap classification reconstructs the 372,139-character corpus, mechanically checks entity presence, and feeds that factual test into semantic cause classification.The design deliberately separates mechanical presence from semantic judgment, but remains circular because both stages rely on model-family decisions.