PaperlayerView exact revision on arXiv

Source-linked AI summary

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra

arXiv:2610.30026v1cs.CLcs.AIcs.IR

TL;DR

Enterprise agents rely on document ingestion, but enterprise documents encode meaning in presentation that a model-readable representation can lose. The paper measures the effect using marked requirements in real policies and tests cue-aware representations. Formatting-blind ingestion performs substantially worse, while the cue-specific loss is concentrated in highlighting and colour.

  • Problem

    Enterprise documents can encode meaning through presentation, yet ingestion must produce a faithful, complete, economical representation for models to read.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates marked requirements in 76 real policy documents against OOXML-derived labels with deterministic matching, and tests formatting cues and semantic-role hints.

    Source: §1 · p. 2, §3.2 · p. 3, §4.1 · p. 4

  • Results

    On 76 real policy documents, micro F1 falls from 0.690 to 0.150 (docling) and 0.075 (plain text), with cue ablation attributing the loss chiefly to highlight (+0.302) and colour (+0.262).

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported loss centers on highlight and colour, while typing cues adds +0.085 beyond preserving them in a pre-registered test.

    Source: §5 · p. 9

    The labels have not yet been validated by human annotators, and PDF support requires a usable text layer because the system does not OCR.

    Source: §5 · p. 9

Abstract

from arXiv · show

Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in the wild silently loses information. We show that one layer is lost by every mainstream converter - the meaning an author encodes in how text looks - and that the loss makes agents confidently wrong rather than visibly incomplete. We build a task from 76 real policy documents: recover the requirements the author singled out, scored against the file's own OOXML properties by a deterministic matcher with no LLM judge. Formatting-blind ingestion collapses from micro F1 0.690 to 0.150 (docling) and 0.075 (plain text), a gap that replicates across six answering models from two vendors - five Claude models spanning the Opus, Sonnet and Haiku tiers, and OpenAI's GPT-5.6. An agent that cannot see the cue guesses, and docling's false-positive rate on requirements the author did not flag is four times ours. A span-level leave-one-out localises the loss to two cues - highlight (+0.302) and colour (+0.262), both Holm-adjusted p < 1e-4 - while bold, italic, monospace and strikethrough each sit within +/-0.003 of zero. The same asymmetry appears in PDF. Colour is also the cue no mainstream converter preserves and no parsing benchmark scores, and it is common: 32.8% of 6,460 documents carry a deliberate colour or highlight, 10.3% on an obligation. A pre-registered test shows that typing a cue for the agent adds +0.085 beyond merely preserving it (Holm-adjusted p = 0.049). We validate the labels against rendered page images through an independent modality (kappa = 0.72) and bound what wording alone can achieve (AUC 0.664). For agent builders the practical finding is that ingestion, not reasoning, is where this reliability is won or lost, and that the fix is cheap and measurable.

1 Introduction

Enterprise agents depend on ingestion layers that can silently discard meaning encoded in document presentation, making reliability depend on what the converter preserves. The paper measures this problem, situates it against existing document and formatting work, and describes a parser that represents formatting cues and assigns semantic-role hints.

1 Introduction

  • Formatting-blind ingestion can silently remove visually marked requirements, leaving agents to guess and raising false-positive risk on requirements authors did not flag.The paper frames this as a common, safety-relevant problem rather than an outage or visibly incomplete output.Source: §1 · pp. 1–2
  • The paper measures formatting-blind ingestion on real documents using file markup as ground truth, localizes cue effects, and tests whether typed cues help agents.It also presents a validated evaluation instrument and reports that progressive disclosure did not reduce token cost.Source: §1 · p. 2

2 Related Work

  • Existing parsing benchmarks focus on text accuracy, reading order, tables, and formulas, while formatting recognition work omits colour and highlighting or stops at attribute detection.The paper positions its work as recovering cues and measuring their effect on answer correctness.Source: §2 · p. 2

3.1 Formatting as First-Class Signal

  • The parser converts document content into formatted spans and attaches semantic-role hints, while serialization uses Markdown, custom cue tags, and structural tags.The role mappings are treated as hints rather than ground truth; medium-confidence mappings are excluded because an incorrect hint can bias an agent.Source: §3.1 · p. 3

3.2 Cue Disambiguation and Recovery

  • Cue interpretation uses context-sensitive rules, including different bold meanings by span or block and non-white highlight as a warning cue.Corpus measurement led to a highlight-rule correction that raised marked paragraphs receiving a salience role from 79.1% to 97.6%; roles attach at block level while spans retain inline markers.Source: §3.2 · p. 3
  • A document-body walk also omits headers, footers, footnotes, document properties, equations, and tracked changes, including accepted revision text.The passage identifies these as content unreachable from that walk.Source: §3.2 · p. 4

4 Evaluation

The evaluation tests whether preserving and interpreting visual cues improves recovery of author-marked requirements, using markup-derived labels and multiple validation settings. Results localize the loss to colour and highlight, while showing that cue typing, external benchmarks, and a shared downstream pipeline provide additional evidence about the approach.

4.1 Corpus, prevalence, and task

  • 32.8% of 6,462 policy documents carry deliberate red text or non-white highlighting, and 10.3% carry such cues on an obligation.The task uses OOXML-marked requirements as ground truth and deterministic lexical matching; a supervised text-only classifier reaches ROC AUC = 0.664.Source: §4.1 · p. 4

4.2 Formatting-blind conversion destroys answerability

  • +0.110 macro F1 separates our representation from docx2python, the strongest preserving baseline, which reaches 0.586.The comparison is significant within the six-arm family (pHolm = 0.022).Source: Table 1 · p. 5, §4.2 · p. 5

4.3 The loss is concentrated in colour and highlight

  • +0.302 for highlight and +0.262 for colour account for the cue-removal effect; bold, italic, monospace, and strikethrough each fall within ±0.003 of zero.The effects are Holm-significant; removing each cue chiefly harms documents that use that cue.Source: §4.3 · p. 5, Figure 1 · p. 6

4.4 Does typing add anything beyond preserving?

  • +0.085 macro F1 comes from typing a preserved cue versus showing the same inline cue untyped (CI [+0.035, +0.145], pHolm = 0.049).Our representation also beats the strongest preserving baseline by +0.107 (CI [+0.047, +0.174], pHolm = 0.014).Source: §4.4 · p. 6

4.5 External validation: Microsoft OCB; 4.6 Generalisation, and what does not replicate

  • Our system leads Microsoft OCB on formatting-dependent items (0.816 against 0.750) and non-formatting items (0.977 against 0.818); adding unreachable document content raised its overall score from 0.853 to 0.959.On ParseBench PDF rules, our cue recovery matches the strongest baseline at 65/126, while plain text recovers 0/126.Source: §4.5 · p. 7, §4.6 · p. 8

4.6 Generalisation, and what does not replicate

  • Plain text reaches only 0.007–0.075 micro F1 across six models; cue preservation replicates overwhelmingly, while typing is positive on five models but replicates on three.The two models without a typing benefit barely attempt the task; Claude Sonnet 5 has recall 0.09 and precision 0.96.Source: Figure 2 · p. 7, §4.6 · p. 8

4.7 Does the advantage survive an identical downstream pipeline?

  • With extraction and rendering held identical across arms, our advantage survives a full pipeline run, although requirement-text retention is similar across arms (0.665, 0.634, 0.607).Only the parsed text varies in this 68-document test.Source: Table 3 · p. 8, §4.7 · p. 8

4.8 Implications for agents deployed in the wild

  • Cue recovery is deterministic and code-only, but costs roughly twice plain-text tokens; missing cues can silently degrade precision or produce extreme conservatism.The paper argues for ingestion-level assertions because standard health, schema, and token-count checks do not detect the loss.Source: §4.8 · p. 9

5 Conclusion

On 76 real policy documents, formatting-blind ingestion sharply reduces requirement recovery, with the loss localized mainly to highlight and colour. The paper also reports scope limits and identifies further validation and recovery work.

5 Conclusion

  • On 76 real policy documents, micro F1 falls from 0.690 to 0.150 (docling) and 0.075 (plain text), a gap replicated across six models from two vendors.Agents answer fluently and wrongly, with four times the false-positive rate on requirements authors did not flag.Source: §5 · p. 9
  • Highlight (+0.302) and colour (+0.262) account for the localized loss (both pHolm < 10−4); other cues are within ±0.003, and the asymmetry also appears in PDF.Colour is also the cue no mainstream converter preserves and no parsing benchmark scores.Source: §5 · p. 9
  • Typing a cue adds +0.085 beyond preserving it in a pre-registered test (pHolm = 0.049).The typed layer is robust on three of six models but shows no benefit on two that attempt the task only sparsely.Source: §5 · p. 9
  • Labels were validated against rendered pages by an independent modality (κ = 0.72), but not by human annotators; PDF support requires a usable text layer because the study does not OCR.The representation costs roughly twice plain text in tokens, while absolute micro F1 is stable only to about ±0.08 across configurations.Source: §5 · p. 9
  • Future work includes human label validation, OCR-backed cue recovery for scanned pages, non-English documents, and diff-based re-ingestion across revisions.Tracked changes already provide reviewer-signed ground truth for the proposed revision-focused direction.Source: §5 · p. 9

S1 Corpus and Prevalence

Scanning real policy documents at the OOXML run level finds deliberate colour or highlighting on 32.8% of documents and on an obligation in 10.3%. The section also details how the parser disambiguates cues, serializes them, and recovers content missed by a body-only walk.

S1 Corpus and Prevalence

  • 32.8% of documents carry deliberate colour or highlight, and 10.3% carry one on a sentence expressing an obligation.The stricter marked-run pattern is 5.6%; another 2.6% carry cues only outside the body. Scanning 6,462 files is performed at OOXML run level.Source: S1 Corpus and Prevalence · p. 11

S3 Tracked Changes

  • Across 28 recorded revisions, the parser preserves both superseded and in-force values with the superseded value marked in 27 cases.Plain text loses the value entirely in 16 cases and surfaces only the superseded value in 3; the forced-choice comparison is too small to support a claim.Source: S3 Tracked Changes · p. 11

S4.1 Formatting Disambiguation

  • Bold becomes IMPLICIT_HEADING for an entirely bold standalone line under 200 characters, or KEY_TERM when only some spans are bold.An all-bold paragraph of 200 characters or more receives no bold-derived role; this occurred in 29 of 29,170 blocks.Source: S4.1 Formatting Disambiguation · p. 12
  • The parser thresholds RGB channels for red rather than matching exact colour codes, while any visible highlight becomes WARNING and red highlight escalates to CRITICAL_WARNING.The highlight rule raised author-marked paragraphs receiving a salience role from 79.1% to 97.6%; 99.7% of corpus red spans were exactly #FF0000.Source: S4.1 Formatting Disambiguation · p. 12

S4.2 Hybrid Serialization Protocol

  • Hybrid serialization uses Markdown for bold, italic, and strikethrough, and custom tags for colour and highlight.It is 15–20% more token-efficient than pure custom-tag serialization; structural tags also encode document metadata such as list membership and table headers.Source: S4.2 Hybrid Serialization Protocol · p. 12

S4.3 Content Unreachable from a Body Walk

  • A body-only walk misses content in separate document parts, including headers, footers, notes, properties, and tracked revisions.The parser walks paragraph children in document order to retain insertions as current text and deletions as retired content; it also handles equations, hyperlinks, and text boxes.Source: S4.3 Content Unreachable from a Body… · p. 13
  • Malformed formatting cues degrade to absent cues rather than making a document unreadable, and XML-level table reading preserves text when merged-cell reconstruction fails.Both robustness fixes followed failures encountered during the 6,462-document scan.Source: S4.3 Content Unreachable from a Body… · p. 13
  • Born-digital PDF cues are recovered geometrically from character-level bounding boxes because PDFs lack run properties.Highlights are filled rectangles behind glyphs, while strikethroughs cross the glyph midline; span-level comparison fails when the renderer merges struck and live text.Source: S4.3 Content Unreachable from a Body… · p. 13

S5 System Architecture and Tiering

The architecture separates deterministic parsing and serialization from downstream extraction and rendering. For long documents, adaptive tiering uses larger-context calls before falling back to overlapping segments and pairwise merging.

S5 System Architecture and Tiering

  • Stage 1 parses and serializes character-level presentation entirely in code with zero LLM calls; extraction and rendering are downstream stages.The separation reserves LLM reasoning for classification, relationship identification, and summarization while keeping deterministic work in code.Source: S5 System Architecture and Tiering · p. 13

S6 Adaptive Tiering by Document Size

  • Documents below 25K tokens use one extraction call, while documents up to 50K add faithfulness validation on a 20% sample.Larger documents are segmented into up to 16 overlapping ∼8K-token chunks and merged pairwise, requiring 2N−1 calls for N segments.Source: S6 Adaptive Tiering by Document Size · p. 13
  • The tiered design uses globally coherent long-context calls to resolve cross-references, unlike uniform 4K-token chunking that would fragment extractions.For a 100-page document, uniform chunking is analytically estimated to need 40–80 calls; this is not a measured baseline.Source: S6 Adaptive Tiering by Document Size · pp. 13–14

S7 From Representation to Agent-Loadable Knowledge

The representation’s advantage persists when all systems use the same downstream extraction and rendering pipeline, and colour or highlighting remains informative in the generated packages. PDF cue recovery reproduces the Word pattern, while results across models show that some gains are not universal.

S7 From Representation to Agent-Loadable Knowledge

  • Marked requirements reach the quick-reference file at rate 0.569 versus 0.245 for unmarked requirements, a 2.3× enrichment.This placement difference is measured over 34 generated packages; the passage notes that typed-extraction accuracy remains future work.Source: S7 From Representation to… · p. 14

S8 Cross-Format Results: PDF

  • PDF cue recovery matches the strongest baseline at 65/126 each, while highlighting favors the proposed system 11/18 to 4/18.Plain text recovers no cues; the cross-format result reproduces broad parity with the colour-class cue as differentiator.Source: S8 Cross-Format Results: PDF · p. 14
  • Scanned-page cue recovery remains out of scope because pixel-only cues require OCR, which the study does not perform.The study also trails pymupdf4llm on text localisation, 75 versus 82 of 126.Source: S8 Cross-Format Results: PDF · p. 14

S9 Equal-Preprocessing Comparison

  • +0.178 over docx2python and +0.422 over docling remain significant when the complete downstream pipeline is held identical across 68 documents.All three arms retain similar shares of gold requirement text, but only the proposed package records which requirements authors singled out.Source: S9 Equal-Preprocessing Comparison · p. 15
  • Absolute package scores are lower than in the direct comparison, and 0.665 retention caps recall after extraction and rendering.On a held-out SOP, all 24 validator-sampled entities trace to verbatim source quotes; 87.5% are fully faithful, with the remainder under-specifying rather than fabricating.Source: S9 Equal-Preprocessing Comparison · p. 15

S10 Cross-Model Generalisation

  • Across six answering models, plain text collapses under every model; cue typing is positive on five of six but significant on three.Beating docx2python replicates on four of six models and is never significantly negative for cue typing.Source: Table 6 · p. 15

S12 Architectural Design Principles

The architecture keeps deterministic document assembly in code and preserves formatting, provenance, and structure in the representation given to the model. Its examples show how those design choices carry author cues and document metadata into typed downstream outputs.

S12 Architectural Design Principles

  • The architecture assigns judgment tasks to LLMs and deterministic parsing, serialization, rendering, and file I/O to code.Its principles also call for preserving formatting as signal, requiring source quotes, using specific entity types, and progressively disclosing output.Source: S12 Architectural Design Principles · pp. 15–16

S13 Complete Formatting-to-LLM Pipeline

  • The pipeline parses and annotates first, then extracts entities and renders a progressive-disclosure package; later stages are downstream utility rather than the measurement target.The diagram shows a five-file package and a parallel vision pass for embedded diagrams.Source: Figure 3 · p. 16

S14 Serialization Tag Set and Attribute Encoding

  • Structural tags encode hierarchy and layout details the model cannot infer reliably from text alone, including list membership, table headers, and heading levels.Tag attributes also carry indentation depth, font size, and Word style name.Source: S14 Serialization Tag Set and… · pp. 16–17

S15 Running Example: Parsing an Onboarding SOP

  • In the onboarding example, red text is serialized with a colour tag while the paragraph receives a block-level CRITICAL_WARNING role.Bold remains Markdown, and structural metadata is retained alongside the cue.Source: S15 Running Example: Parsing an… · p. 17

S16 Handling python-docx Limitations

The extraction pipeline adds XML-level workarounds for enterprise content that python-docx omits or misreads, and uses a 17-type taxonomy plus targeted instructions to preserve distinctions and prevent observed extraction errors.

S16 Handling python-docx Limitations

  • XML-level access recovers text boxes, tracked changes, comments, merged-cell continuations, and OLE objects that python-docx cannot reliably expose.The workarounds add approximately 80 lines of code and are described as essential for complete extraction from real-world enterprise documents.Source: S16 Handling python-docx Limitations · pp. 17–18

S17 Why 17 Entity Types?

  • The 17-type taxonomy avoids forcing distinct content, such as RULEs and PROCEDUREs, into overly broad categories and was validated on enterprise SOPs.Each type has clear, non-overlapping trigger signals.Source: S17 Why 17 Entity Types? · p. 18

S18 The 12-Point Extraction Mandate

  • The 12-point mandate targets observed errors with read-before-extract, checklist classification, WRONG/CORRECT examples, and complete procedural sub-steps.It also prioritizes red-text warnings with explicit consequences for HIGH confidence, even when surrounding content is low-confidence.Source: S18 The 12-Point Extraction Mandate · p. 18
  • The mandate’s remaining rules preserve implicit headings, enumerations, scope, metric units and thresholds, cross-references, diagram confidence, and deduplication precedence.When explicit and diagram-inferred entities duplicate, the explicit entity takes precedence.Source: S18 The 12-Point Extraction Mandate · p. 18

S19 Few-Shot Example Selection

The extraction prompt combines five targeted few-shot examples with a running SOP example and structured tool output. Together, they illustrate multi-entity extraction, source-quote provenance, and validated output handling.

S19 Few-Shot Example Selection

  • Five few-shot examples target formatting cues, numbered procedures, multi-entity passages, nested steps and warnings, and WRONG/CORRECT patterns.The complex-paragraph example yields RULE, EXCEPTION, and FORM_OR_TEMPLATE entities together.Source: S19 Few-Shot Example Selection · p. 18

S20 Running Example: Extraction Output

  • A bold “probationary period” triggers a DEFINITION entity whose output includes a 90-day description, source quote, and HIGH confidence.The example also records the five-day notice during probation.Source: S20 Running Example: Extraction Output · p. 19
  • A compliance-training passage yields a RULE, EXCEPTION, and FORM_OR_TEMPLATE, plus EXEMPTS and REFERENCES relationships in the same pass.The accompanying warning example assigns CRITICAL severity based on red-text semantic role and an explicit consequence.Source: S20 Running Example: Extraction Output · p. 19

S21 Structured Output via Tool Use

  • Tool use forces the extraction target to return the full ExtractionResult payload, which Pydantic validates before the pipeline continues.The receiving model tolerates per-type variation by validating required fields without prescribing a nested schema for each type.Source: S21 Structured Output via Tool Use · p. 19

S22 Tier 3 Merge: Additional Details

The appendix details how segmented outputs are merged and restored, why source-quote verification is used, how diagrams are processed, and how extracted content is packaged for agent use.

S22 Tier 3 Merge: Additional Details

  • Tier 3 walks blocks into segments at an 8,000-token threshold, excludes oversized blocks from overlap, then restores original entity data after merging.For cross-segment duplicates, restoration unions source quotes and merges type-specific fields such as procedure steps.Source: S22 Tier 3 Merge: Additional Details · p. 20

S23 Why Source-Quote Verification Instead of Mutual Dependency?

  • Source-quote verification targets hallucination risk, whereas ZOES Mutual Dependency checks attribute granularity, a problem absent when extraction attributes are predefined.The paper describes quote verification as a simpler, lower-cost approach for its guided-schema setting.Source: S23 Why Source-Quote Verification… · p. 20

S25 Progressive Disclosure: Package Layout and File Roles

  • The five-file package keeps action-critical guardrails and warnings in a <500-line SKILL.md, while full references load on demand and provenance maps entities to source quotes.QUICK_REFERENCE.md surfaces warnings and high-severity rules; sub-files hold full procedures, enumerations, metrics, and definitions.Source: S25 Progressive Disclosure: Package… · pp. 20–21

S24 Diagram Taxonomy and Supporting Protocol Figures

S26 Anthropic-Spec Description Generation

The description algorithm uses imperative triggers to encourage skill loading and selects among three fallback strategies based on available extracted entities. The rendered example shows the resulting frontmatter and how critical content is surfaced.

S26 Anthropic-Spec Description Generation

  • The skill description uses imperative trigger language to counter undertriggering, when an assistant has a skill but does not load it for a query.The specification calls for a “pushy” description.Source: S26 Anthropic-Spec Description… · p. 21
  • Description generation prioritizes audience plus PROCEDURE, RULE, and WARNING names, then falls back to a SCOPE summary or the document title and high-confidence entity names.Document entity names are preferred as triggers because they match users’ query vocabulary better than audience descriptions.Source: S26 Anthropic-Spec Description… · p. 21

S27 Running Example: Rendered Output

  • The rendered SKILL.md uses only name and description in frontmatter, omits entity type labels, links long procedures to sub-files, and presents warnings as CRITICAL blockquotes.The example description names the audience and prompts use for I-9 verification, compliance training, and benefits enrollment questions.Source: S27 Running Example: Rendered Output · p. 23

S28 Open-Schema Discovery

The system adds an optional open-schema mode for information outside its 17-type taxonomy and preserves contradictory source values for human review. Its implementation uses code for parsing and a managed LLM service for extraction and related calls.

S28 Open-Schema Discovery

  • The 17-type taxonomy covers approximately 95% of enterprise SOP content; optional open-schema discovery adds novel types in a first pass and re-extracts consistently with the expanded taxonomy.The mode is intended for structures such as escalation matrices and audit checklists.Source: S28 Open-Schema Discovery · p. 23

S29 Conflict Resolution

  • Contradictions are flagged, both values and source citations are preserved, and confidence is lowered; the system leaves resolution to a human reviewer.Conflicts also appear in a dedicated “Conflicts & Ambiguities” section of the output.Source: S29 Conflict Resolution · p. 24

S30 Implementation Details

  • The implementation spans about 1,800 lines across 10 modules, with inline LLM deduplication replacing embedding-based machinery.Calls use the Strands Agents SDK with Claude Opus 4.6; retryable and non-retryable errors are distinguished, and exhaustion after three attempts raises a contextual PipelineError.Source: S30 Implementation Details · p. 24

S31 End-to-End Example

The appendix traces a real pipeline run on an eight-section password-reset SOP, from formatting-aware parsing through extraction to a structured package. The 579-token document yields 20 entities and 16 relationships, then a five-file output package.

S31 End-to-End Example

S31.2 Stage 1 Output: Hybrid Serialization (excerpt)

  • The parser serializes red warning text with a CRITICAL_WARNING role and annotates bold lockout terms inline; Table 8 details the pipeline’s 11 high-confidence semantic roles.The example’s warning says never to share temporary passwords by email or chat, while its lockout terms include threshold and duration.Source: S31.2 Stage 1 Output: Hybrid… · pp. 24–25, Table 8 · p. 27

S31.3 Stage 2: Tier Selection and Extraction

  • The Opus 4.6 extraction call takes approximately 92 seconds and returns 20 entities, 16 relationships, and 0 conflicts.The call is selected as Tier 1 from an estimated 579-token input.Source: S31.3 Stage 2: Tier Selection and… · p. 25

S31.4 Stage 3 Output: Progressive Disclosure Package

  • The router distributes all 20 entities across five files, including a quick reference for warnings and critical rules and a separate procedure reference.The package also includes concepts and metrics documentation and a source-quote provenance map.Source: S31.4 Stage 3 Output: Progressive… · pp. 25–26
  • A 579-token source document produces a five-file structured package, with a 113-line SKILL.md entry point under the 500-line target and full provenance preserved.The package routes the six-step procedure to a separate reference document.Source: S31.4 Stage 3 Output: Progressive… · p. 26
Loading 2610.30026v1…