Source-linked AI summary
SAGE: Governed Artifact Generation from Enterprise Guidelines
Mohammadreza Sediqin, Shivali Dalmia, Sumukha Thoppanahalli, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji
TL;DR
Enterprise guideline documents take two to three days to convert manually, and current extraction approaches lack a governed workflow for validation, consistency, and traceable artifacts. SAGE coordinates extraction and artifact generation through a shared, versioned rule store and structured evaluation. Across 120 documents, it reports 3.2% hallucination versus 15.7% for an ungoverned one-pass baseline.
Problem
Enterprise guideline conversion takes two to three days per document, while extraction systems lack a combined workflow for validation, consistency checking, and traceable artifact generation.
Source: §1 · p. 1Method
SAGE coordinates deterministic parsing, VLM-based extraction, structured rule modeling, and artifact generation through a shared versioned rule store with schema-validated interstage contracts.
Source: §1 · pp. 1–2Results
3.2% hallucination versus 15.7% for an ungoverned one-pass baseline across 120 documents; SAGE extracted 3,896 rules and produced 812 artifacts.
Source: §1 · p. 2Takeaways & Limitations
Experiments on real-world enterprise documents provide a scalable basis for governed enterprise document understanding.
Source: §5 · p. 8The current evaluation covers English enterprise guidelines only.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Enterprise guideline documents mix narrative text, complex tables, and embedded images, and converting them into structured work artifacts still takes two to three days of manual effort each. Current language and vision-language models extract from such documents but offer no governed workflow beyond extraction: no validation, no consistency checking, no traceable artifact generation. We introduce SAGE, a governed multi-stage LLM pipeline organized around a shared versioned rule store with stable identifiers, schema-validated inter-stage contracts, and end-to-end provenance tracking. Extracted rules undergo deterministic structural validation and LLM-based semantic scoring, then a consistency module that removes duplicates, flags contradictions, and surfaces specification gaps; only uncertain or flagged items reach reviewers, while high-confidence outputs are auto-approved. On 120 documents, SAGE cuts turnaround from days to 20-100 minutes, achieving a 96% document-level success rate with 3.2% hallucination, extracting 3,896 rules and producing 812 artifacts ready for human review; without governance, hallucination rises to 15.7%.
1 Introduction
Enterprise guideline conversion is manual, slow, and difficult to keep consistent, while extraction systems lack the validation and traceability needed for production. SAGE coordinates governed extraction and artifact generation; on 120 documents, it reports 3.2% hallucination versus 15.7% for an ungoverned baseline.
1 Introduction
- Manual guideline conversion takes two to three days per document, often yields inconsistencies and errors, and must be redone when sources change.The delay affects staffing, launch, and ongoing maintenance.Source: §1 · p. 1
- Positional text, complex or image-only tables, and visual examples make extraction difficult, and parsing errors cascade into downstream rule quality.Critical information in diagrams and edge-case illustrations may be absent from document text.Source: §1 · p. 1
- Production workflows require validation, consistency filtering, version reconciliation, and persona-specific artifacts with provenance, which extraction-only systems do not provide.The authors state that, to their knowledge, no existing system combines these functions in one governed pipeline.Source: §1 · p. 1
- SAGE coordinates multiple LLM stages through a shared versioned rule store keyed by stable rule IDs, with schema-validated objects serving as typed intermediate data.Artifacts trace back to their source rules and originating documents.Source: §1 · pp. 1–2
- The contributions span deterministic parsing, VLM-based extraction, structured rule modeling, dependency-aware artifact generation, and structural plus semantic evaluation.Evaluation supports automated acceptance, targeted regeneration, and selective human review.Source: §1 · p. 2
- Zero-edit approvals are reused as calibration signals for the evaluation judge, with the stated aim of progressively reducing review load.The HITL workflow covers rules, gaps, and artifacts.Source: §1 · p. 2
- 3.2% hallucination across 120 documents, versus 15.7% for an ungoverned one-pass baseline; SAGE extracted 3,896 rules and produced 812 artifacts.The artifacts were produced for human review.Source: §1 · p. 2
2 Related Work
The related work describes persistent challenges in multimodal document extraction and the limits of extraction-only systems. It contrasts SAGE’s shared, schema-validated coordination and selective human review with unstructured messaging and flat review queues.
2.1 Document Parsing and Structured Extraction
- Rasterized tables and visually embedded rules remain difficult; rule-based methods rely on geometric heuristics, while VLMs add latency and instability on dense layouts.These weaknesses compound in enterprise governance, where parsing failures propagate into rule quality.Source: §2.1 · p. 2
- Extraction-focused systems improve prompting or extraction robustness but do not provide validation, contradiction handling, or governed artifact generation (Sainz et al., 2024; Lu et al., 2022; Livathinos et al., 2025; Kim et al., 2022).The paper evaluates against a monolithic baseline sharing SAGE’s structure.Source: §2.1 · p. 2
- SAGE coordinates stages through a shared versioned rule store with stable identifiers, schema-validated outputs, and threshold-driven review routed to QM or PM workbenches by expertise.The passage contrasts this design with existing frameworks’ unstructured messages (Yao et al., 2023; Wu et al., 2023; Wang et al., 2024a).Source: §2.1 · p. 2
2.2 Human-in-the-Loop Workflows
- Unlike flat review queues, SAGE routes rules and artifacts that fail thresholds or carry inferred annotations, contradictions, or duplicates, and reuses zero-edit approvals to calibrate the system.The cited prior work describes human verification and the importance of selective routing, interface design, and escalation policy (Wu et al., 2021).Source: §2.2 · p. 2
3 System Architecture
SAGE uses a versioned, schema-enforced rule store to coordinate pipeline stages and maintain provenance, versioning, and auditability. Its ingestion combines deterministic text extraction with VLM processing of visual content, followed by semantic segmentation for rule extraction.
3 System Architecture
- A central versioned rule store keyed by stable rule IDs connects stages through schema-validated contracts, supporting provenance, version reconciliation, and auditable HITL decisions.The pipeline proceeds through parsing, rule extraction, parallel consistency checking and evaluation, threshold-driven review, and artifact generation.Source: §3 · pp. 2–3
3.1 Document Ingestion and Content Extraction
- Ingestion separates deterministic text extraction from VLM processing to preserve reproducibility while covering text, tables, and visual elements.PDFs use PyMuPDF; DOCX files are parsed from XML; PPTX files are converted to PDF before entering the same PyMuPDF pipeline.Source: §3.1 · p. 3
- Images are deduplicated before VLM processing, while tables are detected through layout analysis and processed with a verbatim-preserving prompt.Qwen2.5-VL handles non-textual elements.Source: §3.1 · p. 3
- Quality score Q aggregates normalized defect rates for extraction noise, encoding corruption, redundant outputs, and missing content.Coverage separately measures the fraction of source content units extracted for pages, figures, and tables.Source: §3.1 · p. 3
- Extracted content is segmented into semantic categories such as task definitions, evaluation criteria, edge cases, compliance requirements, and workflow specifications for downstream rule extraction.Outputs are indexed by document and page and emitted as JSON containing document text and generated image and table descriptions.Source: §3.1 · p. 3
3.2 Rule Extraction
SAGE extracts and normalizes guideline rules, checks them for consistency and quality, and routes flagged items for human review. It also analyzes gaps, infers examples, and applies evaluation criteria tailored to each object type.
3.2 Rule Extraction
- A Qwen-family model extracts candidate rules with source spans, types, and confidence, normalizes them into a 26-field schema, then checks for version alignment, duplicates, and contradictions before evaluation and review routing.Rules flagged by consistency or evaluation signals are routed to the appropriate QM or PM workbench.Source: §3.2 · pp. 3–4
3.2.1 Gap Analysis
- Gap analysis identifies missing, ambiguous, or underspecified guideline content; resolved clarifications may add RuleUnits to the store for downstream artifact generation.Each gap is represented as a GapObject with a targeted clarification question for stakeholder review.Source: §3.2.1 · p. 4
3.2.2 Example Inference
- For each approved rule, SAGE extracts explicit examples or infers contextually grounded ones, then evaluates them before storage for use in annotator guidance and reviewer validation.Inferred examples must adhere strictly to the rule semantics.Source: §3.2.2 · p. 4
3.3 Evaluation Engine
- With no ground-truth annotations, evaluation uses cosine similarity, schema validation, and LLM-as-judge scores; experts manually inspected 10–15 documents, and empirically selected thresholds were fixed before experiments.The authors report that expert inspection found scores consistent with human judgment across extraction, rule quality, and artifact evaluation.Source: §3.3 · p. 4
3.3.1 VLM Benchmarking
- VLM benchmarking measures grounding, hallucination, quality, image throughput, duplication, and category diversity, using cosine and NLI thresholds to identify duplicates and ambiguous or unsupported rules.The passage defines evidence as a source match at cosine ≥0.65, hallucination as no match below 0.40, and ambiguity as 0.40–0.65.Source: §3.3.1 · pp. 4–5
3.3.2 Rule Evaluation Framework
- Objects undergo deterministic schema validation before LLM semantic scoring; routing uses the minimum dimension score, and inferred-source objects always go to human review.L1 failures may be retried, corrected, or rejected according to error severity.Source: §3.3.2 · p. 5
3.3.3 Object-Specific Evaluation
- RuleUnits, ExampleObjects, and GapObjects receive object-specific structural checks and semantic scores tailored to rule quality, example fidelity, and gap-question quality.RuleUnit scores cover clarity, persona fit, completeness, category fit, and severity fit; ExampleObject scores cover alignment, discriminability, and realism; GapObject scores cover question quality, severity calibration, and gap type fit.Source: §3.3.3 · p. 5
3.4 Human-in-the-Loop Review
Human review proceeds in three dependent phases, with approved rules serving as the authoritative specification for later gap and example review.
3.4 Human-in-the-Loop Review
- The QM first approves, rejects, or edits extracted rules; approved rules anchor gap review, and resolved gaps can add rules before examples are reviewed against the finalized specification.Examples are routed to the QM or PM workbench according to the owner of their linked RuleUnit.Source: §3.4 · p. 5
3.5 Artifact Generation and Evaluation
SAGE generates persona-specific operational artifacts from approved rules, resolved gaps, and evaluated examples, then evaluates each artifact before release.
3.5 Artifact Generation and Evaluation
- Pydantic-constrained templates generate nine operational artifacts per persona from approved rules, resolved gaps, and evaluated examples.The generation inputs include RuleUnits, GapObjects, and ExampleObjects.Source: §3.5 · pp. 5–6
- QM artifacts include guidelines, QA materials, reviewer instructions, a gaps document, and an agent specification; PM artifacts include an SOW, job description, and requisition.The listed artifacts serve annotators, reviewers, quality leads, Learning & Development, Recruiting, and crowd contributors.Source: §3.5 · p. 6
- Each artifact is scored for rule coverage, structural conformance, persona appropriateness, and cross-section contradiction, with ART ≥4.0 auto-approved, 3.5–<4.0 sent to review, and <3.5 blocked for regeneration.The L1 metrics each require ≥3/5, while cross-section contradiction passes at ≥4/5.Source: §3.5 · p. 6
4 Results and Discussion
Across 115 documents, SAGE’s governance pipeline outperformed a one-pass baseline on hallucination, duplication, and artifact rule coverage; it also reduced completion time from days to 70–85 minutes for a complex document. Rule extraction, artifact generation, and content parsing achieved the reported quality and review outcomes across a 120-document corpus.
4.1 Dataset
- The 120-document corpus comprised 8 text-only documents, 84 with tables and figures, and 28 that were table- and image-heavy.Processing times ranged from about 20–30 minutes for low-complexity documents to 65–100 minutes for high-complexity ones; modality density drove the variation.Source: §4.1 · p. 6
4.2 VLM Selection and Parsing Evaluation
- Qwen2.5-VL-32B was selected for extraction because it led on evidence rate, hallucination, throughput, and duplication; Qwen3-32B scored higher on quality and category distribution.Qwen3-32B’s higher hallucination and duplication would send more rules into consistency filtering and human review.Source: §4.2 · p. 6, Table 1 · p. 7
- 96% (115/120) of documents succeeded, with 99.2% page coverage, 97.1% figure recall, and 88.3% table recall.The five failures involved VLM timeouts on image-heavy documents or poorly structured tables exceeding the processing budget.Source: §4.2 · p. 6, Table 2 · p. 7
4.3 Rule Extraction Evaluation
- Adding consistency processing discarded 117 duplicates, flagged 113 contradictions, and surfaced 1,040 gaps among the 3,896 extracted RuleUnits.L1/L2 validation and HITL routing gated and routed units without deleting them; consistency processing reduced retained rules to 3,779.Source: Table 3 · p. 7, §4.3 · p. 7
- On 115 completed documents, SAGE extracted 3,896 RuleUnits with an 84.8% evidence rate, 82.6% coverage, and 3.2% hallucination.L1 passed 99.1% of units; L2 auto-approved 71.4%, routed 28.6% to review, and rejected none. The L2 judge achieved 0.957 F1 against expert annotations.Source: Table 4 · p. 7, §4.3 · p. 7
4.4 Artifact Generation Evaluation
- Across 812 artifacts, rule coverage reached 93.7%, cross-section consistency 96.3%, structural conformance 82.4%, and persona appropriateness 81.0%.54.2% of artifacts were auto-approved, 42.8% routed to review, and 3.0% rejected. Coverage is a conservative lower bound, not a completeness guarantee.Source: Table 5 · p. 8, §4.4 · p. 8
4.5 Comparison Against a Monolithic Baseline
- Hallucination rose from 3.2% with SAGE to 15.7% without governance, duplication from 3.0% to 10.3%, and artifact rule coverage fell from 93.7% to 62.8%.The comparison used the same 115 documents, parser, models, and scoring harness.Source: Table 6 · p. 8, §4.5 · p. 8
4.6 End-to-End Effort
- SAGE completed a complex, image- and table-heavy document in 70–85 minutes, compared with an estimated 1,423 minutes of manual work.Managers estimated 2–3 working days per document for manual production.Source: §4.6 · p. 8
5 Conclusion
SAGE coordinates parsing, extraction, evaluation, and artifact generation through a shared versioned rule store to produce validated artifacts with provenance. The conclusion reports effectiveness on real-world enterprise documents and identifies future extensions.
5 Conclusion
- SAGE coordinates parsing, extraction, evaluation, and artifact generation through a shared versioned rule store, producing validated artifacts with full provenance.Threshold-based escalation keeps human review selective, and zero-edit approvals feed back to sharpen the evaluation judge.Source: §5 · p. 8
- Experiments on real-world enterprise documents confirm effectiveness across content extraction, grounding quality, structural validation, and consistency enforcement.The paper presents SAGE as a scalable basis for governed enterprise document understanding.Source: §5 · p. 8
- Future work includes improving table detection on degraded layouts, learning persona adaptation from approved artifacts, and extending SAGE to multilingual and other domains.The named domains are legal, clinical, and regulatory.Source: §5 · pp. 8–9
Limitations
SAGE’s extraction stability, artifact dimensions, calibration, and evaluation scope remain constrained in specific settings.
Limitations
- VLM extraction stability decreases on low-quality scans and borderless or merged-cell tables.These are described as a known challenge across current vision-language systems.Source: Limitations · p. 9
- Persona appropriateness and rule coverage remain the most challenging artifact dimensions.The passage connects this challenge to adapting technical rules for non-expert audiences.Source: Limitations · p. 9
- Calibration depends on accumulating zero-edit HITL approvals, and scoring stability remains limited in early deployment cycles.Source: Limitations · p. 9
- The current evaluation covers English enterprise guidelines only.Source: Limitations · p. 9
A.1 Per-Stage Effort Breakdown
For the evaluated documents, SAGE’s end-to-end effort ranges from roughly 20 to 100 minutes, including machine processing and selective human review. The per-stage breakdown maps manual work to pipeline stages and reports ranges for a complex document.
A.1 Per-Stage Effort Breakdown
- The per-stage mapping replaces guideline reading with ingestion and extraction, rule inference with extraction and evaluation, ambiguity resolution with consistency and gap review, and drafting with artifact generation and review.Source: §A.1 · p. 11
- 20–100 minutes is the total effort range across evaluated documents, with machine processing and selective human review making up the effort.For the 70–85-minute case, roughly 47–54 minutes are machine time and 23–31 minutes are human review; the manual baseline is an expert estimate, so the comparison indicates an order-of-magnitude reduction, not an exact head-to-head.Source: §A.1 · p. 11
- Table A1 reports repeated-run ranges for machine and HITL effort at each stage for one complex, image- and table-heavy document.Source: Table A1 · p. 11