Source-linked AI summary
Contestable Agents for Data Governance: Recording Human Stewardship of AI-Generated Metadata Across Structured and Unstructured Systems
Humza Akhtar, Brian Freeman
TL;DR
Industrial organizations must connect structured operational records to unstructured documents, but governance often separates these sources and relies on scarce domain experts. The paper proposes steward-contestable agent decisions with cited evidence, scoring, and retained interventions; illustrative HVAC cases show reviewer corrections and cross-system conflicts. The authors note that decision-quality and workload effects remain unmeasured.
Problem
Structured operational records and the documents that define them are often governed separately, while classification and linking depend on a small group of domain experts.
Source: §1 · pp. 1–2Method
The proposed workflow has specialist agents produce scored, cited arguments for disputed metadata, lets stewards intervene before commit, and stores evidence, edits, and outcomes in a governance record.
Source: §1 · p. 2, §3.4 · p. 5, §6 · p. 12Results
In the point-classification case, reviewer challenges and added manual evidence lead to a mixed-air commit consistent with documentation; the warranty case illustrates conflicts across service records and revisioned documents.
Source: §4.1 · p. 8, §4.2 · p. 9, §5 · p. 12Takeaways & Limitations
The cases illustrate that retaining proposals, sources, and steward corrections makes changes in reasoning inspectable, while a versioned glossary cannot establish whether a source applies to a particular unit or revision.
Source: §5 · p. 12The point-classification implementation demonstrates only part of the workflow, the cross-system warranty record and promotion gate remain proposed, and their effects on review workload and decision quality have not been measured.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Industrial organizations hold operational knowledge in structured records, such as telemetry, point lists, and transactions, and in unstructured sources, such as manuals, service notes, communications, and inspection reports. Governance programs often manage these sources separately. Classifying, linking, and documenting them depends on a small group of employees with deep domain knowledge. Agents built on large language models can propose metadata, cite documents, and flag quality issues, but unreviewed proposals can spread errors into downstream systems. A perfunctory approval leaves no record of what a reviewer checked. We present contestable governance: specialist agents produce scored, cited arguments for disputed metadata decisions; a steward can accept, reject, modify, or add arguments before commit; and a governance record retains the evidence, interventions, and outcome. A versioned domain glossary supplies model-specific terms and acronym definitions to the agents. We describe the workflow, a triage path that reserves full argumentation for uncertain proposals while preserving review before commit, the scoring and record models, and a gate that controls which verified decisions become precedent. Two HVAC use cases illustrate the pattern: point classification using Brick Schema class labels, supported by an open reference implementation, and an illustrative warranty failure-mode decision linking claim fields and telemetry to service notes and revisioned bulletins.
1 Introduction
Industrial HVAC governance must connect structured records with the documents that define them, but those sources are often managed separately and linked through scarce employee expertise. The paper proposes contestable governance so agents can assist with metadata decisions while stewards can challenge them before commit and preserve the reasoning.
1 Introduction
- HVAC telemetry, test results, and warranty records are managed separately from the manuals, specifications, and service notes that explain them, leaving employees to remember the links.The sources span structured records and unstructured documentation across product operation, testing, and service.Source: §1 · p. 1
- As products and connected sites grow, few employees can classify new data reliably, and a mistaken sensor class could trigger an incorrect fault diagnosis and site visit.A mixed-air temperature sensor labeled as discharge air could make normal mixing appear to indicate a failed heating coil.Source: §1 · pp. 1–2
- LLM agents can propose classifications from manuals and prior decisions, but direct writes can propagate errors, while one-click approval may leave no record of what reviewers checked.Reviewer corrections can also remain in emails or tickets that the agent never sees, allowing the same mistake to recur.Source: §1 · p. 2
- The paper proposes cited agent arguments and pre-commit steward contestation, with a governance record that retains interventions and outcomes.Its requirements include attribution, append-only history, retrievable lineage, a gate for verified precedent, and measurable reviewer interventions.Source: §1 · p. 2
2 Background and Related Work
Related work frames data governance as a human stewardship activity and develops tools for metadata, provenance, and contestable decisions. The paper brings these strands together for metadata decisions across structured and unstructured HVAC evidence.
2 Background and Related Work
- Governance research assigns decision rights across data domains, while stewardship practice produces classifications whose reasoning is often stored separately.The cited work includes Khatri and Brown (2010), Weber et al. (2009), Otto (2011), Abraham et al. (2019), and DAMA International (2017).Source: §2 · pp. 2–3
- Provenance, FAIR, datasheets, and data-quality research motivate recording evidence and responsibility close to the metadata decision.The discussion draws on Buneman et al. (2001), Moreau and Missier (2013), Wilkinson et al. (2016), Gebru et al. (2021), and Wang and Strong (1996).Source: §2 · p. 3
- Building-automation methods map differently named points to Brick classes, but names can mislead and manuals may resolve ambiguity.The cited approaches include expert-example parsing and cross-building transfer.Source: §2 · p. 3
- Freeman et al. (2026) motivate role separation and glossary use, but their judge-based results do not establish improved factual correctness for metadata decisions in this workflow.Their tests used a fixed ERP incident and a separate HVAC troubleshooting query.Source: §2 · p. 3
- Contestability research studies people questioning algorithmic outputs, while argumentation represents claims, support, and attacks that reviewers can challenge individually.The paper draws on abstract argumentation, bipolar frameworks, and language-model-generated arguments (Freedman et al., 2024).Source: §2 · p. 3
3 Our Approach
The approach combines evidence retrieval, specialist argumentation, human review, and attributable storage in a contestation workflow. It adds scoped domain vocabulary, triage, quantitative scoring, a governance record, and a promotion gate for verified precedent.
3.1 Workflow
- The workflow gathers structured inputs, documents, glossary entries, and similar verified decisions, then has specialists submit evidence-backed arguments before a steward reviews the case.The steward may accept, reject, modify, or add arguments; the system rescores after edits, and no decision is committed before review ends.Source: §3.1 · pp. 3–4
3.2 Domain vocabulary
- The proposed glossary scopes each term to equipment and revision, cites its authorizing source, and stores a version and steward owner.Agents should cite both an applicable glossary entry and its source document; a reviewer can challenge an obsolete or ambiguous definition.Source: §3.2 · pp. 4–5
3.3 Triage
- Triage routes low-confidence proposals to full multi-agent argumentation while still sending high-confidence proposals, with evidence, to a steward before commit.The routing threshold requires a named owner and must be stored; a reviewer can escalate a proposal because probability alone gives no specific reasoning to challenge.Source: §3.3 · p. 5
3.4 Argument scoring
- QBAF scoring represents supporting and challenging arguments with numerical strengths, then updates strengths iteratively using signed support and attack contributions.The illustrative rule uses α = 0.3, and its strengths are comparison scores rather than calibrated probabilities; iteration propagates graph effects but does not add evidence or change base scores.Source: §3.4 · pp. 5–6
3.5 Governance record
- The governance record links evidence, arguments, outcomes, scores, and each human edit so one query can retrieve a decision’s lineage.Deployment requirements include append-only event logging, named reviewers, model and prompt versions, and intervention rationales.Source: §3.5 · pp. 6–7
3.6 Promotion gate
- The proposed promotion gate admits only verified records as precedent, records promotions as attributable events, and allows demotion with dependent decisions flagged for review.This limits the risk that an incorrect precedent spreads across similar cases.Source: §3.6 · p. 7
4 Use Cases
The two HVAC use cases apply contestable governance to point classification and warranty failure-mode metadata, with stewards reviewing agent arguments before decisions are committed. The examples show how domain evidence and reviewer interventions affect selected metadata, while the warranty workflow remains synthetic and proposed.
4.1 Use case 1: HVAC point classification
- The synthetic point list routes 229 of 247 points to shorter evidence-based review and 18 ambiguous points to full argumentation; none is committed before review.The implemented workflow is demonstrated on one point, AHU-1-DISCH-T.Source: §4.1 · p. 7
- The steward’s rejection of abbreviation-only reasoning and added manual evidence raise the mixed-air margin to 1.758, and the system commits the documentation-consistent class.The decision record preserves the rejected reasoning and its rejection rationale.Source: §4.1 · p. 8
- The point-classification implementation has uncalibrated, run-variable base scores and lacks triage, append-only history, and the proposed promotion gate.Normalized near-tied scores may appear highly confident, so the passage says raw margins should also be displayed.Source: §4.1 · p. 8
4.2 Use case 2: Governing warranty failure-mode metadata across systems
- The warranty workflow snapshots claim, telemetry, service-note, bulletin, and glossary evidence, then assigns specialist agents different evidence scopes before reconciliation.The reconciler flags contradictions but cannot select a tag or authorize a commit.Source: §4.2 · pp. 8–9
- After review, the Sensor margin falls to −1.016 while Linkage remains 0.839, and the scoring engine selects Linkage for steward authorization.The steward rejects the bulletin argument because its serial range excludes DEMO-2407 and adds an argument from the technician note.Source: §4.2 · p. 10
- Only verified records would become precedent; a later bulletin revision could flag dependent decisions for review.The claim-to-bulletin link is a separate decision and would be rejected on applicability grounds in this example.Source: §4.2 · p. 11
- The warranty decision and its scoring records are proposed for a fully synthetic example, not produced by a deployed warranty system.Evaluation on independently adjudicated synthetic cases would need to measure tag accuracy, citation and applicability validity, reviewer interventions, and consistency across repeated runs.Source: §4.2 · pp. 11–12
5 Discussion
The cases show that reviewing individual arguments and their source scope matters, not just the selected metadata value. The discussion also distinguishes an auditable decision process from proof of correctness and identifies limits in the examples and scoring method.
5 Discussion
- The examples show that a reviewer must check whether evidence applies to the particular unit or revision, not just whether a metadata value appears supported.The glossary helps interpret local terms but cannot establish source applicability.Source: §5 · p. 12
- Use case 1 demonstrates choosing among Brick class labels, not validating a Brick instance graph.The reported three-label margins are not ontology-normalized; an ontology-aware deployment would collapse equivalent labels and validate the resulting graph against the relevant Brick version.Source: §5 · p. 12
- The illustrative warranty scoring does not establish that the selected failure-mode tag is more accurate than an alternative method.Because S1 and S2 both derive from C1, correlated arguments could raise a margin without adding independent support; deployment should test stability across alternative argument graphs and values of τ and α.Source: §5 · p. 12
- An auditable record makes decisions reconstructable, but it does not make them correct; the cross-system warranty record and promotion gate remain proposed and unevaluated.Stewards still need to check document scope, resolve conflicting evidence, and authorize commits; effects on review workload and decision quality have not been measured.Source: §5 · p. 12
6 Conclusion and Future Research
The conclusion presents contestable governance as a way to retain cited evidence, steward interventions, and attributable decisions while triaging uncertain proposals for fuller review. It identifies evaluation of the workflow, its learning loop, and its scoring rule as future work.
6 Conclusion and Future Research
- Contestable governance records cited evidence and steward challenges before commit, while triage reserves full specialist argumentation for uncertain proposals.The workflow still requires review before commit; a risk-based automatic path for reversible, low-risk edits is proposed for future evaluation.Source: §6 · p. 12
- Future research should test whether verified decisions used as precedent improve accuracy on later units and how far incorrect precedent spreads before demotion.It should also examine when firmware revisions make older precedent unreliable.Source: §6 · pp. 12–13
- Separate comparisons should test whether versioned glossary entries and specialist roles improve citation validity and repeated-run consistency over an unstructured prompt.Source: §6 · p. 13
- Scoring-rule validation should combine formal checks with held-out empirical comparisons against DF-QuAD and a specified modular alternative.The proposed tests include assessor agreement, top-class accuracy, ranking stability, and stopping at a stated numerical tolerance.Source: §6 · p. 13