PaperlayerView exact revision on arXiv

Source-linked AI summary

QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda, Naomi Couriel, Yelena Mejova, Mariano G. Beiró, Kyriaki Kalimeri

arXiv:2609.35685v1cs.CL

TL;DR

Human reference spans and language-model extractions can disagree, making trustworthy reconciliation important for annotation layers. QuanReview aligns the streams, applies explicit policies, and sends conflicts to reviewers for adjudication. On a 4,457-record benchmark, it auto-merged 8% of documents and routed 3,131 candidate conflicts to reviewers.

  • Problem

    Human reference annotations can disagree with language-model extractions for reasons including model errors, reference errors, or genuine ambiguities, creating a need for auditable reconciliation.

    Source: §1 · p. 1

  • Method

    QuanReview aligns two annotation streams, applies logged policies to one-sided records, and routes conflicts to browser adjudication for acceptance, field-level hybrids, or re-annotation.

    Source: §2 · p. 2, §3 · p. 2

  • Results

    On 4,457 records, QuanReview auto-merged 8% of documents, applied policies to 1,513 records, and routed 3,131 candidate conflicts to reviewers, averaging 5.4 per reviewed document.

    Source: §6 · p. 3

  • Takeaways & Limitations

    The corrected layer retains the input schema and file layout, allowing it to replace the original annotation files without changing downstream tooling.

    Source: §7 · p. 5

    Agreement measures reviewer consistency rather than independent annotation reliability, and extensibility beyond the single schema family used so far remains un demonstrated.

    Source: §8 · p. 5

Abstract

from arXiv · show

Structured span annotations, such as quantities with their units, uncertainty modifiers, and event classes, are expensive to create and hard to keep trustworthy once language models enter the loop. We present QuanReview, an open-source system for auditing and correcting such annotation layers. QuanReview aligns two annotation streams over the same documents at character level, resolves unambiguous cases by an explicit and logged policy, and routes candidate conflicts to a browser-based adjudication interface where reviewers accept either side, build field-level hybrids, or flag items for re-annotation. A campaign manager assigns documents to multiple annotators with configurable redundancy, computes agreement at document and span level, auto-merges unanimous documents, and exports the corrected layer in the original file format, so that it can replace the original annotation files directly. Applied to a 4,457-record humanitarian benchmark and an LLM extraction stream, the system fully auto-merged 8% of documents, applied automatic policy decisions to a further 1,513 records, and concentrated human attention on 3,131 candidate conflicts, a mean of 5.4 per reviewed document.

1 Introduction

QuanReview addresses disagreements between human reference spans and strong-model extractions as an auditable workflow problem rather than a modeling problem. It is designed to help curators reconcile those layers without corrupting the dataset.

1 Introduction

  • Human reference annotations can disagree with strong language-model extractions, and those differences may be model errors, reference errors, or genuine ambiguities.Distinguishing among them at scale is described as a workflow problem.Source: §1 · p. 1
  • Trustworthy reconciliation requires position-anchored comparison, explicit handling of one-sided records, human decisions on conflicts, and an audit trail.Source: §1 · p. 1
  • QuanReview audits HumSet’s quantitative layer, which binds numbers to units, uncertainty modifiers, and event classes, while allowing configurable schemas for other typed-span tasks.It targets curators, NLP researchers, and domain experts reviewing model output in a browser.Source: §1 · p. 1

2 Related systems

Existing annotation platforms support parts of curation and model-assisted labeling, but QuanReview focuses on post-hoc reconciliation of two complete annotation exports. Its authors report that they have not yet compared it with an established platform on the same curation task.

2 Related systems

  • INCEpTION supports curation across annotators, while Label Studio documents agreement and model-prediction comparison workflows (Tkachenko et al., 2020–2024).doccano and Prodigy focus on annotation and model-assisted labeling; Argilla targets LLM-in-the-loop curation.Source: §2 · p. 1
  • Because presenting recommendations beside human judgments can introduce automation and anchoring biases, QuanReview uses neutral labels, randomized order, reversible decisions, and explicit conflict routing.Source: §2 · p. 1
  • QuanReview combines offline two-stream reconciliation, character-offset alignment, explicit one-sided policies, conflict-only routing, field-level hybrids, reversible logs, and input-format export.The authors leave a controlled head-to-head comparison with an established platform for future work.Source: §2 · p. 2

3 System overview

QuanReview is a Python application with a browser interface that runs on local files without sending document text or annotations elsewhere. Its configurable components align records, apply explicit policies, support conflict adjudication, coordinate annotators, and export corrected files.

3 System overview

  • A deterministic alignment engine pairs records by overlapping character offsets and compares matched fields using a declarative YAML schema.The quantity schema includes quantity, unit, modifier, and event type; the configuration mechanism can specify other typed-span and categorical schemas.Source: §3 · p. 2
  • Reference-only records are kept, while model-only records are deferred for a second annotation round rather than discarded or added to the corrected layer.Automatic policy decisions are logged, and reviewer decisions are reversible.Source: §3 · p. 2
  • Reviewers can accept either candidate, build a field-by-field hybrid, or flag a conflict for fresh annotation when neither stream resolves it.Flagged pairs are withheld from the corrected layer; decisions are saved immediately, with undo, review state, and comments available.Source: §3 · p. 2
  • The campaign manager assigns discrepant documents across configured annotators, computes Fleiss’ κ over adjudication decisions, and auto-merges documents with unanimous decisions.Disagreements are emitted as conflict bundles for resolution.Source: §3 · p. 2
  • Corrected documents retain the input JSON schema and include per-document review-state files and comment logs, so they can replace the original annotation files with provenance for changed records.Source: §3 · p. 2

4 User workflows

QuanReview supports single-reviewer audits, redundant multi-annotator campaigns, and comparisons between model outputs. Its architecture routes conflicts to reviewers while applying policies to one-sided records and merging unanimous reviewed documents.

4 User workflows

  • A single reviewer can compare reference and model directories, inspect the conflict queue, and export the corrected layer.The workflow is presented as a quick way to assess how much to trust an annotation layer.Source: §4 · p. 2
  • In a multi-annotator campaign, configured annotators and redundancy drive document assignments; integration reports agreement, merges unanimous documents, and routes remaining cases to an adjudicator.Source: §4 · p. 2
  • Two model outputs can be loaded to inspect system disagreements before spending human annotation time, using the same alignment, policies, and interface.Source: §4 · pp. 2–3
  • The system comparison identifies QuanReview as the only listed system combining two-stream reconciliation, field-level hybrids, and character-level alignment.“Partial” means support for a related capability without the complete post-hoc two-stream workflow.Source: Table 1 · p. 3
  • The architecture aligns reference and model spans, keeps reference-only records and defers model-only records, routes conflicts to adjudication, and automatically merges unanimous reviewed documents.Flagged pairs and deferred model-only records are queued for a second annotation round.Source: Figure 1 · p. 3

5 Demonstration scenario

The demonstration walks through a miniature humanitarian-report campaign, from schema setup and reviewer sessions to adjudication and export. It illustrates how conflicts and missed records are handled and verifies that corrected files retain the originals’ schema and layout.

5 Demonstration scenario

  • The campaign integrates reviewer decisions and exports corrected files in the originals’ schema and layout, allowing direct replacement.Visitors can inspect the exported files beside the originals and try the interface on the sample dataset.Source: §5 · p. 3
  • Two characteristic conflicts demonstrate adjudication: one differs on event class, while another differs on the uncertainty modifier.The demonstration also shows a model-only record automatically deferred for a second annotation round rather than silently dropped.Source: §5 · p. 3
  • The miniature campaign configures the schema, launches two reviewer sessions, runs integration, and opens the agreement report.Source: §5 · p. 3

6 System validation

QuanReview’s validation documents how the workflow divides work between automatic decisions and human review, and reports reviewer outcomes and agreement. Field-level hybrids were the most common human-selected resolution, while agreement varied across overlap-pair subsets.

6 System validation

  • 8% of 635 documents were fully automerged; 1,513 records received automatic policy decisions, and 3,131 candidate conflict pairs went to reviewers (5.4 per reviewed document).The automatic decisions included 792 exact matches retained, 434 reference-only records retained, and 287 model-only records deferred for re-annotation.Source: §6 · p. 3
  • Eight reviewers from two institutions adjudicated conflicts, and their per-reviewer flag rates ranged from 2.0% to 24.5%.Reviewers were assigned different sets of conflict pairs, so activity rates are descriptive.Source: §6 · p. 4, Table 2 · p. 4
  • 63.2% of outcomes were field-level hybrids, compared with 12.7% reference-only and 0.6% model-only acceptances; 23.6% were automatically retained matches.Hybrids made up 82.7% of the 3,150 outcomes resolved by reviewers.Source: §6 · p. 4, Table 3 · p. 4
  • Across 78 shared pairs, adjudication agreement was Fleiss’ κ = 0.624, versus 0.720 on the 13 pairs no reviewer flagged; event type was the hardest field.The higher unflagged-pair agreement is descriptive given the small subset; reviewers showed essentially no agreement on which pairs to flag (κ = −0.01).Source: §6 · p. 4, Table 4 · p. 5

7 Conclusions

QuanReview presents an offline, auditable workflow for reconciling human and LLM annotation layers and reports its use on a 4,457-record humanitarian benchmark. The paper identifies broader schema support and runtime configuration as future extensions.

7 Conclusions

  • QuanReview aligns human and LLM annotation streams, applies logged policies to one-sided records, sends conflicts for adjudication, and exports corrected files in the input schema and layout.On a 4,457-record benchmark, it automerged 8% of documents, applied policies to a further 1,513 records, and routed 3,131 candidate conflicts to reviewers; field-level hybrids accounted for most decisions.Source: §7 · p. 5
  • The current system’s scope is to be extended with broader schema support and runtime editing of schemas, policies, and redundancy settings.The comparison layer is already schema-driven, but broadening it to additional compatible typed-span tasks and dynamic configuration are identified as future directions.Source: §7 · p. 5

8 Limitations

QuanReview is limited to reconciling existing annotation streams, while its agreement measure and demonstrated schema coverage do not establish independent annotation reliability or domain independence.

  • QuanReview reconciles two existing streams; it does not annotate raw text, and established platforms were used to create the reference layer.Source: §8 · p. 5
  • Current agreement scores reflect reviewer consistency, not independent annotation reliability.Source: §8 · p. 5
  • Only one schema family has been exercised, so extensibility to other typed-span tasks is claimed rather than demonstrated domain independence.Source: §8 · p. 5

Ethics Statement

The ethics safeguards keep sensitive data local, preserve dataset-owner governance, pseudonymize reviewer identities in logs, and reduce anchoring on the reference stream.

  • Local execution keeps document text and annotations on the machine; only annotations and review metadata are stored, leaving data governance with the dataset owner.Source: Ethics Statement · p. 5
  • Exported logs identify reviewers by pseudonymous annotator IDs.Source: Ethics Statement · p. 5
  • Neutral Annotation 1 and Annotation 2 labels and independently randomized left–right order limit anchoring on the reference stream.Source: Ethics Statement · p. 5
Loading 2609.35685v1…