PaperlayerView exact revision on arXiv

Source-linked AI summary

Glyph: A Multi-Strategy Agentic System for Column Description and Sensitivity-Ontology Tagging of Enterprise Data Catalogs

Kostia Kudriavtsev, Parvez Rafi, Sha Sundaram

arXiv:2609.10430v1cs.MAcs.IR

TL;DR

Large enterprise catalogs leave columns undocumented and governance labels incomplete, complicating discovery and compliance. Glyph pairs code-grounded descriptions with value-free tagging from a governed ontology. Its metadata retrieval improved on a held-out split, and the paper presents a deployed, auditable multi-strategy system.

  • Problem

    Enterprise catalogs contain missing or stale descriptions and sensitive-data tags that can lag discovery by months, despite their role in access control and privacy compliance.

    Source: §1 · p. 1

  • Method

    Glyph uses cooperating agents to ground descriptions in source code and assign governed sensitivity labels by fusing complementary, value-free tagging strategies.

    Source: §1 · pp. 1–2, §4.2 · p. 4

  • Results

    NDCG@10 0.55 →0.92 and MAP@100 0.19 →0.90 on a per-tag, column-level held-out split; Glyph also demonstrates a deployed multi-strategy system.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The paper concludes that auditable cataloging at enterprise scale is achievable with a disciplined agentic framework under a constraint against reading values.

    Source: §7 · p. 8

    Existing benchmarks do not measure sensitivity classification in Glyph’s multi-label sense, so SOTAB and GitTables offer only an out-of-distribution probe for its metadata-only encoder.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Enterprise data lakes accumulate tables faster than human stewards can document or classify them, leaving columns with missing descriptions and unassigned governance labels. This documentation debt undermines data discovery, access control, and regulatory compliance. We present Glyph, a production system that frames two coupled problems, column description generation and column type annotation for data classification, as cooperating LLM agents orchestrated as stateful graphs. The Descriptor grounds generation in the pipeline source code that produces each column, retrieved on demand from an enterprise GitHub via a reasoning--acting tool loop (active Retrieval-Augmented Generation). The Tagger assigns labels from a governed 275-leaf Data Classification Ontology by running three complementary strategies in parallel (a description tagger, a line-of-business regex tagger, and a metadata tagger backed by a fine-tuned contrastive encoder over a vector database), then fuses their ranked outputs with Reciprocal Rank Fusion (RRF). We fine-tune a 6-layer MiniLM metadata encoder with an in-batch contrastive objective, lifting same-tag retrieval on an in-distribution held-out split from NDCG@10 0.55 to 0.92 (MAP@100 $0.19 \rightarrow 0.90$) relative to the stock base encoder. We report end-to-end multi-label tagging quality under a recall-weighted F2 objective across three evaluation groups, an ablation isolating each strategy and the RRF fusion, and the engineering decisions that distinguish Glyph from prior column-type-annotation work and from commercial value/regex sensitivity scanners: value-free and code-grounded design, per-tag provenance, and graceful degradation. Together these make multi-agent LLM cataloging auditable and operable as a production service.

1 Introduction

Glyph addresses missing column descriptions and governance labels in catalogs too large for manual stewardship. It combines code-grounded description generation with value-free, multi-strategy tagging for a governed sensitivity ontology.

1 Introduction

  • Missing or stale descriptions and delayed sensitive-data tagging undermine discovery, access control, and privacy compliance.The introduction notes that sensitive-data tagging can lag discovery by months.Source: §1 · p. 1
  • A production catalog spans ∼3.4M columns across ∼98,000 tables; annotating it at one column per minute would require roughly 30 person-years.Its tables span four lines of business and three storage backends, while the catalog drifts faster than a team can keep pace.Source: §1 · p. 1
  • Glyph separates meaning generation from category assignment and targets a governed enterprise sensitivity ontology with 275 leaf annotations.It never reads cell values, and grounds descriptions in producing source code.Source: §1 · pp. 1–2
  • On a per-tag, column-level held-out split, metadata-tag retrieval improved NDCG@10 0.55 →0.92 and MAP@100 0.19 →0.90.The approach fine-tunes a metadata-specific encoder with contrastive learning; the introduction also describes steward-curated evaluation sets and strategy ablation.Source: §1 · p. 2

2 Related Work

Related work spans semantic column annotation, benchmarks, and enterprise catalogs. Glyph adapts annotation methods to governed, multi-label sensitivity tagging from metadata without cell values.

2.1 Semantic Column Type Annotation

  • Classical, supervised, table-encoder, and LLM-era CTA systems mostly target open-domain types, while Glyph combines candidate-constrained generation, RRF over heterogeneous strategies, and metadata-only sensitivity tagging.Prior examples include Sherlock, Sato, Doduo, and ArcheType.Source: §2.1 · p. 2

2.2 Benchmarks

  • Existing benchmarks cover open-domain types rather than sensitivity labels or Glyph-style multi-label classification, and treat cell values as the primary signal.SOTAB and GitTables headers offer a usable out-of-distribution probe for Glyph’s metadata-only encoder.Source: §2.2 · pp. 2–3

2.3 Industrial Catalogs and Data-Security Posture

  • Enterprise catalogs and commercial scanners store or propagate classifications, but often rely on rules, human descriptions, lineage, or value-pattern matching rather than inferring contextual seed labels.Glyph is positioned as an enrichment engine producing descriptions and seed labels for catalogs to store and propagate.Source: §2.3 · p. 3

3 Problem Formulation and the Data Classification Ontology

Glyph formulates column documentation and multi-label sensitivity classification as tasks over a metadata key and optional description. Its governed ontology makes the target distinct from open-domain type annotation.

3 Problem Formulation and the Data Classification Ontology

  • For each column, Glyph may generate a natural-language description from authoritative evidence and assign a subset of ontology annotations, each with confidence and justification.The column is identified by a metadata key and may also have a description.Source: §3 · p. 3
  • The Data Classification Ontology has 275 leaf annotations in a hierarchy, with dimensions such as sensitivity level encoding what a column is and how sensitive it is.Examples include USER_NAME, USER_ID, and EMAIL_ADDR.Source: §3 · p. 3
  • Unlike open-domain CTA, Glyph assigns multi-label governance decisions under a severely skewed distribution, with NON_SENSITIVE accounting for ≈72% of columns.Because reading values may be forbidden before classification, Glyph operates on metadata and code, trading away a signal for deployment on restricted-access tables.Source: §3 · p. 3

4.1 System Overview

Glyph composes independently deployable graph agents for description generation and tagging. It combines established methods and production controls rather than relying on a novel model alone.

4.1 System Overview

  • The agents are graph applications whose nodes operate on validated typed state, with injected dependencies and layered configuration.Both agents decode deterministically and cap token usage.Source: §4.1 · p. 3
  • Glyph uses well-characterized active retrieval, contrastive dense retrieval, multi-label classification, rank fusion, and LLM-as-Judge critique as system components.The contribution is their composition, ontology validation, and service design.Source: §4.1 · pp. 3–4
  • The Descriptor grounds descriptions in producing source code through active retrieval, while the Tagger runs three strategies in parallel and fuses ranked lists with RRF.The agents are independently deployable REST services, and generated descriptions can feed the description tagger.Source: §4.1 · p. 3, Figure 1 · p. 4

4.2 The Descriptor Agent

The Descriptor turns catalog schema into column and dataset descriptions, grounding column descriptions in the application code that produces each column. It also evaluates and refines descriptions when grading is enabled.

4.2 The Descriptor Agent

  • Schema discovery fetches column names, types, existing descriptions, and other table details from the enterprise registry.Source: §4.2 · p. 4
  • GitHub code search grounds descriptions in the application code that produces each column; the Descriptor never reads column values.The LLM autonomously chooses search queries, with a default budget of three calls per column.Source: §4.2 · p. 4
  • The LLM analyzes columns in batches of at most 50 and autonomously selects code-search queries under a default three-call-per-column budget.Calls are capped per batch, with forced finalization.Source: §4.2 · p. 4
  • PII-aware descriptions flag identifiers such as user_id, and per-column results feed a table-level business-purpose summary.PII awareness is scored directly; value-format detectors may miss identifiers represented by opaque hashes.Source: §4.2 · p. 4
  • An optional judge scores plausibility, specificity, completeness, and PII awareness, returning below-threshold columns for up to two refinement passes.The agent re-searches GitHub for the lowest-scoring columns during retries.Source: §4.2 · pp. 4–5

4.3 The Tagger Agent

The Tagger assigns ontology labels by running complementary description, regex, and metadata strategies in parallel, then fusing their ranked outputs. Its outputs include per-tag source and reasoning provenance.

4.3 The Tagger Agent

  • Three parallel strategies produce ranked tag lists that a dedicated fusion node combines, using complementary signals with uncorrelated failure modes.Source: §4.3 · p. 5

4.3.1 Description Tagger

  • The description tagger retrieves similar ontology annotations from a name-and-description query, then has the LLM select and rank candidate tags.Candidates are validated against the ontology; optional grading sends below-threshold results through a bounded refinement loop.Source: §4.3.1 · p. 5

4.3.2 Regex Tagger

  • The regex tagger applies line-of-business-specific expert rules to column names, providing high-precision anchors such as email → EMAIL_ADDR.Source: §4.3.2 · p. 5

4.3.3 Metadata Tagger

  • The metadata tagger retrieves historical tags using the fully qualified catalog.database.table.column key, without requiring a human description.Each selected tag records the contributing source and confidence, providing finer-grained provenance than commercial rule-hit logs or academic systems' bare labels.Source: §4.3.3 · p. 5

4.3.4 Cross-Strategy Fusion with Reciprocal Rank Fusion

  • Reciprocal Rank Fusion combines incomparable ranked lists without score calibration, normalizes confidence to [0, 1], and adjusts final scores by ontology sensitivity tier.A confidence of 1.0 requires unanimous top-rank agreement; tags below a minimum normalized score are dropped.Source: §4.3.4 · pp. 5–6

4.4 Domain-Specific Embedding Fine-Tuning

Glyph fine-tunes a compact encoder for metadata-key retrieval, then evaluates same-tag retrieval on a column-level held-out split. The reported retrieval gain is an in-distribution upper bound, not end-to-end tagging quality.

4.4 Domain-Specific Embedding Fine-Tuning

  • 70,948 balanced same-tag metadata-key pairs train MiniLM with an in-batch contrastive objective that pulls same-tag columns together and pushes different-tag columns apart.Pairs are capped per tag to limit dominance by frequent labels.Source: §4.4 · p. 6
  • +0.71 MAP@100 gain measures the fine-tuned encoder's improved recovery of same-tag neighbors on the column-level held-out split.The table reports retrieval rather than tagging metrics.Source: Table 2 · p. 6, §4.4 · p. 6
  • Because held-out queries may share tables with the retrieval corpus, these results are an in-distribution upper bound; unseen-table tagging is evaluated separately.Source: §4.4 · p. 6
  • The system uses an exact flat index at modest scale; scaling beyond approximately 10^6 vectors would require a quantized or graph-based index.Source: §4.4 · p. 6

5 End-to-End Evaluation

Glyph’s full Tagger achieves its highest overall F2 with multi-strategy fusion, while steward acceptance rises during production operation. Evaluation also frames F2 as a recall-weighted deployment choice and cautions that the results are not directly comparable to published CTA scores.

5.1 Per-Dataset Tagging Quality (F2)

  • 0.890 overall F2 for the best fused configuration exceeds the metadata tagger’s 0.880; the regex-only tagger is near zero across groups.The table reports micro-averaged F2 for each configuration by curated dataset group and overall.Source: Table 3 · p. 7, §5.1 · p. 7

5 End-to-End Evaluation

  • F2 weights recall twice as heavily as precision because missed sensitive columns can defeat access control, while over-tags are remediated in steward review.The evaluation sets are steward-authored, table-disjoint from the training corpus, and cover 1,905 tables across multiple lines of business.Source: §5 · p. 7

5.1 Per-Dataset Tagging Quality (F2)

  • 57.8 s p50 latency accompanies fusion’s F2; under F1 fusion ties metadata-only, while under F0.5 the single strategy leads.The description tagger takes 47.3 s for 0.238 F2, making metadata-only versus fusion the operative deployment choice.Source: §5.1 · p. 7

5.2 Deployed Impact: Steward Acceptance

  • The training pipeline re-exports steward decisions and fine-tunes the metadata encoder weekly.Stewards adjudicate suggestions, and accepted or rejected column–tag pairs are written back to a data store.Source: §5.2 · p. 7
  • 63.3% to 99.8% steward acceptance across about 24,000 adjudicated pairs coincides with 21× decision-volume growth, so the trend is reported as association, not causation.Acceptance reflects only emitted tags and is precision-like; it is blind to false negatives and complements rather than replaces recall-weighted F2.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 Positioning Against the CTA Literature

  • Glyph’s CTA scores are not directly comparable with published systems because taxonomy, label cardinality, and available information differ.The comparison is presented as an in-spirit reference; Table 4 positions Glyph against representative CTA systems and commercial scanners.Source: §5.3 · p. 8, Table 4 · p. 8

6 Discussion and Future Work

Glyph’s discussion argues for redundancy and principled fusion across imperfect signals, while identifying operational choices and future extensions. It proposes collective inference and a reinforcement-learning loop as directions for further work.

6 Discussion and Future Work

  • Glyph’s central design bet is that RRF and ontology validation reconcile a brittle rule engine, a drifting dense retriever, and a hallucination-prone LLM.An autograder turns open-ended generation into a measurable, self-improving task.Source: §6 · p. 8
  • Value-free, code-grounded analysis supports restricted-access tables, while table-level LLM batching dominates latency and cost at catalog scale.The paper identifies these as operational lessons from Glyph’s production use.Source: §6 · p. 8
  • Future work includes collective inference over column and tag correlations, plus a reinforcement-learning policy trained to rank rare tags and internalize steward feedback.The authors note that Glyph currently fuses per-column signals and ignores inter-column and inter-tag correlations.Source: §6 · p. 8

7 Conclusion

Glyph concludes that enterprise-scale cataloging can be automated within a governed sensitivity ontology and a privacy constraint that forbids reading data values. The system grounds descriptions in source code and produces tags with provenance, confidence, and reasoning, while public-benchmark generalization and judge–human agreement remain to be shown.

7 Conclusion

  • Glyph composes established techniques in an agentic framework for automated, auditable enterprise cataloging under a no-values privacy constraint.It adapts column-type annotation to a governed sensitivity ontology.Source: §7 · p. 8
  • Source-code-grounded descriptions, value-free tagging signals, ontology validation, and output grading make generated descriptions and tags explainable and auditable.Every tag carries provenance, confidence, and reasoning.Source: §7 · p. 8
  • Public-benchmark generalization and judge–human agreement remain unshown; the authors identify closing that gap as the next step toward reducing manual stewardship burden.Source: §7 · p. 8

Related papers

Loading 2609.10430v1…