Source-linked AI summary
Artificial Intelligence-Assisted Digital Inventory of Cultural Heritage & Traditional Knowledge: Case for Indonesian Open Digital Library of Culture
Hokky Situngkir
TL;DR
PDBI’s manual cultural-heritage collection is limited by fragmented coverage, mixed source integrity, and shallow records. The paper presents a formalised autonomous harvesting framework built around a five-stage funnel and deterministic orchestration with agentic decisions. Its supported outcome is a design that combines expansion and intensification with architectural integrity, reliability, cultural safeguards, and preserved human control.
Problem
Manual PDBI collection faces coverage, integrity, and completeness barriers because cultural knowledge is dispersed, open sources mix authentic documentation with noise, and many entries remain shallow.
Method
The paper formalises a five-stage autonomous harvesting funnel combining focused crawling, multilingual canonicalisation, novelty testing, evidence-based decisions, reliable publication, and staged human governance.
Results
The framework integrates expansion and intensification as two branches of one decision tree, while enforcing source-boundness, uncertainty auditing, idempotent correction, and protection of human contributions.
Takeaways & Limitations
The framework treats cultural-heritage harvesting as a governed machine-augmentation process in which quality, traceability, cultural sensitivity, and human sovereignty are architectural properties.
Takeaways & Limitations
The framework still requires continuous seed curation, faces costly and brittle access barriers, assumes conditional source independence, and depends on language-model and audit-loop quality.
Abstract
from arXiv · showhide
The Indonesian Digital Library of Culture (Perpustakaan Digital Budaya Indonesia, PDBI; budaya-indonesia.org) is a participatory platform that has collected tens of thousands of entries on Nusantara cultural heritage through public contribution since 2007. Manual contribution faces three structural barriers: coverage (knowledge is scattered across languages and sites), integrity (open sources mix authentic documentation with noise), and completeness (subjects are recorded but their data remain shallow). This paper presents a methodological framework for autonomous, AI-based harvesting of cultural knowledge from the open web, designed to expand corpus coverage while intensifying per-entry data depth. The methodology is organised as a five-stage economic funnel: focused crawling, multilingual extraction and canonicalisation, vector encoding with blocking, agentic decision-making, and idempotent publication, under the principle of deterministic orchestration, agentic decisions. Each stage is formalised: funnel economics and optimal filter ordering; crawl-frontier dynamics as a subcritical branching process that explains the necessity of recurrent re-seeding; fact-level novelty via a containment measure; Bayesian multi-source evidence fusion with elevated publication thresholds for sacred categories; exactly-once effects via idempotent upserts and the transactional outbox; sliding-window inference budgeting with a reservation protocol; statistical quality auditing; and seed selection as submodular coverage maximisation. The framework retains four high-value human roles: curator of direction, escalation approver, quality auditor, and guardian of meaning, while machine autonomy is raised in stages. Ethical, legal, and cultural-sensitivity implications are discussed, including the architectural guarantee that the machine never overwrites human contributions.
1 Introduction
PDBI is a participatory platform documenting Indonesian cultural heritage, but manual collection is constrained by scattered knowledge, unreliable sources, and shallow entries. The paper proposes an autonomous harvesting framework that expands coverage and deepens records while preserving traceability and human sovereignty.
- PDBI has collected tens of thousands of entries across fifteen official cultural categories through participatory documentation.
- Manual collection faces coverage, integrity, and completeness barriers across dispersed sources, mixed-quality claims, and shallow records.
- The framework discovers multilingual cultural artefacts, creates source-bound Indonesian records, tests fact-level novelty, and publishes only genuinely additive content.
- A production prototype operates against the PDBI corpus, while the methodology is presented generically for other cultural-heritage initiatives.
- The paper formalises a five-stage funnel, evidence fusion, exactly-once reliability, inference-budget control, auditing, and staged human-governed autonomy.
2 The Indonesian Digital Library of Culture as Context
PDBI is a participatory web-based system for documenting Indonesia’s cultural diversity, supported by communities and institutional policy. The framework treats it as both an application context and a bridge toward machine-augmented participatory heritage platforms.
- PDBI responds to Indonesia’s cultural diversity through participatory, web-based documentation supported by communities and cultural volunteers.
- Within the framework, PDBI supplies the target schema through its fifteen categories and entry structure.
- The approach extends participatory heritage platforms with machine intelligence while retaining human labels and reviews as training material.
3 Related Work
The framework builds on established work in focused crawling, probabilistic knowledge-base construction, entity resolution, distributed reliability, and quality calibration. It adapts these techniques to multilingual cultural-heritage harvesting with source-bound facts and domain-specific constraints.
- Focused crawling contributes relevance-prioritised frontiers, while this framework omits freshness recrawling and uses open encyclopaedias only as pointers to primary sources.
- Knowledge-base construction informs multilingual extraction and probabilistic evidence fusion, augmented by schema-level rejection of facts without references.
- Classical record linkage, blocking, fact-set containment, and HNSW nearest-neighbour search support entity resolution and deduplication.
- Distributed-systems reliability contributes circuit breakers, idempotent exactly-once effects, transactional outboxes, filter ordering, and queue dimensioning.
- Quality control draws on Wilson intervals, model-confidence calibration, and submodular maximisation for coverage selection.
4 Methodological Framework
The framework separates deterministic workflow orchestration from agentic decisions inside a five-stage funnel. Cheap filters narrow candidates before expensive reasoning, with durable queues and shared relational state supporting auditable, stateless workers.
- The design rule is deterministic orchestration, agentic decisions: fixed workflow mechanics surround AI calls at reasoning-intensive decision nodes.
- The funnel begins with human-curated seeds from the open web and covers cultural relevance, extraction, canonicalisation, novelty, and cross-validation.
- Each funnel stage narrows candidates, reserving expensive reasoning for items that survive cheaper filters; rejected items become calibration labels.
- The funnel’s cost profiles place inexpensive aggressive filters early and expensive stages late, concentrating inference on survivors.
- The five-stage architecture uses durable message queues, one consumer per stage, and a relational database with vector support so workers remain stateless.
5 Formal Models
The framework formalises cultural-heritage harvesting as a budgeted pipeline that models corpus structure, funnel economics, crawl dynamics, and multilingual canonicalisation. Its design places costly processing behind selective filters while using recurrent reseeding to counter structurally finite crawl frontiers.
- 5.1 Corpus formalisation and objective: The corpus consists of source-bound entries with categories, regions, and attributed facts, while the objective maximises validated novel information under budget and precision constraints.An unreferenced fact cannot be represented in the schema.
- 5.2 Funnel economics and filter ordering: Filters should be ordered by ascending cost–selectivity ratio, placing expensive reasoning behind strong filters to reduce publication cost.This ordering supports national-scale harvesting on a small, fixed daily inference budget.
- 5.3 Crawl dynamics: priority, politeness, and frontier extinction: The crawl frontier is a priority queue of normalised URLs governed by relevance, bounded depth, denylist rules, and minimum per-host delays.Index and listing pages serve as discovery-only pages, while encyclopaedia references lead to primary sources without storing encyclopaedia text.
- 5.3 Crawl dynamics: priority, politeness, and frontier extinction: The crawler models productive links as a Galton–Watson branching process whose mean reproduction rate falls below one as URL deduplication and neighbourhood saturation increase.The resulting subcritical process dies out almost surely.
- 5.3 Crawl dynamics: priority, politeness, and frontier extinction: Because frontier exhaustion is structural without freshness recrawling, the protocol injects new seeds when frontier depth falls below a threshold and monitors yield between tranches.Seeds are added one category tranche at a time before widening further.
- 5.4 Multilingual extraction and canonicalisation: Stage 2 maps multilingual documents to canonical candidates containing names, aliases, categories, regions, and source-bound facts.Long documents are segmented before extraction, allowing one source to yield multiple candidates with checkpoints.
- 5.4 Multilingual extraction and canonicalisation: Language normalisation is applied once so matching, reasoning, and publication operate on a single canonical language.This placement keeps downstream processing language-clean.
5.5 Vector representation, blocking, and partitioned neighbour search
The framework separates candidate blocking from similarity scoring and defines novelty at the fact level rather than the subject level. This supports both efficient matching and targeted enrichment while preserving human records and applying graded evidence thresholds.
- Vector representation and blocking: Candidates use shape and content feature blocks, but pairwise comparison occurs only after the blocking predicate passes.The shape block contains category, region, and structural flags; the content block contains names, region tokens, and category-conditioned attributes.
- Partitioned neighbour search: Final similarity combines n-gram cosine similarity, attribute Jaccard, and name match as a convex combination whose weights sum to one.The score is computed from the content block only.
- Partitioned neighbour search: Shape features partition the index rather than enter the score because same-category-same-region pairs grow quadratically as the corpus expands.The shape key remains effective for narrowing the nearest-neighbour search.
- Partitioned neighbour search: The matching stage emits structured neighbour evidence—category-region agreement, attribute overlap, and name match—for downstream reasoning and human auditing.Its output is therefore not an opaque scalar score.
- Data-level novelty: Novelty asks whether the corpus already holds the candidate’s data, not merely whether the subject already exists.Fact containment adapts document containment to aligned fact sets.
- Human-record protection: Machine updates may touch only machine-owned entries, while gaps concerning human-covered subjects receive distinct entries that reference rather than overwrite human records.This rule enforces human contribution sovereignty at the publication gate.
- Human-record protection: Deterministic entry identity consolidates sources about one variant while preserving distinct regional variants as separate entries.The design treats local diversity as information to preserve rather than duplicate noise.
- Data-level novelty: Data-level duplication turns the system into an intensification engine by enabling enrichment and supplementation alongside net-new expansion.The novelty tree uses these branches to distinguish new entries from additional facts.
5.8 Idempotent publication and exactly-once effect semantics
The framework achieves exactly-once publication effects despite at-least-once messaging by combining idempotent upserts with a transactional outbox. Provider failures are classified and handled through bounded retries, pauses, terminal outcomes, circuit breaking, shared sliding-window budgets, and statistical auditing.
- Exactly-once publication: Idempotent upserts make redelivery leave the final publication state unchanged.The transactional outbox writes the stage artefact, follow-on job, and outgoing message atomically before relay to the queue.
- Exactly-once publication: Transient failures use bounded geometric backoff and a dead-letter queue that can be re-driven after capability recovery.Work is delayed rather than discarded.
- Inference-cost control: A shared cost ledger and reservation protocol ensure parallel workers enforce one sliding-window inference cap rather than separate per-worker caps.The window self-clears as old spend leaves it, while bounded concurrency provides backpressure.
- Provider reliability: The provider circuit breaker distinguishes rate limits, budget exhaustion, moderation failures, and consecutive provider failures.Rate limits trigger backoff, budget failures pause a stage, moderation failures terminate a request, and circuit recovery proceeds through cooldown and a successful probe.
- Auditing and feedback: Statistical auditing combines stratified human-review sampling, Wilson confidence intervals for corpus error, expected calibration error, and feedback into future encoders.Low-confidence decisions and sensitive categories receive boosted audit inclusion probability.
- Coverage optimisation: Seed selection uses a monotone submodular coverage function, so greedy tranche expansion achieves a (1 − 1/e) approximation while monitoring marginal yield.The coverage matrix spans category × region cells, and widening proceeds one tranche at a time.
6 Operational Protocol and Autonomy Governance
The operational protocol moves from human-curated seeds through focused crawling, multilingual extraction, encoding and filtering, agentic evidence-based decisions, publication, and audit-driven learning. Autonomy advances from shadow to trickle to full operation under human sign-off and risk gates.
- Operational protocol: The recurring protocol begins with human seed curation guided by a category × region coverage matrix.Seeds are widened by category tranche rather than expanded indiscriminately.
- Operational protocol: Focused crawling applies a priority frontier, politeness, depth and page-budget bounds, and re-seeding thresholds.The frontier is monitored to determine when new seeds are needed.
- Operational protocol: Multilingual extraction and canonicalisation preserve source-boundness before encoding, blocking, similarity filtering, and partitioned neighbour search.Structured evidence is retained for downstream decisions.
- Operational protocol: Agentic decisions apply a cultural-scope gate, novelty tree, evidence fusion, and graded thresholds.The scope includes named Nusantara traditions and an explicit carve-out for diaspora heritage practised as Indonesian culture.
- Operational protocol: Idempotent publication enforces ownership, licensed-media retrieval, and audit-sample emission, followed by error estimation, calibration monitoring, and label write-back.The protocol closes with audit and learning rather than publication alone.
- Autonomy governance: Autonomy progresses through shadow, trickle, and full modes, with human sign-off at each promotion across cultural, corpus-safety, and legal risks.Four human roles remain decisive: curator of direction, escalation approver, quality auditor, and guardian of meaning.
7 Discussion
The discussion presents expansion and intensification as coordinated branches of one data-level novelty decision tree, while treating integrity as code-enforced architecture. It also identifies operational, technical, and governance limitations and makes cultural values governing constraints.
- Expansion and intensification: Expansion adds entries and widens coverage cells, while intensification deepens facts in existing entries through enrich and supplement branches.Both branches derive from the data-level definition of duplication and address coverage and completeness together.
- Integrity: Integrity is enforced through source-bound schemas, probabilistic thresholds, uncertainty flags, audits, remediation, soft deletion, and untouchable human contributions.The paper characterises integrity as an invariant enforced by code rather than a guideline.
- Limitations: The framework remains limited by frontier extinction, costly and brittle anti-bot protection, source-copying that violates conditional independence, regional-language canonicalisation quality, and delayed audit-label closure.These limitations respectively constrain curation, access, evidence fusion, language coverage, and learned-encoder development.
- Ethics and cultural sensitivity: Ethical and cultural safeguards include higher thresholds and multi-source requirements for sacred categories, human review, stakeholder-designated sensitivity lists, crawl politeness, and licence checking.The governing principle is that cultural values constrain the technology.
8 Conclusion
The paper presents a formal framework for autonomous AI-based harvesting that expands and deepens digital cultural-heritage documentation in PDBI. Its backbone combines an economic funnel, data-level novelty, and reliability and governance invariants while preserving human sovereignty.
- Conclusion: The framework targets expansion and intensification of cultural-heritage documentation through autonomous AI-based knowledge harvesting for PDBI.Its stated application context is the Indonesian Digital Library of Culture.
- Conclusion: Its backbone is an economic funnel, data-level novelty, and architectural reliability and governance invariants.Examples include exactly-once effects, sliding-window budgets, sacred-category thresholds, and statistical audits.
- Conclusion: The framework is intended to extend cultural-encyclopaedia reach without displacing human contributors’ sovereignty.Human roles remain curator of direction, approver, auditor, and guardian of meaning.
- Conclusion: Future work includes learned similarity encoding from audit labels, source-kinship estimation, and regional-language canonicalisation evaluation.These areas correspond to unresolved learning, evidence-fusion, and multilingual-quality challenges.
A Notation
This section presents Table 2, which contains the paper’s principal notation.
- Table 2 lists the paper’s principal notation.