PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting LLM-SC to Living Sacred Heritage: Domain-Specific Extraction, Semantic Networks and Human Evaluation on Pashupatinath Temple Reviews

Hari Sitaula, Manish Sakhakarmy, Pujan Pant, Sagar Newpane

arXiv:2610.35685v1cs.CLcs.AI

TL;DR

Generic review schemes may miss culturally specific meanings and mixed experiences at living sacred heritage sites. The study adapts LLM-SC for Pashupatinath with a domain-specific extraction scheme and evaluates components against human annotations. The reported scores differ by component, while relation validation is limited to selected candidates.

  • Problem

    Generic review-analysis schemes may fail to represent sacred-heritage concepts and mixed experiences, while evidence for integrated domain-adaptive LLM-SC in this setting is limited.

    Source: §1 · p. 2, §2.6 · p. 7

  • Method

    The study adapts LLM-SC for Pashupatinath with a 12-type entity taxonomy, four relation types, and nine aspect-level sentiment dimensions, then evaluates outputs against human-adjudicated references.

    Source: §1 · pp. 2–3

  • Results

    28.4% provisional entity micro-F1 (95% CI 25.9–31.0%), 53.6% aspect-presence F1, and 42.6% end-to-end aspect-sentiment F1; 153 of 194 selected relation candidates were supported (78.9%).

    Source: §6 · p. 29

  • Takeaways & Limitations

    The component-wise evaluation shows that entity matching, aspect detection, polarity classification, and candidate-level relation validation reflect different sources of error.

    Source: §6 · p. 29

    Relation support was assessed only for first-listed candidates, so it does not estimate full-output precision, relation recall, or F1.

    Source: §5.5 · p. 28

Abstract

from arXiv · show

Large language model (LLM) pipelines can turn visitor reviews into structured data, but generic schemes may miss rituals, sacred spaces, cultural restrictions and emotionally mixed experiences at living sacred heritage sites. We adapted the LLM-Social Computing (LLM-SC) framework to Pashupatinath Temple, Nepal, by specifying a 12-type entity taxonomy (four original LLM-SC types plus eight sacred-heritage types), four relation types and nine aspect-sentiment dimensions in a GPT-4o extraction prompt, which was applied to a fixed random sample of 2,500 English-language Tripadvisor reviews from a processed corpus of 3,939. The exports contain 11,286 entity, 6,232 aspect-sentiment and 5,590 relation records, covering 2,485, 2,487 and 2,433 reviews, respectively; the eight added sacred-heritage types account for 64.8% of entity records. Against an adjudicated two-annotator reference for 200 reviews, exact normalized surface-and-type matching gave provisional entity scores of 41.0% precision, 21.8% recall and 28.4% micro-F1. Aspect-presence F1 was 53.6% and end-to-end aspect-plus-polarity F1 was 42.6%. Human adjudication supported 153 of 194 first-listed relation candidates (one per review; 78.9%), a candidate-level rate that does not estimate relation recall or whole-graph precision. The contribution is a culturally grounded, domain-specific adaptation with component-wise human evaluation and explicit reproducibility and generalization limitations.

1. Introduction

Pashupatinath reviews combine sacred, cultural, environmental, and operational experiences that generic review categories and document-level sentiment may not preserve. This study adapts LLM-SC to represent those meanings and evaluates the resulting GPT-4o outputs against human-adjudicated annotations.

1. Introduction

  • Pashupatinath reviews can combine rituals, sacred spaces, cultural restrictions, spiritual emotions, and operational concerns, requiring representation of both concepts and their cultural and experiential meaning.A single encounter may carry both positive and negative emotions, which document-level polarity and generic aspects cannot preserve.Source: §1 · p. 2
  • The motivation is to adapt LLM-SC because its original configuration did not sufficiently represent culturally specific entities and experiential dimensions in Pashupatinath reviews.The adaptation preserves the framework's core structure while extending its semantic representation for living sacred heritage.Source: §1 · p. 2
  • The study adapts LLM-SC with a 12-type entity taxonomy and an explicitly guided prompt specifying four relation types and nine aspect-level sentiment dimensions.The structured outputs are represented as a Social Computing network of typed relations.Source: §1 · pp. 2–3
  • GPT-4o outputs from a fixed sample of 2,500 reviews are evaluated against human-adjudicated annotations for a 200-review subset, with component-appropriate measures.The study separates evaluation of entity extraction, aspect-level sentiment, and selected relation candidates; it does not quantitatively rank alternative models or test live retrieval and system performance.Source: §1 · p. 3

2. Related Work

Prior work establishes the value of visitor reviews, aspect-level sentiment, LLM-based extraction, and semantic networks, while leaving gaps in domain-specific representation and integrated evaluation for living sacred heritage.

2.1. User-Generated Content and Online Reviews as Data Sources for Tourism Research

  • Visitor-generated reviews capture experiences in visitors’ own words, but their multiple dimensions require structured extraction beyond simply collecting large volumes of text.Prior examples include tourism priorities in Fujian and layered attraction, service, and emotional relationships in Forbidden City reviews.Source: §2.1 · pp. 3–4

2.2. Sentiment Analysis and Aspect-Level Opinion Mining in Tourism

  • Aspect-level sentiment links opinions to experience dimensions, while living sacred heritage requires domain-specific dimensions for mixed spiritual and operational evaluations.Reviews may express spiritual satisfaction alongside dissatisfaction with access or crowding, or both positive and negative emotions about one sacred experience.Source: §2.2 · pp. 4–5

2.3. Named Entity Recognition in Cultural and Heritage Domains

  • In living sacred heritage, entity representation must include religious practices, actors, places, objects, restrictions, and complex experiences; the reviewed literature offered limited evidence for Himalayan or South Asian settings.Entity choices shape which concepts remain available to downstream sentiment and relation analysis.Source: §2.3 · p. 5

2.4. Large Language Models and Domain Adaptation in Tourism Text Analysis

  • LLMs have supported tourism classification and extraction, but prompt-based domain adaptation is relevant where no directly comparable public annotated resource exists for sacred-heritage categories.The reviewed literature also identifies sensitivity to prompt design, computational cost, and multilingual or metaphorical language.Source: §2.4 · pp. 5–6

2.5. Social Computing and Semantic Network Analysis in Tourism Research

  • Semantic networks represent relationships among concepts in visitor narratives, but their usefulness depends on domain-specific entity types and explicitly classified relations.Prior work combined LLM extraction and Social Computing for heterogeneous networks of attractions, services, concerns, and emotional responses.Source: §2.5 · p. 6

2.6. Research Gaps and Positioning of the Present Study

  • Existing research offers limited evidence of integrated domain-adaptive LLM-SC that jointly specifies sacred-heritage entities, domain-specific sentiment dimensions, and typed relationships.The study also uses a fixed review snapshot rather than testing live retrieval or hourly monitoring, and does not establish transfer to other sites.Source: §2.6 · p. 7

3. Materials and Methods

The methods section organizes the workflow from corpus preparation and domain-adaptive extraction through human-reference evaluation and semantic-network construction.

3. Materials and Methods

  • Figure 1 summarizes the study workflow, spanning corpus preparation, taxonomy and prompt design, extraction, human-reference evaluation, and semantic-network construction.These stages are described in Sections 3.1–3.5.Source: §3 · p. 7
  • GPT-4o served as the extraction model, while other generative AI tools assisted with selected manuscript passages, plotting code, and evaluation-statistics analysis code.The stated assistance concerns manuscript preparation and analysis tools, not the extraction model.Source: §3 · pp. 7–8
  • The methods are divided among corpus preparation, domain-adaptive taxonomy and prompt design, extraction, human-reference evaluation, and semantic-network construction.Sections 3.1–3.3 cover the corpus and extraction pipeline; Sections 3.4 and 3.5 cover evaluation and network construction, respectively.Source: §3 · p. 7

3.1. Data Collection, Preprocessing, and Sampling

The study uses a fixed random sample from a processed Tripadvisor corpus and adapts LLM-SC through a domain-specific taxonomy and extraction prompt. Its scope includes explicit data-rights and post hoc duplicate-handling qualifications.

3.1. Data Collection, Preprocessing, and Sampling

  • Reviews came through a third-party API provider, and the authors lacked documentation that the provider was authorized to supply them or permission for redistribution.Review texts are not shared, and the study makes no claim of platform authorization or redistribution rights.Source: §3.1 · p. 9
  • The fixed analysis sample contains 2,500 English-language reviews randomly sampled without replacement from the 3,939-row processed corpus.The sample file is the authoritative input, and the same review text and metadata were used throughout extraction.Source: §3.1 · pp. 9–10
  • A post hoc duplicate sensitivity analysis filtered one sampled review without replacement or rerunning GPT-4o, so it is not a new primary experiment.The filtered sample contains 2,499 reviews; duplicate handling has a small effect on most aggregate counts but can alter connectivity measures.Source: §3.1 · p. 10

3.2. Domain-Adaptive Taxonomy and Prompt Design

  • Taxonomy development used reviews to identify and refine concepts, complemented by domain knowledge, and the final prompt was fixed before the 2,500-review extraction.Pilot inspection clarified category boundaries, particularly between reviewer-felt spiritual emotion and mixed-valence experience.Source: §3.2 · pp. 10–11
  • The taxonomy retains four original LLM-SC entity types and adds eight sacred-heritage types, including rituals, religious actors, sacred spaces, festivals, cultural rules, and sacred objects.The added categories also distinguish reviewer-attributed spiritual emotion from explicitly mixed positive and negative valence.Source: §3.2 · pp. 10–11
  • The prompt defines 12 entity types, four relation types, and nine aspect-level sentiment dimensions, with rules for contextual terms and explicit spiritual-emotion and dual-valence assignments.Dual valence requires positive and negative responses to the same entity or experience, not separate positive and negative reactions to different visit aspects.Source: §3.2 · pp. 11–12

3.3.1. Model Screening and Extraction Configuration

The authors screened candidate extraction approaches qualitatively and selected GPT-4o for the full run. They held the prompt and request settings constant across batches.

3.3.1. Model Screening and Extraction Configuration

  • GPT-4o was selected after qualitative screening focused on domain-specific entities, spiritual emotions, and mixed-valence experiences.The other screened approaches included Llama-family models, GLiNER with SetFitABSA, and GPT-4-family models; the screening did not support a quantitative ranking.Source: §3.3.1 · p. 12
  • Each review was submitted through the OpenAI Batch API with the same prompt and request settings across batches.Requests used the gpt-4o alias, temperature 0.1, a 900-token maximum, and 45 requests per batch.Source: §3.3.1 · p. 12
  • The dated model snapshot behind the gpt-4o alias was not recorded.Requests were submitted in August 2026.Source: §3.3.1 · p. 12

3.3.2. Structured Output Schema

The extraction prompt specified a structured schema, and the study documents how outputs were processed, evaluated against human annotations, and represented as a network. The evaluation uses component-specific procedures, while the entity scores are reported as provisional because the evaluated snapshot cannot be reproduced by the available post-processing code.

3.3.2. Structured Output Schema

  • The prompt required five output components, including typed entities with short supporting quotations, relations, aspect-level sentiment, review type, and a dominant-experience phrase.It defined 12 entity types, four relation types, and nine sentiment aspects, with polarity labels and anchored scores.Source: §3.3.2 · pp. 12–13

3.3.3. Batch Processing, Parsing, and Export Provenance

  • The exports contain 11,286 entity, 6,232 aspect-sentiment, and 5,590 relation records, with records covering 2,485, 2,487, and 2,433 reviews, respectively.Unparseable responses were excluded, out-of-vocabulary labels were retained in aspect-sentiment and relation exports, and no manual semantic correction was performed before scoring.Source: §3.3.3 · p. 13
  • Entity scores are provisional because the evaluated export is a redacted snapshot that the available post-processing code cannot produce.A third party’s personal name was replaced by a redaction marker.Source: §3.3.3 · p. 13

3.4. Human Evaluation Design

  • Two authors independently annotated 200 reviews, then adjudicated disagreements; entity and aspect references were constructed without inspecting GPT-4o predictions.Entity matching used normalized text and type, while aspect evaluation separated presence from end-to-end polarity agreement; polarity accuracy was computed on co-present aspect pairs.Source: §3.4 · pp. 13–15
  • Relation evaluation checked the first GPT-4o relation candidate in each of 194 reviews with output, supporting candidate-level rates but not full-output precision, recall, or F1.Reviews without relation output were treated as coverage failures; confidence intervals used review-level percentile bootstrap resampling.Source: §3.4 · pp. 14–15

3.5. Semantic Network Construction and Analysis

  • Entity and relation records were transformed into graphs, but the resulting network represents extracted relations rather than verifying their correctness.Records were normalized and aggregated before graph construction and measurement.Source: §3.5 · p. 15

3.5.1. Entity Normalization and Node Reconciliation

Entity names and relation endpoints were normalized to form node identifiers, while variants and type ambiguities were preserved. The full entity inventory remains distinct from the relation-network topology.

3.5.1. Entity Normalization and Node Reconciliation

  • Trimmed, case-folded names with collapsed internal whitespace served as node identifiers, without lemmatization or synonym merging.Inflectional and lexical variants therefore remain separate nodes.Source: §3.5.1 · p. 15
  • Observed name variants and assigned types were retained as node attributes, and names with multiple assigned types were flagged as conflicts.The ambiguity was preserved for later review rather than resolved to a single type.Source: §3.5.1 · pp. 15–16
  • The full entity inventory includes relation-only nodes and entities absent from the relation topology.Relation endpoints missing from the entity table were added with an explicit unknown type.Source: §3.5.1 · p. 16

3.5.2. Typed Relation Aggregation and Direction

Relation records were grouped using normalized endpoints, relation label, and schema status. Aggregation accounts for symmetric relation types while retaining direction for causal and description relations.

3.5.2. Typed Relation Aggregation and Direction

  • Relation groups are keyed by normalized source, target, label, and schema status, with counts of records and distinct reviews plus labels and examples.Source: §3.5.2 · p. 16
  • Endpoint order was canonicalized for co_occurrence and contrast, while causal and description relations retained source-to-target direction.Source: §3.5.2 · p. 16
  • Out-of-schema labels were retained and marked, and self-loop groups remained available for inspection but were excluded from topology measures.Source: §3.5.2 · p. 16

3.5.3. Graph Representations and Network Measures

The analysis uses three graph views to preserve different relation structures and summarize the extracted network. It applies unweighted topology measures while treating them as properties of the extracted representation, not real-world causes or sentiment measures.

3.5.3. Graph Representations and Network Measures

  • Three graph views preserve typed relation groups, summarize unique endpoint pairs in an undirected projection, and isolate causal and description relations in a directional subgraph.The projection combines relation labels and excludes self-loops; the directional subgraph combines parallel groups on the same ordered pair for topology measures.Source: §3.5.3 · p. 16
  • Local clustering assigns zero to nodes with degree below two, while transitivity is the global ratio of triangles to connected triples.The average clustering coefficient includes every node.Source: §3.5.3 · p. 17
  • The undirected projection is summarized by average degree, density, components, isolated nodes, largest-component size and share, and within-component path length; the directional subgraph uses connectivity and degree rankings.The directional measures include weakly and strongly connected components and in-degree and out-degree rankings.Source: §3.5.3 · pp. 16–17
  • All measures use unweighted graphs, and the sensitivity comparison repeats the undirected analysis after excluding out-of-schema relation groups; the measures describe extracted structure, not real-world causal mechanisms.Highly connected nodes are structural hubs in the extracted network, not independently validated causes or measures of visitor sentiment.Source: §3.5.3 · p. 17

4. Results

Across the sampled reviews, the exports covered most reviews and added sacred-heritage categories accounted for 64.8% of entity records. Human-reference evaluation showed different performance across entity, aspect-sentiment and relation components.

4.1. Corpus-Level Extraction Coverage

  • Component coverage was 99.4% for entities, 99.5% for aspect sentiment and 97.3% for relations; added sacred-heritage types accounted for 64.8% of entity records.The exports contained 11,286 entity, 6,232 aspect-sentiment and 5,590 relation records; coverage indicates output presence, not correctness.Source: §4.1 · pp. 17–18, Table 1 · p. 18, Table 2 · p. 18

4.2. Human-Reference Evaluation Sample

  • The 200-review reference comprises 1,714 entity mentions, 1,800 review–aspect decisions and 194 relation candidates; annotator agreement was lowest for relation existence (κ = 0.38).Six reviews had no GPT-4o relation candidate, and relation annotation was candidate-level rather than exhaustive.Source: §4.2 · pp. 18–19

4.3. Entity Extraction Against the Human Reference

  • Under exact normalized surface-and-type matching, entity extraction scored 41.0% micro-precision, 21.8% micro-recall and 28.4% micro-F1 (provisional).The scores describe agreement with the adjudicated reference under a strict matching rule for the evaluated sample.Source: §4.3 · pp. 19–20, Table 3 · p. 20

4.4. Aspect-Level Sentiment Evaluation

  • Aspect presence reached 53.6% F1, while end-to-end aspect-plus-polarity F1 was 42.6%, which also penalizes missed and false-positive aspects.Conditional polarity agreement excludes disagreements about aspect presence; end-to-end scoring requires both the correct aspect and polarity.Source: §4.4 · p. 20, Table 4 · p. 21

4.5. Semantic-Relation Validation

  • Of 194 selected relation candidates, 78.9% were supported and 54.6% were both supported and type-matched; these are not full-output precision estimates.Only one candidate per review was adjudicated, although the evaluation reviews contained 450 relation records.Source: §4.5 · p. 21, Table 5 · p. 22

4.6. Semantic-Network Structure

  • The graph contains 5,926 inventory nodes, 5,110 relation-network nodes and 5,341 typed edge groups; its largest component has 3,269 nodes (63.97%).The unweighted undirected projection has 5,221 connections among 5,110 nodes.Source: §4.6 · p. 22, Table 6 · p. 23

4.7. Post Hoc Duplicate-Handling Sensitivity Analysis

  • Duplicate sensitivity affected connectivity more than most aggregate counts: components rose from 800 to 801 and the largest component lost three nodes, while clustering was unchanged.The four removed review IDs were absent from the human-reference workbooks, so the filtering did not affect the 200-review evaluation sample.Source: Table 8 · p. 25, §4.7 · pp. 25–26

5. Discussion

The discussion presents the sacred-heritage taxonomy as a domain-representation choice, then distinguishes component-specific evaluation results from output volume and aggregate scores. It frames network and monitoring uses cautiously and identifies limits on generalization, evaluation, reproducibility, and comparison.

5.1. Domain Adaptation and Semantic Representation

  • The eight added sacred-heritage categories account for 64.8% of entity records, but their use does not establish accuracy or cultural validity.The taxonomy is a domain-representation design whose performance must be evaluated against human references, including missed, unsupported, or mistyped mentions.Source: §5.1 · p. 26

5.2. Separating Detection, Typing, and Polarity

  • Exact surface-and-type matching penalizes lexical differences and taxonomy disagreement, while output limits recall to at most 53.1% against the reference mentions.The two annotators agreed on only 524 normalized span-and-type mentions (pairwise F1 = 38.9%); conditional polarity accuracy (79.4%) is not overall sentiment performance.Source: §5.2 · p. 26

5.3. Interpreting Semantic Relations and Network Structure

  • 153 of 194 selected relations (78.9%) were human-supported, but this candidate-level rate does not estimate full-output precision, recall, or F1.The graph distinguishes node inventories, typed groups, and an undirected projection; whitespace-and-case normalization leaves variants such as “cremation” and “cremations” as separate nodes.Source: §5.3 · p. 27

5.4. Implications for Heritage-Review Monitoring

  • Review analysis could help triage recurring concerns and identify narratives for human attention, but the study does not demonstrate live-monitoring reliability or operational impact.Uncertain or high-impact interpretations should remain inspectable and be reviewed by people familiar with the site and its cultural context.Source: §5.4 · p. 27

5.5. Limitations

  • The findings are bounded by English-language Tripadvisor reviews from one site and a human reference drawn from 200 of 2,500 sampled reviews.The evaluation subset may overlap with reviews used for taxonomy development and prompt refinement, and that overlap was not tracked.Source: §5.5 · pp. 27–28
  • Entity scores are provisional because available post-processing code cannot reproduce the evaluated entity file from raw model responses.The model snapshot was not recorded, raw responses and evaluation files are not available as a frozen release, and no matched quantitative comparison with other models or the original LLM-SC configuration was conducted.Source: §5.5 · p. 28

5.6. Future Work

  • Future comparisons should use matched reviews, schemas, normalization, and scoring units, with replicated annotation and relation evaluation beyond first-listed candidates.The authors also call for evaluation at related sacred-heritage sites before broader claims are made.Source: §5.5 · p. 28, §5.6 · p. 29

6. Conclusions

The study adapts LLM-SC for living sacred heritage and evaluates its components against human annotations. The results include low, provisional entity scores and distinct aspect and relation outcomes; the authors frame the outputs as human-inspectable decision support, not autonomous management evidence.

6. Conclusions

  • Entity extraction achieved provisional 28.4% micro-F1, while aspect-presence and end-to-end aspect-sentiment F1 were 53.6% and 42.6%.The entity result uses exact normalized surface-and-type matching; relation adjudication supported 153 of 194 selected candidates (78.9%), which does not estimate full-output precision, recall, or F1.Source: §6 · p. 29
  • The workflow integrates a sacred-heritage entity taxonomy, nine aspect-level sentiment dimensions, and a typed relation schema for living sacred heritage reviews.The network analysis distinguishes records, typed relation groups, node inventories, and topology connections under explicit definitions.Source: §6 · p. 29
  • The outputs may serve as traceable decision support for human inspection, but the study does not establish cultural completeness or reliability for autonomous management decisions.The authors identify incomplete entity-export lineage, unavailable raw responses, an unrecorded model snapshot, the 200-review reference sample, and no matched benchmark or live-system evaluation as boundaries.Source: §6 · pp. 29–30
  • Derived records, taxonomy, prompt, and analysis code are available, but the review texts and supporting quotations are not redistributed and full end-to-end regeneration is unavailable.The authors have not established reuse or redistribution rights for the Tripadvisor review texts obtained through a third-party API.Source: §6 · p. 30

Appendix A. Extraction Prompt

The appendix prompt instructs GPT-4o to extract structured entities, relations, and aspect sentiments from Pashupatinath reviews. Its schema defines domain-specific labels and decision rules, then requires JSON output with evidence and review-level fields.

Prompt setup

  • The prompt frames the input as a tourist review of Pashupatinath Temple and requests entities, relationships, and aspect-level sentiments in strict JSON.The cleaned review text is inserted after the prompt’s final line.Source: Appendix A · p. 31

Entity types and rules

  • The entity schema retains four generic types and adds sacred-heritage labels for rituals, religious actors, sacred spaces, spiritual emotions, festivals, cultural rules, sacred objects, and dual-valence experiences.It also specifies that spiritual emotions must be reviewer-felt and dual valence must express positive and negative feelings about the same thing.Source: Appendix A · p. 31

Relations and sentiment aspects

  • The prompt specifies four relation types—co-occurrence, causal, description, and contrast—and nine sentiment aspects, including ritual experience and cultural sensitivity.The aspect labels also include service, facility, environment, spiritual authenticity, access fairness, sacred atmosphere, and crowd management.Source: Appendix A · p. 31

Extraction rules

  • The rules prohibit extracting the review subject or generic nouns, distinguish entity types from sentiment aspects, and constrain sentiment labels and scores.Polarity must be positive, neutral, or negative; scores use a 0.0–1.0 range with stated restrictions on 0.5 and 0.0.Source: Appendix A · pp. 31–32

Output format

  • The required JSON output contains entity names, types, and quotes; relation endpoints, types, and descriptions; aspect sentiments with scores and evidence; and review-level fields.The review-level fields are review_type and dominant_experience.Source: Appendix A · p. 32
Loading 2610.35685v1…