Source-linked AI summary
ECHOES: An Adaptive LLM-Powered AR Glasses Docent for Evidence-Grounded Urban Heritage Interpretation
Bang Xiao, Baosheng Jin, Jinhan Niu, Xingchen Zhang, Hua Shen
TL;DR
Scripted heritage tours leave limited room for situated questions, language preferences, and requests for evidence. ECHOES is an adaptive AR-glasses docent that combines site grounding, historical sources, and session context to tailor responses. A prototype walkthrough around Shanghai’s East Nanjing Road and the Bund demonstrates site-grounded questioning, evidence-aware explanations, nearby-site recommendations, and preference-based follow-up.
Problem
Scripted heritage tours provide predetermined descriptions, leaving limited room for visitors’ situated questions, changing curiosity, language preferences, and evidence-seeking.
Source: §1 · pp. 1–2Method
Results
A prototype demonstration around East Nanjing Road and the Bund shows site-grounded questioning, source-aware responses, nearby-site recommendations, and preference-based follow-up.
Source: §4 · pp. 3–4, §5 · p. 4Takeaways & Limitations
The demonstrated prototype supports question-responsive, source-aware urban heritage conversations instead of fixed-script narration.
Source: §4 · p. 4, §5 · p. 4The current prototype covers a curated set of Shanghai buildings; long-term preference memory, source-cue and uncertainty presentation, and real-route deployment remain future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Urban heritage tours are often delivered through fixed scripts, leaving limited room for visitors' situated questions, language preferences, or requests for evidence. We present ECHOES, an adaptive LLM-powered AR glasses docent for evidence-grounded interpretation of historical buildings. Specifically, the agent adapts along three dimensions: it modifies response structure according to the type of visitor question (identification, architectural detail, evidence-seeking, or route inquiry); it accumulates session-level interest signals from repeated questions and expressed curiosity to shape later explanations; and it tracks language preference and retrieval context across turns to maintain conversational continuity. The core of ECHOES is an LLM agent initialized as a source-aware urban heritage docent that realizes this adaptation through dialogue and preference context maintained across the session. Around this agent, ECHOES integrates visual site grounding, a self-built historical knowledge base supporting retrieval-augmented generation, and concise multilingual AR output designed for glanceable use. Through a working prototype centered on Shanghai's East Nanjing Road and the Bund, we demonstrate how an adaptive LLM-powered AR docent can transform historical tours from scripted narration into situated, question-responsive, and source-aware conversations.
1 Introduction
Scripted heritage tours leave little room for visitors’ situated questions, shifting curiosity, language preferences, or requests for evidence. ECHOES replaces fixed narration with an adaptive, source-aware AR docent that adjusts responses to question type and carries interests and language context across turns.
1 Introduction
- Scripted tours deliver predetermined descriptions but cannot flexibly respond to changing question types, curiosity, language preferences, or evidence-seeking behavior.Visitors may ask for building identification, architectural detail, evidence, or a nearby route.Source: §1 · p. 1
- AR heritage systems and LLM tour guides have been explored, but historical interpretation also requires grounding dialogue in specific sites and sources while adapting to question type and visitor context.An AR interface connected to an LLM alone is described as insufficient for these needs.Source: §1 · pp. 1–2
- ECHOES adapts response structure to question type, uses repeated questions and expressed curiosity to shape later explanations, and tracks language and retrieval context across turns.Its responses are intended to follow visitors’ curiosity while remaining grounded in historical sources and concise for AR display.Source: §1 · p. 2
- The paper’s contributions include a session-aware adaptive agent, a source-traceable historical knowledge base, a taxonomy mapping question types to response structures, and an end-to-end AR prototype.The prototype integrates visual site grounding, retrieval-augmented generation, and an output schema designed for glasses displays.Source: §1 · p. 2
2 ECHOES System
ECHOES combines site grounding, a historical knowledge base, and multilingual AR response generation around an adaptive agent. These components support building-specific, source-aware guidance designed for hands-free, glanceable use.
2 ECHOES System
- The system organizes site grounding, a self-built historical knowledge base, and multilingual AR response generation around its adaptive LLM agent.These components address building identification, historically grounded answers, and short guidance that does not require looking down at a phone.Source: §2 · p. 2
2.1 Site Grounding and Voice Activation
- A YOLO-based module maps the viewed building or selected image to a supported site ID, label, and confidence score, enabling questions tied to the current building.The prototype supports a curated set of historical buildings along Shanghai’s East Nanjing Road and the Bund.Source: §2.1 · p. 2
2.2 Self-Built Historical Knowledge Base
- ECHOES uses retrieval-augmented generation to connect questions with a self-built collection of historical materials and surface source cues for evidence-seeking questions.Official and architectural references support factual claims, while encyclopedic sources and selected urban narratives provide background, names, local memory, and everyday-life stories.Source: §2.2 · p. 3
2.3 Multilingual AR Response
- The AR response component maintains the selected language across turns and formats concise outputs that may include an answer, source cue, follow-up prompt, or next-site suggestion.The design targets limited glasses display space and continued attention to the surrounding site.Source: §2.3 · p. 3
3 Adaptive LLM Agent
The adaptive agent coordinates site context, questions, dialogue history, language preference, retrieved sources, and AR response generation. It maintains session context and changes response structure according to the visitor’s interaction.
3 Adaptive LLM Agent
- The agent combines the grounded site, transcribed question, session context, language preference, and retrieved sources to generate a concise response tied to the current site.Later responses can build on earlier questions and expressed interests.Source: §3 · p. 3
- The agent is initialized as an urban heritage docent instructed to answer around the current site, be concise, avoid unsupported historical claims, and use the visitor’s preferred language.This role is intended to constrain responses to situated guidance rather than broad or overly long explanations.Source: §3 · p. 3
- Session context records the current site, prior questions, language, retrieved evidence, and expressed interests, allowing preferences and evidence-seeking behavior to inform later turns.The context also supports continuity when a conversation moves between buildings.Source: §3 · p. 3
- Different interaction types trigger distinct response structures, from concise identification and facade interpretation to source-aware answers or nearby-site recommendations.Depending on the interaction, the agent may answer directly, ask a follow-up, show a source cue, or suggest a next place to visit.Source: §3 · p. 3
4 Demonstration
The demonstration presents an AR-glasses prototype for historical buildings along Shanghai’s East Nanjing Road and the Bund. Its walkthrough shows site-grounded questions, nearby-site suggestions, and follow-up responses shaped by visitor preferences.
4 Demonstration
- In the Wing On Company Building scenario, visitors ask identification, significance, and facade questions, and the agent retrieves historical materials for concise building-specific AR answers.The demonstration also recommends the Peace Hotel as a nearby site connected through hotel culture, diplomacy, jazz, and cosmopolitan memory.Source: §4 · pp. 3–4
- The prototype walkthrough demonstrates activation, first-person building grounding, concise site-specific responses, extension to a nearby heritage site, and preservation of explanation preferences across turns.Visitors can start the docent by tapping the glasses or using a wake word.Source: §4 · p. 3, Figure 2 · p. 4
- When visitors express a preference for connecting architecture with everyday urban life, ECHOES records it in dialogue context and applies it in later turns.The companion display mirrors the interaction, including the current site, generated answer, source-aware state, and follow-up prompt.Source: §4 · p. 4
5 Conclusion and Future Work
ECHOES coordinates site grounding, visitor questions, dialogue history, language preference, retrieved evidence, and AR-suitable responses to support situated, source-aware heritage tours. The prototype is currently limited to curated historical buildings around East Nanjing Road and the Bund, with broader corpus development and real-route deployment planned.
5 Conclusion and Future Work
- ECHOES coordinates site grounding, user questions, dialogue history, language preference, retrieved evidence, and AR-suitable response generation.It supports open-ended voice questioning, session-level preference learning, multilingual responses, evidence checking, and route recommendations compared with fixed-script tours.Source: §5 · p. 4
- The current prototype focuses on curated Shanghai historical buildings around East Nanjing Road and the Bund.Future work plans to expand the corpus and deploy the system on real urban heritage routes.Source: §5 · p. 4
- Future work will examine how adaptive LLM docents influence visitor agency, trust calibration, attention to the built environment, and historical understanding.The authors also plan to refine the presentation of source cues and uncertainty on AR glasses and improve preference memory beyond a single session.Source: §5 · p. 4