Source-linked AI summary
Lighthouse: On Building a Trustworthy Digital Assistant for Exploring History with Civil Rights Archives
John A. Aydin, Angelica Williams, Steven Dindl, Nathan Saunders, Kate Boyd, Vandana Srivastava, Biplav Srivastava
TL;DR
Large historical archives make effective exploration difficult, and chatbot answers raise concerns about accuracy and trust. Lighthouse combines generative and symbolic approaches for Civil Rights archival questions; the authors conclude that trustworthy historical use requires source grounding and expert validation.
Problem
Large twentieth-century archives are difficult to search, while chatbot responses can be inaccurate and historical collections may contain conflicting perspectives.
Source: §1 · pp. 1–2Method
Lighthouse combines generative capabilities with symbolic mechanisms that prioritize validated answers, cite sources, and avoid unsupported responses.
Source: §6.2 · p. 10Results
The results suggest that LLMs can support more flexible archival access, while trustworthy use in historical domains requires hybrid mechanisms for source grounding and expert validation.
Source: §7 · p. 11Takeaways & Limitations
In the John Henry McCray collection case study, Lighthouse is an early step toward conversational systems that may broaden access to complex archival collections.
Source: §7 · p. 11Rule-based and hybrid approaches require expert-answered questions and answer questions already addressed, but do not enable exploratory information retrieval.
Source: §6.3 · p. 11
Abstract
from arXiv · showhide
Artificial intelligence (AI), as a decision-support aid, provides new tools for studying history effectively using multimodal data, including text, images, and audio. We present, to our knowledge, the first digital assistant named Lighthouse--commonly called a chatbot--for exploring the Civil Rights period of the United States using data from South Carolina. From an AI perspective, the system's novelty lies in its guiding design principles of (a) focusing on trustworthiness of content and (b) by modeling and reducing potential harm to users from generated answers. It implements these with a neuro-symbolic approach that combines neural learning through a large language model (LLM) with rule-based symbolic solving, balancing risk of answers with generality of coverage. We evaluate Lighthouse with both factual objective questions that admit a single acceptable answer as well as subjective questions. For the latter, a dataset of subjective question-answer pairs (QAs) is created by discovering a "gold standard" by aggregating answers from independent experts. Lighthouse has access to over 18k primary sources from a range of collections and can be easily extended.
1 Introduction
Large Civil Rights archives make keyword- and Boolean-based searching challenging, while natural-language assistants could make exploration more flexible. Lighthouse explores this approach while addressing the risks of inaccurate chatbot answers through a trustworthy system and comparisons of LLM-based, rule-based, and hybrid methods.
1 Introduction
- Natural-language assistants could let users ask nuanced questions and refine or broaden searches without restarting with new keywords.The introduction contrasts asking about how South Carolinians fought for voting rights with searching keywords and reviewing a large result set.Source: §1 · p. 1
- Confident but inaccurate chatbot responses and conflicting historical perspectives make accuracy, unbiased treatment, and user assurances central concerns.LLMs can hallucinate, while archives may contain conflicting perspectives requiring careful consideration and contextualization.Source: §1 · pp. 1–2
- Lighthouse explores improving digital-archive search through a trustworthy assistant for Civil Rights collections developed with historians, librarians, and computer scientists.The project allows objective and subjective questions and backend changes to investigate learning-based and rule-based responses.Source: §1 · p. 2
- The paper presents Lighthouse and compares LLM-based, rule-based, and hybrid neuro-symbolic approaches for trustworthy archival question answering.It also discusses issues raised by deploying AI in historical archives.Source: §1 · p. 2
2 Background & Related Work
The background frames Lighthouse against the flexibility of conversational assistants, the complexity of South Carolina Civil Rights history, and questions about trust and historical perspectives. It also describes the digitized collections and the challenges of defining acceptable answers in historical work.
2.1 Conversational Assistants
- Rule-based assistants constrain outputs to pre-authored responses or actions, while LLM chatbots generate flexible responses dynamically.Rule-based systems can behave predictably within their designed scope but are limited on novel questions; LLMs support open-ended requests such as explanation and summarization.Source: §2.1 · p. 2
2.2 South Carolina Civil Rights History
- Traditional Civil Rights narratives often center legal milestones, Martin Luther King Jr., and limited locations, leaving South Carolina stories, women, and local leaders less visible.The South Carolina account includes major legal challenges, the Orangeburg Massacre, and leaders such as Modjeska Simkins and Septima Clark.Source: §2.2 · p. 3
2.2.1 Digital Collections Corpus Background
- The digital collections comprise about 51,000 pages across nine collections, with this paper focusing on the John Henry McCray Papers.The collection was selected for its detailed metadata and reliable transcriptions, among other properties.Source: §2.2.1 · p. 4
2.2.2 Ground Truth and Trust
- Historical answers may need to synthesize multiple perspectives because historians do not treat ground truth as a settled concept.Professional historians may distrust an impersonal chatbot and still examine original documents to meet professional standards.Source: §2.2.2 · pp. 4–5
- Conversational convenience may encourage users to accept a single chatbot result, and automation bias may affect trust in system outputs.The authors identify detailed experiments and full-scale deployment as ways to investigate chatbot value for exploring history.Source: §2.2.2 · p. 5
2.3 Trustworthy AI
- LLM chatbot trust is complicated by users’ trust heuristics and system limitations, including limited developer control and sycophancy.Prior work cited in the passage reports that citations can increase trust even when randomly generated.Source: §2.3 · p. 5
- Lighthouse follows the single-result trend, while supporting multiple nuanced perspectives remains a future-work direction.The passage connects that direction to historians’ dialectical expectations.Source: §2.3 · p. 5
3 Problem
The problem is to provide flexible, context-aware access to archival information while keeping answers anchored in validated knowledge. Lighthouse balances expert-validated answers with broader archival evidence, trading off trust against coverage.
3 Problem
- Keyword-based IR assumes well-specified queries, whereas conversational interfaces can provide information access within broader dialog context.The paper focuses on flexible, context-aware generative access anchored in validated knowledge through symbolic or rule-based mechanisms.Source: §3 · p. 5
- The system must balance a validated question-answer dataset, which offers expert-reviewed answers and provenance, against a broader but less structured archival corpus.The first covers questions anticipated during annotation; the second may contain evidence for unanticipated queries.Source: §3 · p. 6
- A user utterance may be answered from the validated dataset when it matches a represented question or scenario, or from retrieved archival evidence otherwise.This is the proposed strategy for choosing between the two knowledge sources.Source: §3 · p. 6
- A system relying only on expert-validated answers has limited coverage, while one relying only on the archival corpus lacks the same reliability guarantees.The design goal is to balance the two sources.Source: §3 · p. 6
4 Approach
Lighthouse’s approach combines curated archival text and expert-created answers with learning-based, rule-based, and hybrid dialog solvers. Its evaluation gold standard is built through expert responses, aggregation, and review.
4.1 Dataset Selection
- The chatbot is trained on the John Henry McCray collection, selected from nine Civil Rights-era collections for its size, historical significance, and links to other collections.The cleaned corpus contains approximately 5.68 million transcript tokens and 6.46 million tokens including metadata.Source: §4.1 · p. 6, Table 1 · p. 6
4.2 Data Processing and Cleaning
- Cleaning filters unusable or duplicate transcripts, repairs recurring OCR errors, and normalizes text while retaining documents likely to contain usable information.The process uses a vocabulary derived from document titles and descriptions and a Spello correction corpus; the project retains existing OCR transcripts rather than relying on MLLM-generated transcripts, which were less reliable on proper nouns in pilot tests.Source: §4.2 · pp. 6–7
4.3 Creating a Gold Standard for Evaluation
- The gold standard uses ten-question scenarios with five objective and five subjective questions, followed by three independent responses, expert aggregation, and separate evaluation.The evaluator assesses accuracy, comprehensiveness, and readability; questions are designed to be answerable at least in part from the dataset, but expert source choices may leave gold answers misaligned with the documents.Source: §4.3 · p. 7, Figure 3 · p. 7
4.4 AI Approach for Dialog Management
- Lighthouse’s learning-based, rule-based, or hybrid dialog manager can be selected within the SafeGenChat architecture.The dialog manager, also called a solver, determines how to respond to a user utterance.Source: §4.4 · p. 8
4.4.1 LLM-based
- The LLM-based method M1-1 fine-tunes Llama-3.1-8B on the McCray collection and then on 2,000 Google Natural Questions pairs.Both stages use LoRA; models are evaluated with greedy decoding.Source: §4.4.1 · pp. 8–9
4.4.2 Rule-based
- The rule-based variant M2 uses SafeChat, a Rasa-based framework for source-grounded chatbots, trained on McCray scenario questions.Source: §4.4.2 · p. 9
4.4.3 Hybrid
- The hybrid variant M3 pairs Llama-3.1-8B-Instruct as System 1 with SafeChat as System 2 and uses a keyword-based civil-rights-history risk model.Source: §4.4.3 · p. 9
- Lighthouse’s hybrid dialog manager routes each query to an LLM or a rule-based solver according to perceived risk and confidence in the LLM response.Low-risk queries with sufficiently reliable responses stay with System 1; higher-risk or uncertain queries fall back to System 2.Source: §4.4 · p. 8, §4.4.3 · p. 9
5 Evaluation
The evaluation compares Lighthouse models on objective and subjective questions using textual similarity metrics. Rule-based and hybrid systems perform strongly in both settings, while LLM rankings differ by question type and response style.
Evaluation design
- The evaluation asks how well the models answer objective and subjective questions, using six normalized textual-comparison metrics.The metrics are Jaccard similarity, ROUGE-L, Jensen-Shannon Divergence, BLEU, Levenshtein distance, and BERTScore.Source: §5 · p. 9
- AE1 tests 50 objective questions across McCray collection scenarios, while AE2 tests 10 subjective questions from the McCray and NAACP scenarios.AE2 compares outputs with short-form gold-standard answers to reduce noise from longer references given the small sample size.Source: §5 · p. 9
Objective questions
- M2 performs best on objective questions, followed by M3; M1-1 outperforms the M1-2 and M1-3 LLM baselines.The authors attribute M1-2’s low scores in part to its multi-sentence outputs compared with the often few-word references.Source: §5 · p. 9
Subjective questions
- On subjective questions, M2 and M3 perform highly, while M1-2 outperforms M1-1 and M1-3.The authors note that stylistic differences in long-form answers can influence textual-comparison evaluation more than content.Source: §5 · p. 9
6 Discussion
The discussion positions Lighthouse as a supplementary route into archival materials, describes its hybrid design and evaluation framework, and notes limits in the approaches explored and questions answerable.
6.1 Societal Significance
- Instructors can use Lighthouse alongside primary sources and scholarly interpretations, while the chatbot may help students and general audiences find archival materials.The discussion rejects using chatbots as the sole or primary interpreter of primary sources.Source: §6.1 · p. 10
6.2 Technical Significance
- Lighthouse balances generative flexibility with symbolic mechanisms that prioritize validated answers, cite sources, and avoid unsupported responses.The discussion contrasts generative flexibility with the accuracy, provenance, and consistency guarantees limited in purely generative systems, and the advance planning required by purely rule-based systems.Source: §6.2 · p. 10
- Its evaluation separates objective factual questions from subjective questions requiring expert judgment, aggregation, and review.Source: §6.2 · p. 10
6.3 Limitations & Future Work
- The study is a narrow exploration and does not cover all chatbot or information-retrieval approaches.Domain-specific foundational models, specialized RAG retrieval models, and agentic workflows were not explored and may yield better results.Source: §6.3 · pp. 10–11
- Rule-based and hybrid approaches require expert-answered questions and do not enable exploratory information retrieval.Source: §6.3 · p. 11
7 Conclusion
The paper presents Lighthouse as a trustworthy conversational assistant for Civil Rights archives and compares generative, rule-based, and hybrid approaches. It concludes that trustworthy historical use calls for source grounding and expert validation, with conversational access supplementing search-based access.
7 Conclusion
- Lighthouse explores Civil Rights archival collections with attention to trustworthiness, provenance, and reducing user harm.Source: §7 · p. 11
- Using the John Henry McCray collection as an initial case study, the paper compares three approaches and develops expert-informed evaluation for objective and subjective historical questions.Source: §7 · p. 11
- The results suggest LLMs may support more flexible archival access that supplements search, while trustworthy historical use requires source grounding and expert validation.Lighthouse is described as an early step toward broader access that preserves evidence, context, and human expertise.Source: §7 · p. 11