Source-linked AI summary
hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advising
Yoonseo Kim, Seongmin Lee, Joongheon Kim, SeongKu Kang
TL;DR
Profile-dependent advising questions expose a gap in profile-blind retrieval, which can return plausible but inapplicable evidence. proFILL adds profile-aware indexing and progressively acquires only the information needed for each query. It achieves the strongest reported retrieval and generation performance, while remaining effective with open-weight models and receiving high user preference on profile-dependent questions.
Problem
Academic advising questions depend on student profiles, but conventional RAG can retrieve semantically similar sources that are plausible yet inapplicable.
Method
proFILL transforms hoBIT into a profile-aware RAG system using offline profile indexing, query-driven profiling, and evidence-driven profiling with re-retrieval.
Results
proFILL achieves the strongest retrieval and generation performance, including MRR 0.593 versus 0.475 and Source Match 0.749 versus 0.412, while open-weight models remain competitive.
Takeaways & Limitations
Profile applicability can be incorporated into retrieval, and proFILL is preferred most for graduation- and credit-related questions, with 93–98% preference.
Takeaways & Limitations
The corpus and benchmark come from one college, and the profile schema must be adapted to each institution’s curricula, policies, and advising practices.
Abstract
from arXiv · showhide
In university academic advising, identical questions can require different answers depending on a student's department, admission cohort, and degree program, causing profile-blind retrievers to surface plausible but inapplicable evidence. We present proFILL, a method for transforming hoBIT, our college's current rule-based advising chatbot, into a profile-aware retrieval-augmented generation (RAG) system. Rather than requiring a complete user profile upfront, proFILL progressively acquires only the profile attributes needed for each query, guided by both the query intent and the initially retrieved evidence, and uses them to condition retrieval over a profile-aware index. Extensive experiments and a human preference study show that proFILL outperforms diverse RAG baselines, is preferred by target users, and remains effective with open-weight models for cost-effective on-premise deployment.
1 Introduction
Academic advising answers depend on student profiles, so profile-blind retrieval can select plausible but inapplicable sources. proFILL addresses this by indexing evidence with applicability information and acquiring only query-relevant profile fields on demand.
- Academic advising questions may require different answers based on department, admission cohort, major type, and related profile attributes.
- Conventional RAG can retrieve semantically similar documents that are plausible but wrong for a particular student.
- proFILL transforms hoBIT into a profile-aware RAG system by treating evidence validity as part of the index structure.
- Offline preprocessing annotates institutional-material chunks with applicable profile values and the fields needed to interpret them.
- At query time, proFILL infers intent, asks only for missing required fields, and combines the filled profile with the query to retrieve applicable evidence.
2 The hoBIT System
hoBIT extends a rule-driven advising chatbot with profile-aware indexing and on-demand profiling. Its online pipeline acquires profile information through query-driven and evidence-driven stages before filtering, re-retrieving, and presenting supporting evidence.
- hoBIT’s original backend maps user queries to predefined FAQ responses, limiting its handling of profile-dependent advising questions.
- proFILL adds offline profile-aware indexing and online on-demand profiling to hoBIT.
- The default profile schema includes department, major type, grade, admission year, and student status, and can be extended for new advising needs.
- Institutional materials are cleaned, chunked, and annotated with profile values that determine each chunk’s applicability, while stable and frequently updated sources use separate indices.
- Intent Routing: An intent router sends retrieval queries through profiling and retrieval, while greetings, ability questions, FAQs, and smalltalk use lighter predefined or conversational handling.
- Adaptive Profiling: Query-driven profiling gathers missing fields before retrieval, and evidence-driven profiling asks targeted follow-ups and re-retrieves when applicability remains unresolved.
- The system offers predefined profile options and displays selected evidence with generated answers to reduce effort and support user inspection.
3 Experimental Setup
The evaluation uses institutional advising materials and profile-grounded questions spanning diverse student profiles, categories, and query forms. It compares deployment and oracle settings using retrieval, answer-quality, matching, and groundedness metrics.
- Corpus: The corpus contains 515 institutional sources, including regulation and orientation PDFs, department webpages, board notices, and administrator-maintained FAQs.
- Profile-Grounded QA Data: The benchmark contains 1,800 QA instances spanning 60 student profiles, 10 advising categories, and three query types.
- Evaluation Settings: Deployment starts without a profile, whereas oracle supplies the complete profile to assess how effectively proFILL uses available profile information.
- Profile-Grounded QA Data: The advising categories cover major, general-education, graduation, credit, elective, and double-major requirements, among related topics.
- Metrics: Retrieval is evaluated with MRR and Recall@{1, 5, 10, 50}, while generation uses lexical, matching, and LLM-based grounded-correctness metrics.
4 Results and Discussion
proFILL achieves the strongest retrieval and generation performance against the evaluated RAG baselines, is preferred by students, and remains competitive across embedding and language-model configurations.
- Comparison with RAG Baselines: proFILL outperforms the evaluated RAG baselines in retrieval and generation, including higher deployment MRR than oracle-profile baselines.Deployment MRR is 0.593 versus 0.475 for oracle baselines.
- Comparison with RAG Baselines: proFILL achieves Source Match of 0.749 versus 0.412 and the highest Grounded Correctness in both evaluation settings.
- Comparison with RAG Baselines: proFILL requires 6.2 seconds per query, compared with 6.9 seconds for HyDE and 4.4 seconds for oracle proFILL.The additional deployment latency reflects on-demand adaptive profiling; absolute latency depends on hardware and external API response time.
- Human Preference Evaluation: 48 students preferred proFILL on all ten profile-dependent questions, with an aggregate non-tie win rate of 85.3%.The comparison was against conventional dense-retrieval RAG; the study recorded 354 wins and 61 losses.
- Human Preference Evaluation: Preference was highest for graduation- and credit-related questions at 93–98% and lowest for available-major-course questions at 58%.
- Ablation Study: Removing either soft prefix or hard filtering substantially degrades retrieval, while disabling evidence-driven re-retrieval lowers ranking quality despite near-ceiling Recall@50.The ablation trends are consistent across three dense embedding models.
- Results with Varying LLMs: Open-weight models remain competitive with gpt-4o-mini, while Korean-specialized models perform strongly on matching metrics.
5 System Demonstration
proFILL demonstrates query-driven profiling for questions resolved by a sufficient attribute and evidence-driven profiling when retrieved curriculum evidence reveals another required attribute.
- Query-Driven Profiling: For a scholarship query, query-driven profiling obtains the student’s status and answers without additional profiling.
- Evidence-Driven Profiling: For a major-course query, initial retrieval collects department and admission cohort, then triggers profiling for major type when curriculum applicability depends on it.
- Evidence-Driven Profiling: After the additional major-type information is obtained, proFILL re-retrieves relevant documents and returns a profile-specific answer with supporting sources.
6 Related Work
Related work spans retrieval-augmented generation, LLM personalization, and academic-advising chatbots; proFILL distinguishes itself by conditioning retrieval on an explicit profile schema.
- Retrieval-Augmented Generation: RAG grounds generation in retrieved evidence using dense, lexical, or hybrid retrieval, with improvements including reranking, query augmentation, and structured indexing.
- Personalization in LLMs: Prior LLM personalization uses methods such as user representations or personas, whereas proFILL uses an explicit schema-typed profile in retrieval without per-user training.
- Academic-Advising Chatbots: Academic-advising assistants have progressed from intent-based dialogue to retrieval-grounded systems, while the closest cited system does not explicitly condition retrieval on profile information.
7 Conclusion
The paper presents proFILL as a profile-aware extension of hoBIT that combines offline profile indexing with on-demand adaptive profiling for academic advising.
- Contribution: proFILL transforms hoBIT from a rule-based chatbot into a profile-aware RAG system through offline profile indexing and online on-demand profiling.
- Contribution: The system acquires missing information through query-driven profiling and requests additional details through evidence-driven profiling when needed.
- Contribution: Experiments on institutional data provide practical insights into how structured profiles improve RAG for academic advising.
Limitations
The study identifies two scope boundaries: evaluation is confined to one college, and the profile schema requires adaptation to each institution’s local advising context.
- The corpus and benchmark come from a single college of informatics, limiting the demonstrated evaluation scope.
- Although the overall pipeline is domain-general, its profile schema must be adapted to each institution’s curricula, policies, and advising practices.
Ethics Statement
The system is designed to minimize retained profile data and support verification, while its implementation uses several automated retrieval and annotation components. Institutional sources remain the basis for checking answers and binding decisions.
- proFILL acquires only query-required profile attributes, retains them for the current session, and discards them afterward without maintaining a persistent user-profile database.
- Usage logs contain no personally identifiable information and are used only to validate query distributions, while benchmark student profiles are entirely synthetic.
- hoBIT is an assistive rather than authoritative tool; answers cite institutional sources, and binding decisions remain subject to official university regulations.
- The corpus is segmented into semantic chunks, enriched with metadata, filtered for expired notices, consolidated for administrator FAQs, and annotated by an LLM across five profile attributes.
- Static and dynamic collections each maintain sparse and dense indices, combining BM25-style lexical retrieval with text-embedding vectors.
- Hybrid retrieval combines dense and BM25 rankings through reciprocal rank fusion with k=60.
- Time-aware aggregation selects ten chunks, changing the static–dynamic allocation for time-sensitive queries and applying a 90-day recency half-life to dynamic notices.
C Dataset Construction and Supplementary Results
The paper constructs privacy-conscious advising and intent-routing datasets from institutional materials and historical query logs, then reports intent-classification and open-ended advising evaluations.
- Dataset Construction: Dataset construction used academic-affairs guidance, deployed-service query logs, GPT-4o-mini assistance, and no personally identifiable information.
- Dataset Construction: The profile-grounded QA benchmark contains 1,800 instances spanning 60 student profiles, 10 advising categories, and three query types.
- Dataset Construction: Each QA instance has a deterministic gold source and expected keywords, with profile information supplied through the session profile rather than query text.
- Dataset Construction: The intent-routing dataset contains 1,600 queries across greeting, ability, faq, smalltalk, and retrieval intents, combining manually labeled seeds, generated examples, and open-ended advising queries.
- Supplementary Results: The proposed framework achieves an F1 score of 0.990 for retrieval queries in intent classification.
- Supplementary Results: Open-ended advising questions cover 12 academic and student-life categories and are evaluated with Top-3 Precision and Answer Completeness.
D Evaluation Metrics and LLM Judges
The evaluation combines lexical overlap, keyword, source-attribution, and LLM-based measures with a blind human preference study. Grounded Correctness explicitly requires both answer correctness and appropriate evidence grounding.
- Evaluation Metrics: ROUGE-L and Token-F1 measure overlap with reference answers after Korean content-morpheme tokenization.
- Evaluation Metrics: Keyword Match measures expected-keyword coverage, while Source Match scores cited gold sources 1, retrieved-but-uncited sources 0.5, and missing sources 0.
- LLM Judges: Grounded Correctness combines LLM-judged Answer Correctness and Source Match using a geometric mean.
- LLM Judges: Answer Correctness evaluates requested yes/no decisions for verification questions and expected-key-item coverage for general questions.
- LLM Judges: LLM-based scores are averaged across two independent judges on a stratified 600-case subset covering 10 categories and three phrasing types.
- Human Evaluation: The blind pairwise human preference study involved 48 college participants who compared responses to the same ten questions.