PaperlayerView exact revision on arXiv

Source-linked AI summary

What Students Actually Ask: Demand Structure and Automation Potential in a Hybrid Support System

Jinal Gupta, Pavani Ayinampudi, Aditya B. M. V., Prakash Hegade, Rohit Sharma, Sakshi Sharma, Meenakshi V, S. R. S. Iyengar

arXiv:2609.38967v1cs.HC

TL;DR

Large online programmes face onboarding query volumes that grow faster than available support staff. This study analyzes operational records from a hybrid query-resolution platform, finding that demand includes substantial participant-specific status enquiries alongside queries addressed by existing corpus content. The findings describe how demand, support paths, and record limitations shape interpretation of this deployment.

  • Problem

    Large online programmes face onboarding query volumes that outpace staffing, while the composition of operational support demand is not well characterized.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes complete operational records from a single-site deployment and classifies query text alongside dispatch and timing data.

    Source: §3.1 · p. 3, §4.1 · p. 4, §4.2 · p. 5

  • Results

    At least 20.4% of queries concerned pending-submission status, which the assistant served only 1.3% of the time; only 8.0% of person-handled queries duplicated corpus content.

    Source: §5.2 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that making pending-process status visible may address a major source of query volume without adding answering capacity.

    Source: §6 · p. 10

    The findings describe one programme during nine weeks of onboarding, and category volumes are lower bounds because one author classified queries without a second coder.

    Source: §4.2 · p. 5, §5.3 · p. 10

Abstract

from arXiv · show

Large online programmes receive heavy volumes of queries during onboarding, at a scale that grows faster than the number of staff available to answer them. This paper reports a study of an AI integrated query resolution platform that spreads incoming queries across four routes: an AI based assistant, answers from fellow participants, a curated corpus of frequently asked questions, and escalation to administrators. Over nine weeks, from 2 May to 6 July 2026, the system handled 4,093 queries raised by 1,434 participants. Nearly every query reached a recorded resolution, and one query in five closed within an hour. Reuse of 114 corpus entries absorbed 21.3% of the volume, participants resolved a further 12.2% on their own, and 132 participants answered questions for one another at a median of 9 to 15 minutes, showing that peer answering, where it occurred, was fast and broadly shared across the cohort. Classifying the query text shows that demand was narrow rather than varied. A single process step, the submission of a certificate and the offer letter that follows it, accounts for 56.8% of corpus mediated resolutions, and at least 20.4% of queries concern the progress of a pending submission rather than a request for information, a class the assistant served only 1.3% of the time, since a stored answer cannot report an individual's current status. Only 8.0% of the queries handled by a person duplicated content already in the corpus, which indicates that the knowledge base was already well used. Most direct administrative closures occur in synchronous bulk events, a pattern that shapes how the records should be read.

1 Introduction

The paper examines operational support demand in a deployed hybrid system and asks how query type, peer support, and the limits of the resolution record shape service design. It describes a layered approach combining a curated FAQ corpus, query routing, and administrator-reviewed peer answers.

1 Introduction

  • The platform aims to resolve queries quickly without assigning each student a dedicated mentor.The corpus records the procedure for each step, whom to contact, and messages students can expect after actions.Source: §1 · pp. 1–2
  • A curated FAQ corpus records procedures, contacts, and expected messages, and is revised daily as rules and recurring situations change.The pipeline routes queries to relevant entries and combines multiple entries when needed.Source: §1 · p. 2
  • Participants may answer peers by pointing to an existing entry or drafting a new one, but administrators and a senior administrator approve answers before release.This review process keeps quality under control while widening participation and allowing the corpus to grow from observed questions.Source: §1 · p. 2
  • The study uses nine weeks of operational records to characterize 4,093 queries from 1,434 participants and examine peer resolution.It distinguishes demand types because different queries call for answering capacity, wider corpus coverage, or participant-specific status information.Source: §1 · p. 2

2 Background Study

The paper positions its study between research on help seeking, automated and peer answering, and the interpretation of support records. It addresses limited reporting on the composition of operational support demand and connects that composition to service design.

2 Background Study

  • Prior help-seeking research largely concerns academic content, whereas this study examines operational needs such as eligibility, documentation, and scheduling.Research on help seeking also reports that learners may underuse or misuse help and that improved help seeking does not reliably improve outcomes.Source: §2 · p. 2
  • The study addresses a gap: chatbot research rarely reports the composition of incoming demand, while this paper examines operational support queries.Prior work commonly evaluates higher-education chatbots on satisfaction or accuracy.Source: §2 · p. 2
  • Learnersourcing research studies how learners generate content and how contributions are assessed and sustained; this study applies those questions to operational answers.The data allow the authors to report who contributes and how effort is distributed.Source: §2 · p. 3
  • The paper relates demand composition to dispatch, path cost, corpus coverage, and peer contribution rather than treating incoming demand as fixed.Related work also covers reusable answers, process visibility, peer latency, and support-ticket classification.Source: §2 · p. 3

3 Methods and Methodology

The study is a descriptive analysis of a deployed hybrid support system in an online internship programme. It classifies query text and links demand types to dispatch and timing, while examining what operational records can establish.

3.1 Methodology and Study Context

  • This single-site retrospective deployment study analyzes individual queries descriptively, without a comparison condition or configuration changes.The setting is a staggered-cohort internship programme with a fixed sequence of onboarding steps, several requiring staff verification.Source: §3.1 · p. 3
  • The platform routes queries to a verified-corpus chatbot, then to peers when needed, with administrator review and final senior approval for peer answers.Administrators may also answer directly, and a scripted facility handles queries covered by a standard response.Source: §3.1 · p. 3

3.2 Research Questions

  • The research questions ask what demand types emerge, how they are dispatched and handled, and when resolution records fairly evidence demand and contribution.They also examine path closure time and how much demand reaching a person was already answerable from verified content.Source: §3.2 · p. 4

3.3 Research Design

  • The design classifies query text separately from dispatch and timing, then applies the demand categories to those analyses for cross-tabulation.Classification was performed without reference to resolution outcome.Source: §3.3 · p. 4

4 Data Collection and Analysis

The analysis uses complete operational records and combines demand classification with dispatch, timing, corpus-coverage, and repeat-enquiry analyses. The authors describe limits in category precision and in using lexical similarity to estimate corpus coverage.

4.1 Data Source

  • The dataset contains all 4,093 queries from 1,434 participants between 2 May and 6 July 2026, including records from two merged schema variants.The export includes query, resolution, responder, timestamp, FAQ-link, escalation, and acceptance information.Source: §4.1 · p. 4

4.2 Classification of Demand

  • The nine demand categories were derived inductively and assigned by a priority-ordered rule-based classifier tuned for precision.A stratified sample of 108 queries measured precision; one author set category boundaries, with no second coder or inter-rater statistic, so category volumes are lower bounds.Source: §4.2 · pp. 4–5

4.3 Dispatch and Timing Analysis

  • Elapsed time is measured between raising and response timestamps, with medians and upper percentiles reported by path because timing is right-skewed.Records without a closure timestamp are counted as censored.Source: §4.3 · p. 5

4.4 Estimation of Corpus Coverage

  • Corpus coverage is estimated by comparing person-resolved, unlinked queries with their nearest corpus-linked query using TF-IDF and cosine similarity.Corpus-linked queries serve as a proxy because the export lacks answer text; lexical matching is conservative for paraphrase, and a manually validated sample is reported across thresholds from 0.50 to 0.85.Source: §4.4 · p. 5

4.5 Repeat Enquiry and Concentration

  • Repeat enquiries require the same participant to submit a lexically similar query within seven days of a prior closed query, using a cosine-similarity threshold of 0.60.Demand across participants and reuse across corpus entries are summarized with Lorenz curves and Gini coefficients.Source: §4.5 · p. 5

5 Results and Discussion

Results show concentrated procedural demand and substantial reuse of existing corpus content, alongside a distinct class of participant-specific status enquiries. The discussion interprets these patterns as limits on what corpus expansion and operational records can establish.

5.1 Results

  • 56.8% of corpus-mediated resolutions concerned certificate submission and offer-letter issuance, concentrated across three entries.Reuse covered 871 queries (21.3%) through 114 entries; 30 entries were each used once.Source: §5.1 · pp. 5–6, Table 3 · p. 7
  • Within one hour, 70.3% of peer-answered queries had closed, compared with 1.9% of direct-administrator closures and 14.8% of scripted closures.Peer resolutions numbered 285 across 132 authors; one administrator accounted for 72.7% of identifiable human-responder records.Source: §5.1 · pp. 6, 8, Table 2 · p. 6

5.2 Discussion

  • Only 8.0% of person-resolved queries without a corpus link matched existing corpus-linked content at the 0.70 similarity threshold, falling to 6.3% after excluding status enquiries.The authors read 8.0% as a floor because lexical matching conservatively detects paraphrases; 183 identical-text queries took different paths, suggesting a dispatch-consistency margin.Source: §5.1 · pp. 6–7, Table 4 · p. 7, §5.2 · p. 9

5.1 Results

  • The slowest decile comprised 380 queries exceeding 242.4 hours, and 273 were closed by scripted bulk action; repeat enquiries were 0.6% of interpretable queries.Classification precision was 75.0%, and status-enquiry share is a lower bound because misclassification ran into that category but not out of it.Source: §5.1 · pp. 5, 8, Table 5 · p. 8
  • At least 20.4% of queries were status enquiries, and the assistant handled only 1.3% of them.Status enquiries closed at a median of 40.8 hours; they made up 21.8% of self-resolved queries versus 20.4% overall.Source: §5.1 · p. 5, §5.2 · p. 9

5.2 Discussion

  • Because the corpus cannot report an individual's changing submission status, the authors identify process-state visibility as a more direct response than additional answering capacity.Demand was broadly spread across participants: Gini 0.447, with 42.9% raising one query.Source: §5.2 · p. 9
  • Bulk administrative closures complicate interpreting raw resolver concentration and latency; separating them distinguishes a 6.1-hour routine median from a 28.9-hour aggregate.The authors caution that resolver fields represent different roles across administrative and peer paths, and accepted_derived indicates whether resolver and asker differed, not answer acceptance.Source: §5.2 · pp. 9–10

5.3 Limitations

  • The findings describe one programme's nine-week onboarding period, not steady-state support or other programmes.The authors also report rule-based classification without a second coder, a proxy corpus without answer text, no resolution-quality finding, and their own operational involvement.Source: §5.3 · p. 10

6 Conclusion

Across nine weeks, the hybrid platform recorded resolutions for nearly all 4,093 queries, while peer answering was quick and broadly shared. Demand clustered around a certificate-and-offer-letter process and pending-submission status; the authors point to process visibility, clearer authorship records, and further cohorts as next steps.

6 Conclusion

  • 132 participants answered peers’ questions at a median of 9 to 15 minutes, while corpus reuse handled 21.3% of queries and participants self-resolved 12.2%.The study covered 4,093 queries from 1,434 participants; nearly every query reached a recorded resolution.Source: §6 · p. 10
  • 56.8% of corpus reuse concerned certificate submission and the following offer letter, while at least 20.4% of queries concerned pending-submission progress.The assistant served this status-enquiry class only 1.3% of the time because stored answers cannot report an individual’s current status; direct administrative answering was concentrated in one administrator, who handled 91.0%.Source: §6 · p. 10
  • The authors propose making pending-process state visible, recording authorship and approval separately on every path, and repeating the study across cohorts.Source: §6 · p. 10

Related papers

Loading 2609.38967v1…