Source-linked AI summary
A Scenario-Knowledge-Driven Pipeline for Just-in-Time Assistance
Zhiyuan Li, Tatsunori Hara, Jun Ota
TL;DR
Silent kiosk users may need help without explicitly asking, while the scenario knowledge governing when and how to intervene is often implicit. The paper introduces a version-controlled document that configures sensing, constrains LLM reasoning, and separates auditable narration, assessment, and graded proposals; two replayed cases support traceable reporting and measured escalation, while appropriateness, quiet-session restraint, and transfer remain open.
Problem
Kiosk users may struggle without asking for help, while decisions about when and how to assist depend on scenario knowledge often buried in weights, thresholds, and branches.
Method
A human-authored, version-controlled scenario document configures sensing, constrains LLM reasoning, and shapes graded interventions while keeping narration, assessment, and proposals separate.
Results
Across 95 narration updates, every sentence cited its primitive events, while the rule layer detected 12 of 13 and 7 of 7 annotated struggle episodes under a strict criterion.
Takeaways & Limitations
The two cases support checkable reporting and measured escalation, including de-escalation on recovery and a single top-rung intervention under maximally converging evidence.
Takeaways & Limitations
Two struggle-heavy cases do not test restraint without struggle, intervention appropriateness, transfer, or whether explicit scenario knowledge outperforms implicit encoding.
Abstract
from arXiv · showhide
Detecting a silently struggling kiosk user is only the first step; deciding whether, when, and how to help depends on scenario knowledge usually buried in model weights and thresholds. We propose a scenario-knowledge-driven pipeline: a single scenario knowledge document, human-authored and version-controlled, configures sensing, constrains LLM reasoning, and shapes a graded intervention proposal. Narration, assistance-need assessment, and proposal are kept separate for independent audit. As proof of concept, we replay two recorded kiosk sessions offline, chosen before the runs for their struggle evidence and retrospective detail. Both cases support what the design promises: checkable reporting and measured escalation. Across 95 updates, every sentence of the append-only narration cites the primitive events underlying it, and the rule layer detects 12 of 13 and 7 of 7 annotated struggle episodes under a strict criterion. The assessor de-escalates on recovery and reaches the top rung exactly once, under maximally converging evidence. At the decisive help-seeking turn, narration, assessment, and the participants' retrospective accounts converge. The appropriateness of these interventions, the pipeline's restraint on sessions without struggle, and the document's transfer to a new scenario frame the agenda.
I. INTRODUCTION
Silent kiosk users may struggle without asking for help, making timely assistance difficult to provide. The paper responds by making scenario knowledge explicit and auditable across detection and intervention decisions.
- Motivation: Many kiosk users struggle silently, sometimes hesitating, repeating steps, seeking staff, or abandoning the task without asking for help.Recognizing struggle before visible failure is intended to support assistance at the moment of need and receptivity.
- Motivation: Prior work found that temporal variability in facial expression signals help-seeking intention, but detection alone does not determine how a system should act.The relevant signal concerns how facial expression changes over time rather than a single facial configuration.
- Problem: Poorly timed or overbearing assistance can interrupt coping users or erode the autonomy of users who do not need help.Deployment therefore requires explicit knowledge of task structure, signals, assistance states, interventions, and social norms.
- Contribution: The proposed pipeline uses a human-authored, version-controlled scenario document to configure sensing, constrain LLM reasoning, and shape intervention selection.It separates observation narration, assistance-need assessment, and intervention proposal so each inference step can be audited independently.
II. RELATED WORK
Related work shows that nonverbal cues and language models can support timely assistance decisions, but existing approaches generally leave scenario-constrained action underdeveloped. The paper positions explicit scenario knowledge and graded intervention as responses to that gap.
- Assistance-need from nonverbal cues: Prior systems use facial action units, head pose, gaze, and language cues to estimate engagement or help-seeking and decide when assistance may be appropriate.These efforts share the logic of just-in-time adaptive intervention, including the authors’ kiosk work on facial-expression variability.
- Assistance-need from nonverbal cues: Most prior assistance-need approaches stop at detection, leaving action under uncertainty open under scenario-specific constraints.The unresolved question is how to act on an uncertain need estimate rather than merely identify it.
- LLM- and VLM-driven proactive interaction: LLM- and VLM-based systems use natural-language guidance or reasoning to decide when and how to act, including when to remain silent.This paper differs by freezing a scenario knowledge document that selects detectors and bounds inference, with the document replaced for a new scenario.
- Perceptual primitives and appropriateness: Because unsolicited help can undermine agency and reasonable deviation can be misread as error, the paper motivates graded, auditable, explicitly configured intervention.This broadens evaluation beyond perceptual accuracy to the appropriateness of assistance.
III. SCENARIO-KNOWLEDGE-DRIVEN PIPELINE
The pipeline makes scenario knowledge a versioned control layer connecting multimodal sensing, constrained LLM reasoning, and graded assistance proposals. Its architecture separates auditable outputs while using event-triggered processing to balance responsiveness, cost, and traceability.
- Architecture: The encoder converts continuous gaze, posture, facial, operation, and positional streams into incrementally updated observation descriptions with provenance.The architecture is designed to run perception and reasoning within a live interaction’s latency budget.
- Scenario knowledge document: The kiosk_v1 document contains task structure, observable signals, assistance-need states, L0–L3 intervention options, social norms, and a version stamp.Structured sections select extracted signals and bound LLM inference, while free-text norms shape intervention choice.
- Scenario transfer: Adapting to another kiosk may preserve observable signals and the L0–L3 ladder, but changed task structure or norms require edits, recalibration, or new perceptual primitives.Transfer to a setting without a screen sequence has not been run, so its cost is unmeasured.
- Graded intervention: The intervention ladder progresses from silent observation (L0) through a passive cue (L1), light verbal inquiry (L2), and a contextual hint (L3).The ordering follows social exposure; takeover and staff handoff are outside the automated ladder by design.
- Auditability: The three outputs—observation narrative, assistance-need assessment, and intervention proposal—remain separate so each can be independently audited.This prevents a confident proposal from concealing a weakly supported observation.
- Architecture: The instantiated R1 pipeline uses a deterministic rule layer, event-triggered narration, and salience-triggered assessment configured by the frozen kiosk_v1 document.A periodic fallback invokes assessment every third narration update when no salient event batch arrives.
IV. PRELIMINARY CASE STUDY
The preliminary case study replays two offline kiosk sessions with a frozen, version-controlled scenario document and evaluates auditable detection, narration, assessment, and graded intervention. The pipeline achieves complete narration citation, detects 12 of 13 and 7 of 7 annotated episodes, and escalates selectively as evidence accumulates.
- Scope: The study replays two preselected, struggle-rich sessions offline, so it demonstrates auditable operation but does not test restraint without struggle or intervention appropriateness.The scenario document was frozen before the case runs, and retrospective annotations were used only for evaluation.
- Auditable narration: Across 95 narration updates, every sentence was uncited-free, with the append-only narrative tied to primitive events from the deterministic rule layer.The event-triggered narrator and provenance-bearing M1 events support traceable reporting.
- Overall results: 12 of 13 and 7 of 7 annotated struggle episodes were detected and narrated under the strict detection-alignment criterion in Cases A and B, respectively.Table I defines alignment as annotated/detected/narrated with type-matched onset within 5 seconds.
- Case A: Case A shifted from none/L0 to likely/L1 at 162 seconds, sustained an L2 plateau from 263 seconds, and avoided L3 because uncertainty remained too high.An earlier burst was de-weighted because it preceded a forward transition, whereas the later burst occurred at a revisited screen.
- Case B: Case B progressed from none/monitor to likely at 219 seconds, reached L2, and received the sole L3 escalation at 798 seconds when tap, scroll, posture, and gaze evidence converged.The L3 contextual hint occurred 67 seconds before the verbal request and later de-escalated to monitor/L1 after booking completed.
- Retrospective validation: At Case B’s decisive turn, the narrative, sole L3 assessment, and participant retrospective account independently converged on severe unresolved struggle.All 24 participant-marked events were accounted for, though 9 agreed, 11 were partial, 2 diverged, and 2 were out of scope.
V. DISCUSSION AND RESEARCH AGENDA
The discussion identifies auditability, memory, detector provenance, and evidence density as design lessons, while emphasizing that the two-case study leaves specificity, appropriateness, comparative benefit, and transfer unresolved.
- Full narrative history shifted 21 reference-session assessments from 11 to 18 likely verdicts and produced 13 light-inquiry proposals, while recency weighting restored de-escalation.The result shows that memory configuration changes conclusions even with identical events and prompts.
- 12/13 and 7/7 annotated episodes were detected after fixing tap-event provenance and recalibrating the two-second tap window.Initial alignment was 1/13 and 3/7; the audit trail exposed the click-based counting gap.
- Median assessment latency was 21 seconds, so live escalations could trail the moment of need; real-time feasibility remains untested.The authors attribute much of this cost to writing assessment text and identify faster decoding and streaming as future directions.
- Because both preselected sessions contained sustained struggle, specificity, intervention appropriateness, and superiority over implicit scenario knowledge were not evaluated.The study also uses one reasoning model, one threshold set, and single-coder unblinded annotations, leaving model dependence and coder reliability unknown.
- The research agenda calls for negative-control sessions, independent document authoring, cross-scenario transfer, explicit-versus-implicit comparisons, route comparisons, appropriateness evaluation, and robot integration.
VI. CONCLUSION
The paper makes scenario knowledge explicit through a version-controlled document that configures sensing, constrains reasoning, and supports auditable graded assistance. Two cases show traceable narration and restrained assessment, while appropriateness, no-struggle restraint, transfer, and robotic action remain future work.
- A version-controlled scenario document configured sensing, constrained LLM reasoning, and kept inference steps auditable across an end-to-end two-case study.
- The narration stayed traceable to primitive events, while the assessor handled different struggle dynamics with graded restraint.