Source-linked AI summary
GLOSS: Group of LLMs for Open-Ended Sensemaking of Passive Sensing Data for Health and Wellbeing
Akshat Choube, Ha Le, Jiachen Li, Kaixin Ji, Vedant Das Swain, Varun Mishra
TL;DR
Passive sensing enables health and behavior prediction, but holistic understanding is limited and interpretation requires specialized expertise for complex multimodal data. GLOSS is an open-ended, query-based system in which LLM agents collaborate to retrieve and analyze raw passive sensing data using generated code. GLOSS is compared with a Retrieval Augmented Generation (RAG) technique and demonstrated in four use cases.
Problem
Passive sensing enables health and behavior prediction, but holistic understanding is limited and interpretation requires specialized expertise for complex multimodal data.
Source: §1 · p. 2, §6.1 · p. 18, §7.1 · p. 22Method
GLOSS is an open-ended, query-based system in which LLM agents collaborate to retrieve and analyze raw passive sensing data using generated code.
Source: §3 · p. 6, §4 · p. 11Results
GLOSS is compared with a Retrieval Augmented Generation (RAG) technique and demonstrated in four use cases.
Source: §1 · p. 3, §8 · p. 26Takeaways & Limitations
GLOSS provides a foundation for accurate, open-ended, and multimodal sensemaking of passive sensing data in human-facing and automated health and wellbeing applications.
Source: §1 · p. 3, §8 · p. 26GLOSS was evaluated with one LLM, can have latency of a couple of minutes for complex queries, and lacks multi-turn history and follow-up support.
Source: §7.4 · p. 25, §7.5 · p. 26
Abstract
from arXiv · showhide
The ubiquitous presence of smartphones and wearables has enabled researchers to build prediction and detection models for various health and behavior outcomes using passive sensing data from these devices. Achieving a high-level, holistic understanding of an individual's behavior and context, however, remains a significant challenge. Due to the nature of passive sensing data, sensemaking -- the process of interpreting and extracting insights -- requires both domain knowledge and technical expertise, creating barriers for different stakeholders. Existing systems designed to support sensemaking are either not open-ended or cannot perform complex data triangulation. In this paper, we present a novel sensemaking system, Group of LLMs for Open-ended Sensemaking (GLOSS), capable of open-ended sensemaking and performing complex multimodal triangulation to derive insights. We demonstrate that GLOSS significantly outperforms the commonly used Retrieval-Augmented Generation (RAG) technique, achieving 87.93% accuracy and 66.19% consistency, compared to RAG's 29.31% accuracy and 52.85% consistency. Furthermore, we showcase the promise of GLOSS through four use cases inspired by prior and ongoing work in the UbiComp and HCI communities. Finally, we discuss the potential of GLOSS, its broader implications, and the limitations of our work.
1 INTRODUCTION
Passive sensing can provide rich health and behavioral information, but its high dimensionality, missingness, multimodal structure, and technical demands make holistic sensemaking difficult for many stakeholders. GLOSS addresses these barriers with an open-ended, query-based system that uses collaborating LLMs to interpret passive sensing data and support diverse applications.
Conceptual framing
- Sensemaking is described as iterative information seeking and understanding refinement, triggered by a sensemaker’s information need and continuing until the resulting explanation is satisfactory.This framing motivates systems that can support both information retrieval and the development of interpretations.Source: §1 · pp. 1–2
Problem
- Passive sensing supports research on health and behavior, but holistic understanding remains limited by high dimensionality, missingness, multimodal triangulation, and specialized programming expertise.Meaningful interpretation may require combining location, activity, and phone-use streams rather than relying on a single measure.Source: §1 · p. 2
- Existing sensemaking tools are often not open-ended, operate on aggregated data, require domain expertise, and provide limited customization for different stakeholders.These constraints can leave researchers, clinicians, and self-trackers dependent on domain experts or predefined visualizations.Source: §1 · p. 2
System contribution
- GLOSS uses a network of collaborating large language models, function calling, and code generation to retrieve and analyze passive sensing data through user queries.Its design is informed by models of sensemaking from HCI and Organizational Studies.Source: §1 · p. 3
Evaluation and applications
- The authors evaluate GLOSS against Retrieval-Augmented Generation and demonstrate applications spanning researcher analysis, personal informatics narratives, anomaly explanation, and personalized EMA prompts.Detailed evaluations of the individual use cases are left for future work.Source: §1 · p. 3
System contribution
- GLOSS introduces an open-ended, scalable, and extensible query-based sensemaking system for interpreting passive sensing data across diverse stakeholders.The system is designed to support users in expressing information needs through natural-language queries.Source: §1 · p. 3
2 RELATED WORK
Prior work shows that passive sensing supports health and behavioral modeling, but interpreting raw, individualized, multimodal data remains difficult. Existing visualization and LLM-based systems provide partial support yet often restrict open-ended reasoning, raw-data access, or complex triangulation.
2.1 Barriers in Interpreting Passive Sensing Data
- Passive sensing supports prediction of health and behavioral outcomes, but interpreting longitudinal data remains difficult because streams are high-dimensional, incomplete, individualized, and technically demanding.These barriers affect researchers, clinicians, behavioral scientists, and self-trackers.Source: §2.1 · pp. 3–4
2.2 Sensemaking of Passive Sensing Data
- Visualization dashboards and collaborative systems assist interpretation but still rely on user expertise, can lack context, or overwhelm users when triangulation across views is required.Prior systems often facilitate reflection or discussion without performing the full sensemaking process for users.Source: §2.2 · p. 4
2.3 LLMs with Passive Sensing Data
- LLM-based health systems commonly convert sensing data into text and use retrieval or fine-tuning for prediction, question answering, narratives, or personalized insights.These approaches demonstrate the applicability of LLMs to passive sensing while generally targeting specific tasks.Source: §2.3 · p. 5
- Existing systems remain limited by aggregated or precomputed data, template-based queries, and basic statistical operations that constrain open-ended multimodal analysis.PhysioLLM focuses on correlations between aggregated measures, while PHIA uses daily CSV data and restricted code generation.Source: §2.3 · p. 5
- GLOSS addresses these gaps by analyzing raw database data, generating complex code for cross-stream triangulation, requiring no training or fine-tuning, and supporting extensible sensor streams.The system is presented as a fully open-ended sensemaking approach for diverse stakeholders.Source: §2.3 · p. 5
3 SYSTEM DESIGN
GLOSS models human sensemaking with collaborating LLM agents that plan, retrieve, process, interpret, refine, and present answers from raw multimodal sensing data. Its query-driven design supports natural-language interaction, iterative triangulation, extensible data streams, and user-specific output formats.
3 SYSTEM DESIGN
- GLOSS uses interconnected information-seeking and sensemaking loops to iteratively gather, transform, interpret, and refine evidence for a user query.The design is inspired by sensemaking frameworks from HCI and Organizational Studies.Source: §3 · p. 5, §3.1.3 · p. 9
- GLOSS operates off-the-shelf on high-dimensional raw data from multiple streams, generates advanced analysis code, accepts natural-language queries, and supports human-facing or automated tasks.Answers can also be personalized to an individual’s needs.Source: §3 · pp. 6–7
3.1.1 Action Plan Generation.
- The Action Plan Generation Agent maps a query to available databases and models, halting when required data are unsupported and otherwise initiating a step-by-step analysis plan.This gate prevents the workflow from proceeding when the system lacks the requested data.Source: §3.1.1 · p. 7
3.1.2 Information Seeking/Sensemaking Loop.
- During each loop, agents request and retrieve relevant data, generate executable code, triangulate across databases or models, translate outputs into language, and update the global understanding.The Next Step Agent decides whether more information is needed, while global sensemaking records missingness or technical failures.Source: §3.1.2 · pp. 7–9
3.1.3 Answer Presentation.
- The Presentation Agent extracts the answer from the final understanding and formats it for different users, including qualitative summaries or JSON outputs.Presentation occurs after the Next Step Agent determines that the process should conclude.Source: §3.1.3 · p. 9
3.2 Implementation Details
- Code generated by the system runs in a Docker container, while aggregated statistics rather than large raw-data chunks are shared with GPT; the workflow stops after five iterations.The implementation also uses a HIPAA-compliant deployment and does not retain queries for training.Source: §3.2 · pp. 9–10
- GLOSS integrates phone and Garmin streams through MongoDB and pre-coded helper functions, and additional streams or models can be added by supplying descriptions and retrieval functions.The implementation uses helper functions to let agents access raw data and models.Source: §3.2 · p. 10, Figure 3 · p. 11
4 METHOD
The evaluation tests GLOSS on open-ended natural-language queries spanning objective, subjective, and mixed questions. It compares GLOSS with a GPT-4o RAG baseline using accuracy, consistency, and blinded human judgments of subjective responses.
4 METHOD
- The evaluation aims to measure GLOSS’s accuracy, consistency, and subjectivity on open-ended passive-sensing queries.The study evaluates the core query-driven system using objective and subjective questions.Source: §4 · p. 11
4.1 Evaluation Queries
- Seven passive-sensing researchers generated 122 natural-language queries, which became 244 queries after assignment to two participants and matching dates.The testing pool included 12 participants and 913 participant-days, with query date ranges limited to seven days.Source: §4.1 · p. 11
- The 244 queries comprised 196 objective queries, 34 subjective queries, and 14 mixed queries.Two researchers independently coded the queries and resolved disagreements through a third team member.Source: §4.1 · p. 12
4.2 Evaluation Metrics
- The baseline is a GPT-4o RAG system that converts raw streams into natural-language records, stores them in Chroma, and retrieves records for each query.The baseline was implemented because prior comparable systems were closed-source or lacked open-ended multimodal capabilities.Source: §4.2 · p. 12
- Objective queries were evaluated for accuracy and consistency, with accuracy requiring manual Python analysis of the logic used and consistency requiring identical answers across three runs.Accuracy was sampled on 60 queries, whereas consistency was evaluated across 210 objective and mixed queries.Source: §4.2 · p. 12
- Subjective responses were blinded to two annotators, who rated relevance, interpretation, domain knowledge, logic, clarity, and overall reasoning on a five-point Likert scale.GLOSS was instructed to explain subjective answers clearly and in detail.Source: §4.2 · pp. 12–13
5 RESULTS
GLOSS outperformed RAG on objective accuracy and consistency, while subjective evaluations showed stronger interpretation and logic but mixed or nonsignificant differences on other dimensions.
5.1 Objective Evaluation
- On objective queries, GLOSS significantly outperformed RAG in accuracy (t(59) = 8.24, p < .001), attributed to its ability to generate code for calculations and data processing.GLOSS’s consistency was 66.19% versus 52.85% for RAG (t(59) = 3.33, p = .001).Source: §5.1 · p. 13, Table 5 · p. 13
- 66.19% consistency for GLOSS versus 52.85% for RAG, although query ambiguity led GLOSS to use different valid logical paths across runs.For example, GLOSS used either step count or GPS-coordinate differences to identify the day with highest mobility.Source: §5.1 · p. 13
5.2.1 Clarity.
- RAG produced more concise and readable responses for clarity, with no statistically significant difference between systems.GLOSS sometimes displayed extensive raw details, such as every timestamp with heart rate above 120 bpm.Source: §5.2.1 · p. 13, Figure 4 · p. 13, Table 6 · p. 14
5.2.2 Relevance.
- GLOSS slightly better triangulated multiple sensor streams than RAG for relevance, but the difference was not statistically significant.GLOSS examined step count, location, Wi-Fi, and phone-usage patterns, whereas RAG used only location and step count; χ2(1) = 3.44, p = .06.Source: §5.2.2 · p. 14, Table 7 · p. 15
5.2.3 Interpretation.
- GLOSS was statistically better than RAG on interpretation and logic, especially when queries required quantitative reasoning.Interpretation yielded χ2(1) = 12.69, p < .001, while logic yielded χ2(1) = 6.64, p = .001; GLOSS’s code generation reduced calculation mistakes.Source: §5.2.3 · p. 15, §5.2.5 · p. 16
5.2.4 Domain Knowlegde.
- GLOSS was slightly better on domain knowledge, but the difference from RAG was not statistically significant.In an example, GLOSS detected and explained a discrepancy between phone and Garmin step counts that RAG missed.Source: §5.2.4 · pp. 15–16, Table 9 · p. 17
6 POTENTIAL USE CASES OF GLOSS
GLOSS is presented through four potential use cases: supporting non-CS researchers, generating personalized health narratives, triggering context-relevant prompts, and improving interpretability of anomaly-detection models. These demonstrations illustrate GLOSS’s accessibility, multimodal sensemaking, and adaptable presentation, while broader evaluations remain future work.
6 POTENTIAL USE CASES OF GLOSS
- GLOSS is proposed as a promising system for making passive-sensing analysis more accessible and efficient across personal health informatics and wellbeing applications.The paper presents these as potential use cases rather than completed evaluations of each application.Source: §6 · p. 17
6.1 Supporting Non-Computer Science researchers with Sensemaking
- Non-CS researchers face technical barriers, dependence on computer-science collaborators, and delays when processing, analyzing, or interpreting longitudinal passive-sensing data.Participants also described manual coding as labor-intensive and non-scalable for large study populations.Source: §6.1 · pp. 18–19, §6.1.1 · p. 20
- GLOSS’s chat interface lets non-CS researchers query data while generating code, triangulating information, and presenting tailored, interpretable insights.The demonstration involved interviews and think-aloud sessions with three psychologists using a subset of their familiar study data.Source: §6 · pp. 17–18, §6.1 · p. 19
6.1.2 Experience with GLOSS:.
- Participants valued GLOSS for saving time, simplifying complex analysis, and improving transparency, with an average aggregated SUS score of 84.02%.They also requested query history, follow-up questions, prompting guidance, downloadable code and data, and graph generation.Source: §6.1.2 · p. 20, §6.1.3 · p. 20
6.1.3 Improvements to GLOSS:.
- The authors identify comprehensive evaluation across multiple passive-sensing research labs as future work for assessing GLOSS’s broader impact.The current interviews provide an indicator of potential impact rather than a completed multi-lab evaluation.Source: §6.1.3 · p. 21
6.2 Narratives for Personal Health Reflection
- GLOSS can triangulate multiple data streams and tailor qualitative or quantitative personal-health narratives through presentation instructions to support reflection.Figure 6 illustrates different narratives generated from the same query when the requested wellbeing dimension changes.Source: §6.2 · p. 21, Figure 6 · p. 21
6.3 AI-triggered Prompting
- GLOSS can continuously integrate sensor streams to trigger contextually relevant EMA questions and address missing passive-sensing data.Examples include asking about stress after contextual changes and querying gaps caused by absent app-usage or phone-conversation data.Source: §6.3 · p. 22
6.4 Enhancing Interpretability
- GLOSS can augment opaque LSTM-based app-usage anomaly predictions with interpretive insights that researchers can use to formulate actionable strategies.The personalized models analyzed seven days of data, while GLOSS supplied additional explanations and supported follow-up queries.Source: §6.4 · p. 22
7 DISCUSSION
The discussion presents GLOSS as an open-ended, multimodal sensemaking system that broadens access to passive sensing analysis, outperforms RAG in evaluation, and supports diverse applications while retaining important usability and scope limitations.
7.1 GLOSS: Enabling Open-ended Sensemaking
- GLOSS combines query-based interaction, code generation, and multimodal triangulation to analyze high-dimensional raw passive sensing data without requiring programming or domain expertise.Its design is inspired by theories of human sensemaking and supports automated retrieval and analysis.Source: §7.1 · pp. 23–24
7.2 Comparison of GLOSS and RAG Performance
- Evaluation consistency can decline when queries are ambiguous because GLOSS may follow different accurate reasoning paths across runs.The authors propose clarifying ambiguous queries or combining multiple valid approaches.Source: §7.2 · p. 24
- GLOSS responses outperformed RAG in relevance, interpretation, domain knowledge, and logic, although RAG scored higher in clarity and the differences were subtle.The authors caution that RAG explanations may appear clear despite inaccurate results, which is concerning for health and well-being applications.Source: §7.2 · p. 25
7.3 Potential for Future Applications
- GLOSS is presented as a foundational model supporting applications for psychologists, intervention designers, self-health trackers, and researchers, including narratives, EMA prompts, and black-box-model interpretation.The authors also discuss extensions involving speech interfaces, accessibility, and other data types such as images and audio.Source: §7.3 · p. 25
7.4 Future Improvements to GLOSS
- GLOSS currently lacks multi-turn conversation, query history, graph display, and data or code downloads, limiting iterative use and transparency.The authors propose follow-up conversations, prompt examples, and clarification requests to address these issues, especially for ambiguous queries.Source: §7.4 · pp. 25–26
7.5 Limitations
- The evaluation did not test multiple LLMs, used GPT-4o with potentially long latency, focused on phone and wearable modalities, and relied on a small researcher-generated query dataset.The authors identify diverse populations and query distributions as priorities for future evaluation.Source: §7.5 · p. 26
8 CONCLUSION
The conclusion presents GLOSS as a query-based sensemaking system in which LLM agents interact according to human-sensemaking theories. It enables accurate, open-ended, multimodal passive-sensing understanding, supports four use cases, and was highly valued by human subjects.
8 CONCLUSION
- GLOSS enables accurate, open-ended, multimodal understanding of passive sensing data through interacting LLM agents.The system was presented alongside four automated and human-facing use cases.Source: §8 · p. 26