PaperlayerView exact revision on arXiv

Source-linked AI summary

OntoAR-HAR: Ontology-Guided Agentic RAG with Personalized Memory for Human Activity Recognition

Xiangqing Lu, Tao Zhu, Liang Yu, Shuangjian Li, Lulu Ban, Furong Duan, Mingxing Nie, Liming Chen, Yaping Wan

arXiv:2610.15292v1cs.AI

TL;DR

Existing RAG-style HAR commonly searches single-level activity spaces using static population examples, leaving current-user motion patterns unrepresented. OntoAR-HAR combines ontology-guided local decisions, executable signal features, and personalized retrieval; across eight datasets, hierarchy and memory provide additive gains, including under mixed sensing conditions.

  • Problem

    Existing RAG-style HAR commonly uses single-level activity spaces and static source-user examples, which cannot represent the current user’s characteristic motion patterns.

    Source: §I · p. 1

  • Method

    OntoAR-HAR uses an activity ontology to guide local retrieval, combines static and personalized stores, and supplies executable signal features as numerical evidence.

    Source: §I · p. 1, B. Ontology and Feature Knowledge… · p. 3, C. Static Population and Personalized… · p. 4

  • Results

    Across eight datasets, hierarchy and memory provide additive gains; with Gemini 2.5 Flash, average accuracy rises from 84.0 with single-level retrieval to 87.2 with hierarchy and 89.0 after adding memory.

    Source: B. Cross-Subject Results and Component… · p. 7, §V · p. 9

  • Takeaways & Limitations

    The framework connects label-level semantics, source-domain sensor evidence, and current-user observations in one recursive inference process.

    Source: §V · p. 9

Abstract

from arXiv · show

LLMs increasingly support sensor-based human activity recognition (HAR) by linking motion signals with activity semantics and physical knowledge. Retrieval-augmented generation (RAG) grounds decisions in sensor examples and domain features, enabling training-free target-user inference with less dependence on raw numerical arrays. However, existing LLM/RAG-based HAR typically searches single-level activity spaces using static population knowledge. Consequently, they compare many unrelated candidates using the same static population evidence and overlook personalized motion patterns. We propose OntoAR-HAR, an ontology-guided agentic RAG framework combining static population and personalized knowledge. Its tree-structured activity ontology decomposes classification into local decisions among semantically related activities. The static store supplies node-constrained examples, while offline-discovered executable signal features provide numerical evidence for local distinctions. During inference, the personalized store collects selected user observations, which the LLM integrates with static knowledge and signal features while traversing the ontology to an activity label. Cross-subject and heterogeneous-sensing experiments show complementary gains from hierarchical retrieval and personalized memory across multiple LLM backbones, achieving state-of-the-art performance. By explicitly modeling hierarchical relations across activity granularities, OntoAR-HAR also addresses the long-overlooked semantic inclusion problem and avoids treating related labels as independent categories.

I. INTRODUCTION

RAG for HAR places observed motion and signal knowledge in an LLM’s decision context, but common single-level retrieval and static population stores do not represent local label relations or current-user motion. OntoAR-HAR organizes retrieval by an activity ontology and adds a personalized store.

I. INTRODUCTION

  • RAG-based HAR supplies retrieved sensor observations or feature knowledge as context, while earlier work also explores direct prompting and sensor-language interfaces.HARGPT studies prompting with activity descriptions and sensor sequences [1]; RAG-HAR and ZARA retrieve observations or domain knowledge,.Source: §I · p. 1, §II · p. 2
  • Single-level retrieval can mix unrelated labels and make local physical distinctions harder to resolve, while static population examples omit the current user’s characteristic motion.Examples include walking subclasses, stationary postures, and eating activities with different wrist dynamics; the static store cannot represent a user’s amplitude, rhythm, or execution style.Source: §I · p. 1
  • OntoAR-HAR combines an activity ontology and personalized knowledge store, restricting comparisons to semantically coherent groups and supplementing population evidence with user-specific observations.The ontology organizes inclusion and sibling relations, and the personal store uses selected observations rather than parameter updates.Source: §I · p. 1, §II · p. 2

II. RELATED WORK

  • Unlike representation-focused approaches, OntoAR-HAR keeps the sensor encoder fixed and addresses how retrieval evidence is organized, which local features resolve ambiguity, and how user observations enter decisions.Its episodic personalization changes the reference examples available to the LLM, not the sensor-window mapping [24], [25].Source: §II · p. 2

III. METHOD

OntoAR-HAR predicts labels for a held-out user from an ordered sensor stream without gradient updates. It organizes labels in a rooted tree and makes recursive local decisions using static and personalized retrieval.

III. METHOD

  • The task uses labeled source sensor windows and predicts labels for an ordered stream from a held-out user without gradient updates.Each source window includes an activity label and a sensor-view or body-location identifier.Source: A. Problem Formulation and Overview · p. 2
  • Activity labels form a rooted tree whose leaves are output labels and whose internal nodes represent semantic groups.At each node, the decision considers only its children.Source: A. Problem Formulation and Overview · p. 2
  • Offline construction defines the ontology, node-constrained indexes, and local executable features; online inference recursively retrieves evidence and selects one child at a time.The decision context draws on static population and personalized examples and numerical feature evidence.Source: A. Problem Formulation and Overview · p. 2

B. Ontology and Feature Knowledge Construction

The framework manually builds an activity hierarchy from retrieval-only confusions, then constructs executable signal features for ambiguous local decisions. The hierarchy and validated features are fixed before target-user inference.

B. Ontology and Feature Knowledge Construction

  • The hierarchy is manually refined from confusion relationships in class-balanced training samples, grouping strongly confused activities and nesting clearer subgroups.The resulting tree keeps every activity as one leaf and is fixed before evaluation.Source: B. Ontology and Feature Knowledge… · p. 2, Figure 1 · p. 3
  • The confusion matrix guides tree design rather than optimizing an automatic tree objective, and the hierarchy is built from source training data without held-out users.Intermediate nodes group leaves without adding output labels, preserving the original classification task while changing candidate order.Source: B. Ontology and Feature Knowledge… · p. 3
  • At ambiguous nodes, an LLM proposes feature names, physical interpretations, and calculations from child labels and sensor metadata; supported proposals are implemented by rules.The representative feature families are validated separately for each decision node.Source: B. Ontology and Feature Knowledge… · p. 3, Table I · p. 3
  • Executable features are evaluated on retrieval-only errors, and execution failures or residual misclassifications inform revised proposals.Each internal node retains its selected executable features and a description of the local decision; generated functions are fixed before target-user inference.Source: B. Ontology and Feature Knowledge… · p. 3

C. Static Population and Personalized Knowledge Stores

OntoAR-HAR keeps population and personal examples in separate, node-constrained stores, then combines retrieval evidence with numerical feature reports for local decisions.

C. Static Population and Personalized Knowledge Stores

  • The static store supplies source examples, while the per-user store holds accepted query windows and predictions, keeping personal evidence constrained to relevant labels and nodes.Static evidence remains visible as an anchor; personal examples cannot replace the source reference distribution.Source: C. Static Population and Personalized… · p. 4
  • For multiple sensor views, reciprocal rank fusion combines ranked lists, avoiding dependence on similarity scores sharing a common scale.The reported fusion constant is η = 60.Source: C. Static Population and Personalized… · p. 4
  • Retrieved examples define feature reference distributions, and the report gives query values, reference means and standard deviations, and standardized deviations.This lets the LLM compare explicit numerical evidence rather than calculate signal statistics from raw arrays.Source: C. Static Population and Personalized… · p. 4

Retrieval-based inference

  • At inference, fusion determines which examples enter the context, while the numerical report helps interpret them at the active ontology node.Source: Retrieval-based inference · p. 5

D. Recursive Inference and Memory Update

Recursive inference uses node-local evidence to choose a child, follows the ontology to a leaf, and selectively adds accepted observations to user memory.

D. Recursive Inference and Memory Update

A. Setup

The evaluation spans eight heterogeneous HAR datasets in cross-subject settings, with balanced inference samples and controlled comparisons of hierarchy and memory.

A. Setup

  • The eight datasets cover 4–19 activities, 6–30 channels, and waist, wrist, chest, ankle, arm, leg, and pocket sensor placements.Source: A. Setup · p. 5
  • Database and inference subjects are disjoint, and cross-subject inference subsets are sampled without replacement and balanced by activity and user.Accuracy measures leaf-label correctness; macro-F1 weights per-class F1 scores equally.Source: A. Setup · pp. 5–6
  • The semantic-inclusion evaluation reports dataset-level hierarchical F1 and overall metrics, with percentage changes expressed in points.Source: Table III · p. 6
  • The controlled component comparison uses cross-subject evaluation with Gemini 2.5 Flash to isolate decision structure and memory in that configuration.Source: A. Setup · p. 6, Figure 3 · p. 6
  • Dataset-specific segmentation is retained, including windows ranging from 2-second segments to WISDM's 10-second windows and DSADS's predefined 5-second windows.Source: A. Setup · p. 6

B. Cross-Subject Results and Component Analysis

OntoAR-HAR performs strongly in cross-subject evaluation, while component comparisons indicate that ontology-guided retrieval and personalized memory contribute complementary gains. Hierarchy also improves both activity recognition and the semantic proximity of remaining errors.

B. Cross-Subject Results and Component Analysis

  • 88.2 average accuracy and F1 for OntoAR-HAR with Gemini 3.0 Flash compare with 87.7/88.5 for UniMTS and 81.6/81.4 for ZARA.UniMTS uses full-shot optimization, unlike OntoAR-HAR’s retrieval-based inference without target-user updates; ZARA is the closer protocol match.Source: B. Cross-Subject Results and Component… · p. 6
  • Hierarchy is most useful where activities share a broader motion concept, as observed for Opportunity and Shoaib.The component comparison attributes distinct roles to the tree’s local comparisons and the user-specific examples available to them.Source: B. Cross-Subject Results and Component… · p. 7
  • Accuracy rises from 84.0 with single-level retrieval to 87.2 with hierarchy and 89.0 after adding personalized memory.Hierarchy helps most on closely related subclasses, while user examples complement static population evidence.Source: B. Cross-Subject Results and Component… · p. 7
  • Average Hierarchical F1 rises from 89.92 to 91.32, tree distance falls from 0.428 to 0.366, and cross-branch error falls from 5.18% to 4.96%.Accuracy also increases from 84.24 to 87.17, indicating gains in leaf recognition and semantic proximity of residual errors.Source: B. Cross-Subject Results and Component… · p. 7

C. Heterogeneous-Dataset Evaluation

The heterogeneous-dataset evaluation combines harmonized activity labels while retaining mixed sensor placements and collection protocols. OntoAR-HAR leads the reported comparison overall, though retrieval evidence remains affected by representation quality and source coverage.

C. Heterogeneous-Dataset Evaluation

  • The benchmark combines five datasets across six harmonized activity categories, with database and inference users kept disjoint and retrieval performed globally.The evaluation uses no body-location filtering.Source: C. Heterogeneous-Dataset Evaluation · pp. 7–8
  • The mixed placements and collection protocols make this a joint test of activity semantics and sensing heterogeneity.Matching labels across datasets does not ensure matching raw channels or nearest-neighbor geometry.Source: C. Heterogeneous-Dataset Evaluation · p. 8
  • 48.70 accuracy and 42.11 macro-F1 for OntoAR-HAR with Gemini 3.0 compare with 43.0/35.8 for UniMTS.OntoAR-HAR leads on four of five datasets; Mhealth illustrates that representation quality and source coverage still affect retrieved evidence.Source: C. Heterogeneous-Dataset Evaluation · p. 8

D. Backbone and Reasoning Effects

The backbone comparison reports increasing Gemini-family averages, and reasoning improves accuracy beyond retrieval-only ranking. Inference costs are dominated by sequential LLM decisions; an example traces how population, personal, and feature evidence contribute to one decision.

D. Backbone and Reasoning Effects

  • Gemini-family average Acc/F1 increases from 85.41/85.25 to 88.99/88.68 and 90.20/90.09 in the fixed-subset backbone comparison.DeepSeek R1, Grok 4, and GPT-5 confirm compatibility while also showing backbone sensitivity; this comparison does not replace Table II’s ten-sample estimates.Source: D. Backbone and Reasoning Effects · p. 8
  • Mean accuracy rises from 81.51 to 84.40 after LLM reasoning with the same retrieved evidence.The comparison does not separate general reasoning from model-specific evidence integration.Source: D. Backbone and Reasoning Effects · p. 8, Figure 5 · p. 8

E. Inference Cost and Reasoning Example

  • Retrieval takes 0.019–0.092 seconds per query, compared with 1.39–4.96 seconds for each LLM decision.Prompts contain 1.00K–2.71K tokens and cost USD 0.000768–0.002055; sequential LLM calls dominate latency, while each call is limited to the current node’s children.Source: E. Inference Cost and Reasoning Example · p. 8
  • In the illustrated path, a previously accepted sitting episode and subsequent feature comparisons shift the decision away from the static retrieval’s initial elevator-down preference.The case traces evidence from population examples, user-specific episodes, and numerical feature checks; it is an illustrative query, not dataset-level component evidence.Source: E. Inference Cost and Reasoning Example · pp. 8–9

F. Discussion

The discussion presents ontology structure, personalized memory, and executable features as complementary parts of an inspectable inference process. Future work targets scaling ontology construction and making personalization and feature selection more reliable.

F. Discussion

  • The ontology restricts comparisons through label relations, personalized observations change the available evidence, and executable features add signal-level evidence where needed.The LLM integrates bounded candidates, retrieved examples, and feature statistics rather than raw arrays or an unconstrained label list.Source: F. Discussion · p. 9

V. CONCLUSION AND FUTURE WORK

  • Across eight datasets, hierarchy and memory provide additive gains, and the heterogeneous-dataset experiment indicates the structure can help under mixed sensing conditions.The framework connects label-level semantics, source-domain sensor evidence, and current-user observations in recursive inference.Source: §V · p. 9
  • Future work targets automation beyond the current manual, confusion-guided ontology construction, along with more complete memory gates and improved feature selection.The stated aims include generating and validating intermediate concepts for larger vocabularies, modeling confidence and long-term drift, and adding broader feature libraries and robust fallbacks.Source: §V · p. 9
Loading 2610.15292v1…