PaperlayerView exact revision on arXiv

Source-linked AI summary

CHEAR: Confusion-Hierarchy Evidence-Aware Reasoning for Cost-Efficient Human Activity Recognition

Xiangqing Lu, Tao Zhu, Mingxing Nie, Liming Chen

arXiv:2610.15298v1cs.AI

TL;DR

Wearable-sensor HAR systems can retrieve relevant examples and sensor knowledge, yet online inference still organizes which activities to compare and when to consult pair-specific features. CHEAR builds and validates a confusion hierarchy offline, then progressively retains candidates and accesses local knowledge only for remaining activity pairs. Across eight datasets it reports 86.79% mean cross-subject accuracy, alongside reduced candidate and token counts.

  • Problem

    In retrieval-augmented HAR, similar examples and pair-specific sensor features are available, but comparison order and knowledge access remain largely organized online as the activity vocabulary grows.

    Source: §1 · p. 3

  • Method

    CHEAR builds and validates a hierarchy from recurrent source-domain retrieval errors, then progressively retains candidates and consults pairwise feature knowledge only when both activities remain.

    Source: §1 · p. 3, §4.1 · p. 5, §4.3.3 · p. 7

  • Results

    Across eight datasets, CHEAR achieves 86.79% mean cross-subject accuracy, with 75.91% candidate reduction, 96.44% true-activity retention, and 85.15% fewer tokens than the variant without the hierarchy.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    Reproducible confusion evidence can narrow the comparison scope before the LLM reads local knowledge relevant to current candidates.

    Source: §7 · p. 11

    The hierarchy is derived only from reproducible source-domain evidence, and target test labels are excluded from construction, feature selection, and online decisions.

    Source: §3 · pp. 4–5

Abstract

from arXiv · show

Large language models (LLMs) offer a new approach to wearable-sensor-based human activity recognition. Retrieval-augmented generation (RAG) further introduces similar motion examples and discriminative sensor knowledge, enabling decisions grounded in observations relevant to the current query. More evidence, however, does not necessarily mean better-organized evidence. When online prompts and LLM agents primarily orchestrate inference, decisions about which activities to compare and when to invoke discriminative knowledge remain concentrated at inference time. As the activity vocabulary grows, the model must repeatedly process more candidates and associated knowledge, whereas pruning candidates too early risks excluding the true activity. We propose confusion-hierarchy evidence-aware reasoning (CHEAR), which moves part of this knowledge organization offline. CHEAR constructs and validates an activity hierarchy from recurrent cross-subject retrieval errors and builds corresponding activity-pair knowledge. At inference time, the hierarchy orders candidate comparisons; the LLM retains candidates progressively, and local discriminative knowledge is accessed only for the remaining activities. Across eight datasets, CHEAR achieves 86.79\% mean cross-subject accuracy, a 75.91\% candidate reduction rate, and 96.44\% true-activity retention. Per-sample token use is 85.15\% lower on average than in a variant without the hierarchical structure. Mean accuracy across eight cross-dataset transfer directions is 88.81\%. These results show that organizing candidates and discriminative knowledge around observed activity confusions reduces online LLM input while maintaining recognition performance.

Highlights

CHEAR organizes activity comparisons and discriminative sensor knowledge using observed confusions, moving part of that organization offline. Its hierarchy progressively narrows candidates before local evidence is consulted.

Highlights

  • The highlights frame CHEAR as confusion-guided motion-evidence organization, with offline hierarchies and local knowledge supporting on-demand evidence use for retained candidates.The reported outcome is accurate recognition with fewer candidates and lower language-model token use.Source: Highlights · p. 2

1. Introduction

  • Observed motion evidence can be retrieved for a query, but candidate-comparison order and decisions about when to use pair-specific features remain largely organized online.More activity classes bring more candidates and related knowledge to process, while premature pruning may remove the true activity.Source: §1 · p. 3
  • CHEAR constructs and validates an activity-confusion hierarchy from source-domain retrieval errors, then builds local feature knowledge for validated activity pairs.Online, the LLM retains a variable number of candidates at each hierarchy level and accesses pairwise knowledge only when both activities remain.Source: §1 · p. 3
  • 86.79% mean cross-subject accuracy across eight datasets accompanies 75.91% candidate reduction, 96.44% true-activity retention, and 85.15% fewer tokens than the variant without the hierarchical structure.Mean accuracy across eight cross-dataset transfer directions is 88.81%.Source: §1 · pp. 3–4

2. Related Work

Related work spans wearable-sensor representations, cross-modal semantics, direct LLM interpretation, and retrieval-augmented evidence. The design uses source-domain confusion evidence to form candidate groups and limits local feature reports to their registered activity pairs.

2. Related Work

  • Wearable HAR work spans general time-series representations, cross-modal alignment of motion with semantics, direct LLM sensor interpretation, and retrieval of query-relevant motion examples.Examples include MOMENT [4], ImageBind [7], HARGPT, RAG-HAR [2], and ZARA [3].Source: §2 · p. 4

3. Problem Definition and Design Principles

  • The hierarchy groups activities with reproducible above-expected confusion, retaining only groups with strictly positive modularity gains in both source evidence sets.Online, selected groups expand by one level; singleton activities remain unchanged.Source: §3 · pp. 4–5
  • The design derives the hierarchy from reproducible source-domain evidence, and pair-specific reports compare only their registered endpoints rather than acting as global votes.Target test labels are not used in hierarchy construction, feature selection, or online decisions; target windows enter as unlabeled queries after components are frozen.Source: §3 · pp. 4–5

4. Method

CHEAR prepares retrieval stores, a validated activity-confusion hierarchy, and pair-specific feature knowledge offline, then uses the hierarchy to narrow candidates and consults relevant evidence online.

4.1. Overview

  • Offline preparation freezes position-specific retrieval stores, an activity-confusion hierarchy, and local feature knowledge; online inference progressively retains candidates and reads reports only for surviving pairs.The LLM selects one label from the remaining activities using retrieval evidence and any available pair reports.Source: §4.1 · p. 5

4.2. Frozen Encoding and Position-Specific Retrieval

  • Position-specific retrieval summarizes hit counts, similarity quality, and sensor-position results so the LLM can compare candidate units at each hierarchy level.Activity scores sum raw cosine similarities across positions and rank retrieval results; they are not probabilities or group-confidence scores.Source: §4.2 · p. 6

4.3.1. Confusion Matrix and Null Model

  • The confusion matrix retains off-diagonal leave-one-subject-out retrieval errors, while positive excess confusion identifies directed errors beyond a null model preserving in- and outstrengths.A positive value means the observed confusion exceeds expectation, not merely that an error occurred.Source: §4.3.1 · p. 6

4.3.2. Candidate Merging and Independent Validation

  • Candidate groups are formed by repeatedly merging groups with the largest strictly positive modularity gain, stopping when no positive gain remains.Independent source validation may select only construction-generated paths with positive gain; recursive grouping imposes neither fixed depth nor binary-tree structure.Source: §4.3.2 · pp. 6–7

4.3.3. Registering Difficult Activity Pairs

  • Difficult activity pairs require positive bidirectional excess confusion in both construction and validation sets, with validation examples for both classes.Within-group pairs attach to their deepest shared group; cross-group pairs are stored as root-level lateral relationships.Source: §4.3.3 · p. 7

4.4. Local Feature Knowledge for Activity Pairs

  • For registered pairs, offline screening selects feature families and a budget; online inference compares target-window feature values with frozen source statistics and gives pair-local evidence to the LLM.Feature reports are enabled only when the selected budget outperforms a similarity-only baseline; otherwise, the pair uses retrieval evidence alone.Source: §4.4 · p. 7

4.5. Variable-Size Hierarchical Candidate Retention

  • The LLM retains a variable-size subset of root units, expands retained groups one level at a time, and proceeds until the remaining candidates are individual activities.The method does not force a fixed top-N or a single path; after expansion, pair reports compare only their endpoints, and a candidate is removed only when all direct comparisons favor its opponents.Source: §4.5 · p. 7

5. Experimental Setup

The experiments evaluate CHEAR on eight public motion datasets under cross-subject and cross-dataset protocols. Comparisons span prompting, pretrained recognition, and retrieval-augmented reasoning, using classification, candidate-structure, and cost metrics.

5.1. Datasets and Evaluation Protocol

  • Eight public motion datasets are evaluated, with the retrieval encoder, indexing rules, hierarchy, activity-pair registry, and feature budgets frozen before testing.Cross-dataset tests align shared activities and comparable body locations, transfer source-domain hierarchy and pair knowledge, and retrieve only from target-domain reference sets.Source: §5.1 · p. 8

5.2. Comparators and Metrics

  • Comparators cover direct prompting, pretrained recognition, and retrieval-augmented reasoning, including ZARA as the retrieval-based comparator.The main configuration and cross-dataset experiments use Gemini 3 Flash Preview; the multi-model study also tests GPT-4.1 mini, DeepSeek-V3.2, and Qwen3-Next-80B-A3B-Instruct.Source: §5.2 · p. 8
  • Evaluation uses accuracy and macro-F1, candidate counts, reduction and retention, token cost, and latency; cross-subject means average across eight datasets.Cross-dataset accuracy and macro-F1 are equally weighted across eight transfer directions, while candidate metrics aggregate query windows.Source: §5.2 · p. 8

6. Results

CHEAR outperforms the strongest tabulated prior method on mean cross-subject accuracy and macro-F1, and evaluates transfer, ablations, candidate retention, and processing latency. Its results also show that candidate reduction can coexist with high true-activity retention, though retention varies by dataset.

6.1. Cross-Subject Main Experiment

  • CHEAR reaches 86.79% mean accuracy and 86.89% mean macro-F1, versus ZARA with Gemini’s 81.6% accuracy and 81.4% macro-F1.Table 1 reports arithmetic means across eight datasets.Source: §6.1 · p. 8, Table 1 · p. 9
  • CHEAR accuracy exceeds ZARA with Gemini on seven of eight datasets, but Opportunity reverses the comparison (85.00% versus 92.5%).WISDM and USCHAD also show particularly large differences.Source: §6.1 · p. 8

6.3. Ablation Results and Token Cost

  • Ablations compare accuracy and tokens; without the hierarchy, all activities reach the final decision rather than being retained level by level.The no-hierarchy variant retains retrieval evidence and registered activity-pair knowledge.Source: §6.3 · p. 9

6.3.1. Accuracy

  • The full method’s 86.79% mean accuracy exceeds the no-feature variant by 3.31 points, the fixed-rule variant by 3.84 points, and the no-hierarchy variant by 1.40 points.These comparisons assess feature knowledge, LLM candidate selection, and hierarchical inference, respectively.Source: §6.3.1 · pp. 9–10, Table 4 · p. 10

6.2. Cross-Dataset Experiments

  • Across eight transfer directions, CHEAR reaches 88.81% mean accuracy versus 86.60% without transferred feature knowledge; it retains 98.24% of true activities with 69.55% mean candidate reduction.The transfer comparison covers four dataset pairs; candidate metrics are aggregated over query windows.Source: §6.2 · p. 9, Table 3 · p. 10

6.4. Does the Hierarchy Retain the True Activity?

  • Across eight datasets, CHEAR ends with 2.38 candidates, 75.91% reduction, and 96.44% retention; USCHAD retention is 90.42%, where upper-level routing removes some true activities.The paper reads candidate reduction together with true-activity retention.Source: §6.4 · p. 10

6.3.2. Token Use

  • 85.15% mean token savings versus the no-hierarchy variant spans 78.75%–90.68% across datasets; the full method reads pair reports after candidate reduction.Full-method token use ranges from 1,013.4 per sample on Shoaib to 12,143.1 on WISDM and depends on registered pairs, feature reports, and routing rounds.Source: Table 5 · p. 10, §6.3.2 · p. 10

6.5. Local Processing Latency

  • Each measured local operation takes less than one second per sample on every dataset, with retrieval taking 0.005–0.043 seconds.Report generation varies with registered candidate pairs, while feature extraction varies with sensor positions and available modalities.Source: §6.5 · p. 10

7. Conclusion

CHEAR uses stable source-domain activity confusions to order comparisons, then loads validated feature knowledge only for registered pairs that remain. Across eight datasets, it reduces candidates and token use while retaining most true activities.

7. Conclusion

  • Stable source-domain activity confusions determine which activities CHEAR compares first, and validated sensor-feature knowledge is loaded only for registered pairs that remain.The conclusion describes variable-size candidate retention narrowing the set level by level.Source: §7 · pp. 10–11
  • Across eight datasets, CHEAR achieves 86.79% mean accuracy with 2.38 final candidates, 75.91% candidate reduction, and 96.44% true-activity retention.It uses 85.15% fewer tokens on average than the variant without the hierarchical structure.Source: §7 · p. 11
  • Across eight cross-dataset directions, mean accuracy is 88.81%, compared with 86.60% without transferred feature knowledge.The conclusion frames reproducible confusion evidence as a way to narrow comparison scope before the LLM reads local knowledge.Source: §7 · p. 11

A. Dataset Descriptions

The experiments use eight public HAR datasets, with retrieval, hierarchy construction, and inference performed within each dataset’s label set.

A. Dataset Descriptions

  • Eight public HAR datasets are used; retrieval, hierarchy construction, and inference stay within each dataset’s label set.Table A.1 summarizes activity counts, sensor positions, and activity labels.Source: A. Dataset Descriptions · p. 11

B. Dataset Preprocessing

Preprocessing uses dataset-specific windows and query-subject selections, with UCIHAR and DSADS retaining their predefined windows and other signals divided into nonoverlapping windows.

B. Dataset Preprocessing

  • Opportunity uses 30-Hz data, subject 4 queries, two-second windows, and 50 query windows per class; Shoaib uses 50-Hz data, subjects 1 and 9, two-second windows, and 30 per class.Source: B. Dataset Preprocessing · p. 11
  • UCIHAR retains its public train/test split and uses 2.56-second windows with 50% overlap; 40 windows per class are sampled from test subjects.DSADS uses its predefined five-second windows.Source: B. Dataset Preprocessing · p. 11
  • PAMAP2 uses 100-Hz data, subjects 5 and 6, two-second windows, and 20 windows per class; USCHAD uses 100-Hz data, subjects 13 and 14, two-second windows, and 20 per class.Source: B. Dataset Preprocessing · p. 11
  • Mhealth uses 50-Hz data, subjects 1 and 6, two-second windows, and 20 windows per class; DSADS uses subjects 2 and 4 and 10 windows per class.DSADS uses its predefined five-second windows; all other signals are divided into nonoverlapping windows.Source: B. Dataset Preprocessing · p. 11
  • WISDM uses aligned 20-Hz smartwatch hand accelerometer and gyroscope data from 47 subjects, including eight query subjects, with ten-second windows and 16 windows per class.Subjects without alignment anomalies are retained.Source: B. Dataset Preprocessing · p. 11

C. Cross-Dataset Preprocessing

Cross-dataset transfer uses aligned activity labels and transfers only the hierarchy and activity-pair feature knowledge, while queries retrieve from target-domain references. The four dataset pairings use specified sampling and, where noted, downsampling settings.

C. Cross-Dataset Preprocessing

D. Predefined Features

Predefined activity-pair knowledge uses feature families computed from sensor signals, with model-specific selection based on importance and validation.

D. Predefined Features

E. Per-Class Metrics

The appendix reports per-activity accuracy for the full method across eight datasets and records the authors’ contributions.

E. Per-Class Metrics

  • Fig. E.1 presents per-activity accuracy of the full method across eight datasets.Source: Figure E.1 · p. 13

CRediT authorship contribution statement

Loading 2610.15298v1…