PaperlayerView exact revision on arXiv

Source-linked AI summary

More Features Are Not More Evidence: Limits of Training-Free Human Activity Recognition with Jev

Orhan Konak

arXiv:2609.36154v1cs.AIcs.LG

TL;DR

It is unclear whether general-purpose models can interpret deterministic sensor descriptions well enough to replace or complement trained HAR models. The paper evaluates Jev without labeled examples, retrieval context, or HAR-specific updates, across three representations and three datasets. Jev remains far below supervised recognition, and the results indicate that recognition depends on the sensor-to-model interface.

  • Problem

    It remains unclear whether general-purpose models can interpret deterministic sensor descriptions well enough to replace or complement trained HAR models, a possible way to reduce dependence on labeled data.

    Source: §1 · p. 1

  • Method

    The study evaluates fixed Jev on deterministic wearable-motion representations for activity classes, without HAR-specific training, fine-tuning, labeled references, or retrieval.

    Source: §1 · p. 1, §2.3 · p. 3

  • Results

    Jev remains far below trained HAR classifiers; numerical features reduce recognition, semantic rendering partly recovers it, and the small WISDM fusion gain does not replicate elsewhere.

    Source: §1 · pp. 1–2, §8 · p. 11

  • Takeaways & Limitations

    The sensor-to-model interface should be evaluated as part of the model, rather than treated as neutral preprocessing.

    Source: §6 · p. 9, §8 · p. 11

    The evaluation uses class-balanced windows with clean activity boundaries, and WISDM threshold fitting includes the unlabeled evaluation population.

    Source: §7 · p. 11

Abstract

from arXiv · show

General-purpose models promise sensor-based decisions without training a task-specific classifier, which could reduce the dependence of Human Activity Recognition (HAR) on labeled data. Yet it remains unclear whether such models can directly interpret deterministic descriptions of physical sensor signals well enough to replace or complement trained HAR models. We study this question using Jev, a fixed general-purpose probabilistic decision model, on 1,800 class-balanced accelerometer windows from WISDM, UCI341, and PAMAP2. Jev receives no labeled examples, retrieval context, or HAR-specific parameter updates. We evaluate three deterministic sensor representations and compare 5,400 Jev decisions with a generative baseline and three supervised HAR models. Jev remains far below supervised recognition, with its strongest representation reaching macro-F1 of 0.038, 0.118, and 0.089 across the three datasets, compared with 0.686 to 0.907 for the supervised models. More numerical features do not improve Jev. Instead, they reduce recognition on all three datasets, while augmenting the same numerical evidence with a deterministic semantic rendering partially recovers performance, although the experiment does not isolate semantics from the accompanying serialization and redundancy changes. Jev is fast and inexpensive to query, but its probabilities are not reliably calibrated for recognition. A post-hoc fusion analysis finds a small improvement on WISDM that does not replicate on UCI341 or PAMAP2. These results show that training-free sensor decisions depend not only on the information available in the signal, but also on whether the model can use the representation through which that information is exposed. The sensor-to-model interface should therefore be treated as part of the model evaluation rather than as a neutral preprocessing step.

1 Introduction

The introduction asks whether a fixed general-purpose model can recognize activities from deterministic sensor descriptions without task-specific training. Jev performs far below supervised HAR models; changing the representation affects its recognition, motivating evaluation of the sensor-to-model interface itself.

1 Introduction

  • The study evaluates Jev as a fixed general-purpose probabilistic decision model on deterministic accelerometer descriptions, without HAR-specific training or labeled reference examples.The contribution also excludes retrieval and supplies a fixed task instruction and candidate activity set.Source: §1 · p. 1
  • Across three datasets, supervised classifiers remain far ahead of Jev and a generative baseline, while adding numerical features reduces Jev’s recognition and semantic rendering partially recovers it.The authors frame these findings as limits of the tested interfaces, not a universal law about feature counts or general-purpose models.Source: §1 · p. 1
  • Jev’s low observed API cost and subsecond median response time make repeated queries feasible, but poor recognition and imperfect probability validity limit their use.A small post-hoc fusion contribution improves an RF on WISDM but does not replicate on UCI341 or PAMAP2.Source: §1 · pp. 1–2
  • The paper treats the sensor-to-model interface as part of model evaluation, rather than as neutral preprocessing, because representations differ in how usable their information is to Jev.The study characterizes probability validity, calibration, selective prediction, operational measurements, and exploratory post-hoc fusion.Source: §1 · pp. 1–2

2 Related Work

Related work spans supervised HAR and methods that reduce dependence on labeled data, including learned representations, retrieval, and language-model approaches. This study instead examines whether a fixed structured decision model can use deterministic sensor descriptions without sensor-specific training or retrieved examples.

2 Related Work

  • The related-work discussion distinguishes conventional HAR, reduced-label approaches, training-free activity recognition, and general-purpose probabilistic decision models.The paper positions its question as whether a fixed decision model can use deterministic sensor-window descriptions for bounded activity choices.Source: §2 · p. 2, §2.3 · p. 3

2.1 Sensor-Based Human Activity Recognition

  • Prior HAR systems learn from task-specific data, while self-supervised and cross-dataset approaches reduce dependence on labels but still learn representations or adapt parameters.GOAT combines sensor representations with natural-language supervision for cross-dataset and zero-shot activity recognition.Source: §2.1 · p. 2

2.2 Foundation Models and Training-Free HAR

  • Foundation-model HAR includes tuned sensor-language models, learned sensor-language representations, zero-shot methods, and retrieval-based workflows, each retaining some learned or contextual component.Examples include SensorLLM [12], SensorLM [27], IMUZero, RAG-HAR [21], RAG-HAR+ [9], ZARA [11], and STELLA [22].Source: §2.2 · p. 2
  • Training-free HAR varies in whether systems learn sensor representations, use labeled references, or retrieve examples; public benchmarks may also have appeared in model pretraining [7].The study uses no trained target-task sensor encoder or classifier and provides no labeled examples at inference.Source: §2.2 · p. 2

2.3 System One and General-Purpose Decision Models

  • Jev is treated as a black-box structured decision model, and the study tests its activity recognition separately from whether its probabilities convey useful uncertainty.The manufacturer describes Jev as returning typed probabilistic decisions, but its architecture and training data are proprietary.Source: §2.3 · p. 3

3 Setting and Experimental Method

The study tests whether a fixed decision model can recognize activities from deterministic descriptions of five-second sensor windows, and separates representation, probability, and operational questions. Its protocol specifies the datasets, model inputs, baselines, and evaluation procedures used for that comparison.

3.1 Scope and Analysis Provenance

  • The pre-inference protocol was frozen before external-model queries, but was not formal preregistration because its commits lacked independent public timestamps.Additional confidence diagnostics were specified before inference; the hybrid analysis was designed after the main results and treated as exploratory.Source: §3.1 · p. 3

3.2 Datasets, Preprocessing, and Benchmark Selection

  • The matched benchmark uses 50 windows per native class (900 WISDM, 300 UCI341, 600 PAMAP2); UCI341 uses six activities re-windowed from raw recordings.Windows are five seconds with zero overlap; evaluation uses isolated activity segments, not continuous transition detection. Signals preserve gravity and orientation and are resampled to 20 Hz using dataset-specific procedures.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Representations: Information versus Accessibility

  • R1 contains 20 statistical features; R2 adds spectral, correlation, and magnitude features for 32 total, while R3 renders all 32 and appends complete R2 values.R3 qualifiers use unlabeled quantiles fitted on all WISDM windows and reused on the other datasets; the R3–R2 contrast also changes redundancy and serialization, so it does not isolate semantics.Source: §3.3 · pp. 4–5

3.4 Inference and Supervised References

  • Jev receives a fixed instruction, candidate activities, and each of three representations for 1,800 windows, totaling 5,400 requests; Claude Sonnet 5 and three supervised models serve as references.Claude provides labels rather than native class-probability vectors; supervised models use leave-one-subject-out evaluation, with the CNN trained on raw windows.Source: §3.4 · p. 5

3.5 Recognition, Probabilities, and Statistical Uncertainty

  • Recognition scores Jev’s native choices with macro-F1 primary; probability metrics use only valid vectors, and bootstrap intervals are descriptive for the fixed benchmark and responses.The probability analyses report Brier score, clipped negative log-likelihood, and ten-bin ECE; coverage is conditional on valid vectors, and no deployment confidence threshold is fitted.Source: §3.5 · pp. 5–6

3.6 Operational Measurements and Research Assistance

  • Latency measures request-to-response wall time at concurrency one, while token costs use execution-frozen list prices; reported AI research assistance is distinct from evaluated model outputs.Latency excludes sensor acquisition and preprocessing; the available scored summaries do not contain a 99th percentile.Source: §3.6 · p. 6

4 Main Results

Across the main results, Jev remains far behind supervised recognition, numerical enrichment worsens its predictions, and semantic rendering partly recovers performance. Its returned probabilities are not reliably calibrated, while low observed latency and cost do not establish deployment guarantees.

4.1 Standalone Recognition: A Large Supervised Gap

  • 0.0378 (WISDM), 0.1184 (UCI341), and 0.0893 (PAMAP2) were Jev R3 macro-F1 scores, versus RF’s 0.7057, 0.8899, and 0.8041.Even Jev’s strongest condition remained far below all three supervised references; the results do not support using Jev as a drop-in replacement under the evaluated conditions.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Representation: Numerical Enrichment Can Fail

  • R2 includes all R1 features, yet Jev macro-F1 decreases on every dataset and predictions collapse toward walking.R2 assigns walking to 899/900 WISDM, 300/300 UCI341, and 595/600 PAMAP2 windows. R3 partly reverses the loss, but changes presentation of existing information; its gains over R1 on PAMAP2 are not clear, and the mechanism remains unresolved.Source: §4.2 · p. 6

4.3 Probability Validity Does Not Imply Calibration

  • All 5,400 Jev choices are valid, but 86 probability vectors fail the frozen validity criterion, leaving 5,314 for probability evaluation.For R3, ECE is 0.2532, 0.3489, and 0.1294 on WISDM, UCI341, and PAMAP2, while corresponding argmax accuracies on valid vectors are 0.1135, 0.2405, and 0.1658; confidence cannot be read as a validated probability of correctness.Source: §4.3 · pp. 6–7, Table 3 · p. 8
  • R3 AURC values are 0.8600, 0.6306, and 0.8012; selective rejection can change error rates, but the full curves give no general operating guarantee.On WISDM, R1 has lower AURC (0.8408) than R3 despite lower full-coverage recognition; classification representation quality and confidence ranking need not improve together.Source: §4.3 · p. 8

4.4 Cost and Latency: Feasible Requests, Weak Decisions

  • Jev’s condition-level median latencies span 0.291–0.300 s and costs USD 0.035–0.071 per 1,000 windows, compared with Claude’s 1.440–1.558 s and USD 1.545–3.547.The slowest Jev response was 5.335 s, exceeding the five-second window duration; subsecond median latency does not establish deadline guarantees or streaming throughput under load.Source: §4.4 · pp. 8–9, Table 4 · p. 9

5 Post-Hoc Exploratory Probability Fusion

The post-hoc fusion study tests whether Jev probabilities complement RF on fixed complete-case subsets, using a frozen weight grid without retuning. A small WISDM gain does not replicate on the other datasets, and larger Jev weights hurt performance.

5 Post-Hoc Exploratory Probability Fusion

  • The protocol froze all 21 weights before hybrid scoring, with no further Jev inference, retuning, or fitted optimal weight.The analysis was informed by the completed main results and was not part of the pre-inference main evaluation protocol.Source: §5 · p. 9
  • Fusion used one complete-case subset per dataset: 890/900 WISDM, 291/300 UCI341, and 591/600 PAMAP2 windows.Excluded cases had invalid Jev probability vectors; endpoints reproduce RF and Jev probability argmax on those subsets, which need not match the all-case main results.Source: §5 · p. 9
  • At λ=0.20, WISDM gains 0.0070 [0.0024, 0.0125] macro-F1, while UCI341 changes −0.0103 [−0.0253, 0.0034] and PAMAP2 changes −0.0043 [−0.0125, 0.0042].The WISDM improvement is a net seven correct decisions among 890 windows; larger Jev weights substantially degrade all three datasets.Source: §5 · p. 9
  • The favorable WISDM pointwise interval does not validate a deployment weight or establish benefit on new subjects or datasets.The report presents the full curve rather than selecting a retrospectively favorable weight.Source: §5 · p. 9

6 Discussion

The discussion distinguishes information present in sensor descriptors from information a fixed decision model can use, and treats the sensor interface as part of evaluation. It also limits dataset explanations and fusion guidance to hypotheses and the tested setting.

6.1 The Sensor Interface Is Part of the Model Evaluation

  • The tested fixed decision interface fails to exploit much of the useful predictive structure that RF extracts from the same R2 features, while R3 partially recovers recognition.R3 changes how the same measurement-derived information is presented, but the experiment does not isolate semantic language from serialization and other changes.Source: §6.1 · pp. 9–10

6.2 Why Datasets Differ: Interpretations to Test

  • Figure 4’s fixed fusion curves show that increasing Jev weight eventually causes large losses; the small WISDM gain does not replicate on UCI341 or PAMAP2.The proposed explanations for dataset differences and WISDM’s corrections remain hypotheses, not established mechanisms.Source: Figure 4 · p. 10, §6.2 · p. 10
  • UCI341’s higher R3 macro-F1 may reflect its six basic activities and smartphone placement, but dataset characteristics vary together and cannot be isolated from raw cross-dataset scores.PAMAP2’s nine subjects limit interval stability; transferred semantic thresholds were not refit to test whether device or placement differences change their meaning.Source: §6.2 · p. 10

6.3 What HAR Researchers Can Use and Should Avoid

  • For recognition-quality requirements with labeled data, the results favor a task-trained sensor classifier over Jev as the sole recognizer or a trusted pseudo-label generator.Any fusion weight or rejection threshold requires independent validation before operational use; the WISDM weight is not supported for transfer to another dataset.Source: §6.3 · p. 11
  • Interface audits should inspect prediction distributions and errors rather than assume more features help, and real-time evaluation must include more than median API latency.The discussion also calls for preserving physical units and representation provenance and disclosing unlabeled target distributions, segmentation labels, and retrieved examples.Source: §6.3 · p. 11

7 Limitations, Ethics, and Reproducibility

The study’s conclusions are bounded by its tested models, datasets, interfaces, and sampling choices. It reuses public data, reports ethical scope and privacy cautions, and describes artifact provenance for reproducibility.

7 Limitations, Ethics, and Reproducibility

  • The evaluation covers one pinned Jev version, one generative baseline, three public datasets, three frozen interfaces, and one response per case; it does not establish results for future models or untested configurations.Proprietary pretraining data are unknown, so public benchmark contamination cannot be excluded [7].Source: §7 · p. 11
  • Class balancing, clean activity boundaries, five-second windows, and summary features constrain how the results represent everyday activity and temporal detail.WISDM threshold fitting includes unlabeled evaluation data, making that pipeline transductive; complete-case probability analyses may also be selection-biased.Source: §7 · p. 11
  • The study reuses public sensor datasets without recruiting participants or collecting new measurements, but derived motion features may still contain personal information.Omitting identifiers does not establish anonymity, and the analysis does not attempt identification or clinical decision making.Source: §7 · p. 11
  • Recorded model identifiers, serialized requests, dependencies, checksums, and validation manifests support traceability, and the reported results can be rebuilt from prepared artifacts without rerunning queries or tuning.Code, evaluation scripts, and derived artifacts were prepared for public release; the archival record is to be made available separately.Source: §7 · p. 11

8 Conclusion

In the evaluated setting, Jev does not replace trained HAR classifiers, and adding numerical features can reduce recognition. The conclusion treats the sensor-to-model interface as part of evaluation, not neutral preprocessing.

8 Conclusion

  • Jev is fast and inexpensive to query, but it does not replace trained HAR classifiers in the evaluated setting.The small WISDM fusion gain does not replicate on UCI341 or PAMAP2 and does not support a general fusion strategy.Source: §8 · p. 11
  • Adding numerical signal features can reduce recognition, while a deterministic semantic rendering of the same numerical evidence can partially recover it.The study does not isolate semantic rendering from accompanying serialization and redundancy changes.Source: §8 · p. 11
  • The results support evaluating the sensor-to-model interface rather than treating it as neutral preprocessing; more signal descriptors do not guarantee more usable evidence.Fast, inexpensive inference does not remove the need to validate how sensor measurements are represented and interpreted.Source: §8 · p. 11

A Complete Exploratory Fusion Results

The section presents exploratory fusion-weight results for WISDM, UCI341, and PAMAP2. Estimates are paired against each dataset’s fixed complete-case RF endpoint and are not confirmatory evidence or a selected model.

A Complete Exploratory Fusion Results

  • All estimates use one fixed complete-case subset per dataset and are paired against that subset’s RF endpoint.The intervals are post-hoc, exploratory, pointwise, and not multiplicity-adjusted; no model is selected.Source: A Complete Exploratory Fusion Results · p. 13
  • Tables 5–7 present all exploratory fusion weights and paired score differences for WISDM, UCI341, and PAMAP2, respectively.Scores and differences are proportions with 95% subject-cluster intervals.Source: Table 5 · p. 13, Table 6 · p. 13, Table 7 · p. 14

B Artifact Map and Audit Boundaries

The artifact map identifies repository-relative paths, while the source package provides materials to compile the manuscript. The package excludes the original sensor archives and does not include provider model weights.

B Artifact Map and Audit Boundaries

  • The source package supplies the material needed to compile the manuscript but does not redistribute original human sensor archives or provider model weights.Full hashes and provenance are in the accompanying artifact index and frozen manifests.Source: B Artifact Map and Audit Boundaries · p. 15, Table 8 · p. 15
  • The main probability audit contains reliability-bin counts and threshold-level coverage/risk values; paired comparisons, class metrics, and confusion matrices remain in frozen score artifacts.Source: B Artifact Map and Audit Boundaries · p. 15

B.1 Frozen Candidate Vocabulary

The frozen candidate vocabulary consists of dataset-specific strings supplied without definitions. Their spelling and case are part of the interface.

B.1 Frozen Candidate Vocabulary

  • WISDM’s supplied vocabulary includes walking, jogging, stairs, sitting, standing, typing, and additional everyday and hand activities.The listed strings also include teeth, soup, chips, pasta, drinking, sandwich, kicking, catch, dribbling, writing, clapping, and folding.Source: §B.1 · p. 15
  • UCI341’s vocabulary contains six uppercase activity labels, including walking variants, sitting, standing, and laying.Source: §B.1 · p. 15
  • PAMAP2’s supplied labels span postures, locomotion, cycling, household activities, and rope jumping.The list includes lying, sitting, standing, walking, running, cycling, Nordic walking, ascending and descending stairs, vacuum cleaning, ironing, and rope jumping.Source: §B.1 · p. 15

Related papers

Loading 2609.36154v1…