Neuro-Symbolic Graph Transformers for Hierarchical Intention Recognition from Multimodal Human Activity Sequences

1.2 Problem Statement and Contributions

• Multimodal benchmarks often provide flat action labels, while richly hierarchical datasets are often unimodal, small, or domain-limited, making deep intention reasoning difficult to evaluate.This motivates constructing NTU-Hierarchy as a four-level extension of NTU RGB+D.Source: §1.2 · p. 3

1.1 Motivation and Context

• The proposed framework combines multimodal perception with graph-structured attention and differentiable FOL constraints to promote consistency across the behavioral hierarchy.The constraints penalize high-level intentions not licensed by constituent lower-level activities and actions.Source: §1.2 · p. 3, §1.1 · p. 3

1.2 Problem Statement and Contributions

• The NTU-Hierarchy dataset maps atomic actions to activities, Low-Level Intentions, and High-Level Intentions for hierarchical intention reasoning.The benchmark is synthesized from the multimodal NTU RGB+D corpus.Source: §1.2 · pp. 3–4

2 Related Work

• The reported novelty is integrating learnable ontology-derived attention biases with a hierarchical differentiable-logic regularizer in a multimodal transformer for multi-level intention recognition.Differentiable logic itself has precedents in semantic loss functions and Logic Tensor Networks [13].Source: §2 · p. 4

2.1 Multimodal Action and Activity Recognition

• Multimodal recognition has expanded beyond RGB to skeletal, infrared, and depth streams, but most models still target short-term atomic actions.The cited discussion also notes multimodal fusion work such as MMAct.Source: §2.1 · p. 5

2.2 Hierarchical Activity and Intention Understanding

• Prior hierarchical approaches often use staged reasoning or shallow action-to-activity mappings, while existing intention benchmarks remain domain-specific or lack a four-level hierarchy.Examples include Breakfast, pedestrian benchmarks JAAD [22] and PIE, and industrial MCV-Intention.Source: §2.2 · p. 5

2.3 Neuro-Symbolic AI in Sequence Modeling

• Neuro-symbolic methods have largely focused on static VQA or simple spatial reasoning; embedding differentiable FOL in a graph transformer addresses its underexplored use with heterogeneous sequential sensors.The loss penalizes hierarchically inconsistent predictions across continuous temporal sequences.Source: §2.3 · p. 6

3.1 Problem Formulation

• The model jointly predicts atomic actions, composite activities, Low-Level Intentions, and High-Level Intentions subject to Human Reasoning Graph-derived consistency constraints.The input is a sequence of multimodal video features.Source: §3.1 · pp. 6–7

3.2 Hierarchical Transformer Backbone

• A Transformer backbone shares temporal representations across four prediction heads, with intention heads chained through continuous projections rather than discrete earlier predictions.The activity head branches directly from the pooled episode representation, while LLI and HLI predictions use successive continuous projections.Source: §3.2 · pp. 7–8

3.3 Neuro-Symbolic Graph Transformer (NSGT)

• The NSGT adds a learnable Human Reasoning Graph bias to attention logits, encoding semantic and ontological relations between action classes and hierarchical goals.Biases are built from predicted action identities and encode transition, activity-membership, and LLI-reachability relations; the edge-type weights are learned, but the graph topology is fixed.Source: §3.3 · p. 8, HR-G Attention Bias · pp. 8–9

3.4 Differentiable First-Order Logic Loss

• Differentiable FOL penalties encode activity-to-LLI implications, LLI-to-HLI implications, and mutually exclusive activity constraints to penalize ontological violations.Soft probability-based violations provide continuous gradient signals for training.Source: §3.4 · pp. 9–10
• The training objective combines supervised losses for all four heads with a weighted FOL regularizer that penalizes inconsistent joint predictions.The logic term is weighted relative to the classification objective by λFOL.Source: §3.4 · p. 10

Head dependencies and error propagation.

• Only the HR-G bias reintroduces a discrete prediction, using first-pass action identities; an action error can therefore activate incorrect graph edges for that clip.The additive bias is bounded by clamping, while the logic objective penalizes higher-level predictions unsupported by lower levels during training.Source: Head dependencies and error propagation · pp. 11–12

4.1 Hierarchy Construction and Ontology Design

• The benchmark maps NTU RGB+D 120 actions into a four-level ontology and synthesizes temporally ordered, multi-clip episodes for hierarchical intention reasoning.Training episodes use 4–17 clips; all clips in an episode come from one subject, but assembled clips are not naturally recorded continuous trajectories.Source: §4 · p. 12, §4.1 · pp. 12–13

4.2 Multi-Membership and Disjunctive Rule Design

• One-to-many mappings preserve context-dependent pathways, so the FOL rule rewards any valid parent intention rather than requiring a single deterministic mapping.For an activity with multiple valid LLIs, the rule checks the maximum predicted probability across those valid consequents; direct action-to-HLI lookup is therefore non-unique.Source: §4.2 · pp. 13–14

4.3 Human Reasoning Graph and FOL Rule Construction

• The HR-G transition graph generates temporally ordered episodes, while its FOL rules constrain episode-level predictions during training.The training loss applies 487 active constraints: 39 activity-to-LLI implications, 28 LLI-to-HLI implications, and 420 mutual-exclusion constraints.Source: §4.3 · pp. 14, 16
• Label assignment precedes training and the FOL loss reads model outputs, but both encode the same ontology, limiting what higher hierarchy consistency can establish.The post-hoc hierarchy-consistency rate is aligned with the FOL objective and is reported as a cross-level coherence diagnostic, not independent validation.Source: §4.3 · p. 15
• Mutual-exclusion penalties apply only to activity pairs with entirely disjoint valid LLI sets, avoiding penalties for pairs that share valid intention pathways.The synthesized label sets satisfy the Type-C constraints by construction because each episode has one HLI shared by its activities.Source: §4.3 · p. 16
• The attention bias is built from first-pass predicted action identities rather than ground-truth labels, and replacing evaluation labels with random labels leaves model logits identical.The HR-G bias is added to attention logits in a second pass; its action-indexed construction can activate incorrect edges when the first-pass action prediction is wrong.Source: §4.3 · pp. 16–17

5 Experimental Setup

• All conditions use the same frozen-encoder features, subject-disjoint splits, and training protocol, but the baselines are controlled implementations rather than fully optimized family representatives.No equal per-model hyperparameter search was performed.Source: §5 · p. 18

5.1 Implementation Details

• Training combines cross-entropy across four prediction heads with a soft FOL regularizer, whose weight is evaluated at λ_FOL = 0.1 and 0.5.The three FOL rule families receive equal weights, and early stopping selects the checkpoint with the highest validation LLI macro-F1.Source: §5.1 · p. 19

5.2 Feature Extraction

• Each clip uses concatenated 5120-dimensional features from skeleton, RGB, infrared, and depth encoders, all frozen during training.The modality ablation and frozen features keep comparisons focused on the reasoning module; fine-tuning was not evaluated.Source: §5.2 · p. 20

5.3 Baselines

• The NSGT is compared with MLP, bidirectional LSTM, sequence Transformer, and R-GCN baselines, plus a Hierarchical Transformer ablation with HR-G bias removed.The HT ablation is evaluated both with and without FOL; models share the same four-level prediction schema.Source: §5.3 · pp. 20–21

5.4 Evaluation Metrics

• Evaluation reports top-1 accuracy and macro-F1 at each hierarchy level, plus action top-5 accuracy, joint accuracy, and hierarchy consistency.Joint accuracy requires the HLI to be correct and the predicted activity and LLI sets to match exactly; hierarchy consistency checks valid parent–child relations.Source: §5.4 · p. 22

Overall ranking.

• NSGT+FOL matches LSTM on action macro-F1 (78.30% vs. 78.07%) and improves activity, LLI, HLI, and hierarchy consistency over LSTM.Hierarchy consistency is 78.79% vs. 71.16%; the graph-structured model also exceeds the hierarchical transformer in hierarchy consistency with and without FOL.Source: §6.1 · pp. 23–24, Overall ranking · pp. 24–25, Table 5 · p. 25

Contextual positioning.

• NSGT reaches 87.37% top-1 action accuracy while predicting four hierarchy levels; its comparison with flat-label systems is indicative, not equivalent.Top-1 accuracy is 87.32% without FOL and 87.37% with it; the paper attributes the gap to specialized flat-label systems to the multi-task constraint.Source: Contextual positioning · p. 25, Interpretation of action-level results · p. 25, Table 6 · p. 26

6.2 Lookup-Table Baseline

• The lookup baseline exceeds the model by 1.44 pp at activity, but the model leads it by 38.18 pp at LLI and 50.53 pp at HLI.Lookup reaches 27.88% HLI macro-F1; its ontology mapping retains multiple valid parents at lower levels, but needs a resolution rule for one-to-many intention mappings.Source: §6.2 · pp. 26–27, Table 7 · p. 27

6.3 Generalisation to Unseen Compositions

• On the compositional split, NSGT + FOL (λ = 0.5) achieves the highest hierarchy consistency (72.15%) and joint accuracy (13.63%) among the evaluated configurations.FOL raises consistency from 68.84% to 72.15% on this split, compared with 76.40% to 78.79% on the standard split; the paper cautions that consistency alone is not independent evidence of general reasoning.Source: §6.3 · pp. 27–28, Table 8 · p. 27

6.4 Ablation Study: The Role of Joint Training

• Adding lower-level supervision raises LLI and HLI F1 over intention-only supervision, while the full model retains action F1 comparable to action-only supervision.FOL improves hierarchy consistency by approximately 2.6 points on the HT backbone; at λ=0.5, it raises consistency by +2.39 pp while per-level classification metrics change by less than 0.4 pp.Source: Joint training benefits the intention… · p. 28, No action-level cost from joint training · p. 29, FOL effect on hierarchy consistency · p. 29

Where the graph bias contributes.

• At matched logic settings, graph-structured attention improves LLI F1, HLI F1, and hierarchy consistency over HT+FOL while leaving action F1 essentially unchanged.The learned LLI co-membership bias is dominant (βlli ≈ 0.60); in the matched-density random-graph control it falls from 0.601 ± 0.011 to 0.132 ± 0.055, while performance degrades gracefully.Source: Where the graph bias contributes · pp. 29–30, Analysis of learned β parameters · pp. 30–31, Table 10 · p. 31

6.5 Error Structure under Partial Supervision

• Adding lower-level heads raises mean diagonal HLI recall from 69.2 percent to 78.7 percent under the full objective, with residual errors among ontologically adjacent classes.The comparison uses configurations with an actively supervised HLI head; the lookup baseline separately shows that lower-level outputs alone do not recover high-level intention.Source: §6.5 · p. 29, Figure 5 · p. 30

6.8 Modality Ablation

• Restricting input to skeleton reduces action F1 by 38.55 pp and hierarchy consistency by 14.70 pp; RGB-only reaches 70.43% action F1 and 79.36% activity F1.Depth-only and infrared-only reach 74.37% and 68.36% action top-1, respectively, while the strongest unimodal configuration, RGB, trails the full model by 7.77 pp in action top-1 and 3.92 pp in HLI macro-F1.Source: Skeleton as the primary kinematic signal · p. 33, RGB as a complementary appearance signal · p. 34, Table 12 · p. 34

Non-redundant fusion.

• 64.09% hierarchy consistency in the skeleton-only condition is the weakest result, indicating that appearance features support ontological coherence beyond kinematic discrimination.No single stream dominates the fusion, and the full four-modality model outperforms every unimodal condition at every hierarchy level.Source: Non-redundant fusion · p. 34
• Residual errors shared across modality conditions involve semantically adjacent episodes, including physical exercise confused with morning routine, reflecting overlapping transitional sub-intentions.Full fusion attains the highest recall on context-dependent classes, while RGB resolves errors concentrated in those classes under skeleton-only input.Source: Non-redundant fusion · p. 34, Figure 7 · p. 35
• 71.5% meal-consumption recognition and 59.0% return-home recognition are reached with full fusion, versus 70.7% and 53.6% with RGB alone.Skeleton-only recognition is 33.5% for meal consumption and 25.2% for return-home; these classes depend on scene and object context rather than body pose.Source: Figure 7 · p. 35

7.1 Consistency Gains at No Accuracy Cost

• +3.31 pp for NSGT and +1.63 pp for HT on the compositional split, compared with +2.39 pp and +2.60 pp on the standard test split.At λFOL = 0.5, consistency gains are statistically significant on NSGT (p = 0.009), while per-level macro-F1 changes are within seed variation; the consistency diagnostic is ontology-aligned with the loss, and the aggregate ablation does not identify the interaction mechanism.Source: §7.1 · pp. 35–36

7.2 Progressive Architectural Gains

• 64.35% action F1 for MLP rises to 78.07% for LSTM and 78.15% for HT, while the relational-graph baseline trails MLP at action, LLI, and HLI levels.The discussion attributes the progression to temporal modeling, global attention, graph-structured attention, and logic regularization, with structural knowledge alone insufficient for coherent predictions.Source: §7.2 · p. 36
• 78.79% hierarchy consistency: NSGT+FOL (λ=0.5) improves by 7.63 points over LSTM and 1.77 points over HT+FOL, with the suite’s best activity, LLI, and HLI macro-F1.The FOL loss adds consistent hierarchy-consistency gains on both backbones: +2.60 points for HT and +2.39 points for NSGT.Source: §7.2 · p. 36

A worked audit-trail example.

• In the held-out eight-clip audit episode, all actions are predicted correctly and the chain check passes at both parent conditions, yielding a joint exact match.DigitalTasks, whose only parent is OfficeWorkSession, resolves the competing OfficeWorkSession and ReturnHome intentions.Source: A worked audit-trail example · p. 37

7.4 Training Convergence

• Validation macro-F1 remains stable while validation loss rises later in training, and model selection uses the epoch with the highest validation LLI macro-F1.The discussion describes the loss rise as calibration drift rather than an accuracy collapse.Source: §7.4 · p. 37

7.5 The HLI Ceiling; Semantic overlap as a deliberate benchmark property.

• 76–78% HLI macro-F1, versus 12.5% chance for eight classes, is paired with residual errors concentrated in classes sharing transitional intentions.The discussion presents semantic overlap as a plausible explanation, not a proven cause of the performance ceiling; one episode alone may not distinguish intentions that share transitional behavior.Source: §7.5 · p. 37, Semantic overlap as a deliberate… · p. 39

7.6 Synthesized Hierarchical Labels

• The four-level labels are synthesized from NTU RGB+D 120 because no comparably broad multimodal hierarchical intention dataset existed, and they encode the designers’ ontology.The FOL constraints derive from the same ontology, so they are not independent semantic validation; independently human-annotated intention labels remain a future validation step.Source: §7.6 · p. 40

7.7 Graph Architecture and Temporal Scale; 7.8 Multimodal Fusion Strategy

• The benchmark uses curated episodes assembled from staged laboratory recordings of single performers, and HR-G transition structure shows its full benefit on longer sequences spanning multiple activities and LLIs.Early fusion concatenates four modality feature vectors into one 5120-dimensional representation before projection; full fusion outperforms skeleton-only and RGB-only input.Source: §7.6 · p. 40, §7.7 · p. 41, §7.8 · p. 41

8 Conclusion and Future Work

• Ontology-structured attention improves hierarchy consistency by +1.98 points over uniform-attention without logic and +1.77 points with logic, while yielding the best intention-level macro-F1.Action-level discrimination remains comparable across the transformer models.Source: §8 · p. 42
• FOL loss improves hierarchy consistency by +2.60 points for HT and +2.39 points for NSGT, with gains significant across seeds and per-level accuracy unchanged within seed variation.On the compositional split, the full model achieves the suite’s highest hierarchy consistency and joint accuracy.Source: §8 · p. 42
• The benchmark provides subject-disjoint and compositional evaluation with 487 active ontology constraints, but uses designer-defined labels without independently annotated real intentions.Source: §8 · p. 42
• The framework supports controlled study of predictive accuracy and ontology-aligned coherence, but does not establish operational safety in real robotic deployment.Source: §8 · p. 42

Longer sequences and richer hierarchies.

• Episodes average 7.04 clips (7.36 on the compositional split); proposed tests include 10–20-clip HLI sequences and five- or six-level ontologies.Longer episodes would test transition structure at longer horizons; deeper annotations would test whether FOL constraints generalize to richer hierarchies.Source: Longer sequences and richer hierarchies · pp. 42–43

Anticipatory goal inference.

• Future work targets anticipatory assistance by coupling predictions of future goals and actions with robotic planning, alongside user-study validation of human trust.Source: Anticipatory goal inference · p. 43

Online and streaming inference.

• The current framework processes fixed-length episodes offline; online use motivates streaming inference with causal attention and incremental FOL checking.Source: Online and streaming inference · p. 43

• The research was carried out within the LISSI Laboratory at the University of Paris-Est Créteil.Source: Statements and Declarations · p. 43
• The authors declare no financial or non-financial competing interests related to the submitted work.Source: Statements and Declarations · p. 43
• The study involved no human participants, animals, or personal data collection, and used NTU RGB+D 120, collected under its original authors’ institutional ethics approval.Source: Statements and Declarations · p. 43
• The synthesized benchmark and NSGT implementation code are stated to be made available upon manuscript acceptance.The listed benchmark materials include episode manifests, ontology mappings, and FOL rule matrices.Source: Statements and Declarations · p. 43
• F. Soleimani is credited with conceptualization, methodology, software, formal analysis, investigation, and the original draft; the other authors are credited with supervision and/or writing review.Source: Statements and Declarations · p. 43

• The comparison uses R-GCN, Seq-Transformer, and NSGT without the logic objective to isolate the graph-structured architecture’s contribution.Source: Appendix A · p. 44
• The matrices show single-level high-level-intention errors; the logic objective’s effect is instead reported through cross-level hierarchy consistency.Source: Appendix A · p. 44
• Error magnitude decreases from the relational-graph baseline to the sequential baseline to the graph-structured architecture.Source: Figure A1 · p. 44

• Medical episode, friendly social, and hostile social are recognized at 96.5%, 99.7%, and 100.0%, respectively, by the proposed model.Source: The confused classes share the same… · p. 44
• Errors concentrate on morning routine, office work, physical exercise, and return-home, whose low-level intentions include shared transitions such as walking transit.Source: The confused classes share the same… · p. 44
• Walking transit belongs to four high-level intentions, whereas the cleanly recognized classes contain distinctive sub-intentions with a single natural parent.Source: The confused classes share the same… · pp. 44–45

Kinematic symmetry compounds the ontological overlap.

• Return-home and morning routine share transitional motor patterns, including walking, handling outerwear, and picking up or putting down objects.Both also draw on walking-transit and phone-check intentions.Source: Kinematic symmetry compounds the… · p. 45
• Because the kinematics are close to identical, separating these episodes depends on static scene and object context rather than body pose.Source: Kinematic symmetry compounds the… · p. 45
• The four transitional classes remain confused through the joint effect of a many-to-one ontology and symmetric kinematic input.Source: Kinematic symmetry compounds the… · p. 45

The baselines fail on this overlap in a characteristic way.

• 80% of return-home episodes are routed to morning routine by the relational-graph baseline, which also routes over half of physical-exercise episodes there.These pairs share transitional intentions but are otherwise distinct.Source: The baselines fail on this overlap in… · p. 45
• Static message passing over ontological edges leaves the relational-graph baseline without temporal evidence to distinguish episodes with overlapping atomic actions.Its errors consequently spill across ontologically distant classes.Source: The baselines fail on this overlap in… · p. 45
• The Seq-Transformer removes most diffuse errors but still recognises physical exercise and return-home below 50%, concentrating residual errors on transitional classes.It models action sequence but has no relational prior.Source: The baselines fail on this overlap in… · p. 45

The residual limitation of the proposed model is a representational boundary, not an architectural one.

• The graph-structured architecture recognises hostile and friendly social episodes at or above 99.7%.The passage attributes this recovery to distinctions carried by the intention structure.Source: The residual limitation of the… · p. 45
• Remaining errors lie on transitional classes whose separation depends on scene context absent from pooled clip-level video and skeleton features.The representation is underdetermined when high-level intentions share physical sub-actions and differ only by context.Source: The residual limitation of the… · p. 45
• Explicit scene-context priors are identified as a concrete direction for improving recognition of the transitional classes.Source: The residual limitation of the… · p. 45