PaperlayerView exact revision on arXiv

Source-linked AI summary

Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior

Farnaz Soleimani, Abdelghani Chibani, Yacine Amirat, Ghazaleh Khodabandelou

arXiv:2609.17064v1cs.AIcs.CVcs.HCcs.LGcs.NE

TL;DR

Assistive systems must infer human goals from partial observations, while existing anticipation often lacks hierarchical targets and explicit ontological constraints. A compact Hierarchical Planning Decoder attached to a frozen neuro-symbolic encoder predicts future actions, remaining activities and LLIs, and the episode HLI using soft logic losses and hard reachability masks. The final system improved next-action top-5 accuracy over the strongest sequential baseline, reached strong HLI inference at half-episode observation, and eliminated HLI-reachability violations.

  • Problem

    Assistive systems must infer human goals from partial observations, while existing anticipation often lacks hierarchical targets and explicit ontological constraints.

  • Method

    A compact Hierarchical Planning Decoder attached to a frozen neuro-symbolic encoder predicts future actions, remaining activities and LLIs, and the episode HLI using soft logic losses and hard reachability masks.

  • Results

    The final system improved next-action top-5 accuracy over the strongest sequential baseline, reached strong HLI inference at half-episode observation, and eliminated HLI-reachability violations.

  • Takeaways & Limitations

    Neural prediction and symbolic constraints together support coherent hierarchical anticipation across longer horizons while preserving ontological validity.

  • Takeaways & Limitations

    HLI top-1 drops by roughly 24 points on the held-out compositional split, and the absolute numbers do not transfer directly.

Abstract

from arXiv · show

Assistive autonomous systems must anticipate human goals before an observed behavior is complete. This article formulates anticipation as goal inference from a partially observed multimodal episode together with structured prediction of the remaining behavior, rather than exact motor forecasting. A compact Hierarchical Planning Decoder (HPD) is attached to a frozen neuro-symbolic recognition encoder and predicts, at four ontological levels, the next actions, the remaining activities and low-level intentions, and the episode high-level intention(HLI). The decoder is trained with soft neuro-symbolic regularization combining transition-coherence and hierarchical continuity losses, and is decoded with hard reachability masks that enforce ontological validity at inference. On a compositional four-level benchmark of 15,002 multimodal episodes built over NTU RGB+D 120 features, three headline properties are observed together. The advantage over the strongest sequential baseline grows with the anticipation horizon, from +1.7 points at step 1 to +7.3 points at step 3 (top-5). Under compositional generalization, where one parent association per multi-parent low level intention is held out, this advantage widens to +4.9 points at step 1. At the episode level, 96.8% of anticipated trajectories satisfy the joint logic constraints, above the 88.1% strongest-baseline value and the 73.9% ground-truth floor; soft logic terms alone account for a 59.8 to 71.1% relative reduction of HLI-reachability violations, and the hard masks then eliminate them entirely. Neural generation supplies predictive ranking, symbolic constraints supply onto logical validity, and their combination yields coherent hierarchical anticipation while exposing remaining challenges in compositional goal generalization and unordered set prediction.

1. Introduction

The paper reframes anticipation as goal inference from partial multimodal observations plus structured prediction of remaining hierarchical behavior. It combines a compact decoder with soft and hard neuro-symbolic constraints, improving long-horizon and compositional anticipation while enforcing ontological validity.

  • Motivation: The work addresses limitations of flat anticipation outputs and logically unconstrained forecasts that can contradict ontological axioms.The motivation is especially strong for compositional behavior, where goals may admit multiple valid orderings.
  • Method: The Hierarchical Planning Decoder extends a frozen neuro-symbolic recognition encoder with four-level structured generation.Training uses transition-coherence and hierarchical-continuity losses, while inference applies hard reachability masks.
  • Problem Formulation: Anticipation is formulated as partial-observation goal inference with prediction of future actions, remaining activities, remaining LLIs, and the episode HLI.The formulation targets structured remaining behavior rather than exact motor-level forecasting.
  • Contributions and Results: +7.3 points at step 3 (top-5) over the strongest sequential baseline, up from +1.7 points at step 1; the advantage reaches +4.9 points at step 1 on the held-out compositional split.The benchmark contains 15,002 multimodal episodes.
  • Contributions and Results: 96.8% of anticipated trajectories satisfy joint logic constraints, versus 88.1% for the strongest baseline and 73.9% for the ground-truth floor.Soft logic terms reduce HLI-reachability violations by 59.8 to 71.1% before hard decoding masks eliminate them.

2. Related Work

Prior anticipation systems increasingly use transformers, intention conditioning, and language models, but commonly lack explicit hierarchical semantics and symbolic validity constraints. This work positions itself between action anticipation and neuro-symbolic planning by generating constrained four-level futures.

  • Anticipation Approaches: Transformer and multimodal methods improved temporal aggregation, while LLM-based systems introduced commonsense priors and explicit intention conditioning.These approaches span causal attention, multimodal fusion, scenario variables, and language-based sequence generation.
  • Limitations of Prior Anticipation: Existing anticipation methods typically predict flat labels or a single latent scenario rather than an explicit multi-level ontology connecting actions to long-term goals.This limitation recurs across short-term, long-term, and intention-conditioned forecasting.
  • Limitations of Prior Anticipation: Generative decoders can produce fluent sequences that remain unconstrained by symbolic axioms and therefore ontologically inconsistent.The cited gap concerns both language-model anticipation and broader sequence decoders.
  • Neuro-Symbolic Sequence Reasoning: Neuro-symbolic work has improved logic consistency and explainability, but prior methods did not jointly combine multimodal fusion, four-level hierarchy, and constrained anticipatory generation.The companion recognition work supplies the first two components, while this paper extends the framework to anticipatory generation.
  • Positioning of the Present Work: The present framework keeps the decoder data-driven while requiring generated futures to satisfy explicit ontological constraints during training and inference.Its positioning is between action anticipation and neuro-symbolic planning.

3. Methodology

The methodology freezes a graph-based neuro-symbolic encoder and trains a compact Hierarchical Planning Decoder on its prefix representations. The decoder predicts ordered actions and hierarchical set and intention targets, using soft coherence losses during training and hard reachability masks at inference.

  • Frozen Recognition Encoder: A frozen graph NSGT encoder supplies per-clip memory and a pooled HLI state, while the HPD is trained on top without changing the encoder representations.Freezing preserves fixed attribution and enables reuse of the encoder for downstream tasks.
  • Decoder Inputs: The decoder fuses ordered encoder states, observed action labels, and an unordered action-count histogram before producing its conditioning summary.The protocol supplies observed labels to match the strongest label-oracle baselines rather than performing raw video-only anticipation.
  • Decoder Outputs: Four output heads generate up to K = 3 future actions with EOS, the remaining-activity set, the remaining-LLI set, and the episode HLI.The action stream is autoregressive, whereas the remaining-set and HLI heads read the pooled decoder state.
  • Hard Constrained Decoding: Inference-time reachability masks enforce ontological validity for action and set outputs, while transition and canonical-order constraints remain soft or diagnostic.Reachability takes precedence when it conflicts with transition support.
  • Soft Neuro-Symbolic Regularization: Differentiable transition-coherence and hierarchical-continuity losses bias training toward coherent futures and continuity between inferred goals and predictions.Transition support is used as a soft preference because valid test trajectories may contain unseen transitions.

4. Experimental Setup

The study evaluates hierarchical anticipation on a compositional four-level benchmark, using multimodal episode prefixes to predict future actions, remaining sets, and episode intentions. It combines a frozen graph-based recognition encoder with controlled baselines, subject-disjoint splits, and trajectory-level logic audits.

  • Dataset and ontology: 15,002 episodes organize behavior into 8 HLIs, 35 LLIs, 50 activities, and 85 source action classes, with 81 effective action classes evaluated.Episodes average 6.95 actions and use four-level compositional structure.
  • Inputs and models: The benchmark fuses skeleton, RGB, infrared, and depth features into 5120-dimensional vectors, with identical frozen features across compared models.The four modalities contribute 2048-, 1024-, 1024-, and 1024-dimensional features, respectively.
  • Dataset and ontology: The compositional test split withholds one parent association for each multi-parent LLI, requiring transfer to an unseen high-level context.The split contains 3,415 episodes, alongside 7,062 train, 1,519 validation, and 3,006 standard-test episodes.
  • Task definition: Each episode prefix is used to predict up to K = 3 ordered next actions, remaining activities, remaining LLIs, and the episode HLI.An EOS flag handles episodes with fewer than three remaining actions; the primary operating point is r = 0.50.
  • Task justification: Next-action predictability is locally order-sensitive, whereas episode labels are order-insensitive, motivating separate sequential, set-valued, and episode-level targets.Conditioning on the previous action improves next-action top-5 accuracy by +38.7 points and reduces entropy from 5.21 to 3.51 bits.
  • Inputs and models: The frozen graph NSGT recognition encoder is paired with sequential, frequency-based, histogram, symbolic-only, and language-model comparison systems.The ontology-only control satisfies E1 and E2 by construction, while B3 is identified as the strongest sequential baseline.

5. Results and Analysis

The gated HPD improves sequential anticipation over the strongest sequential baseline, especially at longer horizons, while set prediction remains stronger for the histogram baseline. Hard reachability constraints provide near-complete trajectory coherence, but compositional HLI generalization remains a major weakness.

  • Goal inference: HLI top-1 rises from 67.2% at r = 0.25 to 81.4% at r = 0.75, with multimodal features adding goal evidence beyond observed labels at r = 0.50.Step-1 top-5 increases from 79.7% to 89.3%, while top-1 dips at the mid-episode cut because of cut position.
  • Next-action anticipation: +7.3 points at step 3 puts HPD top-5 at 64.4% versus 57.1% for B3, up from a +1.7-point step-1 margin.The widening margin indicates stronger relative benefit as rollout moves farther beyond observed evidence.
  • Set versus exact-position prediction: At r = 0.50, HPD raises step-1 top-5 from 81.5% exact-position to 93.8% set-relaxed on test, and from 81.1% to 95.0% compositionally.The gap reflects that future content is easier to identify than its exact order on order-insensitive episode labels.
  • Complementary strengths: The histogram baseline remains stronger for unordered sets, reaching 38.8% activity and 45.8% LLI mAP versus HPD’s 28.2% and 37.0%.HPD instead performs better where sequencing and goal structure matter.
  • Logic coherence: Soft logic training reduces HLI-reachability violations from 5.0% to 2.0% on test and 7.7% to 2.2% compositionally, while hard masks eliminate E1 and E2.The relative reductions are 59.8% and 71.1%, and E1/E2 remain zero under tested input-quality conditions.
  • Logic coherence: 96.83% of test trajectories satisfy D1, E1, and E2 jointly, versus 88.09% for the best baseline and a 73.92% ground-truth floor.On the compositional split, satisfaction reaches 99.96% against a 93.79% ground-truth floor.
  • Compositional generalization: Under compositional shift, HLI top-1 falls from 77.5% to 53.9%, whereas step-1 top-5 changes from 81.5% to 81.1% and the B3 margin grows to +4.9 points.Local transition structure transfers better than high-level goal inference across unseen parent associations.
  • Robustness: Correlated encoder errors harm HLI inference more than iid noise at the same 17.87% error rate: −10.2 versus −4.0 points.The reachability guarantees remain at 0% despite the input corruption.

6. Discussion and Limitations

The experiments show that structured goal inference is feasible, while soft and hard symbolic constraints jointly improve logical coherence. The synthetic benchmark nevertheless leaves substantial gaps in compositional generalization, naturalistic transfer, and unified handling of sequential and set-valued targets.

  • Core Empirical Findings: 77.5% HLI top-1 at half an episode and 81.4% at three quarters show that goal inference improves with observation.Both values exceed chance and the frozen encoder’s recognition reference after accounting for input parity.
  • Core Empirical Findings: 96.8% of test trajectories satisfy the joint constraint set, exceeding the strongest baseline at 88.1% and the ground-truth floor at 73.9%.The constrained decoder cannot emit unsupported transitions present in some synthesized ground-truth trajectories.
  • Neuro-Symbolic Separation: Soft FOL losses reduce HLI-reachability violations by 59.8% on the test split and 71.1% on the compositional split before masking.Inference-time reachability masks then reduce the remaining violations to 0.00% without degrading accuracy.
  • Limitations: The benchmark uses synthesized episodes with strong compositional and transition structure, so absolute results do not directly transfer to naturally recorded egocentric behavior.The authors also identify unified handling of set-valued and sequential targets and encoder adaptation to prefixes as open directions.
  • Limitations: The 24-point HLI top-1 drop on the held-out compositional split is the dominant unresolved generalization problem.The gap widens to about 24 points at r = 0.50 across models, while sequential next-action prediction remains nearly gap-free.

7. Conclusion

The paper formulates hierarchical anticipation as goal inference from partial multimodal episodes and combines neural generation with symbolic validity constraints. The resulting system improves anticipation while exposing remaining challenges in compositional goal generalization and unordered set prediction.

  • 7. Conclusion: The formulation treats anticipation as goal inference from a partially observed multimodal episode with structured prediction of the remaining behavior.This reframes anticipation away from precise motor-level forecasting.
  • 7. Conclusion: The final system combines a Hierarchical Planning Decoder with a frozen neuro-symbolic encoder, soft hierarchical regularization, and hard reachability masks.The decoder predicts remaining behavior across ontological levels, while inference-time masks enforce ontological validity.
  • 7. Conclusion: The final system improved next-action top-5 accuracy over the strongest sequential baseline, achieved strong HLI inference at half-episode observation, and eliminated HLI-reachability violations.These results summarize the principal benchmark outcomes reported for the four-level compositional setting.
  • 7. Conclusion: The paper reports that symbolic reachability alone is coherent but weak, purely neural decoding is accurate but inconsistent, and their combination is accurate and ontologically valid.Neither component reaches the combined system's overall balance of predictive accuracy and logical validity alone.
  • 7. Conclusion: The immediate limitations are a substantial compositional HLI gap under held-out compositions and weaker autoregressive handling of unordered set-valued outputs than a direct histogram baseline.The stated future agenda includes relational generalization, hybrid set-valued heads, prefix-aware training, and validation in naturally recorded assistive scenarios.

Appendix B. Full Robustness Column Set

The supplied appendix passage identifies a generative-AI declaration section.

  • Appendix B. Full Robustness Column Set: The appendix includes a section titled “Declaration of Generative AI and AI-assisted Technologies.”

Writing Process

The supplied passages document the authors’ use of generative-AI tools during manuscript preparation and describe reporting details for robustness and ablation results.

  • Writing Process: The authors used Gemini and Claude for language refinement, grammar correction, and LaTeX formatting, then reviewed and edited the output.
  • Writing Process: The robustness table reports learned-model results as mean ± standard deviation over three seeds, with deterministic models reported as means.
  • Writing Process: Violation rates D1, E1, and E2 are shown to two decimals, while D2 is report-only.
  • Writing Process: The encoder fine-tuning ablation unfroze either the last HGT layer and fusion or the full encoder using learning rate 3 × 10^-5 and three seeds.
  • Writing Process: EOS F1 is omitted for fine-tuned rows because the ablation targeted next-action and HLI attribution rather than the EOS head.
  • Writing Process: The authors state that they take full responsibility for the published article’s content.
Loading 2609.17064v1…