PaperlayerView exact revision on arXiv

Source-linked AI summary

Structural Inference under Hidden Agents

Zhongben Gong, Xiaoqun Wu, Mingyang Zhou, Hui Huang

arXiv:2609.18045v1cs.LG

TL;DR

When agents are entirely unobserved at deployment, reconstructing their trajectories and inferring their interactions are coupled tasks. SIHA initializes hidden states without structural guidance, then uses NRI-based structure estimates to guide iterative reconstruction; experiments report consistent visible-to-visible structural improvements and benefits in hidden-state reconstruction and prediction.

  • Problem

    Structural inference typically assumes trajectories for all modeled agents are available, while joint recovery of entirely hidden-agent trajectories and their interactions remains underexplored.

    Source: §1 · p. 2

  • Method

    SIHA initializes hidden trajectories from visible observations, infers interactions with NRI, then alternates structure-guided hidden-state reconstruction and state–structure refinement.

    Source: §1 · p. 3

  • Results

    SIHA improves visible-to-visible structural inference across the reported settings, with benefits also reported for hidden-state reconstruction and future prediction.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    The framework is evaluated on three benchmark dynamical systems and recorded motion-capture trajectories with simulated whole-limb occlusion.

    Source: §6 · p. 14

    The current formulation relies on complete trajectories during training to supervise hidden-state reconstruction; structural inference with fully latent hidden agents remains open.

    Source: §6 · p. 14

Abstract

from arXiv · show

Recovering latent interaction structures from multi-agent dynamics is important for understanding and predicting interacting systems. Trajectory-based structural inference has achieved promising performance, but conventional formulations assume that the trajectories of all modeled agents are available. In practice, agents may become unobserved at deployment because of limited sensing, occlusion, or communication failure. Existing studies have considered unseen-node estimation, structural inference under partial observations, and missing-value imputation, yet the joint recovery of hidden-agent trajectories and their interactions remains underexplored. We formulate this problem as structural inference under hidden agents. Its key difficulty is a circular dependency: recovering interactions involving a hidden agent requires an estimate of its trajectory, while trajectory reconstruction can itself benefit from structural information. To address this challenge, we propose Structural Inference under Hidden Agents (SIHA), which combines structure-agnostic initialization with structure-guided iterative refinement. SIHA reconstructs hidden trajectories from visible observations, infers interactions using Neural Relational Inference, and feeds the estimated structure back into hidden-state reconstruction through multi-strength structural attention and iterative state--structure updates. Experiments on three benchmark dynamical systems demonstrate consistent improvements in visible-to-visible structural inference, while also showing benefits in hidden-state reconstruction and future prediction. Motion-capture experiments with simulated whole-limb occlusion further demonstrate its effectiveness in realistic hidden-agent settings.

1 Introduction

SIHA addresses structural inference when agents are entirely unobserved at deployment, jointly reconstructing their trajectories and estimating interactions. It initializes hidden states without structure, then alternates structure-guided reconstruction and interaction inference; results show consistent visible-to-visible structural improvements across the reported settings.

1 Introduction

  • The paper motivates latent interaction recovery by describing how network structure shapes dynamics in physical, biological, social, and economic systems.Examples include gene regulatory networks and the propagation of shocks through production networks.Source: §1 · p. 1
  • Conventional trajectory-based structural inference generally assumes that states or trajectories for all modeled entities are available as input.Entirely absent agents arise in settings such as ecosystem surveys, vehicle or robot sensing, and motion capture.Source: §1 · p. 2
  • SIHA studies deployment with only visible-agent trajectories, reconstructing hidden trajectories and estimating sample-specific interactions without edge-label supervision.Complete trajectories are available during training to supervise hidden-state reconstruction.Source: §1 · pp. 2–3
  • Hidden-state reconstruction and complete structural inference are coupled: interactions need hidden trajectories, while reconstructing those trajectories can benefit from structural information.SIHA initializes hidden trajectories from visible observations, estimates interactions with NRI, then uses multi-strength structural guidance and iterative state–structure updates.Source: §1 · pp. 2–3
  • SIHA achieves higher visible-to-visible structural accuracy than visible-only NRI throughout the reported grid, while visible-agent forecasting varies by system.The evaluation also reports benefits in hidden-state reconstruction and future prediction, and includes motion-capture sequences with simulated whole-limb occlusion.Source: §1 · p. 3

2 Related Work

Related work covers structural inference, incomplete-observation settings, and reconstruction of missing values in known variables. SIHA differs by jointly reconstructing entire absent-agent trajectories and estimating interactions involving those agents.

2.1 Relational and Structural Inference

  • Relational and structural inference methods learn latent relations from trajectories, but generally assume modeled entities have observed states, trajectories, or node attributes.NRI learns discrete latent edge types from trajectories through a predictive decoder without ground-truth edge labels [5].Source: §2.1 · p. 3

2.2 Structural Inference under Incomplete Observations

  • Unseen-node inference, partial-observation structural inference, and missing-value relational inference differ from SIHA in their observation units and inference targets.SIHA jointly reconstructs whole hidden-agent trajectories and estimates interactions involving hidden agents, using complete trajectories for hidden-state training supervision but no edge-label supervision.Source: §2.2 · p. 4

2.3 State Reconstruction and Graph-Guided Imputation

  • Imputation methods reconstruct values or segments over known variables, whereas SIHA addresses an agent entirely absent from deployment input and couples structure estimation with state reconstruction.Estimated structure guides hidden-state reconstruction, while reconstructed trajectories support inference of interactions involving hidden agents.Source: §2.3 · p. 4

3 Problem Formulation

The formulation distinguishes visible and hidden agents, specifies the interaction graph, and defines training and deployment inputs. At deployment, visible trajectories support estimates of hidden histories, future trajectories, and the complete interaction structure.

3.1 Interacting Systems with Hidden Agents

  • The system has N agents: the first Nvis are visible and the remaining Nhid are hidden, with each agent represented by a state in R^d over time.The observed history spans T time steps.Source: §3.1 · p. 4
  • The complete interaction structure is a directed graph encoded by an adjacency tensor, and hidden-agent states and their interactions must both be estimated.The tensor represents K possible interaction types for directed agent pairs.Source: §3.1 · p. 5

3.2 Training and Deployment Setting

  • During training, selected agents are masked from predictor inputs and their complete trajectories supervise hidden-state reconstruction; NRI is pretrained without ground-truth interaction labels.Available ground-truth structures are reserved for offline evaluation.Source: §3.2 · p. 5
  • At deployment, SIHA observes only visible-agent trajectories and uses neither hidden ground-truth states nor interaction labels.It combines reconstructed hidden trajectories with visible trajectories for future prediction and complete-structure inference.Source: §3.2 · p. 5

3.3 Inference Targets

  • The inference targets include historical trajectories for the hidden agents.These are estimated from visible-agent histories at deployment.Source: §3.3 · p. 5
  • A second inference target is the future trajectories of all agents over a prediction horizon.The target includes visible and hidden agents.Source: §3.3 · p. 5
  • The remaining target is the complete interaction structure, including interactions involving visible and hidden agents.The structure is estimated from visible-agent histories at deployment.Source: §3.3 · p. 5

4 Method

SIHA addresses the circular dependence between reconstructing hidden trajectories and inferring their interactions through structure-free initialization followed by structure-guided iterative refinement. Its method combines hidden-state predictors with a pretrained NRI backbone and uses separate training and deployment procedures.

4.1 Framework Overview

  • SIHA first initializes hidden trajectories from visible observations, then uses NRI to estimate interactions and future states, and iteratively feeds the structure back to refine hidden trajectories.During HSP-sg training, a per-sample structure cache is held fixed during warm-up and refreshed periodically; deployment alternates HSP-sg and NRI for fixed rounds.Source: §4.1 · pp. 5–6

4.2 Structure-Agnostic Hidden-State Reconstruction

  • HSP-sa treats visible trajectories as an unordered set and maps them to hidden-trajectory slots without using an estimated graph.Synthetic-system slots are aligned for supervised training and evaluation; motion-capture slots retain predefined masked-joint ordering.Source: §4.2 · p. 7

4.3 Structure-Guided Hidden-State Reconstruction

  • HSP-sg conditions its attention on predicted structure, using visible-to-visible, visible-hidden, and hidden-to-hidden relation blocks to guide corresponding attention operations.Four attention heads use strengths 0, 1, 5, and 109, retaining an unguided path alongside soft biases and an approximate hard mask.Source: §4.3 · pp. 7–8

4.4 Iterative State–Structure Refinement

  • Each refinement updates hidden trajectories under the current cached structure, then recomputes structure from visible and reconstructed trajectories.At deployment, the cache is initialized by HSP-sa and NRI, and recurrence runs for a fixed number of rounds with model parameters fixed.Source: §4.4 · p. 9

4.5 Learning and Inference Procedures

  • Training separately pretrains HSP-sa and NRI, then optimizes HSP-sg with masked-agent trajectory targets and periodic NRI cache refreshes; deployment alternates HSP-sg and NRI.Synthetic hidden slots are Hungarian-matched before MSE evaluation, while motion-capture joints use direct joint-wise MSE in fixed order; the final NRI pass supplies the interaction estimate and future trajectories.Source: §4.5 · p. 9

5 Experiments

The experiments evaluate SIHA against visible-only NRI and an internally controlled HSP-sa baseline on synthetic systems and simulated motion-capture occlusions. They report stronger visible-to-visible structure inference, benefits from structure-guided refinement, and challenges as the hidden-agent count grows.

5.2 Overall Performance

  • Across Springs, Charged Particles, and Kuramoto, HSP-based methods exceed visible-only NRI in ACCV-V throughout the grid; visible-agent forecasting errors are lower on Springs and Kuramoto but comparable or slightly better for NRI on Charged.Table 1 reports NRI / HSP-sa / SIHA (HSP-sg) across hidden-agent counts; visible-only NRI does not provide hidden-related outputs.Source: Table 1 · p. 10, §5.2 · pp. 10–11
  • SIHA (HSP-sg) is better than or equal to HSP-sa across displayed metrics, with the largest differences generally on Springs and smaller differences on Charged and Kuramoto.The internal comparison holds hidden-state supervision and the NRI module fixed while changing structural guidance and refinement.Source: §5.2 · p. 11

5.3 Effect of the Number of Hidden Agents

  • With Nvis = 5 fixed, increasing Nhid degrades hidden-state reconstruction and generally lowers structural accuracy across all three synthetic systems, while HSP-based methods retain higher ACCV-V than visible-only NRI.The experiments configure models for a fixed hidden-agent count; an unknown count is examined separately in D.2.Source: §5.3 · p. 11

5.4 Motion-Capture Case Study

  • Across four subject–occlusion settings, SIHA improves hidden- and visible-agent future prediction over HSP-sa and lowers hidden-history reconstruction error in three, with subject #69 left-arm the exception.Both HSP-based configurations have lower visible-future errors than visible-only NRI in all four settings; these experiments use artificial whole-limb masks on recorded motion-capture trajectories.Source: §5.4 · pp. 11–12, Table 2 · p. 12
  • In the subject #35 left-arm visualization, HSP-sg produces a more concentrated cross-hand pattern than HSP-sa, closer to the full-observation NRI reference.The plotted focus comparisons show left- and right-hand relations; the full-observation structure is a reference estimate, not a ground-truth graph.Source: §5.4 · p. 12, Figure 4 · p. 13

5.5 Model and Supervision Analysis

  • On Springs with Nhid = 3, removing the unguided attention head causes the largest reported degradation, while removing intermediate-strength guidance also reduces performance and removing the hard-mask head leaves point estimates unchanged.The results suggest the main multi-strength benefit comes from combining unguided and softly structure-guided attention.Source: §5.5 · p. 12, Table 3 · p. 13
  • Without direct hidden-trajectory supervision, all three HSP-sa variants perform substantially worse in hidden-state reconstruction and visible-agent prediction, with corresponding structural-inference degradation.The tested variants train only through visible-agent future prediction; the authors identify fully latent training without hidden-state supervision as future work.Source: §5.5 · p. 13

5.6 Additional Analyses

  • Additional analyses cover HSP-sa encoder comparisons, five-run variability for representative Nhid = 3 settings, hidden-count model selection, extra unmodeled hidden agents, and computational costs.These analyses are reported in appendices C.2, C.3, D.2, D.3, and B, respectively.Source: §5.6 · p. 13

6 Conclusion

SIHA reconstructs hidden-agent trajectories and infers complete interactions through coupled state–structure refinement. The NRI backbone infers latent edge types from trajectories and trains its decoder to predict future states, with ELBO regularization.

6 Conclusion

  • SIHA combines structure-agnostic trajectory initialization with NRI-based interaction inference and structure-guided iterative refinement.The conclusion reports higher visible-to-visible structural accuracy than visible-only NRI across evaluated synthetic settings, while also recovering hidden trajectories and hidden-related interactions.Source: §6 · p. 14
  • The formulation relies on complete trajectories during training to supervise hidden-state reconstruction; fully latent hidden agents remain an open problem.Ground-truth edge labels are not required for training or deployment, and tested objectives without direct hidden-state supervision do not match supervised performance.Source: §6 · p. 14

A.1 Neural Relational Inference Backbone

  • NRI encodes trajectories into a distribution over latent edge types, then uses a GNN decoder to predict future states through message passing on the inferred graph.Gumbel-Softmax enables back-propagation through discrete edge sampling.Source: §A.1 · p. 17
  • NRI training maximizes the ELBO, combining state-prediction error with regularization of the posterior against the prior over latent graphs.The prior is typically assumed to be uniform.Source: §A.1 · p. 17

A.2 Set Transformer and Hidden-State Predictor Architecture

The hidden-state predictors use Set Transformer attention to model agent sets, with HSP-sg incorporating structure guidance. Training initializes HSP-sg from separately pretrained HSP-sa and NRI, updates cached structures during training, and iteratively refines them at test time.

A.2 Set Transformer and Hidden-State Predictor Architecture

  • Set Transformer encoders apply stacked attention blocks to represent interactions within an input set, while pooling attention maps representations to a fixed-size output set.HSP-sg retains the HSP-sa encoder–decoder topology and applies structural biases to encoder self-attention, pooling attention, and decoder self-attention.Source: §A.2 · pp. 17–18

B Training and Reproducibility Details

B.1 Pretraining of HSP-sa and NRI on Fully Observed Data

  • HSP-sa and NRI are pretrained separately on complete trajectories; HSP-sa uses masked-agent trajectories as reconstruction targets, while NRI trains without ground-truth interaction labels.Synthetic HSP-sa uses Hungarian-aligned MSE, while motion capture uses direct joint-wise MSE in the predefined masked-joint order.Source: §B.1 · p. 18

B.2 Training of Structure-Guided Predictor (HSP-sg)

  • HSP-sg trains on structure estimates initialized by pretrained HSP-sa and NRI, optimizing hidden-state reconstruction while NRI periodically refreshes a per-sample structure cache.For synthetic systems, the cache is frozen for the first 80 epochs and updated every 10 epochs after warm-up.Source: §B.2 · pp. 18–19

B.3 Evaluation-time Refinement via Iterative Structure Cache

  • At test time, the structure cache is initialized with pretrained models and refined for five rounds, alternating HSP-sg hidden-state predictions with NRI structure updates.Source: §B.3 · p. 19

B.4 System-specific Settings and Embedding Modules

  • The experiments use Set Transformer HSP modules, with MLP trajectory embeddings for Springs and one-dimensional convolutional embeddings for Charged Particles and Kuramoto.These embedding choices follow the corresponding NRI configurations.Source: §B.4 · p. 19

B.5 Motion-Capture Training Configuration

Motion-capture HSP modules use fixed architecture and optimization settings, while the backbone is a static-graph NRI model trained separately for each subject.

B.5 Motion-Capture Training Configuration

  • Both motion-capture HSP modules use hidden dimension 256, four attention heads, history length 49, and six-dimensional position-and-velocity inputs.They train for 500 epochs with batch size 8, learning rate 1e-4, weight decay 1e-6, and direct joint-wise MSE.Source: §B.5 · p. 19
  • Motion-capture HSP-sg uses a 40-epoch cache warm-up, updates the cache every 20 epochs, and applies interaction guidance from edge type 1.Source: §B.5 · p. 19
  • The motion-capture backbone is a static-graph NRI model with two edge types and a Gumbel-Softmax temperature of 0.5, trained for 500 epochs per subject.It uses 10 teacher-forcing steps and batch size 8.Source: §B.5 · p. 20

B.6 Dataset Generation and Scale

The synthetic evaluation uses three dynamical systems with fixed trajectory lengths and dataset sizes. Experiments run on a single RTX 4090 GPU, and the authors report that scripts and configuration files will be released publicly.

B.6 Dataset Generation and Scale

  • The benchmark environments are Springs, Charged Particles, and Kuramoto, with 50 time steps for training and validation trajectories and 100 for testing.Springs and Charged Particles use four-dimensional position–velocity states; Kuramoto uses phase difference, amplitude, and intrinsic frequency.Source: §B.6 · p. 20
  • Each dataset contains 200,000 training samples, 50,000 validation samples, and 50,000 test samples.These sizes exceed those used in the original NRI experiments.Source: §B.6 · p. 20

B.7 Compute Resources

  • A complete training run takes approximately 5–15 hours on a single NVIDIA RTX 4090 GPU with 24 GB memory.Training scripts, data-generation tools, and configuration files will be released publicly.Source: §B.7 · p. 20

C Complete Quantitative Results

The section tests whether hidden-state predictors can be trained without direct hidden-state supervision and compares three visible-future-only variants with the supervised HSP-sa setting. The reported variants perform worse on hidden-state and visible-future errors, and the main experiments retain direct supervision.

C Complete Quantitative Results

  • The three tested variants train with supervision only through visible-agent future states, without direct supervision from ground-truth hidden trajectories.They differ in whether NRI is trained from scratch, pretrained, or frozen.Source: §C.1 · p. 20
  • The main supervised setting uses ground-truth trajectories of masked agents as HSP-sa reconstruction targets.The main experiments retain this direct hidden-state supervision.Source: §C.1 · p. 20
  • All three visible-future-only variants have substantially higher hidden-state and visible-future errors than supervised HSP-sa, while visible-to-visible accuracy deteriorates as hidden-agent count increases.These results cover the tested objectives, not an impossibility result for other objectives or models.Source: §C.1 · p. 20, Table 4 · p. 21

C.2 Ablation on Encoder Modules for HSP-sa

The section compares HSP-sa encoder choices, reports five-run statistics for representative settings, and describes how motion-capture focus graphs are constructed.

C.2 Ablation on Encoder Modules for HSP-sa

  • The Set Transformer has the lowest reported hidden-state prediction MSE among the tested HSP-sa encoder variants.The comparison uses Springs with Nvis = 5 and Nhid = 2; the table does not isolate which architectural property accounts for the difference.Source: §C.2 · p. 21, Table 5 · p. 21

C.3 Repeated-run statistics on representative settings

  • Five-run statistics are reported for Springs, Charged Particles, and Kuramoto, each with Nvis = 5 and Nhid = 3.SIHA, pretrained NRI, and HSP-sa models are independently trained over five random seeds; results are reported as mean ± standard deviation.Source: §C.3 · p. 21, Table 6 · p. 22

D.1 Motion-Capture Focus-Graph Construction

  • Motion-capture focus graphs retain top-k joint edges with scores above 0.75, using k = 8 for hand-focused panels and k = 14 for the others.The full-observation NRI structure is a reference estimate, not a ground-truth graph.Source: §D.1 · p. 21

D.2 Unknown and Variable Numbers of Hidden Agents

The section examines hidden-agent count selection and sensitivity to additional unmodeled agents. Count-prediction accuracy varies substantially by system, while the additional-agent experiment shows progressive metric degradation.

D.2 Unknown and Variable Numbers of Hidden Agents

  • The count-selection procedure applies HSP-sa and NRI models configured for different hidden-agent counts separately, then selects by visible-agent future-prediction MSE.Table 7 reports count-prediction accuracy.Source: §D.2 · p. 22, Table 7 · p. 22
  • Accuracy remains above 83% for Springs, but falls from 92.3 to 45.5 on Charged and from 87.5 to 24.7 on Kuramoto as the true hidden-agent count rises from 0 to 5.The method selects the count yielding the lowest visible-agent future-prediction MSE; this is a boundary analysis, not a resolution of unknown cardinality.Source: §D.2 · p. 22, Table 7 · p. 22

D.3 Additional Unmodeled Hidden Agents

  • When additional unmodeled agents are hidden, the three displayed metrics progressively degrade as their number increases.The Springs experiment hides one to three additional agents while training reconstruction for only the first two; it does not establish robustness to arbitrary hidden-agent configurations.Source: §D.3 · p. 22, Table 8 · p. 23

Related papers

Loading 2609.18045v1…