Predictive Spatio-Temporal Scene Graphs for Semi-Static Scenes

I. INTRODUCTION

• Prior approaches often retain only a present-state hypothesis or trade real-time adaptation for long-horizon forecasting, while future reasoning remains largely unexplored.Change-detection methods discard historical data, whereas prediction-based estimators lack adaptation capabilities.Source: §I · pp. 1–2
• PredictiveGraphs integrates Perpetua∗ with an open-vocabulary scene graph to predict object-level semi-static changes and answer future-location queries.Objects are associated with previously observed receptacles, treated as a finite set of discrete states.Source: §I · p. 2
• Perpetua∗ extends Perpetua with Bayesian model selection and a switching prior to support long-horizon predictions while preserving real-time adaptability.The prior can come from environment-specific observations or a large language model.Source: §I · p. 2

II. RELATED WORK

• The related-work discussion distinguishes semantic scene-graph querying from methods that forecast graph changes, positioning PredictiveGraphs as predictive and open-vocabulary.The cited approaches include past-event queries in spatial maps and graph-change prediction.Source: §II · p. 2

III. PROBLEM STATEMENT

• The problem statement assumes periodic object patterns, distinguishable dynamic objects, and a finite set of receptacles for each object.For a query time, the system estimates object-receptacle presence probabilities and identifies the most likely receptacle or absence.Source: §III · p. 3

IV. PERPETUA∗

• Perpetua∗ models the presence or absence of a specific feature over time and extends Perpetua with Bayesian model selection.Perpetua’s mixture formulation captures multiple persistence hypotheses.Source: §IV · p. 3, A. Preliminaries: Perpetua · p. 3

A. Preliminaries: Perpetua

• Perpetua models persistence using multiple hypotheses over feature survival time, with noisy binary observations informing the feature-state estimate.Its mixture components represent alternative survival-time models, and the observation model accounts for false positives and false negatives.Source: A. Preliminaries: Perpetua · p. 3
• The method computes conditional posteriors for mixture components and updates their weights from the observations to form predictions.The final prediction uses the conditional posterior of the dominant component, selected by maximum posterior weight.Source: A. Preliminaries: Perpetua · p. 3
• Perpetua couples persistence and emergence filters with a heuristic belief threshold, requiring costly simulation between observations; without observations, long-horizon predictions rely on periodic patterns and may diverge from true dynamics.Source: A. Preliminaries: Perpetua · p. 4

B. Method

• Bayesian model selection replaces Perpetua’s heuristic switch, using a switching prior to select between emergence and persistence models.The prior can draw on environment-specific observations or an LLM.Source: B. Method · p. 4
• During prolonged observation gaps, exponential annealing flattens the model likelihood so the switching prior increasingly governs model selection.The data gap decays likelihood influence until observations resume.Source: B. Method · p. 4, Figure 2 · p. 4
• Perpetua* predicts by weighting the persistence and emergence models using their posterior model probabilities.Each model’s prediction uses parameters specific to that model.Source: B. Method · p. 4
• A forgetting factor on measurement-likelihood updates supports rapid adaptation as new observations arrive.The factor is applied to the measurement likelihood updates.Source: B. Method · p. 4

1) Multi-Receptacle Belief:

• Per-receptua* beliefs for each receptacle are combined, then MAP selects one location; a probability threshold can also indicate that the object is absent.The factorization permits multiple simultaneous location assignments before MAP resolves them.Source: 1) Multi-Receptacle Belief · pp. 4–5

V. PREDICTIVEGRAPHS

• PredictiveGraphs attaches Perpetua* estimators to the edges of an open-vocabulary scene graph.The representation combines persistence estimation with semantic scene graphs.Source: §V · p. 5

A. Preliminaries: Open-Vocabulary Scene Graphs

• Scene-graph nodes are built by lifting 2D region proposals from posed RGB-D frames into 3D, merging observations over time, and adding vision-language embeddings for natural-language queries.Nodes also contain a point cloud and feature embedding.Source: A. Preliminaries: Open-Vocabulary… · p. 5

1) Object Classification and Spatial Association:

• CLIP retrieves likely semi-static objects and an LLM filters false positives; receptacle annotations are matched to graph nodes using 3D IoU and the Hungarian algorithm.Receptacles are typically large, stationary objects such as tables, shelves, or countertops.Source: 1) Object Classification and Spatial… · p. 5

2) Building the Edge History:

• The graph connects each semi-static object to every receptacle where it was observed across mapping sessions, with Perpetua* edge weights evolving over time.The accumulated session history provides the basis for dense temporal edges.Source: 3) Temporal Edge Formulation · p. 5, 2) Building the Edge History · p. 5

4) Temporal Queries and Map Propagation:

• At any future time, the map can recompute object–receptacle edge weights; visualization selects the most likely receptacle, while navigation can prioritize the top-k.The top-k receptacles are ranked by edge weight.Source: 4) Temporal Queries and Map Propagation · pp. 5–6
• Perpetua* achieves the best tradeoff between adaptation and prediction quality across switching-prior choices, while FreMEn struggles with shifts and Perpetua reverts to cyclic predictions without data.The comparison concerns persistence beliefs under distribution shifts and observation gaps.Source: Figure 4 · p. 6

5) Perpetua* Belief Updates:

• New robot observations update Perpetua* estimators in real time, enabling PredictiveGraphs to adapt when test-time dynamics differ from training.Updates use presence or absence measurements for known object–receptacle pairs.Source: 5) Perpetua∗Belief Updates · p. 6

VI. EVALUATION

• The evaluation benchmarks Perpetua* for long-term prediction, adaptation, and computational efficiency, then tests PredictiveGraphs on semi-static navigation tasks.It spans ProcTHOR scenes and real-world laboratory deployment.Source: §VI · p. 6

C. Embodied LLM Planning

• PredictiveGraphs with Perpetua* or FreMEn outperforms baselines in predictive navigation under both privileged and noisy perception.The system integrates a temporal object map with an agent and observation updates that revise map beliefs during navigation.Source: C. Embodied LLM Planning · p. 6, Table II · p. 7

2) Simulation Environment:

• Perpetua* outperforms Perpetua and FreMEn with all three switching-prior choices, including accurate long-term predictions without observations.The test set includes a long-weekend schedule shift; Perpetua struggles without observations, while FreMEn degrades when dynamics depart from training.Source: 2) Simulation Environment · p. 7

1) Experimental Setup:

• Queries sample a target, robot pose, and future time up to one week ahead; success requires the object within 1 meter of the final position.Simulation includes predictive and adaptive navigation queries; the real-world experiment runs 25 open-vocabulary queries across three rooms.Source: 1) Experimental Setup · p. 7
• Real-world observations are collected every two hours over three weeks, while simulation uses one-hour intervals and four weeks of data per environment.Simulation data are divided into two training weeks, one validation week, and one test week.Source: 1) Experimental Setup · p. 7

2) Baselines:

• Baselines include ConceptGraphs, language-model-augmented variants with or without observation history, DualMap, and PredictiveGraphs using FreMEn instead of Perpetua*.The history-augmented language-model baseline receives the last 100 timestamped observations of presence or absence.Source: 2) Baselines · pp. 7–8

4) Simulation Results:

• PredictiveGraphs performs best across predictive-navigation query types with both Perpetua* and FreMEn; Perpetua* also outperforms the FreMEn variant in adaptive navigation under shifted dynamics.In real-world navigation, PredictiveGraphs robustly estimates semi-static changes under perception noise and outdated maps, and surpasses DualMap.Source: 4) Simulation Results · p. 8, 5) Real-World Results · p. 8, Table III · p. 8
• In predictive navigation, both PredictiveGraphs estimators outperform baselines across query types; language-model baselines degrade when typical movement patterns differ from environment-specific dynamics.Privileged results use semantic segmentation but remain subject to visual occlusion.Source: 4) Simulation Results · p. 8

3) Training Details:

• The adaptive-navigation comparison attributes better performance under distributional shifts to Perpetua*'s ability to adapt to changing dynamics.Perception errors degrade overall performance, but the Perpetua* version still outperforms the FreMEn variant.Source: 4) Simulation Results · p. 8, Table III · p. 8

5) Real-World Results:

• In the real-world demonstration, PredictiveGraphs anticipates a blocked shortest path and chooses a longer feasible route, avoiding unnecessary exploration.The table caption describes robust navigation amid perception errors, including training on noisy real-world data.Source: 5) Real-World Results · p. 8, Figure 5 · p. 8, Table IV · p. 8

VII. CONCLUSION

• PredictiveGraphs predicts future semantic scene graphs for direct reasoning, using Perpetua* Bayesian model selection between emergence and persistence filters.The paper says future generalist robots may require these capabilities for collaboration and complex tasks.Source: §VII · p. 8

VIII. LIMITATIONS

• Independent object–receptacle edges trade expressivity against real-time inference cost; indistinguishable objects are treated as interchangeable, and LLM hallucinations remain possible.The authors identify improved balance, perception, and map-grounding as future opportunities.Source: §VIII · p. 9

• The algorithm updates likelihood, conditional evidence, and posterior weights for persistence and emergence models, then predicts at an arbitrary query time.The final prediction is returned as p(x_t | Y_1:N).Source: §A.1 · p. 11
• The update routine takes an observation tuple and updates model likelihoods, evidence accumulators, conditional evidence, and posterior weights.The pseudocode also specifies a forgetting factor for each receptacle.Source: §A.1 · p. 11
• The prediction routine evaluates candidate predictions, computes model posterior probabilities, and returns the final prediction for the query time.The listed steps refer to equations (8), (12), and (13).Source: §A.1 · p. 11

B.1 LLM Tools

• The three embodied-planning tools are semantic search, location prediction, and active navigation, each specified by its inputs, outputs, and logic.The section presents the tools as interfaces to PredictiveGraphs.Source: §B.1 · p. 11
• Semantic search uses CLIP embeddings to rank mapped objects against an open-vocabulary query; location prediction queries Perpetua* for receptacle probabilities at a requested time.If no receptacle exceeds the confidence threshold, the predicted object is reported absent.Source: §B.1 · p. 11
• Active navigation plans toward a receptacle, updates PredictiveGraphs with RGBD observations, and checks for the target until it is found or the receptacle is reached.Source: §B.1 · p. 11

B.2 LLM Prompt - Switching Prior

• The switching-prior prompt uses few-shot prompting with Gemini 2.5 Pro to infer probable weekly object schedules from household context and object details.The requested output is a weekly schedule of time intervals.Source: §B.2 · pp. 11–12

B.3 Perpetua* Implementation Details

• The prior is modeled as a mixture of log-normal distributions; γ = 0.99 maintains an effective memory of approximately 250 observations, while α0 ≈ 0.01 causes near-total reversion to the prior after approximately 300 time units.Source: §B.3 · p. 12
• Both persistence and emergence filters search up to five mixture components using AIC; switching-prior variants use FreMEn, an LLM, or an oracle.The FreMEn variant selects from 1,000 Fourier coefficients using a one-week held-out set; the oracle uses ground-truth training dynamics.Source: §B.3 · p. 12

B.4 Schedule

• The shifted test schedule makes Monday and Friday follow weekend-like dynamics, testing real-time adaptation to changed routines.The training schedule has distinct weekday and weekend patterns.Source: §B.4 · p. 12, Figure 6 · p. 12

C.1 Complexity Analysis and Illustrative Example: Perpetua*

• Perpetua* computes predictions and belief updates faster than Perpetua by replacing expensive simulation steps with Bayesian model selection.The comparison reports both prediction and update time.Source: §C.1 · pp. 12–13, Figure 7 · p. 13
• With five mixture components and a 1,000-coefficient FreMEn prior, each edge costs 28 KB, while each additional mixture costs 0.047 KB.Memory scales with discovered object–receptacle relationships rather than deployment time; 1,000 edges add 28 MB.Source: Table V · p. 13, §C.1 · p. 13
• When observations stop, the switching prior continues to evolve log posterior odds while the dominant mixture component stays frozen; final predictions mix persistence and emergence outputs.Bayesian model selection identifies the mixture that best explains observed data as observations arrive.Source: Figure 8 · p. 13

D.1 Hierarchical Scheduling Model

• Simulated receptacle schedules use hierarchical time scales and Markov transitions; the test week shifts Monday and Friday to weekend-like dynamics.Receptacle durations are sampled uniformly with guidance from ProcTHOR semantic priors.Source: §D.1 · pp. 13–14, Figure 9 · p. 14

D.2 Temporal Resolution

• Simulation observations are hourly across four weeks and 15 ProcTHOR scenes, each with at least five rooms and 100–150 m2 of navigable space.The four weeks are partitioned into two training weeks, one validation week, and one test week.Source: §D.2 · p. 14

D.3 Relationship Between Groundtruth and Privileged Data

• Groundtruth records the simulator’s true receptacle assignment, while privileged observations account for camera visibility and mark hidden or invalid receptacles separately.The encoding distinguishes presence, absence, missing data, and receptacles invalid for an object.Source: §D.3 · p. 14

E.1 Real-World Details

• The real-world dataset uses 21 manually relocated objects in a three-room laboratory, with 84 RGB-D scans collected bi-hourly over three weeks.A teleoperated Agilex Ranger Mini 2.0 gathered observations; RTAB-Map provided mapping and pose estimation, and a SAM3-based pipeline generated training labels.Source: §E.1 · pp. 14–15

E.2 Hardware

• The robot used an onboard Jetson Orin for control, but model inference ran offboard on an Nvidia RTX 2080.The LLM-based navigation agent and predictive baselines used gpt-5.2.Source: §E.2 · p. 15

F.1 Simulation results

• PredictiveGraphs with Perpetua* outperforms the FreMEn variant under privileged perception and retains its advantage when perception errors degrade performance.The experiments also show accurate target-location predictions supporting navigation when scene graphs have not been updated for hours; the authors expect performance to scale with improved perception pipelines.Source: §F.1 · pp. 15–16, Figure 14 · p. 17

F.2 Real-World results

• In real-world navigation, the robot reaches a hair brush by routing around a closed door and recovers to find a carrot after its initial predicted location is empty.For the carrot search, PredictiveGraphs predicts a new location rather than reverting to the outdated map.Source: Figure 12 · p. 16, §F.2 · p. 16, Figure 13 · p. 16