Spatio-Temporal LLM: Reasoning about Environments and Actions

• MLLMs often lack temporal modeling of action progression, causal dependencies, and event ordering despite advances in spatial reasoning.Source: §1 · p. 1
• Existing 3D-aware methods can localize agents and answer spatial queries but rely on static observations, limiting reasoning about dynamic events.Source: §1 · p. 1
• REA provides five tasks—relative direction, relative distance, find-my-item, furniture affordance prediction, and action planning—to test joint spatio-temporal reasoning.Source: §1 · p. 2
• 23.85% to 31.46% overall ChatGPT-4o accuracy for existing MLLMs contrasts with 41.89% overall accuracy and 47.32% average categorical accuracy for the STLLM baselines.Source: §1 · p. 2
• The paper develops a dataset collection pipeline and studies STLLM baselines that enhance spatio-temporal understanding.Source: §1 · p. 2

• Image and video LLMs commonly connect visual encoders to language decoders, with video work additionally targeting long-form understanding, streaming, and memory.Source: §2 · p. 2
• 3D LLMs typically project 2D visual features into 3D representations aligned with an LLM because text-aligned 3D datasets remain limited.Source: §2 · p. 2
• Image+3D queries from egocentric viewpoints provide grounded depictions of an agent’s surroundings for downstream tasks.Source: §2 · p. 2
• Video+3D models often treat videos as multi-view images, ignore temporal dynamics, or require unavailable per-frame depth maps, making them unsuitable for this evaluation setting.Source: §2 · p. 3
• Existing spatial benchmarks are not grounded in human actions, whereas REA links holistic environment structure with local dynamic changes for embodied question answering.Source: §2 · p. 3

3 REASONING ABOUT ENVIRONMENTS AND ACTIONS (REA)

• REA uses point clouds and egocentric videos to test relative direction, relative distance, find-my-item, furniture affordance prediction, and action planning.Source: §3 · pp. 3–4

3.1 DATA COLLECTION PIPELINE

• The data collection pipeline samples past-action clips, computes spatial relationships and navigation movement, adds action transitions, and grounds frames in point clouds.Source: §3 · p. 4

3.1.1 QUESTION-ANSWER (QA) GENERATION

• QA generation rephrases task templates and requires video sampling, 3D position estimation, spatial relationship estimation, and navigation movement estimation.Source: §3.1.1 · p. 4
• Person locations use mean registered camera poses, while object locations use VISOR masks projected onto COLMAP point clouds near interactions.Source: §3.1.1 · pp. 4–5
• Relative-distance labels use five world-coordinate poses and a linear-regression slope with a ±0.05 threshold, while action planning uses a 1.5-meter displacement threshold.Source: §3.1.1 · p. 5

3.1.2 POINT CLOUD RECONSTRUCTION

• Dense point clouds are reconstructed with pose-free methods and dense video poses are registered using Reloc3r, reducing registration time relative to COLMAP-based pipelines.Source: §3.1.2 · pp. 5–6

• The STLLM input consists of a colored 3D point cloud, video frames, textual instructions, and intrinsic and extrinsic camera matrices for each frame.Source: §4 · p. 6
• The two baselines share vision and point-cloud encoders while representing complementary trade-offs between direct integration efficiency and compact alignment.Source: §4 · pp. 6–7
• STLLM-3D concatenates projected point-cloud features with image and text embeddings for direct LLM decoding, but its token cost scales poorly for complex scenes.Source: §4 · p. 6
• STLLM-Aligner uses learnable queries and cross-attention to combine point-cloud, video, and text features into a compact spatial representation before decoding.Source: §4 · p. 7
• High-frequency positional encoding adds ray-based geometric cues to image and point-cloud features before alignment.Source: §4 · p. 7

Evaluation

• Evaluation uses two LLM judges to assess whether predictions semantically match ground-truth answers rather than requiring exact-match output.The judges verify correctness instead of solving the spatio-temporal tasks themselves.Source: §5 · pp. 7–9

Results on Standard QA Metrics

• STLLM baselines improve upon existing models on standard sentence-level and n-gram QA metrics, but spatio-temporal reasoning remains unsolved.Source: §5 · p. 8

Results on LLM-Judge

• 41.89%/46.50% LLM-Judge accuracy is achieved by STLLM-Aligner, exceeding directly finetuned LLaVA-Video-7B-Qwen† at 36.99%/42.37%.The two values correspond to ChatGPT-4o and Gemini 2.0 Flash, respectively.Source: §5 · p. 8, Table 2 · p. 8

Task difficulty

• Find My Item and Action Planning are generally harder because they require open-ended answers combining spatial and temporal reasoning.Relative Direction and Relative Distance instead demand strong spatial understanding.Source: §5 · p. 8

Judge comparison

• ChatGPT-4o and Gemini 2.0 Flash largely agree on model rankings, although Gemini reports roughly 1.5× higher scores on more open-ended tasks.This supports the robustness of comparative evaluation across judges.Source: §5 · p. 8

Cross-Dataset Evaluation

• REA-finetuned models transfer to SQA3D, with LLaVA-Video-7B-Qwen improving from 43–45% to 52–53% average accuracy.LLaVA-OV-Qwen2-7B also improves from 42–44% to 44–49%.Source: §5 · p. 9, Table 3 · p. 9

6 CONCLUSION

• REA studies joint reasoning about unfolding events from egocentric observations and allocentric environment representations for embodied AI.Source: §6 · p. 9
• The dataset collection pipeline produces REA, a five-task benchmark for studying this capability.Source: §6 · p. 9
• The authors show that classic MLLMs struggle with spatio-temporal prompts, while equipping them with point-cloud understanding improves results significantly.Source: §6 · p. 9
• The work remains a first step, leaving open which training data and MLLM components best improve spatio-temporal reasoning.Source: §6 · p. 9
• Spatio-temporal reasoning may benefit embodied AI but could also enable unnoticed mass surveillance, requiring careful deployment.Source: §6 · p. 10

A.1 DATASET

• Human verification reviewed 100 test samples, with two expert annotators marking a sample correct only when both agreed.Source: §A.1 · p. 14
• QA quality requires correct spatial relation, temporal logic, semantic correctness, and question clarity, while spatio-temporal reasoning must use both layout and action sequence.Source: §A.1 · p. 14
• 80% of REA questions require point-cloud access because query objects are often absent from the query video.Source: §A.1 · p. 14, Table 4 · p. 14

A.2 LLM-JUDGE

• LLM-Judge prompts were iteratively refined against human judgments until agreement exceeded 97%, with stricter handling for open-ended tasks.Source: §A.2 · p. 14

B LLM JUDGE PROMPT

• Task-specific judges compare predictions with ground truth using criteria tailored to relative direction, distance, affordance, item finding, and action planning.Source: B LLM JUDGE PROMPT · pp. 15–16

C INFERENCE INSTRUCTION PROMPTS FOR VIDEOLLMS

• Existing VideoLLMs receive 25 multi-view images as static scene descriptions and 32 query-video frames, with prompts explicitly directing attention to both streams.Source: C INFERENCE INSTRUCTION PROMPTS FOR… · p. 17

D TRAINING DETAILS

• Training uses next-token prediction, frozen modality encoders, trainable projectors, and LoRA adaptation of the LLM decoder; STLLM-Aligner also updates alignment.Source: D TRAINING DETAILS · p. 17
• The models train on REA for one epoch using four NVIDIA H200 GPUs, balancing efficiency and generalization.Source: D TRAINING DETAILS · p. 17
• LoRA enables efficient parameter updates while mitigating overfitting to fixed answer templates and preserving broader output generalization.Source: D TRAINING DETAILS · p. 18
• STLLM-Aligner uses Nquery learnable embeddings, while STLLM-3D training defines local point-cloud groups through sampled centers, radius, and neighborhood size.Source: D TRAINING DETAILS · p. 18, Table 5 · p. 18, Table 6 · p. 18

D.1 SYSTEM PROMPT

• The system prompt states that the first 1024 tokens encode learnable queries for global 3D scene objects and locations, followed by egocentric action tokens.Source: §D.1 · p. 18

E QUANTITATIVE RESULTS

• Table 7 reports LLM-Judge accuracy in percent, with higher values better, under ChatGPT-4o and Gemini 2.0 Flash judges.Source: Table 7 · p. 19
• 41.43% / 45.87% overall accuracy is achieved by STLLM-Aligner (w/ Pos. Enc.) with LoRA, versus 35.38% / 36.37% without it.The paired values correspond to ChatGPT-4o and Gemini 2.0 Flash, respectively.Source: E QUANTITATIVE RESULTS · p. 19
• STLLM-Aligner‡ (w/ Pos. Enc.) with 1024 Nquery outperforms its 32-query version, although the improvement is relatively modest.Source: E QUANTITATIVE RESULTS · p. 19
• Across the reported settings, STLLM baselines outperform directly finetuned LLaVA-Video and LLaVA-OV counterparts.The comparison is reported in Tables 7 and 8.Source: E QUANTITATIVE RESULTS · p. 19, Table 7 · p. 19, Table 8 · p. 19

F QUALITATIVE RESULTS

• The proposed model correctly answers examples involving relative direction, distance, item localization, next-object prediction, and action planning.Examples include the microwave’s position, stable distance to the hob, tarragon retrieval, hob interaction, and washing the knife next.Source: F QUALITATIVE RESULTS · pp. 20–22
• Baseline models frequently give incorrect explanations or predictions when spatial relationships must be grounded across actions.Errors include denying the microwave relation, inferring movement toward the hob, mislocalizing tarragon, selecting the oven, and proposing the wrong next action.Source: F QUALITATIVE RESULTS · pp. 20–22

F.1 LLM JUDGE FAILURE CASE

• ChatGPT-4o can mark a prediction incorrect even when it matches the ground-truth object and provides a reasonable justification.The reported example concerns predicting the fridge as the next interaction object.Source: §F.1 · pp. 22–23
• Differences between ground truth and prediction explanations can be speculative while preserving the same core intent and outcome.Source: §F.1 · p. 22

F.2 MODEL FAILURE PREDICTION

• A separate failure example shows the model incorrectly judging the hob and freezer as being on the same side of the person.Source: §F.2 · p. 23

G.1 DATASET STATISTICS

• REA dataset statistics are reported by task and split, with separate tables covering task counts and overall dataset properties.Source: §G.1 · p. 23, Table 9 · p. 23, Table 10 · p. 23

G.2 DATASET GENERATION PIPELINE RUNTIME

• The pipeline reports runtimes for generating 5,000 VQA samples on a single NVIDIA RTX 4090 GPU; scene reconstruction takes 5 seconds per scene, and registration under 4 hours total.Source: §G.2 · p. 24

G.3 QA TEMPLATES

• QA templates cover relative direction, relative distance, find-my-item, furniture affordance, and action-planning questions with structured or free-form answers.Source: §G.3 · pp. 24–25

G.4 ANSWER GENERATION PROMPTS

• Answer-generation prompts constrain outputs to use known directions, avoid unsupported inferences and invented layouts, and describe locations, next interactions, or actions naturally.Source: §G.4 · p. 26

G.5 LICENSES

• The source datasets EPIC-KITCHENS, EPIC-FIELDS, and VISOR are licensed under CC BY-NC 4.0.Source: §G.5 · p. 28

H EXTENDED LIMITATIONS

• Fixed answer templates for Relative Direction and Relative Distance risk overfitting to format instead of encouraging diverse responses.The authors suggest LLM-based paraphrasing or refinement as a mitigation for better generalization to non-templated QA pairs.Source: H EXTENDED LIMITATIONS · p. 28

I LLM USAGE

• LLMs assisted with language editing, generated and refined QA pairs, and served as automated judges, while the authors retained responsibility for the research and claims.Source: I LLM USAGE · p. 28