PaperlayerView exact revision on arXiv

Source-linked AI summary

MARPLE: A Benchmark for Long-Horizon Inference

Emily Jin, Zhuoyi Huang, Jan-Philipp Fränken, Weiyu Liu, Hannah Cha, Erik Brockbank, Sarah Wu, Ruohan Zhang, Jiajun Wu, Tobias Gerstenberg

arXiv:2410.01926v1cs.LG

TL;DR

MARPLE addresses the lack of benchmarks for long-horizon, multimodal inference in complex everyday scenarios. It evaluates culprit identification from simulated household histories using vision, language, and audio, and finds that human participants outperform Monte Carlo and GPT-4 baselines.

  • Problem

    Existing benchmarks focus mainly on short-horizon reasoning or visual stimuli, leaving long-horizon multimodal inference in complex everyday scenarios undercovered.

    Source: §1 · p. 1

  • Method

    MARPLE uses procedurally generated household environments and agent behaviors to evaluate culprit identification from visual, language, and audio evidence.

    Source: §1 · pp. 1–2, §4 · p. 6

  • Results

    Human participants outperform both Monte Carlo and GPT-4 baselines, which struggle respectively with future-state prediction and generalization, and with understanding environmental changes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark’s long-horizon multimodal inference tasks remain challenging for current AI models.

    Source: §7 · pp. 9–10

    The benchmark focuses on two-agent “whodunit” scenarios, although the simulator can support multiple agents acting in parallel.

    Source: §7 · p. 9

Abstract

from arXiv · show

Reconstructing past events requires reasoning across long time horizons. To figure out what happened, we need to use our prior knowledge about the world and human behavior and draw inferences from various sources of evidence including visual, language, and auditory cues. We introduce MARPLE, a benchmark for evaluating long-horizon inference capabilities using multi-modal evidence. Our benchmark features agents interacting with simulated households, supporting vision, language, and auditory stimuli, as well as procedurally generated environments and agent behaviors. Inspired by classic ``whodunit'' stories, we ask AI models and human participants to infer which agent caused a change in the environment based on a step-by-step replay of what actually happened. The goal is to correctly identify the culprit as early as possible. Our findings show that human participants outperform both traditional Monte Carlo simulation methods and an LLM baseline (GPT-4) on this task. Compared to humans, traditional inference models are less robust and performant, while GPT-4 has difficulty comprehending environmental changes. We analyze what factors influence inference performance and ablate different modes of evidence, finding that all modes are valuable for performance. Overall, our experiments demonstrate that the long-horizon, multimodal inference tasks in our benchmark present a challenge to current models.

1 Introduction

MARPLE targets a gap in evaluating long-horizon, multimodal inference in everyday scenarios, using simulated household activities and a human comparison to test who caused an environmental change.

1 Introduction

  • Existing benchmarks largely emphasize short-horizon reasoning or visual stimuli, leaving long-horizon multimodal inference in complex everyday scenarios undercovered.Humans readily combine sensory modalities when inferring what happened.Source: §1 · p. 1
  • MARPLE tests whether a model can identify which of two agents caused a household state change from evidence about their behavior and the environment.The evidence can include visual, language, and audio cues available up to a given timestep.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • MARPLE extends Mini-BEHAVIOR with hierarchical agents whose household interactions generate visual, language, and audio evidence in procedurally generated environments.The simulator supports high-level behavior research with fast prototyping and training.Source: §1 · p. 2
  • Human participants outperform both Monte Carlo and GPT-4 baselines, which struggle respectively with future-state prediction and generalization, and with understanding environmental changes.The benchmark also contributes training and inference data and evaluation metrics for these tasks.Source: §1 · p. 2

2 Related Work: Cognition-Inspired AI Inference Benchmarks

Cognition-inspired benchmarks cover different kinds of physical, agent, and social reasoning, but long-horizon multimodal inference in complex everyday scenarios remains a gap.

2 Related Work: Cognition-Inspired AI Inference Benchmarks

  • Cognitive-science frameworks for causal reasoning include force dynamics, mental models, causal models, and counterfactual simulation.The paper positions its work alongside both search-based and LLM-based inference methods [41] [34] [42].Source: §2 · pp. 2–3
  • Table 1 compares visual reasoning, causal reasoning, and cognition-inspired benchmarks, with MARPLE characterized as long-horizon, high-level, and supporting vision, text, and audio.The table’s comparison dimensions include average stimulus length, object diversity, and annotated data generation.Source: Table 1 · p. 3
  • Prior benchmarks cover physical events, agent behavior, and multi-agent social behavior, while several provide human judgments or performance baselines for comparison [1] [7] [14] [43].Most use visual stimuli, though some support multimodal input [18] [20], including audio and vision.Source: §2 · p. 3
  • Earlier benchmarks primarily emphasize short-term reasoning or single-modality stimuli, rather than long-horizon inference across multimodal everyday scenarios.MARPLE is framed as a way to evaluate recent inference methods on this undercovered setting.Source: §2 · p. 3

3 MARPLE Benchmark

MARPLE poses agent attribution as inference from multimodal histories: determine which of two agents most likely caused a state that appears uniquely in one agent’s trajectory.

3 MARPLE Benchmark

  • A “whodunit” query selects a state unique to one of two agents’ trajectories, and models use multimodal observations from their activity histories to identify its cause.For example, a laundry-machine-on state can distinguish the agent whose mission was doing laundry.Source: §3 · p. 3
  • The formalization decomposes missions into mid-level subgoals and low-level actions, with each subgoal depending on past completion and enabling future steps.The problem is represented as a POMDP with states, actions, rewards, transitions, observations, an observation function, and a discount factor.Source: §3 · p. 4
  • The target is the probability that an agent caused the query state at an intermediate timestep, with the available observation horizon affecting inference difficulty.Inference becomes trivial when observations extend to the final timestep.Source: §3 · p. 4
  • Solving the task requires world, observation, and agent-policy models; agent policies are unknown to the inference model and must be learned from training trajectories.The setup represents each agent’s mission preferences as a prior distribution used to sample missions.Source: §3 · p. 4
  • Evaluation measures the probability of correctly choosing the responsible agent, with stronger models expected to reach high accuracy earlier in the evidence sequence.Scenario difficulty, environment complexity, similarity between agent behaviors, and inference horizon also affect performance.Source: §3 · p. 4

4 MARPLE Household Simulator

The MARPLE Household Simulator procedurally generates household settings and agent behaviors, then provides multimodal activity evidence across five inference scenarios.

4 MARPLE Household Simulator

  • The simulator extends Mini-BEHAVIOR with multimodal stimuli, procedural generation of agent behaviors, and a human-experiment interface.It abstracts away low-level physical details to support efficient prototyping of high-level, long-horizon inference models.Source: §4 · pp. 4–5
  • Language describes an agent’s next intended subgoal, revealing future information whose usefulness for identifying the mission varies by scenario.When agents share many subgoals, language may distinguish their missions only near the end.Source: §4 · p. 5
  • Audio maps actions to sounds, but shared sounds for actions such as navigation provide only partial evidence about an agent’s low-level behavior.Audio can help resolve uncertainty when visual evidence, such as agent proximity, is inconclusive.Source: §4 · p. 5
  • A hierarchical planner samples a mission, decomposes it into subgoals with a finite-state machine, and uses a low-level planner to produce an action sequence.The simulator backend represents missions as subgoals and procedurally generates environments from mission definitions and configuration files.Source: Figure 3 · p. 5
  • MARPLE defines five inference scenarios by pairing ten long-horizon household missions and selecting a meaningful subgoal unique to one agent as the query state.The experiments use the simplest assignment: each agent performs one mission.Source: §4 · p. 6

5 Inference Methods and Baselines

The simulation baseline uses learned agent policies to roll out possible futures and estimate which agent could reach the query state. It is evaluated alongside GPT-4 and human participants, with simulation variants using different combinations of visual, audio, and language evidence.

5.1 Simulation with Learned Agent Models

  • MCTS rolls out learned agent policies from the current state, estimates each agent’s chance of reaching the query state, then normalizes predictions with softmax.The agent policies are learned from past behavior, and the simulator supplies the world model.Source: §5.1 · p. 6
  • The four simulation variants condition action prediction on vision alone, vision plus audio, vision plus language, or all three modalities.Audio refines the vision-only action distribution; language supplies a subgoal, and audio can refine the language-conditioned prediction.Source: §5.1 · p. 6

5.2 Additional Baselines

  • GPT-4 predicts which agent caused the query state from scene-graph observations, while human participants incrementally compare visual trajectories.The GPT-4 baseline uses a zero-shot step-by-step prompt; selected experiments also include examples from two trajectories.Source: §5.2 · pp. 6–7
  • Figure 4 presents baseline performance across scenarios in increasing difficulty order, including simulation models trained in-distribution and out-of-distribution.Its error bands represent 95% confidence intervals across tested trajectories.Source: Figure 4 · p. 7

6 Experiments and Results

Across the long-horizon inference scenarios, human participants make correct inferences with less evidence than the model baselines. Simulation models struggle to generalize to unseen environments, while language and audio evidence improve performance in complementary ways.

6.1 Benchmarking Model Performance in Long-Horizon Inference Scenarios

  • At 50% of the “Who picked up the plant?” trajectory, simulation reaches 0.9 accuracy by using ToggledOn(light) as a precursor to Pickup(plant), while GPT-4 infers incorrectly.Simulation methods generally outperform GPT-4 and reach 1.0 accuracy by the end of the trajectory.Source: §6.1 · p. 8, Figure 5 · p. 8
  • GPT-4 fails to converge on “Who turned on the shower?” and “Who turned on the laundry?”; its reasoning was biased toward changes in agent state rather than environment state.In-context learning improves performance on these tasks but does not resolve the failure to converge.Source: §6.1 · p. 8
  • Humans outperform all models, reaching 0.8 accuracy with an average of 48% of the evidence and requiring 10% and 47% less evidence than the best MCTS variant in-distribution and GPT-4, respectively.The comparison focuses on how early methods infer correctly; accuracies generally rise over time and converge at the trajectory’s end, except for GPT-4.Source: §6.1 · pp. 7–8

6.2 Benchmarking Generalization Capabilities of Simulation Models

  • Out-of-distribution, the human–best-simulation evidence gap grows from 10% to 33% less evidence, as simulation accuracy drops in unseen environments.The passage attributes the simulation drop primarily to learned agent models that generalize poorly, reducing action-prediction and rollout accuracy.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 Benchmarking in Multimodal Settings

  • For “Who turned on the shower?”, the vision, language, and audio variant consistently outperforms the other simulation variants; language appears more valuable than audio.Audio slightly improves on vision alone, but its benefit is limited when only one timestep of audio is used for one action prediction. The language-conditioned policy reaches 0.92 inference-trajectory action accuracy versus 0.86 for vision-only.Source: §6.3 · p. 9, Figure 6 · p. 9

6.4 Additional Benchmarking Experiments

  • As agents’ preferences for their own missions converge toward preferences for the other mission, vision-only and audio-augmented models need more trajectory evidence to reach 0.6 accuracy.With primary-mission preference 1.0, both reach 0.6 accuracy at around 40% of the trajectory; at preference 0.6, audio-augmented requires 70% and vision-only 85%.Source: §6.4 · p. 9

7 Limitations and Conclusion

MARPLE remains challenging for current AI models, which fall short of humans on long-horizon multimodal inference. The authors identify boundaries in the setup while presenting the benchmark as a foundation for inference research.

7 Limitations and Conclusion

  • The current setup features only two agents, although the simulator can support multiple agents acting in parallel.The passage also suggests future scenarios in which either agent could cause the queried change, but its wording is unclear.Source: §7 · p. 9
  • Current AI models, including Monte Carlo tree search and LLM methods, fall short of humans in leveraging multimodal stimuli and performing long-horizon inference.The authors also note that the current setup remains challenging because it requires understanding diverse agent behaviors and generalizing across environments.Source: §7 · pp. 9–10
  • The simulator does not yet support agent interactions, which would add complexity to the inference process.Source: §7 · p. 9

Supplementary for MARPLE: A Benchmark for Long-Horizon Inference

The supplementary material details MARPLE’s household scenarios, multimodal simulator, hierarchical planner, and data generation. It also describes the benchmark’s difficulty factors and computational trade-offs.

A.1 Overview

  • MARPLE pairs ten household missions into five inference scenarios, each asking which of two agents most likely reached a query state unique to one mission.The scenarios span increasing difficulty based on average inference horizon and mission similarity.Source: Table A.1 · p. 13

A.2 Inference Scenario Setup

  • Each scenario pairs two household missions and selects a query state unique to one agent; missions comprise subgoals specifying target objects, locations, actions, and states.The inference question asks which agent is more likely to have performed the action corresponding to that query state.Source: §A.2 · p. 13

A.3 Inference Scenario Difficulty

  • Inference difficulty depends on horizon length and mission similarity; longer horizons increase prediction uncertainty, while similar mission trajectories make attribution harder.The chosen scenarios span a range of mission similarities.Source: §A.3 · pp. 13–14, Table A.2 · p. 14

B.1 Simulator: A Multimodal GridWorld Environment

  • The simulator builds on Mini-BEHAVIOR’s procedurally generated GridWorld and adds visual, language, and audio evidence for household activities.Language includes agent intent and testimony; audio uses one-second recordings mapped to actions.Source: B MARPLE Household Simulator: Details · p. 15, §B.1 · p. 15

B.2 Planner: A Hierarchical Planner for Agent Behavior Generation

  • A hierarchical planner samples missions, selects the next uncompleted feasible subgoal, then uses A-star search to plan actions toward it.Infeasible missions terminate; the simulator updates the environment as actions execute.Source: §B.2 · p. 16

B.3 Usage: Ensuring diversity and complexity

  • Scenario configuration files specify required initial conditions and optional environment constraints; extra elements are randomly placed before the planner generates trajectories.Environment size and numbers of rooms and objects can be scaled, while the state representation grows exponentially with array size.Source: §B.3 · pp. 16–17

C.2 Data Generation

  • Each scenario has 500 paired test trajectories across ten generated rooms and two training sets of 5,000 paired trajectories each.One training set reuses the ten test environments; the other uses 5,000 procedurally generated environments.Source: §C.2 · p. 18

D.1 MARPLE Simulator: Computational Resources

  • The simulator runs at 600 FPS and needs one frame per primitive action, enabling faster experiments than detailed physical simulators that are unnecessary for this high-level inference setup.The authors contrast this with BEHAVIOR at 60 FPS and 100 frames per primitive action.Source: §D.1 · p. 19

D.2 Experiment Resource Requirements

The appendix reports the resources and statistical procedure used for experiments, along with the Monte Carlo algorithm and learned-agent model implementation. It records substantial differences in trial time and cost across baselines.

D.2 Experiment Resource Requirements

  • Each mental-simulation trial took 1.5 hours, while each GPT-4 trial took 1 minute and cost $44.00.Across 50 trials, GPT-4 cost $2,200; the mental-simulation variants required 300 total hours, while the human study took roughly three hours.Source: §D.2 · p. 19

D.3 Statistical Significance

  • The experiments use 50 trials, plotting inference accuracy with 95% confidence intervals calculated as x̄ ± σ√n, with n = 50.The authors say the resulting error bars are small enough to draw meaningful conclusions.Source: §D.3 · p. 19

E.1 Algorithm

  • Algorithm E.1 uses Monte Carlo sampling with learned agent models to estimate which agent caused the query state.The supplied algorithm description states that it is used for simulation with Monte Carlo sampling and learned agent models.Source: §E.1 · p. 20, §D.3 · p. 20

E.2 Implementation Details

  • The learned agent policies have vision-only, audio-augmented, language-conditioned, and audio-augmented language-conditioned variants.The vision policies use a ViT encoder and MLP action predictor; language-conditioned variants use a ViT encoder with four decoders.Source: §E.2 · p. 20
  • Agent models are trained for all ten missions on two behavior dataset types, using batch size 64, learning rate 1e-4, Adam, and 20 epochs.Training uses four warmup epochs followed by cosine annealing, with gradient accumulation.Source: §E.2 · p. 20

F Ablations of Simulation with Learned Agent Models

The modality ablations find that the full vision, language, and audio baseline consistently outperforms the other variants. The appendix also includes a simplified GPT-4 prompt template.

F Ablations of Simulation with Learned Agent Models

G Prompts for GPT-4

H Additional Results of Open-Source LLMs

The open-source LLMs struggle on MARPLE’s inference task. Llama-3.1’s accuracy progression varies by scenario, while Qwen2’s does not improve as more of the trajectory is shown.

H Additional Results of Open-Source LLMs

I Additional Results of GPT-4 with In-Context Learning

In-context learning improves GPT-4’s performance on two scenarios where zero-shot prompting failed to converge, but it still does not converge. The paper analyzes GPT-4’s reasoning and describes how human participants incrementally judged agent trajectories.

I Additional Results of GPT-4 with In-Context Learning

J Analysis of GPT-4 Reasoning

  • In the laundry example, GPT-4’s reasoning favors agent-state cues over the laundry’s state change, a tendency the authors identify in both zero-shot and in-context prompting.The relevant environmental evidence is the laundry’s ToggledOn state changing from False to True.Source: J Analysis of GPT-4 Reasoning · p. 25

K Details on Human Experiments

  • Two human experts each completed 50 trials after habituation, viewing paired agent trajectories step by step and giving judgments at 11 timesteps on a 0–100 Agent A-to-Agent B scale.The study diagram shows participants advancing through Gridworld scenes with an arrow and responding with a slider.Source: K Details on Human Experiments · p. 26, Figure K.1 · p. 26

Related papers

Loading 2410.01926v1…