PaperlayerView exact revision on arXiv

Source-linked AI summary

The N-Body Problem: Parallel Execution from Single-Person Egocentric Video

Zhifan Zhu, Yifei Huang, Yoichi Sato, Dima Damen

arXiv:2512.11393v3cs.CV

TL;DR

The paper asks how to predict a feasible parallel execution from one single-person egocentric video, rather than relying on multiple demonstrations of the same task. It formalizes the N-Body Problem and uses structured prompting to guide a VLM’s reasoning about spatial, object, and causal constraints. Guided Gemini 2.5 Pro is reported to achieve high coverage with low constraint violations on the evaluated videos.

  • Problem

    The N-Body Problem asks how to reallocate work from a single-person video across multiple workers while respecting real-world feasibility constraints.

    Source: §1 · p. 1

  • Method

    The paper formalizes parallel execution from one video and uses structured VLM prompting to reason about spatial, object, and causal constraints.

    Source: §1 · pp. 1, 3

  • Results

    91.3% action coverage, with collision, object, and causal conflicts at 4.2%, 0.23%, and 18%, respectively, is reported for guided Gemini on 80 HD-EPIC videos.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The experiments report that Gemini 2.5 Pro can be guided to produce executions with high coverage and reasonable speed-up while maintaining low constraint violations.

    Source: §5 · pp. 14–15

    There is no ground-truth parallel execution, so evaluation identifies invalid executions rather than matching plans to a single reference.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Humans can intuitively parallelise complex activities, but can a model predict this from observing a single person? Given one egocentric video, we introduce the N-Body Problem: predicting how N individuals, can hypothetically perform the same set of tasks. The goal is to maximise speed-up, but naive assignment of video segments to individuals often violates real-world constraints, leading to physically impossible scenarios like two people using the same object or occupying the same space. To quantify this, we formalise the N-Body Problem and propose a suite of metrics to evaluate both performance (speed-up, task coverage) and feasibility (spatial collisions, object conflicts and causal constraints). As a proof of concept, we introduce a structured prompting strategy that guides a Vision-Language Model (VLM) to reason about the 3D environment, object usage, and temporal dependencies, producing a viable parallel execution. On 100 videos from EPIC-Kitchens and HD-EPIC, for $N = 2$, our structured prompt improves action coverage by 45% over a baseline prompt for Gemini 2.5 Pro, while simultaneously slashing collision rates, object and causal conflicts by 51%, 52% and 55% respectively.

1 Introduction

The N-Body Problem asks whether a multi-agent parallel execution can be predicted from one single-person egocentric video while respecting real-world constraints. The paper formalizes this task, evaluates its performance and feasibility, and uses structured VLM prompting to guide plan generation.

1 Introduction

  • The motivating question is whether a model can predict parallel execution from observing one person, whose egocentric activity may already pursue multiple goals in parallel.The passage situates this question in prior work on unscripted egocentric videos.Source: §1 · p. 1
  • The N-Body Problem predicts a multi-agent parallel execution from one video showing a single-agent execution trace.Plans must respect spatial, object, and causal constraints, not merely maximize concurrent work.Source: §1 · pp. 1–2
  • A structured prompt guides the VLM to reason about 3D environments, object use, and task dependencies for training-free scheduling.The setting assumes one video and no prior task knowledge.Source: §1 · p. 2
  • 91.3% action coverage, with collision, object, and causal conflicts at 4.2%, 0.23%, and 18%, respectively, is reported for guided Gemini on 80 HD-EPIC videos.The evaluation covers 100 long videos from two egocentric datasets and does not compare plans against a single ground-truth parallel execution.Source: §1 · p. 3

2 Related Works

Prior work studies egocentric activity understanding, concurrent intentions, collaboration, and procedural learning, but does not predict a feasible parallel execution from one demonstration. The paper frames this as scheduling dependent tasks across limited workers and elicits spatial reasoning from VLMs through prompting.

Egocentric Videos

  • Egocentric vision supports passive activity understanding along one timeline, but prior work does not reallocate the same work across concurrent workers while preserving spatial, object, and semantic feasibility.Task-graph annotation [28] can enforce causal order but does not capture physical constraints of space and objects.Source: §2 · p. 3

Task Parallelisation

  • Prior parallelisation work describes or segments concurrent intentions, while no prior work aims to predict a parallel execution from a single demonstration.The Lemma dataset records one-person and two-person task completion, but its videos are short-term (2 minutes); procedural learning typically uses several videos of the same task [28].Source: §2 · p. 4

Spatial Reasoning in Vision-Language Models (VLMs)

  • The spatial prompt discretises the environment and aligns time segments with spatial zones to guide collision avoidance without modifying VLM weights.This task-driven approach responds to reported spatial-reasoning weaknesses in VLMs [16] [43].Source: §2 · p. 4

3 Problem Formulation

The formulation maps a single-person video into non-overlapping segments assigned across agents, then evaluates parallel plans by their coverage and speed-up while minimizing physical and logical violations. It treats feasibility as multi-dimensional, with spatial, object, and causal constraints.

3.1 Input and Output Representations

  • The input is a single-person video divided into non-overlapping assignable segments, which need not match standard action boundaries and may leave some footage unused.Multiple actions may belong to one segment when one agent performs them together.Source: §3.1 · p. 5
  • Each segment is assigned a start time in one agent’s execution, keeps its source duration, and cannot be assigned to another agent.Segment order may be shuffled, and agents may wait in idle states for viable tasks.Source: §3.1 · p. 5

3.2 Goals

  • The goals are to maximize task coverage and speed-up, defined as sequential execution time divided by the parallel execution time of the longest-working agent..Source: §3.2 · pp. 5–6

3.3 Constraints

  • No single measure captures plan quality; the formulation treats goals as quantities to maximize and constraints as quantities to avoid or minimize.Different parallel executions with the same performance measures are considered equivalent.Source: §3.3 · pp. 6–7
  • Spatial collision scoring distinguishes relocatable actions from actions tied to fixed locations, counting colliding frames involving restricted locations relative to total plan duration.Agents sharing space during relocatable actions are not counted as spatial collisions.Source: §3.3 · p. 6
  • Object conflict measures concurrent access by multiple agents to the same unique object, excluding replaceable items from the unique-object set.The rate counts conflicting frames relative to the parallel execution duration.Source: §3.3 · p. 6
  • Causal constraints preserve required action order or duration; CVR counts causal pairs that are reversed, performed in parallel, or missing a prerequisite.The rate is the number of violated pairs among all causal pairs.Source: §3.3 · pp. 6–7

4 Experiments

The experiments evaluate parallel executions on long kitchen videos using separate performance and feasibility metrics, then test progressively structured VLM prompts and other scheduling baselines. Spatial prompting lowers spatial collision rates while improving coverage, and increasing the agent count brings greater speed-up alongside more constraint violations.

4.1 Datasets and Annotations

  • On 100 long videos from HD-EPIC and EPIC, the evaluation uses dataset annotations to measure action coverage and physical feasibility without a ground-truth parallel execution.The set comprises 80 HD-EPIC and 20 EPIC videos across 24 scenes, with an average duration of 25.3 minutes; the authors argue that multiple valid solutions make a single ground-truth parallel execution ill-defined.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 Evaluation Metrics

  • Evaluation separates frame and action coverage and speed-up from spatial collision, object conflict, and causality violation rates.Action coverage requires temporal overlap above 0.5 IOU; speed-up is measured separately to avoid rewarding solutions that omit work, while SCR considers collisions at restricted locations.Source: §4.2 · pp. 8–10

4.3 Model Setup

  • Gemini 2.5 Pro is tested with progressively expanded prompts, culminating in explicit zone-and-time information to discourage concurrent assignments to the same zone.The spatial prompt divides the ground plane into zones and supplies start time, end time, and zone number; the model uses 1 FPS sampling. Comparisons also include Qwen2.5-VL-72B, simple and HEFT-style schedulers, and post-processed HiERO.Source: §4.3 · pp. 10–11

4.4 Main Results

  • Spatial prompting consistently lowers SCR and increases coverage, with slightly lower speed-up; goals-only prompting achieves the highest speed-up and significantly improves coverage while decreasing CVR.The base prompt yields 1.58x speed-up and 62.9% action coverage, with 8.7% restricted-location collision and 40.6% CVR; the EPIC evaluation videos show a similar trend.Source: §4.4 · pp. 12–13, Table 1 · p. 12, Table 2 · p. 12
  • The naive and HEFT action-scheduler baselines have high collision rates, while HEFT without privileged action timestamps has higher OCR; Qwen2.5-72B returns no output for 42 of 100 videos and has poor coverage on its 58-video subset.HiERO post-processing reaches 1.72× speed-up on HD-EPIC and 1.84× on EPIC, but its spatial collision rates are 10.10% and 25.3%, respectively.Source: §4.4 · p. 13
  • Across three analyzed properties, prep is usually performed by the same agent as the corresponding step, and one agent often walks substantially farther than the other.The analysis also compares agents’ cooking-task time against non-cooking work; qualitative examples show parallel preparation for different ingredients or concurrent distinct tasks.Source: §4.4 · pp. 13–14, Figure 3 · p. 13, Figure 4 · p. 14
  • Scaling from N = 2 to N = 3 increases speed-up but also raises SCR, OCR, and CVR on HD-EPIC.Reported speed-up changes from 1.40 to 1.51 on HD-EPIC and from 1.35 to 1.64 on EPIC; the authors describe constraint reasoning as more challenging with more agents.Source: §4.4 · p. 14

5 Conclusion and Future Work

The paper introduces the N-Body Problem and explores structured prompting to generate parallel executions from a single egocentric video. Experiments on 100 videos report high coverage, reasonable speed-up, and low constraint violations, while highlighting unresolved reasoning challenges and future research directions.

5 Conclusion and Future Work

  • Guided Gemini 2.5 Pro produced executions with high coverage, reasonable speed-up, and low constraint violations in experiments on 100 videos.The conclusion also describes prompting for spatial, object, and causal constraints.Source: §5 · pp. 14–15
  • The model still struggles with precise temporal requirements and some task dependencies, relies heavily on explicit spatial prompting, and may over-restrict shared large objects.Examples include boiling durations, grinding beans before brewing, and two people placing food on one tray.Source: §5 · p. 15
  • Future work could improve causal modeling for unseen activities and spatial reasoning, test other scenarios, and synthesize full parallel executions.The proposed synthesis could be in pixel space or as fine-grained 3D interactions.Source: §5 · p. 15

Appendix

The appendix provides supplementary material on the teaser figure, spatial jump distance, spatial-prompt variants, recipe dependency annotations, object types, and the full VLM prompt.

Appendix

  • Section A expands the teaser figure and explains the supplementary videos.Source: Appendix · p. 19
  • Sections B and C introduce a secondary spatial jump metric and report experiments with different ways of providing spatial information to the VLM.Source: Appendix · p. 19
  • Sections D–F cover recipe dependency annotations, replaceable and unique objects, and the full VLM prompt.Source: Appendix · p. 19

A Qualitative Figure and Videos

The appendix expands the teaser figure with the segment splits and agent locations for two- and three-person executions, and points to supplementary videos with additional results.

A Qualitative Figure and Videos

B Spatial Jump Distance

The appendix measures relocation distances between an agent’s consecutive assigned segments rather than inserting travel-time gaps, which would require assumptions and could alter the proposed schedule.

B Spatial Jump Distance

  • Rather than inserting travel gaps, the method measures spatial distances in the predicted distribution, avoiding assumptions about walking speed and schedule shifts that could overlap object use.Source: B Spatial Jump Distance · p. 19
  • The Jump metric averages the spatial relocation distance between consecutive segments assigned to each agent.It summarizes potential “jumps” between the end location of one segment and the start location of the next.Source: B Spatial Jump Distance · p. 19
  • The appendix reports jump distances alongside the HD-EPIC and EPIC results in Tables S1 and S2.Source: Table S1 · p. 21, Table S2 · p. 21
  • The reported jumps are described as within a reasonable range, with a 0.5-metre step given as a natural example.Source: B Spatial Jump Distance · p. 21

C Spatial Prompt Variants

The spatial prompt tests trajectory input and zone-based representations to reduce collisions while preserving parallel speed-up. The results show a trade-off between larger zones and speed-up.

C Spatial Prompt Variants

  • The comparison tests raw trajectories, equal-sized zones at multiple scales, and GMM-clustered zones with dynamic sizes.The proposed prompt uses 120x120cm zones.Source: C Spatial Prompt Variants · p. 21
  • Larger zones reduce collision rates but increase speed-up, meaning the execution speeds up less as agents reserve more space exclusively.Raw trajectories provide only a small collision-rate reduction at 1.48× speed-up; GMM with 5 components and equal-sized 40 × 40cm zones achieve the same speed-up with much lower collision rates.Source: C Spatial Prompt Variants · p. 21, Table S3 · p. 22

D Recipe Causal Dependency Annotations

The appendix describes manually annotated recipe dependencies and reports causal-violation results by error type. The spatial prompt has causal performance comparable to Goal-Only among Gemini 2.5 methods.

D Recipe Causal Dependency Annotations

  • Manual annotation covers 56 recipes and 204 step-step dependencies; with 728 prep-step dependencies, the reported CVR covers 932 constraints across 60 videos.The step-step annotations were selected from 80 videos; 20 lacked recipes or recipe step dependencies.Source: D Recipe Causal Dependency Annotations · pp. 22–23
  • Among Gemini 2.5 methods, + Goal-Only has the lowest total causal error, while the spatial prompt performs comparably.Naive Half-Half and HEFT 1-min have the fewest Miss errors but most Reverse errors; HEFT GT start-end and Qwen2.5-VL-72B show the opposite pattern.Source: Table S4 · p. 23, D Recipe Causal Dependency Annotations · p. 23
  • Causal violations are broken down by prep-step versus step-step constraints and by reversed segment pairs versus missing causes.The manual annotations include only dependencies judged definitely necessary; doubtful dependencies are not labeled causal.Source: D Recipe Causal Dependency Annotations · pp. 22–23
  • The proposed prompt has the lowest normalized reversal and missed rates across videos in Figure S3.The figure normalizes Reverse Error and Miss Error by their respective total counts.Source: D Recipe Causal Dependency Annotations · p. 23

E Replaceable Objects

The appendix describes how objects are classified as replaceable or unique for the N-Body setting. The prompt bases replaceability on whether standard home kitchens typically have substitutes.

E Replaceable Objects

  • The prompt classifies objects by whether standard home kitchens are likely to have replacements, treating numbered names such as knife2 as general replaceable objects.It contrasts standard-shape clean knives with less readily duplicated butter knives or scales.Source: E Replaceable Objects · p. 24
  • Coffee cups, forks, glasses, knives, plates, and spoons are examples of replaceable objects.Source: E Replaceable Objects · p. 24
  • Air fryers, electric kettles, and coffee machines are examples of unique objects.Source: E Replaceable Objects · p. 24

F The Complete Prompt

The complete prompt asks a VLM to divide one egocentric video into two agents’ work while preserving full coverage, spatial feasibility, and task semantics. It specifies zone-based scheduling, dependency checks, and a single-agent fallback.

F The Complete Prompt

  • The planner rearranges original video clips into two agents’ streams to find efficient parallel execution while respecting dependencies and avoiding spatial conflicts.Each job corresponds to a segment from the original video.Source: F The Complete Prompt · pp. 25–26
  • Spatial scheduling uses trajectory zones: tasks in the same zone must be serialized, and agents are assigned different zones.The prompt says spatial reasoning should rely exclusively on the trajectory CSV and still respect task semantics.Source: F The Complete Prompt · p. 25
  • The prompt guards against semantic dependency disorder, including reversing coffee preparation steps or using the mixer or trash bin concurrently.Source: F The Complete Prompt · p. 25
  • Every segment of the original video must be covered, with no skipped duration and 100% total coverage.Source: F The Complete Prompt · pp. 25, 27
  • The output is JSON with P1 and P2 job lists; each job includes a new start time, original start and end times, and clip-summary text.Source: F The Complete Prompt · pp. 25–26
  • When constrained space makes multi-agent scheduling difficult, the prompt prefers a single-agent plan over a plan with high potential spatial collision.Source: F The Complete Prompt · p. 27
Loading 2512.11393v3…