Source-linked AI summary
Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulations
Yi Fei Cheng, Fan Yang, Iremsu Bas, Koichiro Niinuma, Narishige Abe, David Lindlbauer
TL;DR
LLM human simulators are increasingly used in applications where faithfully reproducing real behavior matters, but long-horizon real-world activity remains underexplored. The paper introduces a multi-scale fidelity framework and evaluates six trace-conditioned approaches on a 43-hour office dataset. Statistical priors align activity and sequence distributions most closely with real behavior, while over-segmenting days and reducing within-person variability.
Problem
Long-horizon real-world behavioral fidelity is largely unexplored despite the use of LLM simulators in downstream applications requiring faithful behavior.
Method
The paper introduces an evaluation framework spanning temporal granularities and analysis levels, then evaluates six trace-conditioned simulation approaches on a 43-hour multi-camera office dataset.
Results
Statistical priors reduce divergence on activity distributions and local transition structure but over-segment days and produce less day-to-day variation within individuals.
Takeaways & Limitations
Behavioral fidelity is non-uniform, so evaluation should span multiple metrics, temporal granularities, and individual- and population-level analyses.
Takeaways & Limitations
The findings are specific to one multi-room office, one work week, five individuals, six activity labels, and a fixed social configuration.
Abstract
from arXiv · showhide
As LLM-based human simulators are increasingly used for policy, evaluation, and training, they must faithfully reproduce real behavioral patterns. While prior work has examined behavioral fidelity in survey responses and dialogue, longer-horizon real-world activity remains largely unexplored. We introduce a framework for evaluating behavioral fidelity in long-horizon activity simulations across temporal granularities and levels of analysis. As a case study, we collect a 43-hour multi-camera dataset of in-the-wild office activity and compare trace-derived conditioning mechanisms: persona descriptors, few-shot exemplars, and statistical transition and time-of-day priors. We find that behavioral fidelity is not uniform across metrics: statistical priors bring activity and sequence distributions closest to real behavior, yet over-fragment routines and suppress within-person variability. These findings motivate a more holistic evaluation that spans multiple metrics, temporal granularities, and levels of analysis.
1 Introduction
The paper addresses the largely unexplored fidelity of LLM simulations for long-horizon real-world activity. It introduces a multi-scale evaluation framework and applies it to office activity simulations conditioned on real-world behavioral traces.
- Long-horizon behavioral fidelity remains largely unexplored compared with fidelity in simulated surveys and dialogue.
- The framework evaluates fidelity across activity-level, time-of-day, and day-level temporal granularities and individual- and population-level analyses.
- The case study uses a 43-hour multi-camera office dataset capturing in-the-wild activities of 55 people.
- Six simulation approaches vary in their use of real-world traces, including persona descriptors, few-shot full-day exemplars, and statistical activity and time-of-day priors.
- Statistical priors improve alignment with real activity and sequence distributions but produce over-segmented days and reduced within-person variability.
- Persona-based conditioning approaches differ only slightly by comparison, and population-level agreement can mask individual-level errors.
2 Related Work
Prior work has used LLM-based agents to generate believable behavior through cognition, embodiment, and scaling, but this paper emphasizes fidelity to real-world physical activity. It targets open-ended, long-horizon behavior rather than isolated questions or dialogue.
- LLM-based agents have been scaffolded with memory and planning to perform complex multi-step tasks across domains.
- Generative-agent research models everyday activities and social interactions, while related work adds needs, emotion, personality, physical grounding, and embodiment.
- Other studies explore scaling human simulations to large multi-agent societies.
- Believable behavior is insufficient for policy evaluation, which requires simulations to faithfully reproduce real human behavioral patterns.
- This work focuses on open-ended, long-horizon behavioral patterns grounded in traces of real-world physical activity.
3 Long-Horizon Activity Simulation
The paper models long-horizon human activity as autoregressive episode generation by an interactive agent. Each next activity depends on prior episodes, environmental observations, and a persona describing behavioral attributes.
- A long-horizon human activity simulator autoregressively generates activity sequences within an interactive environment over a day.
- Each activity episode is one uninterrupted stretch of a single labeled activity with a start and end time.
- At each step, the agent samples an activity conditioned on episode history, prior environment observations, and a persona.
- Rolling out the policy for M steps produces a contiguous trajectory of episodes spanning the simulated day.
4 Evaluating Behavioral Fidelity
The evaluation framework defines fidelity as similarity between real and simulated activity distributions across temporal granularities and levels of analysis. It combines trajectory-based reference data with distributional, count, duration, and sequence metrics.
- Behavioral fidelity is the extent to which simulated activity trajectories reproduce patterns observed in real human activity traces.
- Temporal granularity: The framework captures local episode dynamics, time-of-day allocation and frequency, and day-level activity patterns and switches.
- Level of analysis: Fidelity is assessed at individual and population levels because aggregate agreement can coexist with collapsed individual differences.
- Statistical framework: Real and simulated behavior are represented as distributions P and Q, with fidelity defined by how closely Q approximates P.
- Statistical framework: The framework estimates real behavior from human activity trajectories and simulated behavior from policy rollouts conditioned on personas representing individuals.
- Metrics: Distributional representations use TV, JS, and forward and backward KL divergence, while count and duration statistics use L1 distance and signed bias.
5 Data Collection
The study collected 43 hours of multi-camera office activity and manually annotated a six-label vocabulary, focusing longitudinal analysis on five individuals with sufficient repeated observation.
- 43 hours of continuous video were recorded with 11 synchronized cameras across a four-room shared office.The study was approved by an IRB.
- The activity vocabulary contains six labels: use computer, engage in conversation, have meal, take a break, use phone, and out of room.The same label space was used for annotation and simulation.
- All labels in the real corpus were produced through manual annotation of the recordings.
- The final real-activity corpus contains trajectories from 55 unique individuals across 43 recorded hours.Five individuals contributed substantially more longitudinal data than the remaining captured individuals.
- The five longitudinally observed individuals averaged 4 ± 1 days and 7 ± 1 hours per day in the office, supporting long-horizon persona construction.The remaining 50 individuals were typically observed for 1 ± 1 day and 1 ± 1 hour per day.
6 Experiments
The experiments instantiate a shared perceive–plan–act agent architecture in a scene-graph office environment and compare six conditioning approaches, from no persona to statistical and hybrid priors.
- Each method generated a five-agent, eight-hour simulation mirroring the recorded office and modeled the five longitudinally observed individuals.Simulated and real corpora were compared to assess conditioning approaches.
- Agent Architecture: The simulation environment is represented to agents as a hierarchical scene graph of rooms, areas, objects, and spatial relationships.The 3D rendering is used only for visualization.
- Agent Architecture: All variants share a perceive–plan–act architecture integrating perception, memory, planning, and action.
- Conditioning Approaches: The six approaches are no-persona control, authored persona, inferred persona, persona plus few-shot exemplars, statistical prior, and hybrid prior.They differ in how real behavioral traces contribute to persona descriptions, exemplars, or activity-selection priors.
- Conditioning Approaches: Statistical-prior activity selection samples and scores candidates using empirical first-order transitions and time-of-day activity distributions.Episode durations are also sampled from empirical per-activity duration distributions, while day-plan initialization and grounded action execution remain LLM-based.
7 Results and Analysis
Statistical priors most closely matched measured activity and sequence distributions, but produced more fragmented, less variable days; persona differences were smaller and metric-dependent, while population agreement could mask individual errors.
- Overall Interpretation: No single metric sufficiently characterizes behavioral fidelity across the evaluated temporal granularities and levels of analysis.
- Distributional Fidelity: Statistical priors reduced day-level time-allocation divergence from JS = .19–.24 for B0–C3 to .010 for C4–C5.The smallest contrast between a persona approach and C5 was ∆JS = .18 [.09, .27].
- Distributional Fidelity: Statistical priors reduced transition divergence from JS = .31–.36 for B0–C3 to .107 for C4 and .091 for C5.The 3-gram and 4-gram distributions showed the same pattern, with all 16 contrasts excluding zero.
- Distributional Fidelity: Across all four time-of-day windows, statistical priors produced lower time-allocation and activity-frequency divergence than B0–C3.Every contrast of C4 and C5 against B0–C3 excluded zero in every window.
- Day-Level Variability: Statistical-prior and hybrid methods generated ∆sw = +38.0 and +32.3 more activity switches than real days, respectively.Persona-based and control conditions instead produced 10.8 to 15.0 fewer switches.
- Day-Level Variability: Real within-person variability was V real = .097, compared with V sim = .018 for C4 and .026 for C5.Every approach was less variable than real data, with statistical priors producing the most uniform sessions.
- Persona-Based Conditioning: Persona-based approaches differed less from one another than statistical-prior approaches differed from them, with persona contrasts spanning |∆JS| = .008–.056 versus .179–.235.Within persona methods, inferred descriptors had lower divergence in 38 of 39 contrasts, but only 13 excluded zero.
- Level of Analysis: All 13 individual-level versus population-level JS contrasts excluded zero, including ∆ind−pop = +.020 for time allocation and +.069 for transitions.Population-level agreement can therefore mask errors in individual routines.
8 Conclusion
The paper introduces an evaluation framework for long-horizon human simulation and applies it to six approaches in an office-activity case study. Statistical priors reduce divergence in several behavioral distributions but over-segment days, while population agreement can conceal individual-level errors.
- The framework evaluates behavioral fidelity in long-horizon human simulation.
- Statistical priors reduce divergence in activity distributions and local transition structure relative to other conditioning approaches.
- Statistical priors over-segment days and produce less day-to-day routine variation than real individuals.
- Persona-based conditioning approaches differ little by comparison, with effects depending on the evaluation metric.
- Population-level agreement can mask individual-level error, motivating holistic evaluation of LLM-based human proxies.
9 Limitations
The framework is designed to generalize across activity-sequence settings, but the experiments are confined to one office case study and require broader replication. The evaluation measures outward activity rather than intent, while several simulator-design choices remain unvalidated.
- The metrics apply to arbitrary categorical activity sequences without assuming a particular label vocabulary, physical layout, or social configuration.
- The experiments cover one multi-room office, one work week, five individuals, six activity labels, and a fixed social configuration.
- Replication is needed across environments, longer observation periods, richer vocabularies, and varied social configurations.
- The annotations capture outward activity sequences but not mental state or intent, limiting downstream uses that depend on intent and counterfactual inference.
- The effects of day-plan quality, higher-order priors, persona prompting, and backbone choice remain insufficiently investigated.
10 Ethics Statement
The paper addresses ethics through privacy-preserving data release and explicit warnings about workplace surveillance and misuse. It releases derived activity information rather than raw imagery, while documenting annotation and metric procedures used in the study.
- Only derived activity sequences, room labels, and estimated 2D positions linked to anonymized identifiers are released; raw video and identity-linkable metadata are withheld.
- Trajectory data may still pose re-identification risks for people familiar with the environment.
- Workplace camera instrumentation creates power asymmetries because visibility and opt-out capacity vary across occupants and visitors.
- Researchers are asked not to treat the dataset as precedent for monitoring uninformed occupants without an opportunity to opt out.
- The paper warns that improved simulation fidelity should not license replacing people in high-stakes decisions and recommends independent ethical review and empirical grounding.
- Activity annotations showed high inter-rater reliability, with Fleiss’ κ = 0.86 and unanimous agreement on 87% of timestamps.
D Backbone Analysis
Backbone analysis suggests that the qualitative advantage and trade-offs of statistical priors persist across the three tested backbones, although effect magnitudes vary. Backbone choice has a larger effect on persona-based approaches for several distributional metrics, but the causes remain unresolved.
- Statistical priors outperform control and persona-based approaches qualitatively across backbones on several activity, transition, motif, and time-window metrics.
- Statistical priors also produce more activity switches than real days and less within-person day-to-day variation across backbones.
- The cross-backbone JS spread for day-level activity time allocation is .044 for persona-based approaches versus .006 for statistical priors.
- For transitions, the corresponding cross-backbone ranges are .063 for persona-based approaches and .026 for statistical priors.
- The analysis does not determine why cross-backbone differences occur or support conclusions about particular models.
E Leave-One-Session-Out Evaluation
Leave-one-session-out evaluation reproduced the main pattern: statistical priors reduced divergence from held-out real activity and sequence distributions, while producing more fragmented, less variable days. The results also show that persona-based differences were small and that population-level agreement can conceal individual-level errors.
- Held-out results: Statistical priors reduced held-out activity time-allocation divergence to .071–.075, versus .28–.30 for persona-based approaches and the no-persona control.All eight statistical-prior contrasts against B0–C3 excluded zero; the weakest was C1 vs. C5 at ∆JS = .209 [.106, .370].
- Held-out results: Statistical priors generated more activity switches than held-out real days, whereas B0–C3 generated fewer.C4 and C5 produced 32.5 and 27.2 more switches, while B0–C3 produced 16.6–19.6 fewer.
- Held-out results: Statistical-prior sessions had lower within-person variability than real sessions and were farther from the real variability value.V sim was .022–.023 for C4–C5, compared with .038–.059 for B0–C3 and V real = .097; their gaps to real variability were L1(JS) = .074 and .073.
- Conditioning comparisons: Persona-based conditioning approaches differed little by comparison and their advantages depended on the metric.Across 78 contrasts among B0–C3, 12 excluded zero; B0 was lower than C1 on six metrics and higher on seven.
- Analysis levels: Population-level divergence was lower than individual-level divergence in 76 of 78 method–metric combinations, including C4–C5 transition divergence falling from .331 to .198.This supports examining both population and individual levels when evaluating behavioral fidelity.
- Scope and limitations: The evaluation remains bounded by one backbone, one generation per fold, five routine sessions, and new sessions from known rather than unseen individuals.Cross-individual and cross-setting generalization remain future directions, and the small cohort limits broader generalization.