PaperlayerView exact revision on arXiv

Source-linked AI summary

FileGram: Grounding Agent Personalization in File-System Behavioral Traces

Shuai Liu, Shulin Tian, Kairui Hu, Yuhao Dong, Zhe Yang, Bo Li, Jingkang Yang, Chen Change Loy, Ziwei Liu

arXiv:2604.04901v1cs.CVcs.AI

TL;DR

File-system personalization lacks scalable multimodal behavioral data, memory-centric evaluation, and architectures grounded in operational traces. FileGram addresses these gaps with a trace-generation engine, a diagnostic benchmark, and bottom-up memory from atomic actions and content deltas. FileGramOS reaches 59.6% accuracy, while existing baselines top out at 48–50% and multimodal methods reach 44.7%.

  • Problem

    Personalized file-system agents lack scalable multimodal behavioral data, dedicated memory-centric benchmarks, and methods that capture operational traces beyond dialogue summaries.

  • Method

    FileGram combines profile-conditioned trajectory generation, four-track benchmark evaluation, and FileGramOS, which organizes atomic actions and content deltas into procedural, semantic, and episodic memory channels.

  • Results

    59.6% accuracy is achieved by FileGramOS, compared with 48–50% for context-based and narrative-first baselines and 44.7% for multimodal methods.

  • Takeaways & Limitations

    FileGram provides a data, evaluation, and structural foundation for developing memory-centric personalized AI coworkers.

  • Takeaways & Limitations

    Accuracy on human screen recordings drops to single digits across all evaluated methods, exposing a gap between clean simulated traces and noisy real-world video behavior.

Abstract

from arXiv · show

Coworking AI agents operating within local file systems are rapidly emerging as a paradigm in human-AI interaction; however, effective personalization remains limited by severe data constraints, as strict privacy barriers and the difficulty of jointly collecting multimodal real-world traces prevent scalable training and evaluation, and existing methods remain interaction-centric while overlooking dense behavioral traces in file-system operations; to address this gap, we propose FileGram, a comprehensive framework that grounds agent memory and personalization in file-system behavioral traces, comprising three core components: (1) FileGramEngine, a scalable persona-driven data engine that simulates realistic workflows and generates fine-grained multimodal action sequences at scale; (2) FileGramBench, a diagnostic benchmark grounded in file-system behavioral traces for evaluating memory systems on profile reconstruction, trace disentanglement, persona drift detection, and multimodal grounding; and (3) FileGramOS, a bottom-up memory architecture that builds user profiles directly from atomic actions and content deltas rather than dialogue summaries, encoding these traces into procedural, semantic, and episodic channels with query-time abstraction; extensive experiments show that FileGramBench remains challenging for state-of-the-art memory systems and that FileGramEngine and FileGramOS are effective, and by open-sourcing the framework, we hope to support future research on personalized memory-centric file-system agents.

1 Introduction

FileGram addresses data, evaluation, and methodology bottlenecks in personalizing file-system coworkers by grounding memory in behavioral traces and content deltas. Its framework combines scalable trace generation, a dedicated benchmark, and bottom-up memory, while experiments show substantial difficulty for existing systems and higher performance from FileGramOS.

  • File-system personalization requires behavioral traces and content deltas to capture users’ operational preferences and produced outputs.
  • Real-world multimodal, long-trajectory data is difficult to collect because of privacy constraints and limited scalable collection strategies.
  • Existing benchmarks emphasize conversational recall or isolated GUI success, while memory architectures rely mainly on top-down dialogue summaries.
  • FileGram combines FileGramEngine, FileGramBench, and FileGramOS for scalable trace generation, memory-centric evaluation, and bottom-up profile construction.FileGramBench covers four tracks and 16 attributes spanning procedural, semantic, and episodic memory.
  • 59.6% accuracy is achieved by FileGramOS, compared with 48–50% for context-based and narrative-first baselines and 44.7% for multimodal methods.

2 Related Work

Prior agent benchmarks largely measure conversational recall or task execution, and existing memory systems remain disconnected from users’ operational file-system behavior. Recent multimodal and trajectory methods capture selected dimensions but often model them in isolation or constrained environments.

  • Conversational benchmarks emphasize static semantic retrieval from long dialogues, stripping away procedural context from real workflows.
  • Execution-driven benchmarks place agents in operating systems or web interfaces but measure memory indirectly through objective task success.
  • Existing memory architectures primarily extract facts and relations from conversational histories rather than operational environments.
  • Multimodal perception and trajectory tracking methods typically model temporal or visual dimensions in isolation or within constrained simulation settings.

3 FileGramEngine: Behavioral Data Generation

FileGramEngine synthesizes profile- and task-conditioned file-system trajectories in isolated workspaces, converting tool use into canonical atomic actions and rich content artifacts. Its controlled profile, task, and perturbation design supports realistic behavioral variation and benchmark-scale trace generation.

  • FileGramEngine pairs user profiles with tasks to generate controlled, long-term file-system behavioral trajectories.
  • The pipeline filters and canonicalizes raw tool traces, producing standardized behavioral traces with aligned text, document, and visual views.
  • Profile Design: Profiles encode six behavioral dimensions with three L/M/R tiers, covering consumption, production, organization, iteration, curation, and cross-modal behavior.
  • Task Design: Tasks span six types and 32 text-centric or multimodal workflows designed to elicit trace-observable behavioral signals.
  • Behavioral Perturbation: Behavioral perturbation shifts one task-relevant dimension by one tier in five trajectories per profile, supporting persona-drift evaluation.
  • FileGramEngine: 640 profile–task combinations run in isolated sandboxes, where tool-using agents execute continuous think–act–observe loops.
  • Dataset Composition: The dataset contains 640 trajectories, 20,028 atomic actions, and approximately 2.5K generated files, with procedural logs interleaved with content deltas.

4 FileGramBench: Evaluation Framework

FileGramBench evaluates memory-centric personalization across profile understanding, reasoning, behavioral drift detection, and multimodal grounding. It converts controlled trajectories and screen recordings into structured QA items while restricting answer generation to retrieved memory.

  • FileGramBench contains 4.6K memory-targeted QA pairs across nine sub-tasks and four tracks covering procedural, semantic, and episodic channels.
  • The benchmark converts FileGramEngine trajectories into structured evaluation items through a template-based pipeline.
  • MCQ construction uses profile-based distractors differing in only 1–2 dimensions to test behavioral discrimination.
  • The dataset includes 640 trajectories, approximately 10K output files, and 20,028 atomic actions for evaluation-scale behavioral traces.
  • Open-ended answers use profile-derived ground truth and per-attribute Likert 1–5 scoring by an LLM judge.
  • Real-world evaluation uses human screen recordings produced from task descriptions, behavioral profiles, and simulated guidance videos.
  • QA Taxonomy: The four tracks cover profile understanding, pattern reasoning, behavioral-drift detection, and multimodal grounding across trajectories and rendered outputs.
  • Evaluation Protocol: Methods ingest raw trajectories and content deltas, while answer models access only retrieved memory and not profiles, definitions, or perturbation tags.

5 FileGramOS: Bottom-Up Memory Framework

FileGramOS builds memory bottom-up from raw action traces and file-content changes, preserving procedural, semantic, and episodic evidence until query-time retrieval.

  • Stage 1: Per-Trajectory Encoding: Each trajectory is encoded into an Engram containing procedural statistics, semantic content, and segmented episodic structure.Parallel extraction streams process actions, file snapshots, edit diffs, and event boundaries before convergence.
  • Stage 1: Per-Trajectory Encoding: Procedural extraction counts actions, computes behavioral metrics, and compresses over 50 features into a dense 17-dimensional fingerprint.The fingerprint captures mechanics such as browse ratios and average output lengths.
  • Architecture Overview: Figure 6 presents the complete three-stage pipeline from trajectory encoding through cross-engram channel consolidation to query-adaptive retrieval.The figure also includes an LLM verifier for variation-versus-outlier decisions.
  • Stage 2: Cross-Engram Consolidation: Cross-engram consolidation routes procedural, semantic, and episodic units into specialized channels within a unified MemoryStore.The architecture aggregates behavioral fingerprints, ingests semantic content, and preserves temporal episodes across sessions.
  • Stage 2: Cross-Engram Consolidation: The episodic channel clusters trajectories into behavioral modes and uses centroid distance to identify anomalous sessions for further judgment.An LLM-based Anomaly Judge distinguishes task-dependent variation from genuine behavioral shifts, with τ=1.5 used for flagging.
  • Stage 3: Query-Adaptive Retrieval: At query time, keyword extraction selects relevant procedural, semantic, and episodic clues before a final language-model generation step composes the answer.This defers final interpretation rather than collapsing all evidence into an ingestion-time narrative.

6 Experiments

Experiments evaluate FileGramOS and twelve comparison methods across text, multimodal, and real-world settings on FileGramBench. Results favor bottom-up behavioral structure, while exposing persistent difficulties in shift attribution, multimodal grounding, and sim-to-real transfer.

  • Experimental Setup: 640 trajectories are evaluated under Text, Multimodal, and Real-World settings, with PDFs or images and human screen recordings introducing progressively broader inputs.Behavioral event logs remain identical across the first two settings, and Gemini 2.5-Flash is shared across settings.
  • Results Analysis: FileGramOS achieves 59.6% accuracy, outperforming the strongest narrative baseline EverMemOS at 49.9%.Narrative-first ingestion can erase action counts, directory depth, and edit granularity before querying.
  • Track and Channel Overview: Track 1 and Track 2 are partially solvable, whereas Track 3 reveals a detection-versus-explanation gap in anomaly and shift analysis.AnomDet methods that aggregate behavioral norms exceed 70% accuracy, while flat memory systems remain near random at 21–26%.
  • Results Analysis: Existing models detect overall deviations but fail to attribute changes to specific behavioral dimensions or directions.FileGramOS improves this analysis through channel-wise procedural cues, while semantic methods remain more competitive.
  • Results Analysis: Full Context matches FileGramOS on TraceDis by preserving complete action chains but underperforms on cross-session outlier comparison requiring structured aggregation.VisRAG captures layout cues in semantic tasks but lacks the behavioral abstraction needed for procedural tasks.
  • Results Analysis: Multimodal memory systems do not outperform the strongest text-only baselines because rendered images omit operation-level statistics and file-system structures.Vision-based methods can succeed on formatting cues while failing on file counts, output lengths, edit frequencies, directory depth, and naming conventions.
  • Limitations and Research Frontiers: Accuracy on human screen recordings drops to single digits across all evaluated methods, exposing a substantial sim-to-real gap.The paper attributes this difficulty to noise, variable pacing, and unstructured visual input in real-world recordings.

7 Conclusion

FileGram unifies trajectory generation, diagnostic evaluation, and bottom-up memory to make file-system behavioral personalization measurable and reproducible. Its comparisons emphasize that prior systems lack joint coverage of procedural, semantic, and episodic file-system evidence.

  • FileGram combines FileGramEngine, FileGramBench, and FileGramOS to make file-system behavioral personalization measurable and reproducible.
  • Prior dialogue-based systems primarily extract semantic facts, while multimodal systems process video or audio without ingesting file-system traces.
  • No existing system jointly models procedural, semantic, and episodic channels from file-system evidence.
  • FileGramOS uses aggregate statistics from 17-D fingerprints for procedural retrieval, trading graph flexibility for deterministic reproducibility and higher retrieval efficiency.
  • The framework retains 12 cleaned atomic action types and removes 10 simulation metadata types accounting for 74.3% of raw events.

B.2 Procedural Fingerprint Specification

FileGramOS constructs complementary procedural, semantic, and episodic representations from behavioral traces, then combines them through query-time retrieval. Its procedural channel uses deterministic fingerprints and cross-trajectory statistics, while the other channels preserve content and recurring episode structure.

  • B.2 Procedural Fingerprint Specification: The procedural fingerprint compresses each trajectory into a fixed-length 17-feature vector spanning six behavioral dimensions.
  • B.2 Procedural Fingerprint Specification: Cross-session consolidation z-score normalizes fingerprint dimensions and stores mean, median, standard deviation, minimum, and maximum statistics.
  • B.2 Procedural Fingerprint Specification: The 17 procedural features use deterministic counting-based computations that require no LLM calls and produce reproducible, interpretable outputs.
  • Semantic Channel: The semantic channel combines file metadata, representative filenames, content embeddings, and VLM-generated descriptions of formatting and detail level.
  • Episodic Channel: The episodic channel segments trajectories into 2–5 coherent episodes, clusters recurrent themes, and separately groups behavioral modes using 17-D fingerprints.
  • Query-Time Abstraction: At query time, retrieval concatenates procedural patterns, top semantic content chunks, and episodic consistency information in a fixed order.

C.2 Task Pool and File Type Statistics

The benchmark includes 32 tasks across six desktop activity types, with both text-centric and multimodal inputs. Its scope covers common OS-agent activities but excludes several operational domains.

  • Task Pool: The task pool contains 32 tasks across six types: Understand, Organize, Create, Synthesize, Iterate, and Maintain.
  • Task Pool: The benchmark divides tasks evenly between 16 text-centric and 16 multimodal tasks using audio, image, or video inputs.
  • Task Representativeness: The six task types cover core desktop activities from OSWorld, OfficeBench, and OS-Copilot while adding curation and cross-modal dimensions.
  • Task Representativeness: Code development, real-time collaboration, and system administration are outside the benchmark scope.

C.3 Evaluation Pipeline

The evaluation pipeline generates difficult reconstruction questions and controls comparisons across memory systems. It also tests whether FileGramOS’s behavioral signal depends on the QA backbone.

  • Question Generation: MCQ distractors share at least three dimensions with the target, while Profile Reconstruction uses 1–5 Likert scoring with randomized attributes and calibration examples.
  • Cross-Backbone Validation: Inter-backbone variance remains below 2.0 percentage points across three QA backbones, supporting model-agnostic behavioral signal.
  • Baseline Setup: All 12 baselines and FileGramOS use Gemini 2.5-Flash, identical cleaned logs and output files, published defaults, and no per-baseline hyperparameter sweeps.

D.2 Ablation Studies

The ablations show that FileGramOS depends most on its procedural channel, while compact semantic context supports content-sensitive tracks. The architecture is intended for real OS-level deployment, but current integration still faces operational challenges.

  • Channel ablation: Removing the procedural channel causes the largest overall accuracy drop, confirming that it carries the dominant behavioral signal.The drop is −11.1 pp, compared with −5.5 pp for the semantic channel and −4.2 pp for the episodic channel.
  • Channel ablation: Each memory channel most strongly affects its corresponding question type, indicating that procedural, semantic, and episodic signals are complementary.
  • Parameter sensitivity: Track 2 remains near-invariant across configurations because Trace Disentanglement relies on procedural statistics alone.
  • Parameter sensitivity: Track 1 is content-sensitive: displaying 800 characters offers the best trade-off, while removing content previews reduces accuracy by −4.7 pp.
  • Deployment: FileGramOS is designed to run atop real OS-level file-system monitors, using native APIs for common event types and application hooks for read events.A production deployment chains local event collection, periodic encoding, and on-demand consolidation and retrieval.
  • Deployment: Deployment remains bounded by interleaved multi-application streams, duplicate or out-of-order cloud-synchronized events, and per-directory privacy controls.

E.2 Ethical Considerations

The ethical discussion identifies limitations from synthetic data, benchmark scope, and sim-to-real transfer, while emphasizing privacy risks and consent requirements for deployment. The benchmark is intended for research use rather than surveillance or unauthorized profiling.

  • Synthetic data and bias: Synthetic traces avoid direct privacy concerns but may inherit model biases and cannot capture the full complexity of real-world file-system interaction.The traces are generated by Claude Haiku 4.5 and span 20 profiles validated by human verifiers.
  • Privacy: File-system traces can expose working hours, priorities, and organizational habits, so deployment requires consent, minimization, deletion rights, and access control.The procedural channel stores 17-D aggregate fingerprints and the semantic channel stores descriptors, but episodic temporal patterns may remain re-identifiable.
  • Scope limitations: The benchmark uses one LLM, single-tier behavioral shifts, 32 tasks, 20 profiles, and 640 trajectories, limiting diversity, drift realism, task coverage, and scale.
  • Scope limitations: Real-world transfer remains an open challenge because accuracy on human screen recordings drops sharply from structured trace settings.
  • Intended use: FileGramBench is released for research use and is not intended for surveillance, employee monitoring, or profiling without explicit consent.
Loading 2604.04901v1…