HippoCamp: Benchmarking Contextual Agents on Personal Computers

Zhe Yang; Shulin Tian; Kairui Hu; Shuai Liu; Hoang-Nhat Nguyen; Yichi Zhang; Zujin Guo; Mengying Yu; Zinan Zhang; Jingkang Yang; Chen Change Loy; Ziwei Liu

Source-linked AI summary

HippoCamp: Benchmarking Contextual Agents on Personal Computers

Zhe Yang, Shulin Tian, Kairui Hu, Shuai Liu, Hoang-Nhat Nguyen, Yichi Zhang, Zujin Guo, Mengying Yu, Zinan Zhang, Jingkang Yang, Chen Change Loy, Ziwei Liu

arXiv:2604.01221v1Apr 1, 2026cs.AIcs.CV

TL;DR

Agents still lack a standardized test for reasoning over massive, personalized multimodal file systems. HippoCamp benchmarks this capability at realistic device scale and finds substantial gaps from human performance, with profiling reaching only 48.3 Acc overall.

Problem
No standardized benchmark evaluates agents’ understanding, recall, and reasoning over massive, personalized, heterogeneous multimodal file systems at personal-computing scale.
Method
HippoCamp constructs realistic device-resident personal file systems and evaluates evidence-grounded factual retention and profiling through search, perception, and reasoning.
Results
Current methods remain far below human performance across tasks; ChatGPT Agent Mode achieves 21.0 F1 / 48.3 Acc overall profiling.
Takeaways & Limitations
Personalized multimodal memory remains unsolved, with post-retrieval evidence discrimination, grounding, entity binding, and verification as dominant bottlenecks.
Takeaways & Limitations
None of the evaluated methods performs an explicit final-stage check that answers remain traceable to a minimal, coherent evidence set.

Abstract

from arXiv · show

We present HippoCamp, a new benchmark designed to evaluate agents' capabilities on multimodal file management. Unlike existing agent benchmarks that focus on tasks like web interaction, tool use, or software automation in generic settings, HippoCamp evaluates agents in user-centric environments to model individual user profiles and search massive personal files for context-aware reasoning. Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files. Building upon the raw files, we construct 581 QA pairs to assess agents' capabilities in search, evidence perception, and multi-step reasoning. To facilitate fine-grained analysis, we provide 46.1K densely annotated structured trajectories for step-wise failure diagnosis. We evaluate a wide range of state-of-the-art multimodal large language models (MLLMs) and agentic methods on HippoCamp. Our comprehensive experiments reveal a significant performance gap: even the most advanced commercial models achieve only 48.3% accuracy in user profiling, struggling particularly with long-horizon retrieval and cross-modal reasoning within dense personal file systems. Furthermore, our step-wise failure diagnosis identifies multimodal perception and evidence grounding as the primary bottlenecks. Ultimately, HippoCamp exposes the critical limitations of current agents in realistic, user-centric environments and provides a robust foundation for developing next-generation personal AI assistants.

1 Introduction

HippoCamp addresses the absence of a standardized benchmark for agents reasoning over massive, personalized, multimodal file systems by evaluating search, perception, and long-horizon reasoning in realistic personal computing environments. It contributes device-scale heterogeneous data, dense evidence-grounded supervision, and tasks spanning factual retention and user profiling.

Task overview: Tasks require grounded retrieval, cross-file pattern inference, and long-horizon reasoning over tens of gigabytes of user-specific assets across diverse modalities.These capabilities are designed to evaluate personalized multimodal understanding in large-scale personal file systems.
Motivation: HippoCamp fills the gap in evaluating agents’ ability to understand, recall, and reason over massive, personalized, multimodal file systems, beyond document-level retrieval or personalized tool-use planning.The benchmark targets the scale and heterogeneity of personal computing environments that existing efforts do not cover.
Realistic personal computing environments: The benchmark constructs three realistic, file-intensive personal computing environments that capture long-term continuity, idiosyncratic folder structures, and interconnected digital assets.These environments are archetypal instantiations of a high-dimensional profile space rather than a literal three-way categorization.
Device-scale corpus with dense supervision: The corpus contains over 2000 heterogeneous files totaling 42.4 GB, 581 evidence-grounded user-need queries, and 46.1K fine-grained annotations.The supervision supports evaluation across varying depths and perspectives with multiple levels of granularity.
Comprehensive agent capability evaluation: HippoCamp evaluates factual retention and user profiling through multi-step integration of file-system search, multimodal evidence perception, and personalized reasoning across interrelated files over time.The benchmark therefore assesses more than simple retrieval, requiring agents to retrieve specific information and infer user preferences.

2 Related Work

Prior work advances multimodal retrieval, agentic memory, and personalization, but evaluations remain limited in scale, modality, or realism. HippoCamp addresses this gap with a personalized multimodal file-system environment supporting cross-file and cross-modal reasoning at real-world scale.

HippoCamp: HippoCamp introduces a personalized multimodal file-system environment spanning all five modalities, enabling cross-file and cross-modal reasoning at real-world scale.This setting positions personal file systems as the benchmark environment for multimodal contextual agents.
Benchmarks for Multimodal Contextual Agents: Existing contextual retrieval benchmarks range from text-centric datasets to multimodal and agentic settings requiring richer evidence and cross-modal grounding.MultimodalQA supports grounding over text, images, and tables, while WebQA studies retrieval from distractor-containing candidate pools; document benchmarks address layout and mixed text–image evidence.
Agentic Systems with Memory and Personalization: Prior agentic systems use trajectory memories, user-specific records, knowledge graphs, preference embeddings, and long-term histories to support context integration and personalization.Examples include PersonaAgent’s episodic and semantic records and Telemem’s narrative and multimodal episodic memories for personalized retrieval.
Agentic Systems with Memory and Personalization: Existing evaluations are often small-scale or synthetic and restricted to narrow modalities, failing to represent heterogeneous, evolving personal-computing context spanning five modalities.LoCoMo contains 300 text-only personal questions, while EgoLifeQA/Ego-R1-Bench provides approximately 20% personalized samples and is limited to video/audio single-modal context.

3 The HippoCamp Benchmark

HippoCamp benchmarks evidence-grounded memory-augmented agents in realistic, multimodal personal file systems. Its hierarchical supervision evaluates factual retention and user profiling through search, multimodal perception, and long-horizon reasoning.

3 The HippoCamp Benchmark: The benchmark’s hierarchy progresses from localized atomic grounding and action traces to structured trajectories and task-level questions, culminating in user-level memory and profiling.Factual retention relies on localized evidence and intermediate reasoning, while profiling requires higher-level synthesis.
3 The HippoCamp Benchmark: HippoCamp evaluates personalized file understanding as open-ended, evidence-grounded question answering over realistic, device-resident multimodal personal file systems.The setting treats personal computing as a long-tail information space organized by file-system structure and temporal metadata.
3 The HippoCamp Benchmark: Factual retention requires retrieving and reasoning over verifiable file-grounded facts, whereas profiling synthesizes grounded facts across time into coherent user-level inferences.Profiling covers preferences, behavioral patterns, scheduling, retrospective reflections, and workflows; both tasks require search, perception, and reasoning.
3 The HippoCamp Benchmark: HippoCamp derives from interviews with 100+ participants and produces anonymized, minimally edited, unindexed personal file systems after multi-stage source selection.The resulting “haystack” systems remove system-generated non-user artifacts and anonymize sensitive identifiers for stress-testing personalized multimodal agents.
3 The HippoCamp Benchmark: A hybrid expert–LLM pipeline creates and curates diverse questions, then records each QA pair with answers, rationales, localized atomic evidence, file metadata, and capability labels.Human review enforces factual correctness, contextual grounding, intent and pattern diversity, modality balance, and evidence-set balance; labels decompose search, perception, and reasoning.

4 Experiment

Across profile-local HippoCamp evaluations, current methods remain substantially weaker than human experts, with multimodal retrieval, long-horizon reasoning, and evidence grounding as persistent bottlenecks. ChatGPT Agent Mode achieves the strongest reported overall scores, but remains computationally expensive and operationally unstable.

Autonomous Agent Systems: ChatGPT Agent Mode achieves the strongest reported overall scores, with 21.0 profiling F1 / 48.3 Acc and 35.3 factual-retention F1, yet remains far below human performance.It reaches 55.0% profiling accuracy on Profiles (b) Adam and (c) Victoria.
RAG methods: RAG performs poorly on profiling, with Standard RAG at 18.4 F1 / 26.7 Acc and Self-RAG at 15.2 F1 / 10.0 Acc overall.Self-RAG achieves 0.0 Acc on Profile (c) Victoria, reflecting brittle retrieval and weak cross-file aggregation.
Evaluation and results: Factual retention improves only modestly to 30.0–31.9 overall F1, mainly on direct-lookup cases, while models often rely on filenames or directory strings instead of file content.The evaluation covers question-answering quality and evidence retrieval accuracy using automatic metrics and LLM-based judgment.
Search agent methods: Search agents improve factual retention but transfer little to profiling: ReAct reaches 55.3 Acc on Profile (b) Adam, while Search-R1 reaches only 10.8 profiling F1 / 5.0 Acc overall.Search-R1 reaches 58.0 F1 on factual retention for Profile (b) Adam but 0.0 Acc on another profiling profile.
Autonomous Agent Systems: Autonomous agents provide only moderate gains over search-based methods but remain computationally expensive and operationally unstable, often requiring 10–15 minutes per query and re-execution.Incomplete outputs and missing file references increase run-to-run variance and make performance sensitive to execution instability.

5 Analysis

HippoCamp’s dominant failure source is post-retrieval: agents often find partially relevant files but fail to discriminate, ground, integrate, and verify evidence under profile-local, cross-modal, and temporally extended conditions. Analysis localizes these failures across capability gaps, a five-stage error pipeline, and design-relevant success patterns.

5 Analysis: Post-retrieval processing, rather than evidence retrieval itself, is HippoCamp’s dominant failure source across profile-local, cross-modal, and temporally extended conditions.Methods frequently locate partially relevant files but fail to discriminate, ground, integrate, and verify them.
5.1 Metric decoupling and bottleneck localization: 56.5% profiling accuracy with 28.9% search F1 shows that locating candidate files is necessary but not sufficient for correct answers.Search-centric agents achieve higher profiling retrieval F1—36.3% for ReAct (Qwen3) and 34.8% for Search-R1—but lower profiling accuracy of 22.2% and 22.1%.
5.1 Metric decoupling and bottleneck localization: Perception is the most universal bottleneck, with profiling accuracy ranging from 13.2% for Self-RAG to 28.5% for ChatGPT Agent Mode.The strongest system’s perception accuracy is roughly half its search accuracy, 28.5% versus 56.5%, while reasoning inherits errors from prior evidence discrimination.
5.2 Canonical failure pipeline: The recurring failure pipeline progresses through retrieval mismatch, grounding avoidance, hard evidence hallucination, entity misattribution, and verification deficit.These failures include semantically related but contextually irrelevant retrieval, generic answers despite relevant files, fabricated paths or metadata, wrong referents, and no explicit final traceability check.
5.2 Canonical failure pipeline: Iterative file-system exploration enables ChatGPT Agent Mode to recover from early errors, reaching 55.0% profiling accuracy and 90.3% factual accuracy on Adam.Its advantage comes from revisiting directories and candidate files, refining evidence hypotheses, and narrowing the support set rather than stronger one-shot retrieval.
5.3 Design principles: True personalization requires multimodal, cross-file reasoning over implicit behavioral signals scattered across heterogeneous file types, not merely retrieval of explicit persona statements.HippoCamp frames local digital-life modeling as synthesis across diverse personal records.

6 Conclusion · Appendix

HippoCamp evaluates search, perception, and reasoning over realistic multimodal personal file systems, finding that post-retrieval processing is the dominant bottleneck. The appendix documents benchmark construction, annotation, task design, and evaluation procedures.

6 Conclusion: HippoCamp benchmarks agents’ ability to search, perceive, and reason over realistic multimodal personal file systems.
6 Conclusion: The dominant bottleneck lies after evidence retrieval, involving evidence discrimination, multimodal grounding, entity binding, and final verification.
6 Conclusion: Profiling requires a qualitatively different capability composition from factual retention, highlighting the distinct demands of personalized multimodal memory.
Appendix: The appendix details participant selection, interviews, archetype aggregation, privacy filtering, external augmentation, and file-system statistics.
Appendix: It specifies the trajectory schema, evidence-unit design, human-in-the-loop question-answering pipeline, and agreement procedures.
Appendix: It expands the task taxonomy, difficulty definitions, and representative user-profile examples.
Appendix: It describes evaluation settings, budgets, metrics, robustness checks, and extended result summaries.

A Dataset Construction and Profile Aggregation … A.6.1 Modality Composition by Profile

HippoCamp constructs privacy-preserving, multimodal personal-device profiles through screened participant data, protocol-guided interviews, coherence-checked aggregation, and controlled augmentation. The resulting archetypes preserve heterogeneous evidence distributions while supporting grounded, auditable user-profile reasoning.

A.1 Participant Pool and Source Selection: Profiles are built from devices containing at least 500 files, at least four of five modalities, at least 10 extensions, and at least three months of activity.Sources also require sustained full-time study or professional use and auditable cross-file, temporal, and organizational evidence.
A.2 In-Depth Interview Protocol: Protocol-guided 60–90-minute interviews scope devices, reconstruct workflows, identify ambiguity, elicit task needs, and validate extracted structures before aggregation.Interview evidence informs source filtering, aggregation checks, trajectory construction, QA design, and final consistency review.
A.3 Archetype Aggregation: Three archetypal profiles are formed by distribution-preserving partitioning and coherence checks over modality, file-type, organizational, temporal, entity, and workflow patterns.Conflicts are repaired through source reassignment or minimal edits, followed by human validation of consistency, privacy, and native organizational signals.
A.4 Privacy, Filtering, and Anonymization: Privacy processing combines opt-in consent, withdrawal rights, controlled handling, artifact filtering, identifier redaction or pseudonymization, and participant approval.The pipeline removes clearly system-generated artifacts while preserving evidence-bearing cues and admits only participant-approved processed data.
A.5 External Benchmark used for Data Augmentation: External documents supplement underrepresented professional forms in the Law and Finance profiles, but adapted questions and sanitized content remain governed by the HippoCamp pipeline.FinanceBench (Islam et al., 2023) and LegalBench-RAG (Pipitone and Alami, 2024) materials are screened, rewritten, grounded, and capped so augmented sources remain a minority.
A.6 File-System Statistics: The benchmark reports file-system statistics through profile-level modality and file-type distributions, with augmented documents constrained to preserve those distributions.These statistics describe evidence composition rather than storage burden.
A.6.1 Modality Composition by Profile: Figure 7 shows intentionally heterogeneous modality compositions across Bei Weiwei, Adam Turner, and Victoria Anne Clarke rather than uniformly balanced profiles.The figure reports percentage breakdowns across five modalities by file count; Bei Weiwei has the broadest modality spread.

A.6.2 Storage Footprint and File-Type Burden … B.9 AU–Evidence Mapping

HippoCamp combines storage- and time-diverse personal files with grounded trajectory annotations that explicitly represent support files, localized multimodal evidence, rationale steps, and atomic evidence units. Its schema and validation rules enable reproducible capability, difficulty, and evidence-level evaluation across heterogeneous modalities.

A.6.2 Storage Footprint and File-Type Burden: Storage burden is asymmetric: a few large video, audio, or long-form document files can dominate footprint, increasing indexing, conversion, retrieval, and evidence-localization costs beyond modality proportions.Bei is especially costly because of high-volume .mp4 and .mkv files, Adam combines compactness with extension diversity, and Victoria is dominated by large PDFs, structured artifacts, and audio.
A.6.3 Temporal Coverage of Files: File activity concentrates in 2024–2025 while retaining legacy artifacts back to 2012, supporting both recent-activity modeling and temporal cross-checks for factual retention.Creation and modification timestamps jointly capture long-term accumulation and recent interaction across the three profiles.
B Annotation Schema and Quality Control; B.1 Trajectory JSON Schema; B.2 JSON Record Overview: Each trajectory couples a task with its normalized answer, minimal supporting files, capability labels, localized evidence, and stepwise rationale traces for downstream evaluation and diagnosis.The released JSON is designed to support parser reimplementation, capability-wise breakdowns, difficulty statistics, and evidence-level evaluation.
B.3 Field Semantics and Interpretation Rules: Field semantics distinguish file-level aggregates from item-level evidence, define concise normalized answers, and encode required capabilities rather than model behavior.file_path denotes the smallest annotated support set, file_modality summarizes supporting files, modality_type describes each localized evidence item, and agent_cap uses constrained reasoning labels.
B.4 Evidence Object Specification: Evidence objects preserve directly checkable support through unique trajectory-local IDs, faithful evidence text, modality-normalized types, and locators such as document pages or media timestamps.Rationale steps cite these IDs, while evidence text preserves answer-supporting content rather than summarizing an entire file.
B.5 Rationale Trace Specification: Rationale traces normalize reasoning into planning, navigation plus reading, and integration plus verification, requiring evidence links for every substantive step except abstract planning.This step-level groundedness connects intermediate operations, observations, conclusions, and final answers to localized support.
B.6 Schema Validation Rules: Schema validation enforces parseability and reproducibility by checking evidence-ID resolution, file-path resolvability, and locator validity within document or temporal bounds.These checks stabilize bookkeeping for downstream evaluation across released records.
B.7 Designing Atomic Units for Evidence; B.7.1 Motivation; B.7.2 Definition; B.8 AU Generation Procedures; B.9 AU–Evidence Mapping: Atomic units provide modality-aligned, independently verifiable evidence segments, mapping text, pages or document regions, image regions, and audio-video intervals into a common evaluation representation.AU matching tolerates small temporal slack and spatial overlap differences, while the current release primarily records page- and timestamp-level locators.

B.10 Usage in Evaluation, Diagnosis, and Training … C.1 Task Taxonomy

HippoCamp combines AU-based fine-grained evaluation and diagnosis with a human-in-the-loop pipeline that constructs, structures, reviews, and validates grounded benchmark trajectories. Its task taxonomy covers factual retention and profiling through balanced, multimodal, long-horizon instances, while bounded LLM assistance remains provisional and human oversight preserves grounding, privacy, and schema consistency.

B.10 Usage in Evaluation, Diagnosis, and Training: AU-normalized grounding complements file-level retrieval by measuring decisive evidence localization and exposing wrong-page, timestamp, or visual-region errors.AUs primarily support fine-grained evaluation and diagnosis rather than a standalone headline score.
B.11 Human-in-the-loop QA Construction: HippoCamp constructs benchmark instances through five stages: question proposal, candidate consolidation, de-duplication and balancing, trajectory structuring, and bounded model assistance.The pipeline transforms candidate information needs into finalized, trajectory-annotated items.
B.12 Two-Source Question Proposal: Manual proposals capture realistic profile-specific information needs, while restricted-metadata LLM proposals expand coverage of modality combinations, evidence-set sizes, and long-tail task patterns.Synthetic proposals remain candidates rather than defining benchmark semantics.
B.13 Candidate Consolidation and Screening; B.14 De-duplication and Coverage Constraints: Human screening retains candidates that are grounded, non-trivial, and non-redundant, then balances modality combinations, evidence-set sizes, task families, factual retention, profiling, and long-tail cases.De-duplication removes superficial rewordings and repeated solution or evidence configurations.
B.15 Trajectory Structuring: Each accepted question receives a minimalist gold trajectory containing supporting files, localized evidence objects, rationale steps, and capability labels for search, perception, and reasoning.The compact support structure justifies the answer while allowing agents to discover longer or alternative valid paths.
B.16 Model Assistance Protocol; B.17 Prompt Families for LLM-assisted Proposal: LLM assistance is bounded to curated metadata, local file batches, directory cues, timestamps, and seed examples, excluding unrestricted files, personal identifiers, and external knowledge.Across prompt families, outputs remain provisional; sensitive financial cases may be authored directly by humans, with models limited to normalization or rationale drafting.
B.18 Inter-Annotator Agreement and Quality Control; B.18.1 Annotator Setup and Sampling Protocol; B.18.2 Unified Review Protocol; B.18.3 Adjudication Protocol: Domain-aware annotators review all three profiles, task families, modality configurations, and difficulty bands using unified checks, shortest plausible solution paths, and adjudication for disagreements.Profiling, legal, and financial items receive focused review when longitudinal interpretation or specialized constraints create ambiguity.
B.18.4 Quality Control and Automated Sanity Checks; B.18.5 Post-hoc Audit and Final Validation; C Tasks and Difficulty; C.1 Task Taxonomy: Final release combines rule-based and human quality control, privacy and consistency checks, stratified auditing, and schema validation, ensuring records remain auditable for long-horizon multimodal evaluation across Factual Retention and Profiling.Records are accepted, revised, or rejected based on grounding, clarity, validity, privacy, and structural requirements.

C.1.1 Factual Retention

Factual retention evaluates whether agents recover precise facts from device-resident files and keep answers traceable to explicit evidence. It spans atomic retrieval, file localization, temporal comparison, normative extraction, and multimodal verification.

Definition: Factual retention covers atomic fact retrieval, document-level localization, temporal or comparative recovery, and normative clause extraction.Across these regimes, answers must remain fully traceable to explicit file-grounded evidence.
Illustrative examples: Figures 11 and 12 illustrate grounded factual-retention answers with supporting file lists and evidence visualizations.Figure 11 presents cross-modal asset retrieval, while Figure 12 presents document–video compliance verification.
Illustrative examples: Cross-modal asset retrieval requires locating a vlog script, extracting requirements, and matching them to photos using document and visual evidence.The example tests precise file localization, structured fact extraction, and visual verification under explicit constraints.
Illustrative examples: Document–video compliance verification checks an advertisement’s logo placement against a clearspace rule extracted from a brand manual.The example combines rule extraction from the manual with spatial inspection of video frames to support grounded correctness.

C.1.2 Profiling … C.6.1 Definition

HippoCamp evaluates profiling through weak, distributed evidence and characterizes benchmark difficulty using evidence breadth, modality breadth, reasoning depth, and a coupled scalar score. Its queries span multimodal, longitudinal, and constraint-aware reasoning demands that expose challenges beyond single-source retrieval.

C.1.2 Profiling: Profiling infers user-level attributes by integrating weak signals across files, modalities, and time into consistent, actionable, evidence-traceable outputs.It requires longitudinal evidence integration, event-to-trait abstraction, and context-aware personalization rather than reliance on one decisive statement.
C.1.2 Profiling: The five profiling subtasks are preferences, behavioral patterns, scheduling information, retrospective reflections, and workflows, covering trait inference, periodicity recovery, planning, historical reconstruction, and procedure synthesis.Representative cases require grounding photo-editing preferences, a stress-regulation routine, conflict-aware scheduling, a graduation itinerary, and a recurring legal-aid workflow.
C.2 Complexity Axes and Marginal Distributions: HippoCamp characterizes complexity along evidence breadth, modality breadth, and reasoning depth, corresponding to increasingly demanding retrieval, multimodal perception, and multi-step integration.These axes provide interpretable first-order indicators but do not fully capture coupled constraints in realistic file systems.
C.3 Evidence Breadth: 354 overall questions require only one evidence file, yet the heavy-tailed distribution also includes substantial multi-file queries, such as 59 two-file questions.This distribution measures retrieval breadth in the benchmark’s realistic file-system haystack.
C.4 Modality Breadth: 65 overall queries require two modalities, with additional cases requiring three or more, creating a meaningful multimodal tail beyond dominant unimodal queries.These queries require heterogeneous-format interpretation, evidence localization, cross-file dependency handling, and temporal integration.
C.5 Reasoning Depth: Reasoning depth peaks at 6–8 annotated steps while retaining a long tail of deep multi-step queries; in Bei, 57 questions require 5 steps and 87 require 6.The step counts provide an explicit proxy for reasoning depth under structured trajectories.
C.6 Question Difficulty: Because marginal axes miss coupled constraints such as cross-file dependencies and cross-modal grounding, HippoCamp introduces a separate scalar difficulty score in Appendix C.6.The score approximates effective hardness beyond individual breadth and depth statistics.
C.6.1 Definition: The difficulty score combines eight normalized factors with weighted base terms, interaction coupling, a hard-case bonus, and sigmoid mapping to [0, 100].Log–quantile normalization uses the benchmark-wide 90th percentile and clipping to control heavy tails, while interactions model multi-file, multimodal, and long-horizon challenges.

C.6.2 Distribution

HippoCamp spans moderate-to-high overall difficulty with a substantial hard tail, while profiling is markedly harder than factual retention. Comparable profile-level distributions show that this difficulty reflects diverse personal ecosystems rather than a single regime.

Distribution: Factual retention peaks near score 47, whereas profiling peaks near score 97, separating fact-level retrieval and verification from user-level synthesis.The factual-retention peak contains 115 questions, while the profiling peak contains 26.
Distribution: Profiling is substantially harder than factual retention, with means of 89.1 versus 53.8 and ≥70 ratios of 93.3% versus 19.0%.Profiling requires aggregating weak signals across time and files into coherent user-level inferences.
Distribution: Overall difficulty is comparable across Bei, Adam, and Victoria, with means of 55.9, 58.4, and 58.6, respectively, despite differing distribution shapes and profiling tails.Their ≥70 ratios are 27.2%, 27.6%, and 25.6%, respectively.
Distribution: Mean LLM-judge scores are generally low-to-moderate and decline as difficulty increases, with pronounced degradation in the hard tail.This question-level relationship is reported across profiles and the full benchmark for nine methods using 5-point difficulty bins.

C.6.3 Correlation between Difficulty and Performance … D.4.1 Answer Quality

HippoCamp links increasing question difficulty to declining model performance and uses representative multimodal, evidence-grounded cases to expose retrieval, verification, and cross-modal reasoning demands. Its controlled, profile-isolated evaluation compares heterogeneous agent regimes with explicit budgets, failure handling, and complementary answer-quality metrics.

C.6.3 Correlation between Difficulty and Performance: Average judge scores generally decrease across higher difficulty bins, while remaining modest even at low-to-mid difficulty, supporting the benchmark’s difficulty definition and revealing broad performance limitations.The analysis uses 5-point difficulty bins and per-query LLM-as-a-judge scores on a 0–5 scale across profiles and the merged benchmark.
C.7 Profile Example Set; C.8 Representative Example from Profile (a) Bei Weiwei: Representative cases require agents to extract normative constraints, retrieve personal files, and verify multimodal evidence, as in the Japanese visa-photo example.The visa example checks 45 mm×45 mm size, front-facing presentation, no headwear, and a white background against Identity/Id photo_1.jpeg.
C.9 Representative Example from Profile (b) Adam Turner; C.10 Representative Example from Profile (c) Victoria Anne Clarke: Other examples test evidence-grounded legal drafting and cross-quarter corporate attribution by integrating correspondence or filings with supporting precedent and earnings-call transcripts.The legal case grounds a response in VYR v VYS ([2022] SGHCF 24) and Rule 100(2)(a), while the corporate case tracks Walmart revenue-growth drivers from Q3 FY2025 to Q2 FY2026.
C.11 Edge Cases and Challenging Examples: Rare edge cases occupy complexity tails, requiring scalable search, candidate pruning, deduplication, cross-file corroboration, and genuine cross-modal grounding beyond single-document retrieval.These realistic personal-workflow instances complement the benchmark’s broad mid-range coverage with deep compositional tests of retrieval, grounding, and verification.
D Evaluation Protocol and Robustness; D.1 Shared Evaluation Constraints and Framework; D.1.1 Pipeline Overview; D.1.2 Profile-Isolated Evaluation; D.1.4 Allowed and Disallowed Channels: The evaluation harness standardizes dataset loading, invocation, recording, normalization, and metric computation while preserving profile-local information boundaries and each method’s native interaction paradigm.Each query exposes only its corresponding simulated profile; methods receive the natural-language query and use permitted local interfaces rather than external information sources.
D.1.3 Execution Interfaces; D.2 Method Regimes and Tooling; D.2.1 Native Retrieval Setting; D.2.2 Vacuum Docker Agent Setting; D.2.3 Docker Tooling and Multimodal Return Path; D.2.4 Official Hosted Agent Setting: HippoCamp compares retrieval-native, vacuum-Docker terminal, and official hosted-agent regimes, with shared local access but non-identical tooling and backend-specific multimodal serialization.The hosted configuration is not fully tool-parallel to Docker agents, yet supplies a practically relevant upper-bound-style reference point.
D.3 Budgets, Retries, and Randomness; D.3.1 Resource Budgets; D.3.2 Retries and Failure Handling; D.3.3 Randomness and Determinism: Resource allocation is method-appropriate rather than globally matched, while retries address malformed, empty, tool-failed, or timed-out executions and successful incorrect answers remain failures.Randomness is controlled where possible, but interactive search ordering, decoding, tool branching, and backend behavior prevent uniform determinism.

D.4.2 Evidence Retrieval Metrics · D.4.3 Capability-wise Metrics

HippoCamp evaluates not only answer correctness but also whether agents retrieve the minimal supporting evidence and exercise search, perception, and reasoning capabilities in a grounded, auditable manner. Its metrics separate evidence-retrieval failures from reasoning failures and support diagnostic comparisons across heterogeneous skills.

D.4.2 Evidence Retrieval Metrics: File-level precision, recall, and F1 compare each method’s retrieved or referenced files with the annotated minimal supporting file set.Reported metrics average instance-level values across the evaluation split.
D.4.2 Evidence Retrieval Metrics: File Hit Rate corresponds to mean file-level recall and measures coverage of required supporting evidence despite potentially retrieved spurious files.
D.4.2 Evidence Retrieval Metrics: For profiling, file-level F1 uses annotated minimal supporting files rather than all potentially relevant files, capturing required evidence coverage instead of exact human reasoning paths.This distinction matters because profiling evidence often consists of weak signals distributed across multiple files and time points.
D.4.2 Evidence Retrieval Metrics: Retrieval metrics diagnostically distinguish failures to search or cover evidence from failures to reason over content that was already retrieved.
D.4.3 Capability-wise Metrics: Capability-wise performance groups human-annotated agent_cap labels into search, evidence perception, and reasoning, aggregating statistics from constituent subcategories.Each instance may contribute to one or more capability bins.
D.4.3 Capability-wise Metrics: Capability-family accuracy is the unweighted arithmetic mean of subcategory accuracies, while family-level F1 similarly averages subcategory file-level or evidence-level F1.Unweighted averaging prevents frequent subcategories from dominating heterogeneous capability families.
D.4.3 Capability-wise Metrics: Avg Latency averages valid per-instance runtimes with positive recorded duration and serves as a secondary measure of practical capability–efficiency trade-offs.Latency is not treated as a primary quality metric.
D.4.3 Capability-wise Metrics: Together, the metric suite requires correct answers, appropriate evidence retrieval, and grounded use of search, perception, and reasoning capabilities.This supports auditable evaluation aligned with HippoCamp’s benchmark philosophy.

D.4.4 LLM-as-Judge Robustness

HippoCamp uses a controlled LLM-as-a-judge protocol to evaluate semantic correctness for open-ended, evidence-grounded answers. Judgments combine binary decisions and graded scores with prompt controls, orthogonal signals, and human auditing.

D.4.4 LLM-as-Judge Robustness: The judge evaluates semantic match from the question, ground-truth answer, and model prediction rather than exact-string overlap.Paraphrases and non-conflicting elaborations are allowed, while omission of key information is penalized.
D.4.4 LLM-as-Judge Robustness: Each judgment returns a binary correctness decision and an integer quality score from 0 to 5, providing hard and graded answer-level signals.These signals are used throughout answer-level evaluation.
D.4.4 LLM-as-Judge Robustness: The protocol normalizes inputs into a fixed prompt and withholds hidden annotations, rationale traces, gold evidence sets, and explicit model identity from the judge.These restrictions reduce irrelevant variation and prevent direct disclosure of which model produced the candidate response.
D.4.4 LLM-as-Judge Robustness: Judge outputs are interpreted alongside file-level retrieval metrics and capability analyses, then checked through a stratified human audit across profiles, task families, modalities, and difficulty bands.The audit is designed to control judge-side variance rather than treat the LLM judge as a standalone oracle.

D.5 Extended Metric Summary

Table 8 shows that answer quality, retrieval quality, and runtime efficiency diverge substantially across HippoCamp methods. ChatGPT Agent Mode leads final answer metrics, while retrieval-oriented and RAG methods expose trade-offs between evidence coverage, specificity, reasoning, and latency.

D.5 Extended Metric Summary: ChatGPT Agent Mode achieves the best accuracy and judge scores across profiling, factual retention, and overall benchmark averages, while Terminal Agent (GPT-5.2) ranks second in answer quality.ChatGPT Agent Mode records 48.3/56.8/55.9 accuracy and 5.3/5.9/5.8 judge scores across profiling/factual retention/overall; Terminal Agent reaches 30.0/45.7/44.1 accuracy.
D.5 Extended Metric Summary: Retrieval advantages do not reliably translate into judged correctness: ReAct variants and Search-R1 achieve strong file-retrieval metrics yet remain substantially behind autonomous agents on final answers.ReAct (Qwen3-VL-8B-Instruct) leads overall File F1 at 36.9 and File Precision at 36.2, while ReAct (Gemini-2.5-flash) leads overall File Recall at 74.5.
D.5 Extended Metric Summary: Factual retention is easier than profiling across nearly all methods, with Terminal Agent improving from 30.0 to 45.7 accuracy and ChatGPT Agent Mode from 48.3 to 56.8.The difference reflects factual-retention questions more often requiring explicit user-local facts, whereas profiling requires abstraction.
D.5 Extended Metric Summary: Standard RAG and Self RAG trade high overall File Recall for limited File Precision and modest accuracy, indicating broad evidence retrieval without sufficiently precise evidence selection.Standard RAG records 69.3 recall, 21.7 precision, and 27.5 accuracy; Self RAG records 61.3 recall, 24.5 precision, and 23.8 accuracy.
D.5 Extended Metric Summary: Runtime reveals a capability–efficiency trade-off: Standard RAG is fastest, whereas ChatGPT Agent Mode is by far the slowest despite achieving the strongest answer-level performance.ReAct (Gemini-2.5-flash) and Search-R1 remain relatively efficient compared with full autonomous agents.