PaperlayerView exact revision on arXiv

Source-linked AI summary

Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Xia Hu, Honglei Zhuang, Brian Potetz, Alireza Fathi, Bo Hu, Babak Samari, Howard Zhou

arXiv:2602.19001v2cs.CVcs.AIcs.CLcs.IR

TL;DR

Existing multimodal personalization benchmarks primarily target concept recognition, despite expectations that assistants reason over users’ histories. Life-Bench evaluates broader evidence scopes and four retrieval paradigms; accuracy drops sharply as scope broadens, and performance beyond concept recognition remains modest.

  • Problem

    Existing multimodal personalization benchmarks primarily target concept-level recognition, while personal-history questions can require reasoning across events, relationships, timelines, and visual evidence.

    Source: §1 · p. 1, §4.1 · p. 7

  • Method

    Life-Bench provides over 11,800 human-verified synthetic question-answer pairs across 10 tasks, and the paper evaluates four retrieval families and proposes source-linked graph retrieval in LifeGraph.

    Source: §1 · p. 2

  • Results

    Accuracy declines with evidence scope, from 0.58–0.78 on concept identification to 0.30–0.57 on event understanding and below 0.40 on aggregated reasoning, with no single retrieval paradigm dominating.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LifeGraph’s structured, source-linked retrieval shows particular promise on event and aggregated tasks, while multimodal personalization beyond concept recognition remains an open challenge.

    Source: §1 · p. 2

    Performance beyond concept recognition remains modest for all evaluated methods, including LifeGraph.

    Source: §1 · p. 2

Abstract

from arXiv · show

As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people to understanding events to aggregating patterns, yet existing benchmarks primarily target concept-level recognition. We introduce Life-Bench, a fully synthetic, human-verified multimodal benchmark of over 11,800 question-answer pairs across 10 tasks, organized by required evidence scope: concept identification, event understanding, and aggregated reasoning. The benchmark's photo-centric personal histories are distributionally aligned with real user accounts under embedding statistic. The interconnected structure of personal data invites graph-based solutions; we propose LifeGraph, a personal knowledge graph framework providing structured retrieval with on-demand access to source visual evidence, showing particular promise on event and aggregated tasks. Systematic evaluation of four retrieval paradigms on Life-Bench demonstrates that accuracy degrades sharply with evidence scope, falling below 0.40 on aggregated tasks, and that no single paradigm dominates across categories. Performance beyond concept recognition remains modest for all evaluated methods, establishing personalization over multimodal histories as an open challenge and Life-Bench as a testbed for future progress.

1. Introduction

Personal assistants are expected to reason over multimodal personal histories, but benchmarks have largely emphasized concept recognition. Life-Bench evaluates broader evidence scopes; results show performance drops as scope expands, and remains modest beyond concept recognition.

1. Introduction

  • The paper frames the challenge as searching and reasoning across context that grows from concepts to events and then evidence aggregated across time.As scope expands, more records must be searched, more modalities aligned, and more evidence composed into a coherent answer.Source: §1 · p. 1
  • Life-Bench provides over 11,800 human-verified question-answer pairs across 10 tasks, organized into concept identification, event understanding, and aggregated reasoning.Its synthetic images, social networks, and event histories support public release without exposing real users’ data; generation is seeded with scenarios derived from YFCC100M photographs.Source: §1 · p. 2
  • 0.58–0.78 on concept identification falls to 0.30–0.57 on event understanding and below 0.40 on aggregated reasoning across four retrieval families.No single paradigm dominates; the paper identifies cross-modal alignment, relational and temporal navigation, and synthesis over dispersed records as challenges no evaluated method handles consistently.Source: §1 · p. 2
  • LifeGraph encodes concepts, relations, and events in a graph with links to source records, enabling graph traversal and on-demand access to multimodal evidence.It achieves the best overall retrieval-based performance and leads on several event-level and aggregated tasks, while performance beyond concept recognition remains modest across evaluated methods.Source: §1 · p. 2

2. Related Work

Prior personalization work often uses personal data for stylistic alignment or builds memory from dialogue, while multimodal systems commonly store visual content as text. Life-Bench spans photo-centric histories and evidence scopes, and LifeGraph retrieves linked multimodal records through a knowledge graph.

2.1. Personalization in LLM Era

  • Personalization research includes preference-based generation and long-term memory, but conversational archives remain common and captioning multimodal inputs can discard fine-grained visual evidence.Personal photo and social-media archives preserve timestamped records, while natural queries can require event reasoning or integration across a history.Source: §2.1 · pp. 2–3
  • Existing benchmarks largely emphasize recognition and preference, while real personal collections face release barriers and open synthetic alternatives have been text-only or smaller multimodal sets.The paper identifies a gap in publicly releasable benchmarks combining photo-centric histories, structured social networks, evidence-scope tasks, and source-record provenance.Source: §2.1 · p. 3

2.2. Structural and Graph-based Retrieval

  • LifeGraph organizes user-specific concepts, relations, and events in a knowledge graph, links them to supporting records, and retrieves original multimodal records on demand.The related work describes prior graph retrieval as typically returning text, including personal knowledge graphs and graph-based memory systems.Source: §2.2 · pp. 3–4

3. Life-Bench

Life-Bench is a synthetic benchmark for multimodal personalization, designed to avoid releasing real personal records while testing questions that require progressively broader context. Its construction combines synthetic personal histories, task-specific questions, and multi-stage verification, with distributional comparisons to real accounts.

3. Life-Bench

  • Life-Bench contains over 11,800 human-verified question-answer pairs across 10 tasks, covering concept identification, event understanding, and aggregated reasoning.All data is synthetic, and textual descriptions generated from real-world photos seed the histories to retain realism while preserving privacy.Source: §3 · p. 4

3.1. Benchmark Design Rationale

  • Synthetic histories address privacy and legal barriers to publicly releasing real people’s photos, relationships, and long-term records.The task design separately organizes questions by required context: personal concepts and relations, specific events, or patterns across the history.Source: §3.1 · p. 4

3.2. Task Categories and Definitions

  • The 10 tasks use text-only or text-and-image inputs and multiple-choice, binary, or open-generation outputs, with many requiring cross-modal evidence chains.Concept questions resolve entities and relations; event questions locate and inspect specific events; aggregated questions combine evidence across dates and events.Source: §3.2 · pp. 4–5

3.3. Benchmark Construction

  • Each Vaccount combines a social network of 3–5 concepts with timestamped images and descriptions; task-specific questions are grounded in linked records and images.The pipeline builds account seeds, concepts, and event histories, then uses retained source information to generate question-answer pairs.Source: §3.3 · pp. 5–6

3.4. Quality Control and Benchmark Validity

  • 0.56 is the Fréchet distance between Vaccounts and sampled YFCC accounts, compared with a real-to-real reference of 0.53; pairwise cosine distance is 0.91 versus 0.96.Questions were typically 0.07 closer in embedding space to a non-gold context than to the gold context, indicating direct semantic matching does not reliably identify the gold context.Source: Table 3 · p. 6, §3.4 · p. 7
  • Over 100 annotators reviewed every image and question-answer pair; uncertain or rejected items were adjudicated or corrected before release.The verification process checks image fidelity and answer support, and removes questions answerable without personal context through generic logic or visual shortcuts.Source: §3.4 · p. 6

4. LifeGraph for Personalization

LifeGraph addresses the need to retrieve and reason over personal histories whose relevant evidence spans text, images, relationships, and events. It organizes that history as a source-linked knowledge graph and retrieves graph paths alongside multimodal records when needed.

4.1. Problem Formulation

  • LifeGraph targets cross-modal personalization, where captions can omit visual details and unorganized records force query-time reconstruction of personal context.Relevant evidence can include appearance, actions, and spatial relations as well as relationships and events distributed across a history.Source: §4.1 · p. 7

4.2. LifeGraph Construction

  • LifeGraph builds a directed graph with six predefined entity types and open-ended relations, encoding facts as triples that can carry auxiliary attributes.For complex n-ary facts, a primary binary relation is paired with remaining arguments as edge attributes.Source: §4.2 · p. 7
  • Construction first builds a concept subgraph, then uses it to ground concepts in multimodal records and add event-centered triples linked through shared entities, dates, or objects.A two-turn extraction protocol identifies typed candidate entities before generating triples among them.Source: §4.2 · p. 7

4.3. Graph Retrieval with Multimodal Evidence

  • At query time, LifeGraph returns traversed relational paths and follows provenance links to source records when visual details are absent from the graph.Retrieval jointly ranks relation–entity candidates, checks whether the accumulated context suffices, and can stop before reaching the depth limit.Source: §4.3 · p. 8, §4.2 · p. 8

5. Experiments

The experiments compare retrieval methods across Life-Bench’s tasks and find that performance depends on evidence scope and the method’s indexing strategy. LifeGraph’s source-linked graph helps on event and aggregated tasks, but retrieval and reasoning remain limiting factors.

5.2. Main Findings

  • Average accuracy across retrieval methods declines as required evidence scope and reasoning complexity increase: 0.58–0.77 on Concept Identification, 0.30–0.57 on Event Understanding, and 0.26–0.40 on Aggregated Reasoning.The best retrieval results reach only 0.33 on Relational Temporal Reasoning and 0.20 on Frequency and Counting; even full-context achieves only 0.38 on Frequency and Counting.Source: §5.2 · p. 10
  • On all 10 Vaccounts, the method ordering holds, and per-account spread is smaller than the gaps between methods.Figure 3 groups category-wise accuracy by Vaccount; bars show means and dots show individual accounts.Source: Figure 3 · p. 9, §5.2 · p. 11

5.1. Experimental Setup

  • All methods use Gemma-3 12B as the common backbone for controlled comparison, with retrieval context sizes k∈{3, 5} and LifeGraph depths d∈{2, 3}.RAP and R2P are also reported with their original backbones at k=3; the study additionally reports reference settings and validates its automatic judging on 2,090 samples.Source: §5.1 · pp. 8–9

5.2. Main Findings

  • Retrieval and reasoning both constrain performance: oracle evidence raises Gemma’s four-event-task average to 0.74 from 0.57 with the strongest retrieval, while oracle performance differs by 0.19 across backbones.The oracle results are 0.95 for Gemini-3.5-Flash and 0.76 for Gemma-3-12B.Source: §5.2 · p. 10
  • No retrieval paradigm dominates: RAP and R2P rank last overall, BM25 is stronger over the shared caption corpus, and LifeGraph leads overall with its largest margins on event and aggregated tasks.RAP leads Preference & Persona, RAG-Cap leads Fine-Grained Scene, and HippoRAG2 leads visual-concept scores.Source: §5.2 · p. 10
  • From k=3 to k=5, RAG-Cap recall rises from 0.39 to 0.47 and HippoRAG2 recall from 0.36 to 0.45, while their accuracy gains are only 3.3 and 5 points.The passage reports that simply increasing retrieval budget cannot substitute for precise localization and organization of personal evidence.Source: §5.2 · pp. 10–11

5.3. LifeGraph Analysis

  • Removing source-image retrieval causes LifeGraph to collapse on visually grounded tasks while retaining strong relational and logical reasoning.The ablation indicates complementary roles for graph-structured personal knowledge and fine-grained visual evidence from source records.Source: Figure 4 · p. 11, §5.3 · p. 11
  • LifeGraph accuracy saturates at depth d=2, and its average effective depth stays below 1.9 even when the limit rises to 5.Accuracy peaks at width k between 3 and 5, then degrades for larger k as added records become increasingly irrelevant.Source: Figure 5 · p. 11, §5.3 · p. 11

6. Conclusion

Life-Bench evaluates personalization across concept, event, and history-wide reasoning, and results show that broader evidence scope remains difficult for current methods. LifeGraph shows promise on event and aggregated tasks, but multimodal personalization remains an open challenge.

6. Conclusion

  • Across four retrieval paradigms, accuracy falls below 0.40 on aggregated tasks, no paradigm dominates, and both retrieval and reasoning remain bottlenecks.LifeGraph shows particular promise on event and aggregated tasks, but performance beyond concept recognition remains modest.Source: §6 · p. 11

A. Life-Bench Details

  • The appendix describes Table 6 as summarizing the 10 tasks’ reasoning focus, input/output formats, and question counts, with Figure 6 showing modality distributions.The passage specifies the figures’ and table’s contents but supplies no distribution values.Source: A. Life-Bench Details · p. 16

A.1. Benchmark composition.

The released collection combines human-verified evaluation questions over personal histories with additional diagnostic and development data. Its task counts reflect the different scales of event archives and personal concept networks.

A.1. Benchmark composition.

  • The core evaluated benchmark contains 11,811 human-verified QA pairs across 10 Vaccounts, and the total released collection contains 13,458 QA pairs.An additional 1,647 single-hop concept questions are released as a diagnostic set, excluded from primary evaluation because evaluated methods nearly saturate on them.Source: §A.1 · p. 16
  • Event-grounded tasks draw on hundreds of timestamped moments and images per Vaccount, while concept- and persona-level tasks are bounded by social networks of 3–5 concepts.Figure 6 presents the suite’s input- and output-format distributions without specifying their values in the supplied passage.Source: §A.1 · p. 16, Figure 6 · p. 16
  • The development set adds comparable-scale personal context for 40 Vaccounts, generated and validated through the same pipeline but without human annotations or task-specific QA pairs.It is provided for method development and tuning without risk of overfitting to the evaluation set.Source: §A.1 · p. 16

A.2. Task Definitions

Life-Bench defines tasks by the personal-history evidence they require, and its supporting materials detail benchmark construction and LifeGraph’s graph representation and retrieval. The task set includes concept, event, and history-level reasoning, while retrieval combines graph traversal with source-record access.

A.2. Task Definitions

  • The 10 tasks span concept, event, and history-level reasoning, with definitions specifying their operational requirements and evidence scope.Tasks include concept QA and recognition, scene and person-centric event questions, preference and persona inference, frequency and counting, and relational temporal reasoning.Source: §A.2 · pp. 16–17, Table 6 · p. 17
  • The public release adds 1,647 single-hop direct concept-recognition questions, bringing the total to 13,458 QA pairs.The added questions cover Text Concept QA, Visual Concept Recognition, and Concept VQA.Source: §A.2 · p. 17

A.3. Seeds Pool for Life-Bench Construction

  • Life-Bench’s seed pool uses 1,296 concept seeds and 8,681 event seeds derived from YFCC100M captions.The event seeds span 167,823 image-level captions; original YFCC images are used only for captioning, not in benchmark construction.Source: §A.3 · p. 17

B.1. Construction details

  • LifeGraph’s schema uses six entity types and open-ended relations, with n-ary hyper-relational edges retaining multi-argument facts without auxiliary nodes.The entity types are PersonAnimal, Event, Date, Location, Activity, and Object.Source: §B.1 · pp. 17–18
  • Event-history extraction batches chronological records within the VLM’s effective reasoning capacity, using batch size 5 in the experiment.The greedy binpacking strategy aims to balance construction throughput with extraction quality.Source: §B.1 · p. 18

B.2. Retrieval details

  • LifeGraph retrieves from multiple query-relevant entities using bidirectional traversal, with average effective depth below 2 even at depth limit 5.The retrieval walkthrough combines traversed paths with fetched multimodal source references to answer a personal query.Source: Figure 9 · p. 18, §B.2 · p. 18, Figure 8 · p. 18

C.1. LLM-as-Judge Agreement Details

Two judges agreed on most sampled open-ended answers, and the method ranking remained the same when GPT-5.4 graded more strictly. The authors report that Gemini 3.6 Flash did not show self-preference bias toward Gemini-family responses.

C.1. LLM-as-Judge Agreement Details

  • The judge comparison used identical prompts on a stratified sample across seven generation tasks, 10 Vaccounts, and five evaluation settings.Three concept-identification tasks using exact-match scoring were excluded.Source: §C.1 · p. 18, Table 8 · p. 19
  • GPT-5.4 assigned lower accuracy across methods but preserved the full method ranking, and the larger LifeGraph–RAG-Cap gap under GPT-5.4 argues against Gemini judge self-preference.Agreement was at least 97.6% for Closed-book and Oracle Gemini-3.5-Flash settings.Source: Table 7 · p. 19, §C.1 · p. 19
  • 96.2% of 2,090 sampled open-ended instances received the same Correct/Wrong judgment, with task-level agreement from 89.3% to 100%.Agreement was 89.3% for Relational Person-Centric and 100% for Frequency and Counting.Source: §C.1 · p. 18, Table 8 · p. 19

C.2. Per-task Breakdown of Vaccount

Across Vaccounts, task-level patterns largely match category-level patterns, and differences between methods exceed within-method user-level variation.

C.2. Per-task Breakdown of Vaccount

  • Method rankings remain largely consistent across Vaccounts at both category and individual-task levels.The individual-task breakdown is provided in Figure 10, while Figure 3 groups accuracy by task category.Source: §C.2 · p. 19
  • Within-method cross-Vaccount variance is substantially smaller than the accuracy gaps between methods.Table 9 reports cross-Vaccount macro mean ± std for N=10.Source: §C.2 · p. 19
  • Table 9’s cross-Vaccount macro means can differ slightly from Table 4’s pooled per-task accuracy, but relative rankings and conclusions remain identical.The two tables aggregate results differently: one characterizes user-level dispersion and the other reports pooled task accuracy.Source: §C.2 · p. 19

C.3. Per-task Recall

Per-task recall is reported separately from computational-cost analysis, with aggregated reasoning excluded from recall because it requires cross-history synthesis. Retrieval approaches trade query-time cost against retrieval and reasoning accuracy.

C.3. Per-task Recall

  • Aggregated Reasoning tasks are omitted from recall evaluation because they synthesize across the full timeline rather than retrieve localized gold events.Table 10 provides the per-task recall breakdown for included tasks.Source: §C.3 · p. 19

C.4. Computational Cost and Efficiency

  • BM25, RAG-Cap, and RAP use no VLM calls at retrieval time, while R2P and HippoRAG2 use a fixed number per query.R2P and HippoRAG2 call the VLM for query analysis and candidate scoring, independent of traversal depth.Source: §C.4 · p. 19
  • LifeGraph uses three VLM calls per traversal depth plus one initial topic-extraction call, so retrieval cost grows linearly with depth.Its average effective depth remains below 2 in practice.Source: §C.4 · p. 19
  • The full-context baseline processes over 200 images per query, consumes 30–50× more input tokens, and cannot scale to archives of thousands of photographs.The authors identify lower-cost structured traversal that retains accuracy benefits as a future direction.Source: §C.4 · p. 19

D. Scope and Limitation

Life-Bench evaluates bounded personal histories at controlled scale, while its scope excludes long-horizon identity change and continual learning. The appendix also documents evaluation, data-generation, graph-construction, and retrieval prompts.

D. Scope and Limitation

  • Life-Bench omits long-horizon identity evolution and continual-learning evaluation, and its accounts are smaller than real archives that may contain tens of thousands of photographs.The authors describe this as a tradeoff among benchmark scale, construction cost, and verification rigor.Source: D. Scope and Limitation · pp. 19–20
  • Figure 10 shows per-task retrieval accuracy at k=5 and LifeGraph d=2, with bars for means and dots for individual Vaccounts.The plotted means aggregate 10 Vaccounts.Source: Figure 10 · p. 20
  • Ten Vaccounts average 289 images across 38 events; full human verification bounds the feasible account count, while 40 additional development accounts support broader-scale method development.The 10 evaluation accounts span varied social structures, event compositions, and personal contexts.Source: D. Scope and Limitation · p. 20
  • LifeGraph is a directional exploration rather than a production-ready solution, with higher inference cost from multi-call retrieval and dependence on general-purpose VLM extraction.The authors identify lower-cost structured traversal and stronger relational and temporal extraction as open needs.Source: D. Scope and Limitation · p. 20

E. Prompt Templates

Loading 2602.19001v2…