PaperlayerView exact revision on arXiv

Source-linked AI summary

ICM-Bench: Person-Level Identity Reasoning in Multimodal Agents with Long-Term Memory

Shidu Ren, Yunze Liu, Xing Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen

arXiv:2609.04438v2cs.CV

TL;DR

Broad memory benchmarks do not isolate whether multimodal agents maintain recurring person identities and reason over their cross-time relations. ICM-Bench pairs a theme-configurable synthetic video memory with traceable identity-focused questions; the strongest automated system scores 74.0% overall but 60.3% on Profile questions, and results remain a controlled diagnostic rather than evidence of transfer to unconstrained personal recordings.

  • Problem

    Existing broad memory evaluations do not isolate whether agents maintain recurring person identities and reason over their cross-time relations.

    Source: §1 · pp. 1–2

  • Method

    ICM-Bench combines synthetic long-video memories with open-ended questions and annotations tracing target identities and supporting clips or shots.

    Source: §1 · pp. 2–3

  • Results

    74.0% overall accuracy and 60.3% on Profile questions are achieved by Gemini 3.1 Pro, while current systems recover many events but struggle to organize distributed evidence around stable people.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Current systems are less reliable when evidence must be accumulated around a stable person than when recovering individual events.

    Source: §5 · p. 9

    Because ICM-Bench uses synthetic content and omits real-video challenges, its results are a controlled diagnostic, not evidence of transfer to unconstrained personal recordings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Long-horizon multimodal agents should remember not only what happened but also who participated. This capability depends on linking recurring faces, voices, names, person-associated objects, events, and social relations to consistent identities over time. Existing long-video and multimodal-agent benchmarks measure broad memory question answering, but they do not isolate the ability to maintain recurring person identities and reason over their cross-time relations. We introduce ICM-Bench (Identity-Centric Memory Benchmark), which, to the best of our knowledge, is the first benchmark specifically designed to evaluate identity-centric reasoning over long video memories in multimodal agents. The benchmark contains 839 synthetic clips spanning 141 minutes and 1,217 open-ended questions about six recurring adults in a one-year life album. A theme-configurable pipeline generates the video collection and associates each question with its target identities and traceable supporting evidence. We compare direct caption-memory baselines, memory-augmented agents, and graph-retrieval systems. Gemini 3.1 Pro achieves the highest overall accuracy of 74.0%, yet its score falls to 60.3% on questions that require long-term identity profiles. The results show that current systems recover many event-level memories but remain less reliable when evidence must be accumulated around a stable person.

1 Introduction

Long-horizon agents need to link recurring people across modalities and time, yet broad memory benchmarks do not isolate whether systems maintain stable identities. ICM-Bench targets that gap with traceable questions about recurring people and cross-time relations, where strong models still struggle with long-term profiles.

1 Introduction

  • Identity-centric memory requires binding faces, voices, names, objects, events, and social relations to stable people to answer questions across time.The paper notes that reminders such as “what she promised last week” are under-specified unless the intended person can be resolved from multimodal context [26] [18] [24].Source: §1 · p. 1
  • Broad long-video and memory benchmarks mostly score event understanding or evidence localization, leaving stable identity reasoning unmeasured by aggregate QA scores.Those scores cannot show when identity mechanisms are necessary, helpful, or insufficient.Source: §1 · p. 2
  • ICM-Bench evaluates whether agents can locate and aggregate evidence linked to recurring people and their cross-time relations.Its questions and annotations identify target people and supporting clips or shots, while the timeline portion available is controlled for each question.Source: §1 · pp. 2–3
  • 74.0% overall accuracy and 60.3% on long-term identity-profile questions are achieved by Gemini 3.1 Pro, the strongest direct baseline.The results also report weaker open-source direct baselines, gains for several memory-augmented and retrieval settings over corresponding open-source baselines, and substantially worse ASR-only controls.Source: §1 · pp. 2–3

2 Related work

Prior long-video and multimodal-memory benchmarks cover extended context and diverse memory tasks, but recurring person identity generally remains context rather than the basis for selecting and connecting evidence. ICM-Bench focuses evaluation on persistent person-centered reasoning across an episodic timeline.

Long-video and multimodal-memory benchmarks

  • Existing long-video and memory benchmarks primarily organize questions around events, activities, references, entities, or world states, not recurring-person evidence links.Structured diagnostic benchmarks such as CLEVR and GQA provide a precedent for targeting specific reasoning capabilities with verifiable structure [17] [16].Source: §2 · p. 3

Long-horizon multimodal agents and memory organization

  • Long-term multimodal systems vary in how they organize and retrieve episodic, semantic, visual, and graph memories, but broad QA scores cannot establish stable person identity reasoning.Examples include M3-Agent, WorldMM, Vgent, and HippoRAG2 [24] [41] [35] [15].Source: §2 · p. 3

Person identity grounding across modalities

  • Face, speaker, and cross-modal grounding methods evaluate identity attribution within bounded clips or conversations, rather than persistent person-centered memory across distant events.ICM-Bench targets retrieval, relation reasoning, and profile inference across an extended memory [4] [20].Source: §2 · pp. 3–4

3 ICM-Bench

The section motivates identity-focused evaluation with a diagnostic showing that broad QA accuracy can persist without explicit identity components, then defines and constructs ICM-Bench to test person-linked evidence across time.

3.1 Motivating identity-processing diagnostic

  • 55.3% accuracy without face processing and 53.0% without face–voice linking exceed the full pipeline, while Audio only retains 52.2%.Broad QA can remain largely unchanged without dedicated visual identity processing because questions may be answered from transcripts, local events, or co-occurrences; ICM-Bench centers recurring people and cross-time relations instead.Source: §3.1 · p. 4

3.2 Task definition and benchmark annotations

  • Identity Recall and Cross-Episode Retrieval use a permitted timeline prefix, while Profile questions use the full timeline to infer accumulated roles, habits, preferences, or relations.Each item records its question, answer, category, supporting clips or shots, involved people, and latest accessible clip; evidence annotations are withheld from systems.Source: §3.2 · p. 4

3.3 Theme-configurable benchmark construction

  • A theme-configurable pipeline plans timelines, recurring cast, and identity cues; reuses reviewed visual anchors and recurring voices in rendered clips; and retains records linking questions to intended identities, relations, and supporting shots.The one-year Li Ming’s Life Album, from May 1, 2025 to May 1, 2026, has six recurring adults and 839 clips totaling 141.0 minutes.Source: §3.3 · p. 5, Figure 2 · p. 6

3.4 QA construction and quality control

  • 1,217 open-ended questions comprise 400 Identity Recall, 500 Cross-Episode Identity Retrieval, and 317 Long-Term Identity Profile Inference items.Candidates are generated from plans, events, dialogue, timeline facts, and relations, then filtered and manually reviewed for answerability, evidence, and cutoff validity.Source: §3.4 · pp. 6–7

4 Experiments

Experiments show that automated systems retrieve many event-level memories more reliably than they infer stable person-level profiles, while identity-specific ablations and qualitative errors expose distinct failure modes. The evaluation uses a shared open-ended QA protocol across the benchmark’s question families and system types.

4.1 Experimental setup

  • The evaluation covers 1,217 questions across Recall, Retrieval, and Profile, with question-specific cutoffs for Recall and Retrieval and the full timeline for Profile.It compares direct caption-memory models, memory agents, graph-retrieval systems, transcript-only controls, and human answers under a shared semantic-judging protocol.Source: §4.1 · p. 7

4.2 Main results

  • 73.4% for Gemini 3.5 Flash direct captions exceeds HippoRAG2’s 63.4% and M3-Agent’s 58.7%, while memory organization can lower Profile accuracy despite overall gains.For Qwen3.5-9B, Profile accuracy falls from 44.2% to 31.2% with Vgent and 27.4% with HippoRAG2.Source: §4.2 · p. 7
  • 93.8% human overall accuracy exceeds Gemini 3.1 Pro’s 74.0% overall and 60.3% Profile accuracy, the best automated results.Table 4 reports accuracy on all 1,217 open-ended questions under the shared judging protocol.Source: §4.2 · p. 7, Table 4 · p. 8
  • 45.4% overall for M3-Agent compares with 42.9% without equivalence linking and 41.0% without face processing, while a documented case shows incorrect identity merging.The ablation indicates identity-related components affect downstream QA, but is not a complete assessment of M3-Agent.Source: §4.2 · p. 8
  • 73.4% to 38.7% is Gemini 3.5 Flash’s overall accuracy change when caption memory is replaced by ASR-only memory, showing transcripts alone are insufficient for many questions.ASR-only controls omit visual captions, frames, face cues, and image-derived event descriptions.Source: §4.2 · p. 9
  • 7.0% of Qwen3.5-9B responses are abstention-like under HippoRAG2, versus 5.7% for Qwen3-VL-8B-Instruct; Profile rates are 6.9% and 1.9%, respectively.Because the retrieval framework is fixed, the difference suggests some errors arise from answerer response calibration rather than retrieval alone.Source: Figure 3 · p. 8, §4.2 · p. 9
  • 60.3% is the best automated Profile score, versus 89.6% for humans; Profile requires aggregating identity, role, and relationship evidence across the album.Most open-source or framework settings remain below 46% on Profile questions.Source: §4.2 · p. 9

4.3 Qualitative Failure Analysis

  • Profile errors include relation misinterpretation, retrieval distractors, and over-abstention, despite systems describing individual scenes.The examples indicate failures in relation interpretation, evidence selection, or cross-episode aggregation rather than one common stage.Source: §4.3 · p. 9

5 Conclusion

ICM-Bench evaluates recurring identities and cross-time relations using reviewed synthetic life-album clips, traceable annotations, and open-ended questions. Results show a gap between event recovery and stable person-level inference, with transfer to unconstrained personal recordings outside the demonstrated setting.

5 Conclusion

  • ICM-Bench combines reviewed synthetic life-album clips, traceable person-level annotations, and 1,217 open-ended questions across identity recall, cross-episode retrieval, and profile inference.The strongest automated system reaches 74.0% overall and 60.3% on Profile questions, while transcript-only controls perform substantially worse.Source: §5 · p. 9
  • Current systems recover many individual events but still struggle to organize distributed audio-visual evidence around stable people over time.The strongest automated system reaches 74.0% overall and 60.3% on Profile questions.Source: §5 · p. 9
  • Because ICM-Bench uses reviewed synthetic content and omits real-video challenges, results are a controlled diagnostic rather than evidence of transfer to unconstrained personal recordings.The listed omitted challenges include occlusion, camera motion, changing illumination, background noise, and overlapping speech.Source: §5 · p. 9

A Implementation details

The implementation uses a shared evaluation interface while allowing each framework to build and access memory through its native procedure. Construction artifacts illustrate how recurring identities and shot-level plans are prepared for video rendering.

A Implementation details

  • All methods answer the same 1,217 questions with concise-answer instructions, while evaluator-side references, target identities, and evidence annotations remain hidden.Recall and Retrieval use clips up to question-specific cutoffs; Profile uses the complete timeline, and video methods may use a common face–voice calibration clip.Source: A Implementation details · p. 13
  • Gemini direct baselines caption complete clips, while Qwen baselines caption 12 sampled frames per clip with timestamped, speakerless ASR text.Both produce chronological clip-level records and answer without a retrieval stage.Source: A Implementation details · p. 13
  • Vgent, M3-Agent, and HippoRAG2 use distinct native memory and retrieval procedures; the M3-Agent ablations disable face processing or face–voice equivalence linking.Vgent retrieves five top-ranked nodes, while HippoRAG2 removes records beyond the permitted timeline before forming answer context.Source: A Implementation details · p. 13
  • The primary Gemini 3 Flash semantic judge sees the question, reference answer, and response; abstention-like outputs include “Unknown” or explicit claims that evidence is insufficient.ASR-only controls retain timestamped transcripts but remove visual information, and a second judge is used for the response analysis.Source: A Implementation details · p. 13

B.1 Main-theme construction artifacts

  • Canonical images establish the six recurring identities, while shot anchors specify planned cast, wardrobe, setting, blocking, and composition for visual references.Table 5 presents records from actual shot plans, including dialogue, audio, and camera instructions.Source: §B.1 · p. 13

B.2 Second-theme construction example

The authors apply the construction process to a second theme, a one-month community-theater series, to illustrate construction with a new cast and setting. The pilot includes recurring characters across planned scenes but is excluded from benchmark evaluation.

B.2 Second-theme construction example

  • The Backstage in São Paulo pilot contains six recurring adults, 14 scene groups, 30 clips, and 84 planned shots, and is excluded from benchmark QA and evaluation.It demonstrates construction under a second theme with a new cast and setting.Source: §B.2 · p. 13
  • Table 6 presents the structured record for a stage-rehearsal anchor, while Table 5 captions describe shot-plan excerpts by cast, blocking, dialogue, audio, and camera fields.The supplied passages identify these tables as shot-plan examples, not evaluation results.Source: §B.2 · p. 13, Table 6 · p. 14, Table 5 · p. 14
  • Figure 5 shows Renata recurring across control-booth, rehearsal, and outdoor rest-day settings, and Carlos in a canonical reference and rest-day anchor.Figure 4 illustrates canonical images and selected anchors used before video rendering.Source: Figure 4 · p. 14, Figure 5 · p. 15

C QA examples

The QA examples span local-event Recall, cross-clip Retrieval, and person-level Profile questions. The section also describes question curation, human answerability and judge-reliability checks, and evaluation prompts.

C QA examples

  • Recall targets a local memory, Retrieval connects evidence across multiple clips, and Profile aggregates repeated person-level evidence over time.Questions are open-ended and paired with lightweight evidence metadata; some identify the queried person through context or a family relation.Source: Table 7 · p. 15, C QA examples · p. 15

D QA validation and curation protocol

  • Candidates undergo automated temporal and category checks, then human review for answerability, target identity, supporting evidence, cutoff, ambiguity, and plausible alternatives.Only items passing every check enter the final set of 1,217 questions.Source: D QA validation and curation protocol · p. 16

E Human answerability reference

  • Human evaluators’ mean accuracy is 93.8% overall, with 96.8% on Recall, 94.2% on Retrieval, and 89.6% on Profile.Two independent evaluators answered using only the audio-visual memory permitted for each question, without reference answers or metadata.Source: E Human answerability reference · p. 16

F Judge reliability protocol

  • Judge–human agreement for M3-Agent is 96.6% for Gemini 3 Flash and 95.7% for GPT-5.4-mini; for Gemini direct, it is 96.1% and 95.3%, respectively.The audit compares two independent model judges with two human evaluators across complete answer runs.Source: F Judge reliability protocol · p. 16

G Evaluation prompt templates

  • Three prompt templates support reproducibility: caption-memory construction, open-ended answering, and semantic judging.System-specific implementations use minimal wrappers for each framework interface.Source: G Evaluation prompt templates · p. 16

G.1 Caption generation

  • The caption-generation prompt requests a JSON object covering clip, caption, visual details, speech, audio events, people, objects, places, and time or event.It directs the captioner to use visible or transcript-supported information and avoid inventing unclear speech.Source: §G.1 · p. 17

G.2 Open-ended answering

  • The answering prompt asks for a concise likely answer from the evidence and permits reasonable guesses when information is insufficient.It also instructs the assistant not to answer “Unknown” or provide reasoning.Source: §G.2 · p. 17

G.3 Open-ended semantic judge

  • The semantic judge evaluates whether an agent answer supports or implies the ground truth in context, allowing semantic equivalence rather than requiring exact wording.The prompt returns only “Yes” or “No.”Source: §G.3 · p. 18

H M3-Agent identity-linking diagnostic

The diagnostic shows how M3-Agent’s face-and-voice identity linking can merge distinct speakers into one persistent character identifier. The section also notes three Profile failures, tracing each from evidence to system output.

H M3-Agent identity-linking diagnostic

  • M3-Agent links face and voice features into persistent character identifiers, but one run returns both <character_0> and <character_17> for Li Ming.The described mechanism predicts equivalence relations between observations before merging linked features.Source: H M3-Agent identity-linking diagnostic · p. 19
  • In the observed clip, M3-Agent merges voice observations from different speakers, then represents both visible participants as <character_0> and assigns Sarah’s name to that identifier.The figure shows two distinct face observations and three voice segments; the saved equivalence explicitly merges <voice_3886> with <voice_3053>.Source: H M3-Agent identity-linking diagnostic · pp. 18–19, Figure 6 · p. 18

I Detailed Failure Cases

I.1 Relationship-specific profile miss

The case concerns a Profile question whose answer depends on connecting Li Ming’s repeated photographs of Wang Lin to their relationship. M3-Agent retrieves photography-related memories but misses her as the recurring subject.

I.1 Relationship-specific profile miss

  • Three retained episodes show Li Ming photographing Wang Lin’s spontaneous reactions and interactions with her surroundings.Answering the question requires connecting the repeated subject to their relationship, not merely recognizing visual themes.Source: §I.1 · p. 19
  • M3-Agent retrieves memories about macro lenses, city views, leaves, parks, and cats, then answers with a generic preference for candid street photography.The semantic judge marks that answer incorrect.Source: §I.1 · p. 19
  • The diagnosis is that retrieval misses Wang Lin as the recurring subject, so the answer gives a generic photographic style instead of Li Ming’s repeated preference for candid photographs of her.Source: §I.1 · p. 19

I.2 Retrieval distractor

The postcard question requires linking the same object across three separated episodes. Vgent retrieves travel-related distractors rather than those episodes and consequently repeats a receipts-and-tickets distractor.

I.2 Retrieval distractor

  • The answer depends on recognizing postcards across three separated episodes of examining, purchasing, and later organizing the cards.No individual episode alone captures the recurring-object pattern.Source: §I.2 · p. 19
  • Gemini 3.1 Pro direct and both HippoRAG2 variants answer with postcards, while Vgent variants give incorrect travel-related answers.The Vgent + Qwen3.5-9B answer is “Travel receipts or tickets”; Vgent + Qwen3-VL-8B-Instruct answers “Memory cards.”Source: §I.2 · p. 20
  • Vgent’s five retrieved memories omit all three annotated postcard episodes and instead include travel and record-keeping distractors, leading it to answer with tickets or receipts.The final retrieved memory explicitly mentions train tickets and European receipts.Source: §I.2 · pp. 19–20

I.3 Over-abstention under repeated evidence

The weekend-morning routine must be inferred from repeated episodes of togetherness during separate quiet activities. Some systems capture that shared routine, while one retrieval system abstains because it does not aggregate ordinary events.

I.3 Over-abstention under repeated evidence

  • The recurring pattern is the couple sharing a space while pursuing separate quiet activities, including reading and stretching.The episodes occur in the living room or study, but none states the routine directly.Source: §I.3 · p. 20
  • Gemini 3.1 Pro describes quiet independent activities in shared space; Vgent + Qwen3.5-9B proposes alternating collaborative activities and individual relaxation, while HippoRAG2 + Qwen3.5-9B says evidence lacks information.HippoRAG2 + Qwen3-VL-8B-Instruct lists stretching, cleaning, and preparing for outings.Source: §I.3 · p. 20
  • HippoRAG2 + Qwen3.5-9B abstains despite relevant evidence, which is individually low-salience and spread across ordinary events.The diagnosis is that this system does not aggregate those events into a stable social routine.Source: §I.3 · p. 20

Related papers

Loading 2609.04438v2…