Source-linked AI summary
Personal AI Agent for Camera Roll VQA
Thao Nguyen, Krishna Kumar Singh, Donghyun Kim, Yong Jae Lee, Yuheng Li
TL;DR
Personal camera-roll VQA requires reasoning over large, personalized, long-horizon visual memories that existing retrieval and long-context approaches do not adequately support. The paper introduces the camroll dataset and camroll-agent, which combines hierarchical memory with dedicated tools for navigation. The agent outperforms the evaluated baselines and methods, while the paper identifies hierarchical memory, iterative retrieval, and domain-specific tool use as critical.
Problem
Existing systems and datasets do not adequately support open-ended, personalized reasoning over large camera rolls, where exhaustive visual context is impractical.
Method
The paper constructs camroll from human-annotated personal-camera-roll VQA and designs camroll-agent with hierarchical memory and a minimal dedicated tool set.
Results
camroll-agent outperforms the evaluated RAG, memory, and agent baselines, achieving 4.11 free-form performance with approximately 3.2k tokens.
Takeaways & Limitations
Hierarchical memory, iterative retrieval, and domain-specific tool use are critical for long-horizon personalized visual reasoning.
Abstract
from arXiv · showhide
We study the personal camera roll visual question answering setting. In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant photos to answer queries, ranging from simple factual questions (e.g., ``Name of the food I tried yesterday?'') to more open-ended ones (e.g., ``Recommend some dishes I have never eaten before''). Given the vast nature of the personal camera roll (i.e., multiple years, hundreds to thousands of photos), a successful AI assistant needs to understand a long-horizon, highly personalized visual content stream in order to navigate and locate the correct and/or relevant information. To support this, we collect and manually annotate questions that mimic real-world usage. The final dataset, camroll, contains 50 users, 31,476 images, and 2,500 QA pairs. We further design camroll-agent, a conversational AI agent equipped with hierarchical memory and a minimal set of tools for efficient navigation over large, personalized visual memory. Experimental results show that camroll-agent outperforms numerous baselines and methods for long-context understanding AI agents system. Together, the camroll dataset and camroll-agent highlight the gap in AI agents' long-context reasoning: personalized visual memory requires different approaches from standard long-context textual memory, especially when consistency, visual details, and user-specific context are present.
1 Introduction
Personal camera rolls are large, visually redundant archives that existing chronological and surface-level retrieval systems do not organize around human memory or support efficiently. The paper introduces camroll and camroll-agent to study and navigate personalized, long-horizon visual question answering.
- Motivation: Smartphones accumulate thousands of personal images that serve as cues for revisiting autobiographical experiences.The motivating examples range from everyday meals to significant life events.
- Motivation: Chronological organization and basic similarity search do not match context-, experience-, goal-, or event-based memory structures.Existing systems largely operate as surface-level retrieval modules.
- Technical challenge: 1–10 millions tokens can be required to place thousands of HD photos into an MLLM context, making exhaustive processing impractical and degrading long-context understanding.A single HD photo costs approximately 1–3k tokens.
- Research gap: Existing datasets lack a standardized framework for open-ended, personalized reasoning over long-horizon visual memory.Prior work covers text-only personalization, generic visual retrieval, or simple retrieval queries over real photo collections.
- Contributions: camroll contains 31,476 photos and 2,500 personalized VQA pairs from 50 real user camera rolls, while camroll-agent combines hierarchical memory with dedicated navigation tools.The agent is designed for efficient search and navigation over large-scale visual memory.
2 Related Work
Related work spans personal photo-album understanding, long-context multimodal models and retrieval-based memory, and autonomous agents. The paper positions personal camera-roll VQA as requiring domain-specific tools and interaction patterns beyond existing systems.
- Personal photo albums: Personal photo-album research has progressed from organization and event recognition toward broader album-based image manipulation tasks.Earlier work also addressed representative or interesting image selection.
- MLLMs with long-context understanding: Long-context MLLM performance consistently degrades as context length increases, motivating context extension, efficiency, and retrieval-augmented memory methods.These methods address pure text, interleaved multimodal sequences, and image collections.
- AI agents: AI agents combine a language or multimodal model with tools and memory to reason, plan, act, and maintain context across interactions.The related-work description frames memory as supporting long-term consistency and personalization.
- AI agents: Different domains require substantially different tools and interaction patterns, limiting the generalization of domain-specific agents toward truly general-purpose systems.Coding agents can sometimes generalize, but their tools are designed for well-defined environments.
3 Camroll: Personal Camera Roll Dataset
camroll is a large personal-camera-roll VQA dataset assembled from real and curated photo sources, with human-posed semantic and episodic questions. Its annotations and analyses reveal strong user-specificity and a heavy-tailed, highly diverse answer distribution.
- Dataset: camroll contains 50 users, 31,476 images, and 2,500 QA pairs from camera rolls spanning 2–6 years.The dataset draws from in-house user data and YFCC-curated data.
- Data collection: The dataset combines YFCC-100M with in-house real-user data because YFCC is outdated and professionally curated, whereas in-house images better reflect incidental and redundant mobile capture.The in-house source is intended to align more closely with contemporary personal camera rolls.
- Data collection: YFCC filtering retains users with more than 500 photos, at least two years of temporal coverage, and Creative Commons licensing, followed by metadata and LLM-ensemble screening.The process samples 20 users and yields 15,927 images from YFCC.
- Annotation: Human annotators are prioritized over synthetic query pipelines to capture higher-level and longitudinal questions that owners uniquely contextualize.Questions are organized into semantic and episodic types corresponding to general facts and time- or place-situated events.
- Personalization: 16.5% episodic-question kNN user purity is 8× the 2% random baseline, while semantic-question purity remains near baseline at 2.1%.Answer purity is lower: 4.3% for episodic answers and 1.9% for semantic answers.
- Personalization: 90.2% of 1,875 distinct gold answer strings occur in only one user’s roll, alongside 66.9% of content tokens and 88.1% of answer bigrams.These statistics indicate strong user-level disjointness at multiple answer granularities.
- Answer distribution: 32.0% of camroll answer tokens are covered by the top 10% of its vocabulary, compared with 89.9% in VQA and 65.9% in LLaVA-1.5-mix-665k.At the top 0.1%, camroll covers 2.96% of answer occurrences, versus over half in VQA and LLaVA.
4 Camroll-agent: A Personal Camera Roll Agent
camroll-agent organizes personal camera rolls into linked pixel, caption, and event representations, then exposes them through retrieval and access tools. Its interface separates retrieval strategy from access depth and supports iterative, budget-aware interaction.
- Overview: camroll-agent builds hierarchical personal memory and exposes it through dedicated tools for answering questions over a user’s camera roll.The design is intended to support long-horizon navigation over personal visual data.
- Hierarchical memory: The three-level memory pyramid preserves raw pixels, personalized image captions with metadata, and compact event summaries linked across adjacent levels.Event summaries group chronologically contiguous images with natural-language summaries and date or location metadata.
- Memory construction: Memory layers are constructed chronologically, with personalized captions conditioned on the user’s identity and a recent image window.This conditioning grounds captions in who appears and what preceded the image.
- Memory construction: Event segmentation incrementally chooses ADD, UPDATE, or NO_OP to create, extend, or leave unchanged the current episodic event summary.The decision uses the current caption, timestamp, recent captions, and current event summary.
- Tools: The tool interface factorizes access by retrieval paradigm and depth, distinguishing semantic, lexical, and metadata filtering from previews, full records, and raw pixels.This design yields search, grep, list, get, and view.
- Interaction: A ReAct loop combines model reasoning with tool calls under an explicit remaining-budget reminder, while the modular interface permits swapping models, captioners, or retrieval backends.The agent loop and tool interface remain unchanged when components are replaced.
5 Experiments
Experiments compare retrieval, memory, and agentic approaches for personal camera-roll QA, showing that iterative, domain-specific navigation improves both answer quality and efficiency. Ablations and error analyses indicate that structured memory, semantic search, and better agent decisions are central to performance.
- Comparisons with baselines: Camroll-agent achieves the best results among evaluated methods, outperforming baselines including bare MLLM with full image captions.Table 3 identifies camroll-agent as the top-performing system across the quantitative comparison.
- Comparisons with baselines: 86.4% multiple-choice accuracy is reached with oracle evidence, while all images score 5.01 and all captions 3.82 on freeform QA.The full-image and full-caption settings require approximately 750k and 150k tokens, respectively, making them impractical despite their context access.
- Comparisons with baselines: 40+% versus 30% shows that RAG and memory-layer methods improve over no context, but remain below the 86.4% oracle and 63.4+% full-context settings.The authors attribute the remaining gap to limited one-time retrieval and loss of fine-grained visual details in textual representations.
- Comparisons with baselines: 4.11 freeform performance at approximately 3.2k tokens lets camroll-agent exceed ClaudeCode’s 3.77 at approximately 59k tokens and approach all captions’ 3.82 at approximately 150k tokens.The comparison highlights selective exploration enabled by structured memory and a dedicated tool set.
- Analysis: 48% of QA episodes remain active at Turn 5, with agents moving from broad search, grep, and list calls toward get and view for inspection and verification.Later-turn difficult questions trigger renewed coarse retrieval, while final budget-constrained turns favor symbolic retrieval or raw-pixel inspection.
- Analysis: 17.5% of camroll-agent failures are attributed to poor VLM ability, while most failures arise from agent decisions such as missed retrieval or insufficient image inspection.The analysis categorizes failures into six mutually exclusive groups and suggests dedicated post-training for memory-agent tasks.
- Analysis: 53.6% of camroll-agent’s tool budget goes to semantic search versus 25.2% to image viewing, compared with ClaudeCode’s 45.3% search and 51.9% Read usage.ClaudeCode uses approximately 59.0k tokens, whereas camroll-agent uses approximately 3.2k, linking domain-specific tools to more efficient investigation.
- Ablations: 4.22 is the full-memory overall score, while removing captions reduces overall performance to 2.29 and removing search produces the largest tool-ablation drop.The ablations also report 4.03 with generic captions and no event, and identify search as the most impactful tool.
6 Conclusion and Discussion
The paper presents camroll and camroll-agent for long-horizon personalized visual reasoning, emphasizing memory structure, retrieval, and specialized tools while identifying limitations and future directions.
- camroll-agent is designed for long-horizon personalized visual reasoning over personal camera rolls.
- Hierarchical memory, iterative retrieval, and domain-specific tool use are critical for this task.
- The work is primarily a benchmark and analysis effort rather than dedicated end-to-end memory-agent training.
- Future work should investigate learning-based retrieval, joint training, and stronger privacy-preserving personalization.
A.1 Broader Impacts
Personal camera-roll reasoning may support personalized assistants, memory support, and multimedia retrieval, but persistent multimodal memory introduces substantial privacy and security concerns.
- Personal camera-roll reasoning has potential applications in personalized AI assistants, memory support, and multimedia retrieval.
- Personal photo collections contain sensitive information about identities, relationships, locations, and daily activities.
- Persistent multimodal memory raises risks of unauthorized retrieval, profiling, and memorization of personal content.
- Future deployments should prioritize user consent, controllable memory management, secure storage, and privacy-preserving mechanisms.
A.2 Data Statistic
The appendix characterizes Camroll’s temporal and geographic coverage, question and answer structure, evidence signals, schemas, and agent error categories.
- Demographics and coverage: Camroll spans 24 years of personal photo-taking, from 2002–2026, across five continents and roughly 25 countries.
- Question analysis: Episodic questions average 15.9 words versus 7.3 for semantic questions, while semantic questions are dominated by what-style queries at 57%.
- Answers analysis: The median gold answer length is 2 tokens, and 72.9% of answers are multi-word phrases.
- Error categorization: Error analysis uses mutually exclusive categories evaluated top-to-bottom, including retrieval failure, perception or reasoning failure, hard-budget exhaustion, and premature stopping.
- Schemas: Question types classify answer shapes, whereas condition types encode constraints that scope the search; a question may carry multiple conditions.
A.3 Prompts
The prompts maintain structured long-term memory for a user’s photo library by processing images chronologically, generating captions, and updating episodic events.
- The prompt processes each current image alongside a user profile photo used only as an identity reference.
- The current image alone supplies the image caption, operation, and event reasoning.
- Images are processed strictly from oldest to newest, and only the most recent event row is updated when appropriate.
- An event is an episodic memory unit that can span consecutive photos belonging to one coherent real-world episode.
- The structured output records ADD, UPDATE, or NO_OP, an image caption, and an optional event with its name, description, date, and image paths.