Source-linked AI summary
Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale
TL;DR
Embodied VLMs struggle to maintain consistent object descriptions across viewpoints because prior approaches decouple exploration, association, and caption learning. EPOS-VLM unifies these functions with tokenized episodic memory in an autoregressive agent, improving object-level captioning and semantic consistency while supporting scalable scene representation.
Problem
VLMs often describe the same object inconsistently across viewpoints, limiting persistent object identity and semantic representations for embodied reasoning.
Method
EPOS-VLM jointly predicts data associations, object captions, and navigation actions from current observations, an explored map, and tokenized episodic object memory.
Results
The method improves object-level caption accuracy, cross-view semantic consistency, long-horizon association, and computational scalability over prior methods.
Takeaways & Limitations
Persistent object memory supports long-horizon semantic reasoning in embodied agents rather than relying only on transient visual representations.
Takeaways & Limitations
EPOS-VLM relies on an external instance segmentation model and is evaluated in photorealistic static environments with ideal sensors.
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.
1 Introduction
Embodied agents need memory and repeated viewpoint exploration to maintain stable object semantics, whereas static-image VLMs often produce inconsistent descriptions. EPOS-VLM unifies memory-conditioned association, captioning, and exploration to resolve ambiguities during navigation.
- Motivation: Repeated observations across diverse viewpoints and memory support stable object representations in embodied perception.The paper contrasts dynamic human exploration with independent visual inputs.
- Motivation: Static-image VLMs can describe the same physical object inconsistently across viewpoints, undermining persistent object identity for embodied reasoning.Viewpoint, distance, and occlusion changes can produce contradictory descriptions.
- Approach: End-to-end joint learning lets exploration revisit semantically unstable objects and collect informative viewpoints for ambiguity resolution.This formulation connects captioning, association, and action selection during navigation.
- Approach: EPOS-VLM jointly learns data association, object-level captioning, and action prediction within a unified Vision-Language model.Its structured episodic memory stores per-object caption histories and 3D positions.
- Approach: Tokenized episodic memory enables the pretrained VLM backbone to reason over long-horizon object histories while the agent gathers multi-view observations.Object-level pseudo-captions aggregated from memory histories supervise fine-tuning of the same VLM.
2 Related Work
Related work spans cross-view caption consistency, object-level memory for embodied representation, and action-driven semantic learning. EPOS-VLM combines these directions by embedding episodic object memory into caption generation and jointly optimizing aggregation, association, and exploration.
- Cross-View Semantic Consistency in Captioning: Cross-view consistency research addresses the mismatch between strong single-image VLM performance and inconsistent object descriptions under viewpoint, occlusion, or scale changes.These VLMs are commonly trained on independent image-text pairs without explicit cross-view consistency mechanisms.
- Object-level Memory for Embodied Representation: Embodied memory systems use episodic or spatial-temporal representations to support long-horizon reasoning under partial observability.EPOS-VLM instead embeds object-level memory directly into the VLM architecture.
- Action-Driven Semantic Learning: Action-driven semantic learning selects informative viewpoints, with prior work using caption disagreement or exploration policies to improve semantic acquisition.EPOS-VLM extends this direction by jointly optimizing exploration with cross-view aggregation and data association.
3 Embodied Persistent Object Semantics Visual Language Model (EPOS-VLM)
EPOS-VLM unifies visual observation, episodic object memory, data association, captioning, and navigation in one autoregressive Vision-Language model. Its memory preserves persistent object identities and caption histories while self-supervised pseudo-captions enforce viewpoint-invariant descriptions.
- Unified model: EPOS-VLM jointly predicts object associations, object-level captions, and navigation actions from visual observations and episodic memory.The model links current detections to persistent memory objects or initializes new identities within the same autoregressive policy.
- Object association: Transient visual identifiers are randomly reinitialized each frame, while explicit memory-based matching maintains persistent identity across observations.This separation prevents shortcut learning through identifier memorization and makes temporal linking an explicit model decision.
- Episodic memory: The episodic memory stores persistent object identifiers, estimated 3D positions, caption histories, and occurrence counts for long-horizon reasoning.Memory entries are updated when observations are associated with known objects, including position refinement and caption-frequency updates.
- Memory tokenization: Memory is serialized as structured textual object blocks containing identifiers, discretized positions, captions, and caption frequencies within scene delimiters.This representation injects object-level episodic information directly into the pretrained Vision-Language model context without architectural modifications.
- Self-supervised captioning: 3D-CPS aggregates caption histories and geometrically selected, complementary image crops to generate concise pseudo-captions containing multi-view-supported object attributes.Perspective-dependent details are omitted, and the resulting pseudo-caption supervises every frame in which the object appears.
4 Experimental Setup
The evaluation examines EPOS-VLM across captioning, semantic consistency, object association, scalability, and action-policy effects using simulated embodied-navigation data and established comparison methods and metrics.
- Evaluation tasks: EPOS-VLM is evaluated on pseudo-caption accuracy, object-level caption accuracy, cross-view semantic consistency, data association, computational scalability, and action-policy impact.The tasks assess both language quality and persistent object semantics during navigation.
- Data collection: Experiments use Habitat simulations with HM3D and Gibson scenes, scene-level splits, and episodes capped at 400 steps.Simulation provides depth, pose, and instance identifiers needed for object association and consistency metrics at scale.
- Baselines: The study compares pseudo-captioning against IC3, ECO-score, and LD-CPS, and object captioning against Qwen3-VL-2B, BLIP-2, and InternVL-3B.The supplied setup identifies aggregation, ranking, pseudo-captioning, and Vision-Language baselines for comparison.
- Metrics: Caption accuracy is measured with BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, while semantic consistency uses SBERT cosine similarity and IQR.Object tracking evaluation additionally includes association accuracy, F1-Match, F1-New, identity switches, and track fragmentation.
5 Results
EPOS-VLM improves pseudo-captioning, object-level captioning, cross-view consistency, generalization, data association, and exploration-policy outcomes while using episodic memory for persistent object semantics.
- Pseudo-Captioning Evaluation: 3D-CPS consistently outperforms prior pseudo-captioning methods across metrics, improving SPICE and caption similarity over IC3 and LD-CPS.It also exceeds ECO-score by +0.62 in CI and +6.10 p.p. in CS.
- Object-level Captioning and Semantic Consistency: +9.76 p.p. in SP and +0.89 in CI versus Intern-VL establish EPOS-VLM’s advantage over the best-performing off-the-shelf VLM.It also surpasses the multi-stage method by +0.65 in CI, +2.04 p.p. in SP, and +7.12 p.p. in CS.
- Object-level Captioning and Semantic Consistency: +29.22 p.p. in Mean CS versus BLIP-2, with the lowest IQR, demonstrates stronger cross-view semantic consistency.EPOS-VLM also improves Mean CS by +7.39 p.p. and IQR by -6.08 versus the consistency-based method.
- Generalization to Unseen Environments: +0.58 in CI, +3.46 p.p. in SP, and +7.65 p.p. in alignment versus the multi-stage method show improved captioning and alignment on Gibson after HM3D training.Mean and median caption similarity are higher and dispersion is lower, despite Gibson’s domain shift and lower mesh quality.
- Data Association and Computational Scalability: EPOS-VLM achieves comparable data association to the point-cloud method while representing objects with estimated 3D positions rather than dense point clouds.The reported differences are +0.02 p.p. in Acc, +0.02 p.p. in F1-M, and -0.01 p.p. in F1-N.
- Action Policy Evaluation: +9.22 p.p. in Mean CS and -6.14 in IQR versus Frontier Exploration show that the learned policy improves cross-view consistency by revisiting semantically inconsistent objects.EPOS-VLM also exceeds Frontier Exploration by +0.39 p.p. in CI and +4.38 p.p. in SP.
6 Conclusion
EPOS-VLM combines episodic object memory, object-description association, and action selection to support consistent long-horizon embodied captioning. Experiments report improvements in caption accuracy, cross-view consistency, data association, and scalability, while validation remains limited to photorealistic static environments with ideal sensors.
- EPOS-VLM integrates object-level episodic memory, description association, and action selection in one autoregressive architecture.
- The model processes object locations and descriptions across time and viewpoints for long-horizon semantic reasoning.
- EPOS-VLM improves object-level caption accuracy, cross-view semantic consistency, long-horizon data association, and computational scalability compared with prior methods.
- External instance segmentation can produce inaccurate masks that cause data-association and captioning mistakes.
- Validation focused on photorealistic static environments and assumed ideal sensors.
A.1 Architecture
The architecture serializes visual observations, explored-map information, and episodic object memory for a unified autoregressive sequence. It outputs persistent object associations, captions, and navigation actions, updating memory as objects are matched or newly discovered.
- EPOS-VLM uses RGB-D observations, an explored map, and prior episodic memory to predict associations, captions, and actions.
- Current-frame detections receive transient random IDs, preventing reliance on temporal ID continuity.
- The textual input contains serialized object blocks with persistent IDs, caption histories, and spatial information.
- The structured output includes association decisions, object captions, and a navigation action parsed from special tokens.
- The model emits MATCH decisions linking frame detections to memory objects or NEW_ID for unseen objects.
- Matched detections update episodic entries, while NEW_ID creates entries initialized with spatial and semantic information.
B Data Collection Details
The training-data pipeline detects and captions objects, maps semantic disagreement, and selects viewpoints balancing inconsistency against travel cost. Parameter-efficient fine-tuning and recovery procedures support scalable trajectory collection.
- Mask2Former detects instances, while Qwen3-VL-2B captions object crops and SBERT embeds those captions.
- The disagreement map accumulates semantic instability from caption statistics projected into the top-down navigable map.
- High-disagreement regions are thresholded, clustered, filtered by area, and ranked for target selection.
- Candidate viewpoints are generated around target boundaries and filtered for navigability and obstacle safety.
- Viewpoints balance disagreement priority against travel distance using coefficient α.
- The agent navigates with Habitat’s shortest-path planner while updating disagreement statistics for visible objects.
- A failure-recovery policy detects insufficient displacement over τs steps and samples nearby target positions when the agent is stuck.
- LoRA adapts selected attention and language-modeling layers while reducing trainable memory and computational cost.
D.1 Memory Scalability
Memory token count increases alongside actively tracked objects during exploration, indicating that the representation scales primarily with object cardinality. The proposed tokenization therefore provides a bounded alternative to geometry-centric accumulation.
- Memory token count and actively tracked object count show similar increasing trends during exploration.
- Tokenization scales predominantly with object cardinality rather than trajectory length.
- Example trajectories include successful multi-view object revisits and one partial navigation failure involving a wall collision.
- The tokenization scheme yields a predictably bounded memory representation unlike dense point-cloud accumulation.
D.2 Qualitative results
The qualitative results show exploration trajectories with repeated object revisits and compare caption predictions for the same objects across multiple views.
- Exploration trajectories: Exploration episodes revisit one or more objects from different vantage points, typically focusing on one or two objects within 400 steps.Some trajectories also include collisions with nearby walls while revisiting objects.
- Caption consistency: Figure 7 compares vanilla Qwen3-VL and fine-tuned EPOS-VLM captions for randomly selected objects across multiple observed views.The vanilla model predicts captions independently for each crop, producing view-dependent descriptions or minor semantic variations.
- Evaluation measures: Captioning metrics include BLEU, METEOR, ROUGE-L, CIDEr, SPICE, and SBERT-based cosine similarity.Figure 7 highlights caption errors in red.
E Additional pseudo-captioner comparisons
The paper compares Qwen3-VL-30B with two similarly sized vision-language models for pseudo-captioning on the manually annotated HM3D test set. Qwen3-VL-30B performs best across all reported metrics and is selected as the 3D-CPS backbone.
- Additional comparisons: The pseudo-captioner is evaluated against two large vision-language models with similar parameter counts on the manually annotated HM3D test set.The comparison is reported in Table 8.
- Additional comparisons: All compared models achieve strong pseudo-captioning results, supporting the strategy across different vision-language model backbones.The passage presents this as an overall comparison outcome.
- Backbone selection: Qwen3-VL-30B achieves the best performance across all metrics and is adopted as the 3D-CPS backbone.Its pseudo-captions are described as more accurate relative to human annotations and more semantically consistent.