ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

Javier del Pino; Salvador Rodríguez; Alejandro Garabito; Javier Álvarez; Chema Garabito

Source-linked AI summary

ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

Javier del Pino, Salvador Rodríguez, Alejandro Garabito, Javier Álvarez, Chema Garabito

arXiv:2609.03756v1Sep 3, 2026cs.CVcs.AI

TL;DR

Text-guided segmentation remains vulnerable to identity loss, fragmentation, and semantic confusion between targets and look-alikes. ENEAS unifies memory-based instance tracking with cascaded embedding and VLM verification, achieving a fourfold F1 improvement over the state-of-the-art baseline while preserving strict precision on its evaluated capture.

Problem
Text-promptable segmentation can confuse visually similar artifacts with target entities, lose targets after disappearance, and fragment objects during extreme close-ups, limiting reliable segmentation.
Method
ENEAS uses text-driven SeC initialization for instance tracking and a discovery cascade combining region proposals, SigLIP 2 filtering, and selective Qwen3-VL verification.
Results
A fourfold F1 improvement over the state-of-the-art baseline is reported, while ENEAS 2B enforces 94.7% precision on the evaluated semantic-discovery capture.
Takeaways & Limitations
ENEAS supports semantic tracking and discovery across ordered video and unordered image collections while distinguishing true instances from visually similar artifacts.
Takeaways & Limitations
Semantic-discovery recall is bounded by region proposals, VLM verification depends on contextual cues, dense scenes raise cost, and discovery does not assign persistent cross-frame identities.

Abstract

from arXiv · show

We present ENEAS, a unified, text-promptable method for instance tracking and semantic discovery. Text-promptable segmentation models, including the latest foundation models such as SAM 3, still suffer from temporal hallucinations, spatial fragmentation, and semantic misclassification: they fail to report target absence when an object leaves the field of view, segment local textures instead of the complete object during extreme close-ups, and prioritize visual features over ontological reality, so that visually similar artifacts such as statues, paintings, or reflections are segmented as target entities. ENEAS works two ways from a single method: precise tracking and high-quality segmentation of a unique instance, and open-concept discovery of every instance a text query names, resolved by a semantic verification layer. For tracking, we extend the geometrically robust SeC architecture, previously limited to point interactions, with a text-prompting adapter and leverage its temporal memory, so that the target is held through disappearance without drifting to distractors and kept whole even when it fills the entire view. For discovery, the verification layer combines high-speed visual embedding matching with conditional VLM refinement, invoking semantic reasoning only for ambiguous candidates, which filters out the ontological errors that visual-only models cannot distinguish while keeping latency low. Designed with 3D reconstruction in mind, where a single misclassified distractor corrupts the asset, ENEAS unlocks high-quality semantic tracking and segmentation of video, of broad libraries, and of collections of temporally or spatially unordered data, together with the discrimination to tell true instances from their doppelgangers: things that look alike but are not the same. The code and models are available at https://github.com/speridlabs/eneas

01 · Introduction

ENEAS addresses persistent temporal, spatial, and semantic failures in text-guided segmentation with one framework for instance tracking and category discovery. It combines text-driven memory tracking with selective semantic verification and reports improved robustness on hyper-realistic statues.

Motivation: Text-guided segmentation still loses target identity after disappearance, fragments objects during close-ups, and mistakes visually similar artifacts for target entities.These failures reflect geometrically accurate masks that are semantically incorrect.
Approach: ENEAS unifies specific-instance tracking and category-level discovery for ordered video and unordered image collections.The same grounding and segmentation components support both behaviors through a shared interface.
Instance tracking: Text-driven initialization extends SeC into a tracker that handles disappearance without drifting and preserves spatial integrity under extreme scale changes.Its temporal memory maintains the target identity across challenging views.
Semantic discovery: Selective semantic verification combines embedding matching, prompt ensembles, and a conditional VLM judge only for uncertain candidates.This design targets high semantic precision while keeping VLM activation and latency low.
Evaluation: ENEAS removes false positives from SAM 3 on hyper-realistic statues while preserving most of its recall.Ablations examine component contributions and the precision–latency trade-off.

02 · Related Work

Related work spans universal promptable segmentation, open-vocabulary detection, referring segmentation, memory-based video propagation, vision-language embeddings, and VLM judging. ENEAS combines these lines while targeting reliability under linguistic prompts.

Promptable and universal segmentation: Promptable segmentation evolved from fixed-vocabulary detectors to models accepting points, boxes, masks, and noun phrases for image and video segmentation.SAM introduced broad promptability, SAM 2 added streaming memory for video, and SAM 3 introduced promptable concept segmentation.
Open-vocabulary detection and segmentation: Open-vocabulary detection transfers vision-language knowledge into phrase-grounded detectors and segmentation systems for open-set concepts.Grounding DINO and related methods fuse language and vision features, while pixel-level methods align embeddings with text.
Referring and reasoning segmentation: Referring segmentation and multimodal mask decoders use language to localize or generate masks for objects described by expressions.SeC instead uses a VLM to maintain concept-level memory of a tracked object.
Video object segmentation with memory: Memory-based video object segmentation propagates masks through space-time memory, with later systems adding longer-term, motion-aware, or distractor-aware memory.Other approaches decouple open-vocabulary image segmentation from temporal propagation or associate detections across time.
Vision-language embeddings: Sigmoid vision-language matching scores image–text pairs independently, while prompt ensembles and calibration address robustness and score interpretation.ENEAS uses this family of techniques to filter candidate crops before semantic judging.
Vision-language models as judges: VLMs can judge image content but may hallucinate, and explicit reasoning improves accuracy at additional token cost.ENEAS uses constrained, selective judging rather than relying on unrestricted reasoning for every candidate.

03 · Method

ENEAS maps a natural-language prompt and ordered or unordered frames to binary masks through shared grounding and segmentation operators. It either propagates one initialized instance or cascades candidate discovery from proposals through embeddings and selective VLM verification.

Problem formulation: ENEAS accepts a natural-language prompt and a frame set, then returns binary masks using Florence-2 grounding and a Segment Anything mask generator.The frame set may be temporally ordered or unordered.
Problem formulation: For a specific instance, ENEAS outputs one mask per frame and sets the mask empty whenever the object is not visible.Category prompts instead produce a variable number of instance masks per frame.
Instance tracking: Text-driven initialization adapts SeC’s concept-level memory tracker, propagating the target while enforcing identity through disappearance and re-entry.The reference frame can be chosen anywhere in the sequence, and points or natural language can initialize interaction.
Semantic discovery: Semantic discovery re-evaluates every frame and uses a cascade whose later, more expensive stages process only unresolved candidates.This allows new instances entering the scene to be found without re-prompting.
Region proposal: Florence-2 proposes permissive candidate regions per frame, merging duplicates so later stages can remove distractors.The proposal stage prioritizes recall over precision.
Embedding verification: SigLIP 2 independently scores each candidate against an ensemble of category phrasings, and thresholds divide candidates into accepted, rejected, and uncertain groups.The thresholds trade latency against semantic rigor.
Semantic verification: Qwen3-VL judges only uncertain candidates, with neighboring candidates masked and binary output constrained to keep verification costs low.Accepted candidates are passed to SAM 2 for one mask per instance.

04 · Experiments & Results

ENEAS is evaluated for 3D-reconstruction-oriented tracking and semantic discovery, targeting identity drift, spatial fragmentation, and ontological false positives. Across qualitative and quantitative analyses, it preserves object identity and completeness while improving semantic precision through selective verification.

Method: ENEAS uses a cascade of embedding matching and conditional VLM verification, restricting expensive semantic reasoning to ambiguous candidates.This design aims to improve semantic precision while reducing verification latency to approximately one second on resource-constrained hardware.
Instance Tracking: Temporal memory lets ENEAS report target absence without drifting to curtains or other distractors when objects leave view or become occluded.The method also continuously re-evaluates frames, enabling newly entering instances to be detected without user re-prompting.
Instance Tracking: ENEAS maintains a complete mask during extreme close-ups, avoiding the spatial fragmentation that causes competing models to segment only local painting details.Its output preserves the painting as a holistic object even when it occupies the entire view.
Semantic Discovery: ENEAS filters semantic clutter in multi-instance discovery, rejecting boxes, tables, walls, paintings, and sculptures when prompts name chairs or living people.The qualitative analyses show reduced false positives while focusing on valid target subjects across sequences.
Quantitative Results: 94.7% precision is achieved by the default ENEAS 2B configuration, while the 4B variant reaches an F1-Score of 87.6% with improved recall and near-perfect semantic purity.The configurations expose a precision–recall–latency trade-off, with 2B positioned as an efficient baseline and 4B for more rigorous offline processing.

05 · Ablation Study

The ablations test component necessity, threshold sensitivity, efficiency, and model capacity, showing that semantic verification improves precision while introducing accuracy–latency trade-offs. Recall remains bounded by the initial proposal stage.

Component Necessity Analysis: Church Statues provides the primary testbed because COCO, LVIS, and video segmentation suites lack comparable ontological ambiguity.The experiments evaluate whether systems distinguish living subjects from realistic artifacts under the prompt “person”.
Component Necessity Analysis: Embedding-only filtering exposes a precision–recall trade-off: permissive thresholds admit statues, whereas strict thresholds discard valid instances.The baseline region proposal network has high theoretical recall but catastrophic precision, while strict visual filtering loses instances under difficult lighting or occlusion.
Component Necessity Analysis: The conditional VLM recovers true positives from the strict filter while maintaining minimal false positives; the 4B variant reaches baseline recall and the highest overall F1, at higher latency.The 4B model fully recovers recall to the baseline level and achieves near-perfect precision, but the paper reports a significant latency penalty.
Limitations: Semantic-discovery recall is inherently bounded by the initial region proposal network because missed, small, distant, or occluded instances cannot be recovered downstream.The contribution therefore focuses on maximizing the precision of detected proposals and semantic reliability.
Threshold Sensitivity and Trade-off Analysis: The Robust configuration uses a 0.10–0.90 uncertainty interval and strict acceptance to reject nearly all artifacts while achieving the highest overall F1-Score.Thresholds are configurable to adapt the system to different latency and precision targets.
Scenario Adaptability and Efficiency: Fast Mode on Moving Boxes reaches 1.14 seconds per frame, 27.0% VLM activation, and 98.0% F1 with zero False Positives.Compared with robust execution on Church Statues, latency falls from 3.29 seconds per frame as visual embeddings resolve most standard-scene instances.
Model Capacity: The 4B judge achieves 87.6% F1 at 5.02 seconds per frame, so ENEAS retains 2B as its default and exposes 4B for offline maximum-rigor use.The larger model increases latency by 52%, while the modular design supports different efficiency–accuracy requirements.

06 · Conclusion

ENEAS unifies text-guided instance tracking and semantic discovery across ordered video and unordered image collections. Its verification design raises Church Statues F1 substantially while exposing a precision–recall–latency boundary and scope limitations.

Conclusion: ENEAS handles specific-instance tracking and semantic discovery over ordered video and unordered image collections within one method.Tracking uses text-driven initialization for SeC, while discovery combines proposals, embedding filtering, and conditional VLM verification.
Conclusion: On Church Statues, F1 rises from 19.5% for SAM 3 to 82.8% with a 2B judge and 87.6% with a 4B judge, while precision remains above 94%.On SA-Co/VEval, ENEAS matches or improves SAM 3’s overall tracking profile.
Limitations: Semantic-discovery recall is bounded by region proposals, contextual judging can fail on cue-poor crops, verification cost grows with candidate count, and discovery does not assign persistent identities across frames.Dense scenes such as crowded streets fall outside the regime for which the method was designed.