Source-linked AI summary
SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
Haoyu Huang, Jinfa Huang, Zhongwei Wan, Xiawu Zheng, Rongrong Ji, Jiebo Luo
TL;DR
Agentic MLLM tool-use chains create sequential dependencies that raise latency and limit concurrency. SpecEyes uses agentic-level speculative reasoning with lightweight models and cognitive gating, reducing latency by up to 3.35× while remaining comparable in accuracy and improving throughput.
Problem
Tool-use chains impose data dependencies that create a fundamental barrier to per-query latency and system-level concurrency.
Method
SpecEyes uses a lightweight, tool-free model for speculative agentic reasoning, cognitive gating based on answer separability, and a heterogeneous parallel funnel.
Results
Across three image-understanding benchmarks, SpecEyes reduces end-to-end latency by up to 3.35×, remains comparable with the agentic baseline in accuracy, and consistently improves throughput under concurrent serving.
Takeaways & Limitations
SpecEyes converts speculative single-pass answers into system-level throughput gains while preserving full accuracy for queries that bypass the tool-use loop.
Takeaways & Limitations
Probability-based confidence can produce false acceptance because token probabilities may be spuriously high and do not measure separation from strong competitors.
Abstract
from arXiv · showhide
Agentic multimodal large language models (MLLMs) (e.g., OpenAI o3 and Gemini Agentic Vision) achieve remarkable reasoning capabilities through iterative visual tool invocation. However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead. This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level concurrency. To this end, we propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck. Our key insight is that a lightweight, tool-free MLLM can serve as a speculative planner to predict the execution trajectory, enabling early termination of expensive tool chains without sacrificing accuracy. To regulate this speculative planning, we introduce a cognitive gating mechanism based on answer separability, which quantifies the model's confidence for self-verification without requiring oracle labels. Furthermore, we design a heterogeneous parallel funnel that exploits the stateless concurrency of the small model to mask the stateful serial execution of the large model, maximizing system throughput. Extensive experiments on V* Bench, HR-Bench, and POPE demonstrate that SpecEyes achieves 1.1-3.35x speedup over the agentic baseline while preserving or even improving accuracy (up to +6.7%), thereby boosting serving throughput under concurrent workloads.
1 Introduction
Agentic MLLMs gain capability through iterative visual tool use, but stateful dependencies create latency and concurrency bottlenecks. SpecEyes addresses this by speculatively bypassing unnecessary tool chains with a gated small model and parallel serving architecture.
- Agentic MLLMs iteratively invoke perception tools during reasoning, supporting fine-grained inspection and multi-step visual tasks.
- Agentic depth D creates linear per-query latency growth and severe batching limits because each tool-use step depends on the preceding state.
- SpecEyes lifts speculative acceleration from tokens to the agentic level, bypassing entire tool-use loops for queries answerable without deep tool reasoning.
- Cognitive gating uses top-K logit answer separability as a label-free, scale-invariant criterion for trusting the small model or escalating.
- A heterogeneous parallel funnel runs the stateless small model concurrently and forwards low-confidence queries to the agentic model, converting acceptance into multiplicative throughput gains.
2 Related Work
Prior efficiency methods reduce token or perception costs within a fixed model and leave the serial agentic loop intact. SpecEyes instead targets the agentic level by bypassing complete tool-use loops through gated lightweight speculation and parallel execution.
- Token-level speculative decoding accelerates generation steps but leaves the multi-turn perception-and-reasoning loop fully serial.
- Multimodal token pruning, compression, sparsification, and merging reduce per-step computation without eliminating repeated tool invocations.
- SpecEyes differs by speculatively bypassing entire tool-use loops with a cognitively gated lightweight model and enabling high-throughput parallel execution.
3 Methodology
SpecEyes models the stateful sequential bottleneck of agentic MLLMs and uses a gated, heterogeneous funnel to bypass unnecessary tool chains while retaining an agentic fallback.
- Stateful Bottleneck: Agentic MLLMs maintain state across perception and reasoning steps, making each subsequent tool choice depend on prior observations and forcing sequential execution.The state transition applies the selected tool's observation to the next state, and agentic depth D denotes the number of reasoning steps.
- Stateful Bottleneck: Batch throughput is dominated by the slowest trajectory, so converting a βα fraction of queries to stateless inference reduces the effective agentic residual from B to (1−βα)B.The slowest or deepest query can stall the batch because each query's tool-use chain remains sequential.
- SpecEyes Pipeline: SpecEyes uses a four-phase pipeline: judge tool necessity, predict with the small model, gate by answer separability, and fall back to the full agentic model.Queries judged answerable from the global image proceed to speculative prediction; low-confidence or tool-dependent queries use the agentic fallback.
- SpecEyes Pipeline: The small model produces stateless predictions and logits concurrently, while the large model handles rejected queries through the complete stateful perception-reasoning loop.The full agentic model retains access to all tools and provides a safety net for low-confidence predictions.
- Heterogeneous Parallelism: When βα is large, lightweight front-end costs dominate expected latency, while heterogeneous parallelism converts speculative acceptance into system-level throughput gains.The architecture runs the stateless stage concurrently and forwards only low-confidence queries to the sequential agentic stage.
- Cognitive Gating: Answer separability replaces raw softmax confidence by measuring the decision margin between the top prediction and strong competitors.The gating score is intended to avoid softmax miscalibration and spuriously high token probabilities that can cause false speculative acceptance.
4 Experiment
SpecEyes is evaluated across three multimodal benchmarks and two agentic backbones, with experiments covering accuracy, speedup, gating, batching, and draft-model choices. The results show strong accuracy–efficiency gains when separability-based routing bypasses unnecessary tool trajectories, while tool-intensive HR-Bench remains a bottleneck.
- Experiment Setup: Experiments cover V*, HR-Bench, and POPE using Qwen3-VL-2B as the speculative model and DeepEyes or Thyme as agentic backbones.All benchmarks use accuracy, and the agentic models are capped at five tool-use steps per query.
- Main Results: SpecEyes (min) achieves 1.73× average speedup with DeepEyes while improving average accuracy from 81.39% to 84.26%.On V*, Relative Position rises from 82.89% to 89.47% at 1.90× speedup, while POPE reaches 2.13–2.19× speedup with accuracy above baseline.
- Main Results: SpecEyes (min) with Thyme delivers 1.42× average speedup while raising accuracy from 82.29% to 83.99%.POPE benefits most at 1.70–1.78×, whereas HR-Bench reaches only 0.95–1.01× because high-resolution inputs suppress speculative and acceptance rates.
- Main Results: SpecEyes avoids the POPE collapse observed when agentic backbones answer without tools, routing tool-required queries back to full agentic reasoning.DeepEyes without tools drops from 78.43% to 46.90% on POPE Adversarial, whereas SpecEyes preserves strong accuracy with speedup.
- Gating Analysis: Smin_sep provides the largest confidence-distribution separation, enabling a single threshold to preserve accuracy while maximizing speculative acceptance.Its incorrect-sample scores form a low peak and correct-sample scores a sharp high mode, unlike the more overlapping alternatives.
- Ablation Study: Lowering the gating threshold increases acceptance and speedup while degrading accuracy, with V* and POPE remaining near or above baseline across a broad range.HR-Bench is more sensitive because more queries require tool-assisted inspection.
- Ablation Study: Increasing batch size improves end-to-end speedup without affecting accuracy, but gains diminish as the stateful agentic fallback becomes the bottleneck.Increasing K similarly improves speedup while degrading accuracy; K=64 is used as a balanced default and yields 1.94× on Relative Position.
- Ablation Study: A larger Qwen3-VL-8B speculative model improves selected accuracies but reduces speedup to 1.67× versus 1.73× for Qwen3-VL-2B.Qwen2.5-VL-7B likewise lowers speedup to 1.18× with DeepEyes and 1.04× with Thyme, making speculative-model cost a bottleneck.
5 Conclusion and Future Work
SpecEyes lifts speculative acceleration from token-level generation to the entire agentic pipeline by using a tool-free model, separability-based gating, and heterogeneous parallel serving. It reduces latency while maintaining comparable accuracy, but its current depth-zero speculation limits gains on tool-intensive benchmarks.
- Conclusion: SpecEyes speculatively answers tool-free queries with a lightweight model, gates them using answer separability, and forwards uncertain cases to the agentic pipeline.Its heterogeneous parallel funnel converts per-query latency savings into system-level throughput gains.
- Conclusion: SpecEyes reduces end-to-end latency by up to 3.35× while remaining comparable with the agentic baseline in accuracy and improving concurrent-serving throughput.The result holds across V*, HR-Bench, and POPE.
- Future Work: Current depth-zero speculation limits speedups on benchmarks such as HR-Bench, where most queries genuinely require tool assistance.The paper proposes bounded multi-depth speculation as future work.