Source-linked AI summary
Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making
Amirhosein Ghasemabadi, Ruichen Chen, Bahador Rashidi, Di Niu
TL;DR
Reliable agents need to decide whether to continue, defer, seek information, use tools, or abstain, but existing approaches often rely on costly input-side mechanisms. Multi-Head Latent Control attaches lightweight heads to frozen models’ hidden-state traces to predict capability and resolution decisions, improving deployment behavior and quality–cost tradeoffs across evaluated settings.
Problem
Reliable agentic deployment requires deciding whether to continue reasoning, defer, seek information, invoke tools, or abstain beyond next-token prediction.
Method
Multi-Head Latent Control attaches Capability and Resolution Heads to frozen language or vision-language models, reading hidden-state trajectories to emit deployment-time control signals.
Results
Across collaboration, long-horizon execution, resolution decisions, tool use, and prefix-time prediction, the signals improve deployment behavior while preserving frozen-model efficiency and reusability.
Takeaways & Limitations
Hidden-state traces can provide a scalable substrate for lightweight control interfaces rapidly attached to new foundation models during deployment.
Takeaways & Limitations
Control-signal quality directly affects system efficiency, motivating further improvements in their quality, robustness, and calibration.
Abstract
from arXiv · showhide
Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.
1 Introduction
Reliable agent deployment requires latent decisions about capability, information needs, tool use, and abstention—not only next-token generation. Multi-Head Latent Control addresses this with lightweight heads over hidden-state trajectories from frozen models, enabling deployment-time routing and intervention without backbone modification.
- Motivation: Agentic LLMs must assess whether they can solve a task, defer to a stronger model, request information, invoke tools, or abstain.These decisions arise in long-horizon, multi-step, and tool-augmented settings where generation alone is insufficient.
- Motivation: High token consumption in commercial agentic systems motivates self-awareness that can reduce cost by assessing capability during inference.The introduction identifies inefficiency even on relatively simple tasks as evidence of potential savings.
- Prior Work: Existing routing and orchestration methods mainly use external mechanisms or input-side signals, limiting instance-level assessment of model adequacy.Decoding-side efficiency methods optimize generation after invocation, whereas this work targets how often stronger models are invoked.
- Method: Multi-Head Latent Control attaches Capability and Resolution Heads to frozen language or vision-language models, reading hidden-state trajectories to emit actionable control signals without fine-tuning.The interface supports both model selection and intervention decisions during inference.
- Method: The Capability Head predicts whether the current model is adequate, enabling instance-level retention or escalation to a stronger model.This forms the model-selection level of the proposed two-level self-awareness mechanism.
2 Related Work
Prior work addresses routing, collaboration, tool intervention, decoding efficiency, and latent reliability through separate mechanisms. This paper instead proposes a unified, lightweight latent control interface for deployment-time decisions from a frozen LLM or VLM.
- Model routers and adaptive selection: Model routers and cascades use input-level or pre-generation signals for coarse cost-quality tradeoffs rather than assessing whether the current model fits a specific instance.The paper reads control signals directly from the model’s own generation process.
- Multi-model collaboration and agentic systems: Multi-model collaboration coordinates models, tools, or roles, but typically depends on external orchestration, fixed assignments, or dedicated training.Examples include multi-agent debate, MetaGPT, mixture-style systems, and trained orchestration models.
- Tool use and intervention decisions: Tool-use research improves invocation behavior or determines whether tools are necessary, while speculative decoding accelerates inference without reducing large-model invocation frequency.These lines address tool-calling decisions and decoding speed as separate objectives from the paper’s control setting.
- Reliability based on latent signals: Latent-reliability methods estimate correctness or hallucination from internal signals, whereas this work uses latent signals for unified next-action control during agentic inference.The proposed interface equips a frozen LLM or VLM with lightweight deployment-time decisions beyond routing, tool use, decoding acceleration, or scalar self-verification.
3 The Multi-Head Latent Control Mechanism
Multi-Head Latent Control treats deployment-time control as latent decision-making over hidden-state trajectories from a frozen backbone. Separate lightweight heads infer model adequacy and within-model intervention needs to route or act at inference time.
- Latent representations: The Capability Head reads the final-layer trace, while the Resolution Head reads a selected middle-layer trace, allowing each decision to use its most separable depth.Both variable-length traces are compressed into fixed-budget representations before decoding.
- Control outputs: The Capability Head outputs pcap ∈ [0, 1] for retaining control with m1 or transferring it to stronger model m2.The Resolution Head outputs sres = [sinfo, stool, scant] for within-model interventions, and only the lightweight heads are trained.
- Inference-time execution: At inference, the system hands off when pcap < τcap; otherwise, the Resolution Head selects among clarification, tool use, or abstention interventions.Direct answering is represented implicitly when no explicit resolution score exceeds the intervention threshold.
- Capability training: The Capability Head learns a scalar adequacy signal from judged comparisons between frozen-backbone outputs and reference answers.Weighted mean squared error is the default regression loss, with example weights compensating for imbalanced adequacy labels.
- Resolution training: The Resolution Head is trained on WHEN2CALL to recover the appropriate action from latent trajectories rather than relying on the backbone’s potentially incorrect surface behavior.Its action space includes direct answering, tool invocation, additional-information requests, and abstention.
4 Experiments and Results
Experiments across diverse language and vision-language backbones show that latent control improves score–cost tradeoffs, resolution decisions, and selective web-search escalation. The Capability and Resolution Heads provide useful intervention signals even when models’ native behavior is inadequate.
- Experimental scope: Evaluations span Qwen3-VL, Qwen3.5, and Gemma backbones from 2B to 32B, including thinking and non-thinking variants.This tests transfer across model families, scales, and inference modes.
- Capability-guided routing: 90.7% paid API cost reduction accompanies Qwen3-VL-4B →32B score improvement from 0.47 to 0.60.Qwen3.5-9B →27B also improves score from 0.51 to 0.56 with an 85.8% cost reduction.
- Resolution decisions: Qwen-VL-2B Resolution Head augmentation raises F1 from 37.3 to 49.0 and accuracy from 52.7 to 65.1 on WHEN2CALL.On Qwen3.5-4B, F1 rises from 43.5 to 54.5 and accuracy from 57.9 to 69.5.
- Selective web search: Capability-Head augmentation improves TriviaQA score and reduces missed-needed web calls across backbones, including strong large models.The results indicate improved escalation quality rather than blanket suppression of tool use, because some configurations make more web calls.
- Early intervention: Prefix-trained heads better recover full-trajectory routed performance at 200 tokens while preserving substantial paid-cost reductions versus always using the large model.This supports detecting model adequacy early enough for useful routed inference.
5 Conclusion
Multi-head latent control equips frozen foundation models with a lightweight deployment-time interface for deciding what to do next during inference. Small heads read hidden-state traces to predict model adequacy and resolution decisions, improving deployment behavior while preserving frozen-model efficiency and reusability.
- Multi-head latent control adds a lightweight deployment-time control layer to frozen foundation models for inference-time decisions.The backbone remains unmodified.
- Small control heads read hidden-state traces to predict model adequacy and resolution decisions.This provides a practical control interface without modifying the backbone.
- Across multi-model collaboration, long-horizon agentic execution, structured resolution decision-making, tool-use decisions, and prefix-time prediction, the signals improve deployment behavior while preserving frozen-model efficiency and reusability.
A Training Data and Label Construction · A.1 Capability Head Data · A.2 Resolution Head Labels from WHEN2CALL
The appendix constructs training signals from frozen-model latent trajectories: a broad 120K-example mixture supervises capability estimation, while WHEN2CALL examples receive externally judged resolution labels. This enables heads to predict model adequacy and correct resolution decisions without relying on the backbone’s explicit action choice.
- A.1 Capability Head Data: 120K examples train the Capability Head to estimate whether a frozen model is adequate for each instance under the current setup.The design targets generalization across modalities, topics, and task styles rather than a verifier specialized to a narrow regime.
- A.1 Capability Head Data: The Capability mixture spans visual QA, science and diagram reasoning, chart and document understanding, screen understanding, UI grounding, multimodal and text-only reasoning, factual QA, and tool use.Figure 3 characterizes the mixture as covering visual QA, reasoning, parametric knowledge, grounding, tool use, and agentic interaction across vision and text.
- A.1 Capability Head Data: For each prompt, an external LLM evaluator scores the frozen backbone’s generated output against the task reference, producing a scalar target in [0, 1].The recorded supervision pairs the evaluator score with the backbone’s aligned hidden-state trajectory.
- A.1 Capability Head Data: VQAv2, ScienceQA, ChartQA, and DocVQA contribute grounded QA, scientific reasoning, chart reading, and OCR- or layout-sensitive document understanding.These sources broaden the mixture beyond a single visual-question-answering regime.
- A.1 Capability Head Data: ScreenQA, A-OKVQA, AI2D, and InfographicVQA add screen-centered QA, knowledge-intensive visual QA, diagram reasoning, and infographic understanding.The listed sources combine visual interpretation with knowledge, OCR, and layout understanding.
- A.2 Resolution Head Labels from WHEN2CALL: WHEN2CALL evaluates tool-calling decisions but releases preference-oriented data without the exact per-example labels required by the Resolution Head.Its decision scope includes tool calls, follow-up questions, and cases unanswerable under the current tool setup.
- A.2 Resolution Head Labels from WHEN2CALL: An external judge LLM derives gold labels in A = {info, tool, cant}, and the Resolution Head trains on the frozen backbone’s latent trajectory regardless of its explicit action correctness.This construction is intended to recover the correct resolution decision from latent generation traces rather than copy the backbone’s stated choice.
B Additional Experiments … B.3 Model Token Confidence vs. Latent Adequacy Signal
The additional experiments evaluate routed behavior, prompt-level self-switching, token confidence, and layer choices for latent control. They show that latent signals support more effective handoff and adequacy assessment than surface-level alternatives.
- B.1 End-to-End Effect of Prefix-Time Capability Prediction: Table 6 evaluates task score and paid API cost when the Capability Control signal drives actual routing.It extends signal-quality evaluation to end-to-end routed behavior and examines transfer decisions under deployment conditions.
- B.1 End-to-End Effect of Prefix-Time Capability Prediction: Training directly on partial trajectories improves the prefix-time adequacy signal for routing.
- B.2 Prompt-Level Self-Switching as a Baseline: Prompt-level self-switching remains near the small-model baseline, whereas latent-control routing recovers substantially more of the stronger model’s performance.This comparison uses the Qwen3.5-9B →Qwen3.5-27B-Thk setting on ScreenSpot-Pro and MMLU-Pro.
- B.2 Prompt-Level Self-Switching as a Baseline: Prompt-level abstention causes severe under-escalation, while the Capability Head provides a more effective transfer signal for deployment-time control.
- B.3 Model Token Confidence vs. Latent Adequacy Signal: Model token confidence shows substantial overlap between correct and incorrect cases, making it a weak adequacy signal.Figure 4 compares mean token probability over the last 100 response tokens with Capability Head scores on Qwen3.5-9B over ScreenSpot-Pro.
- B.3 Model Token Confidence vs. Latent Adequacy Signal: The final layer is the best overall choice for the Capability Head across average ROC-AUC, AUPR-C, AUPR-I, and ECE.This layer is therefore used by default in the Qwen3-VL-4B-Thk ablation.
- B.3 Model Token Confidence vs. Latent Adequacy Signal: The middle layer provides the strongest resolution-intervention signal on WHEN2CALL and is used by default for the Resolution Head.
C Ablations · C.1 Layer Choice for the Capability Head · C.2 Layer Choice for the Resolution Head
The ablations show that the best latent layer depends on the control task: the final layer is strongest for capability prediction, while a middle layer is strongest for resolution decisions. These findings motivate the layer choices used in the main system.
- C.1 Layer Choice for the Capability Head: For the Capability Head, the ablation compares early Layer 5, middle, and final hidden-state traces using Qwen3-VL-4B-Thk.The same head is trained on traces from each representative layer.
- C.1 Layer Choice for the Capability Head: The evaluation reports average ROC-AUC, AUPR-C, AUPR-I, and ECE across all benchmarks.These metrics assess model adequacy prediction across the evaluated tasks.
- C.1 Layer Choice for the Capability Head: The final-layer trace provides the strongest overall signal for model adequacy prediction.This result motivates using the final-layer trace for the Capability Head.
- C.2 Layer Choice for the Resolution Head: For the Resolution Head, the ablation compares early Layer 5, middle, and final traces using Qwen3-VL-4B on WHEN2CALL.The same Resolution Head is trained on traces from each representative layer.
- C.2 Layer Choice for the Resolution Head: The evaluation reports F1 and accuracy on the validation split.These metrics assess resolution decision quality on WHEN2CALL.
- C.2 Layer Choice for the Resolution Head: The middle-layer trace provides the strongest resolution decision signal.This result motivates the default middle-layer choice used in the main paper.
C.3 Training-Data Breadth
Capability Head training breadth is necessary for transferable adequacy estimation: a full 120K mixed dataset outperforms visual-math-only training when evaluated on the distinct ScreenSpot Pro regime. The result supports heterogeneous training exposure as a deployment requirement rather than a generic benefit of more data.
- Motivation: The Capability Head aims to estimate model adequacy across varied tasks, so its training mixture spans distinct decision regimes.The full mixture contains 120K examples designed to reflect heterogeneous deployment behavior.
- Experimental setup: The breadth ablation changes only the training distribution: one head uses the full 120K mixed dataset, while the other uses visual-math data alone.Both heads share the frozen Qwen3VL-4B backbone, architecture, and optimization setup.
- Transfer test: ScreenSpot Pro tests transfer beyond visual-math by emphasizing visually grounded understanding with grounding- and action-oriented structure.This evaluation distinguishes broad adequacy signals from task-specific confidence heuristics.
- Results: The visual-math-only head transfers poorly to ScreenSpot Pro, showing weaker discrimination and much worse calibration than the full mixed-data head.The full mixed-data head remains substantially stronger and more reliable on this out-of-regime evaluation.
C.4 Prefix-Length Ablation for the Capability Head · D Cost Estimation Details
The sections examine how early capability assessment can be recovered from hidden-state traces and specify how API costs are estimated across model families and inference modes. The ablation varies observed-generation prefix length, while costing uses token counts and family-matched pricing or a proxy when necessary.
- C.4 Prefix-Length Ablation for the Capability Head: The Capability Head ablation tests whether model adequacy is recoverable from hidden-state traces at different stages of generation.The study evaluates how early the capability signal becomes reliably recoverable.
- C.4 Prefix-Length Ablation for the Capability Head: Prefix-based Capability Head variants are trained on Qwen3-VL-4B-Thk using the first K generated tokens.The tested prefix lengths are K ∈{50, 200, 1000, full}.
- C.4 Prefix-Length Ablation for the Capability Head: Each prefix-length variant is evaluated on the same benchmark suite.This keeps the evaluation setting consistent across observed-generation lengths.
- C.4 Prefix-Length Ablation for the Capability Head: Performance is summarized using average ROC-AUC, AUPR-C, AUPR-I, and ECE across benchmarks.These metrics quantify capability-signal quality across the benchmark suite.
- D Cost Estimation Details: API cost estimation uses the observed input and output token counts from each model call.The calculation uses corresponding per-1M-token input and output pricing.
- D Cost Estimation Details: Qwen3-VL and Qwen3.5 costs use official Alibaba pricing matched by model scale and thinking/non-thinking mode.Pricing is selected for the corresponding model family and inference mode.
- D Cost Estimation Details: Gemma costs use Qwen3-VL pricing as a proxy because comparable public API pricing is unavailable for the experimental models.The proxy is also matched by scale and inference mode.
E Limitations · F Broader View of Latent Control
The paper identifies control-signal quality, robustness, and calibration as key limitations, especially because small improvements can compound in long-horizon settings. It also frames hidden-state trajectories as a reusable substrate for broader lightweight, trainable inference-time control beyond the two studied heads.
- E Limitations: Control-head quality directly affects overall system efficiency at deployment time.The paper treats this dependence as a central limitation of the current approach.
- E Limitations: In long-horizon settings, modest improvements in adequacy or intervention prediction can compound across steps.Such improvements can produce better routing decisions, fewer unnecessary escalations, and stronger quality–cost tradeoffs.
- E Limitations: Future work should improve the quality, robustness, and calibration of latent control signals.
- F Broader View of Latent Control: The two studied heads are concrete instances, not an exhaustive set of latent-control interfaces.The broader claim is that hidden-state trajectories provide a reusable substrate for lightweight inference-time control.
- F Broader View of Latent Control: Hidden-state trajectories are proposed as a basis for control, extending beyond prior diagnostic uses such as hallucination or correctness estimation.
- F Broader View of Latent Control: Additional heads could support escalation, clarification, tool use, abstention, confidence shaping, safety filtering, and application-specific routing.This direction targets different deployment needs, user preferences, and application settings.
- F Broader View of Latent Control: Multi-head latent control points toward frozen models exposing lightweight, trainable control interfaces for real-