Source-linked AI summary
Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs
Sandeep Mishra, Devichand Budagam, Anubhab Mandal, Bishal Santra, Pawan Goyal, Manish Gupta
TL;DR
Inline completion in multimodal conversations needs to account for visual context as well as partially typed text. The paper defines MAC, adapts two dialog datasets into benchmarks, and introduces Router-Suggest to select models dynamically. Textual models and VLMs show different strengths, while Router-Suggest reports a 2.3-10× speedup over the best models and user-study benefits for VLM completions.
Problem
Inline auto-completion for multimodal dialog is absent, although images can shape conversational intent and real-time predictions face latency-efficiency trade-offs.
Source: §1 · p. 2Method
The paper defines MAC over partially typed text and multimodal dialog history, adapts MMDialog and ImageChat into benchmarks, and proposes a router that selects among models.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
Textual models excel with known prefixes, while VLMs maintain high trigger rates and better TES and robustness in new conditions; Router-Suggest achieves 2.3-10× speedup with competitive partial-F1.
Source: §6 · p. 7Takeaways & Limitations
The user study finds that TES aligns with subjective satisfaction ratings and that VLM completions better meet user expectations than textual-model outputs.
Source: §6 · p. 7The datasets cover only single-image contexts, limiting generalization to real-world multimodal settings with evolving or multiple visuals.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Real-time multimodal auto-completion is essential for digital assistants, chatbots, design tools, and healthcare consultations, where user inputs rely on shared visual context. We introduce Multimodal Auto-Completion (MAC), a task that predicts upcoming characters in live chats using partially typed text and visual cues. Unlike traditional text-only auto-completion (TAC), MAC grounds predictions in multimodal context to better capture user intent. To enable this task, we adapt MMDialog and ImageChat to create benchmark datasets. We evaluate leading vision-language models (VLMs) against strong textual baselines, highlighting trade-offs in accuracy and efficiency. We present Router-Suggest, a router framework that dynamically selects between textual models and VLMs based on dialog context, along with a lightweight variant for resource-constrained environments. Router-Suggest achieves a 2.3x to 10x speedup over the best-performing VLM. A user study shows that VLMs significantly excel over textual models on user satisfaction, notably saving user typing effort and improving the quality of completions in multi-turn conversations. These findings underscore the need for multimodal context in auto-completions, leading to smarter, user-aware assistants.
1 Introduction
Multimodal intent makes inline completion challenging: existing text-only completion is underdeveloped for multi-turn dialog, while visual context can affect what a partial prefix means. The paper defines Multimodal Auto-Completion, builds benchmarks, evaluates models, and proposes dynamic routing.
1 Introduction
- MAC predicts inline user input from partially typed text and multimodal dialog history, addressing the absence of inline completion for multimodal dialogues.The task must handle partial-input disambiguation, align predictions with visually salient cues, and balance latency against efficiency.Source: §1 · p. 2
- The authors adapt MMDialog and ImageChat into standardized MAC benchmarks, filtering for strong visual relevance.Source: §1 · p. 2
- The evaluation compares recent vision-language models with textual baselines on MAC, examining multimodal understanding and completion-quality trade-offs.Source: §1 · p. 2
- Router-Suggest chooses at each character between a lightweight textual model and a more expressive VLM based on the dialog context’s visual significance.Source: §1 · p. 2
- The user study reports substantial gains over text-only methods when evaluating Typing Effort Saved and user satisfaction.The study quantifies both measures and the reported results show gains over text-only methods.Source: §1 · p. 2
2 Related work
Prior work spans query suggestions and inline text completion, but dialog research has focused more on next-utterance prediction than mid-turn input. Multimodal inline completion remains underexplored despite images influencing conversational intent.
2 Related work
- QAC systems use popularity rankings, spatial and temporal patterns, and session co-occurrence, with methods ranging from classical models to neural approaches.Source: §2 · p. 2
- TAC presents one continuation inline rather than ranked suggestions, a design suited to conversational flow; methods include subword language models [Kim, 2019], GPT-2 fine-tuning [Lee et al., 2021], and reinforcement learning [Chitnis et al., 2024; Li et al., 2024].Source: §2 · p. 2
- Dialog research largely emphasizes next-utterance prediction, leaving multimodal mid-turn completion underexplored even though images can influence intent.The paper positions MAC as connecting full-turn response generation with real-time typing assistance.Source: §2 · p. 2
3 Methods for MAC
The methods define multimodal auto-completion, construct visually relevant benchmarks, compare textual and vision-language models, and introduce adaptive routing to balance completion quality and latency.
3.1 The MAC Task Definition
- MAC predicts a contextually appropriate continuation from a partial text prefix and multimodal dialog history.The generated continuation is textual and should form a fluent, contextually coherent message when joined to the prefix.Source: §3.1 · p. 3
3.2 Benchmark Construction for MAC Evaluation
- The benchmarks adapt MMDialog and ImageChat, filtering for dialogs where images are essential to predicting the user's next input.GPT-4V performs relevance filtering, with single-image conversations used to assess visual relevance.Source: §3.2 · p. 3
3.3 Models for the MAC Task
- The model suite contrasts textual trie- and n-gram-based approaches with vision-language models on the MAC task.The textual approaches include MPC, MPC++, and QueryBlazer; the VLMs include MiniCPM-V, PaliGemma, and Qwen2-VL.Source: §3.3 · p. 3
3.4 The Proposed Router-Suggest Framework
- Router-Suggest uses a lightweight classifier to select a model for each prefix, with training that incorporates model quality and latency costs.Prefix representations support router training, and partial-F1 selects the ground-truth model; the combined objective balances correct classification against expected cost.Source: §3.4 · pp. 3–4
4 Experiments and Results
The experiments assess inline completion with MAC-specific metrics, benchmark text models and VLMs, and evaluate whether routing can balance completion quality with latency. VLMs outperform textual baselines on unseen prefixes, while Router-Suggest offers competitive performance with lower latency.
4.1 Evaluation Metrics
- MAC-specific metrics assess completion accuracy, usability, and syntactic fluency because ranked-list and sequence-overlap metrics do not fit inline continuation.The metrics include Partial Precision, Partial Recall, Partial F1, Trigger Rate, Typing Effort Saved, and Syntactic Match.Source: §4.1 · pp. 4–5
4.2 Finetuning Setup
- The dataset preparation progressively unrolls dialog context and splits the final utterance into a prefix and target completion; VLMs were fine-tuned with LoRA while text models followed QB-style settings.VLM training used five epochs, batch size 8 per device, and a learning rate of 0.0001; the base model was frozen while LoRA parameters were updated.Source: §4.2 · pp. 5–6
4.3 Performance on MAC Benchmarks
- On unseen prefixes, VLMs outperform text models across both datasets; MiniCPM-V achieved the best overall TES (0.2136), while ImageChat showed a narrower performance gap.VLMs maintained Trigger Rates around 0.99 and stable PR metrics; text models collapsed on MMDD, including MPC’s SM of 0 and TES of 0.0015.Source: §4.3 · p. 6
4.4 Evaluation of Router-Suggest
- Router-4-L matched MiniCPM-V’s PR-F1 on MMDD at 5× faster response time, while Router-2-L achieved 10× speedup over MiniCPM-V and neared Qwen2-VL’s PR-F1.Router-4-P reached PR-F1 0.281 against a 0.356 upper bound at one-third of MiniCPM-V’s latency.Source: Table 3 · p. 6, §4.4 · p. 6
5 User Study
The user study compared completion systems in 190 sessions and found that MiniCPM-V achieved the highest TES and user rating among the tested models. TES and user ratings rose together, though interactive suggestions can influence the text used to calculate TES.
5 User Study
- Across models, higher TES was associated with higher user ratings.Users rated completions from 0 to 9 after interacting with a randomly selected model without knowing its identity.Source: §5 · p. 7
- Interactive TES may be inflated because users adapt their typed continuations to the system’s suggestions.The study calculated TES against the final user query at the time the rating was submitted.Source: §5 · p. 7
- MiniCPM-V achieved the highest TES and unnormalized user rating, while Router-2-L also achieved similar scores.The study included 190 sessions across MPC, QB, MiniCPM-V, and Router-2-L.Source: §5 · p. 7
6 Conclusion
The paper introduces Multimodal Auto-Completion (MAC) and benchmarks for inline predictions grounded in visually grounded conversations. Experiments report advantages for VLMs in new conditions and a router that reduces latency while retaining competitive partial-F1.
6 Conclusion
- Router-Suggest provides competitive partial-F1 with a 2.3–10× speedup over the best models, and the paper also provides a low-resource setup.The user study reports that VLM completions better meet user expectations than textual-model outputs.Source: §6 · p. 7
- Textual models excel with known prefixes but struggle with new ones, while VLMs maintain higher trigger rates, better TES, and robustness in new conditions.The user study finds TES aligns with subjective satisfaction ratings.Source: §6 · p. 7
- MAC is introduced as a task for predicting user input in visually grounded conversations, with standardized benchmarks from MMDialog and ImageChat and an inline-autocompletion evaluation protocol.Source: §6 · p. 7
7 Limitations
The benchmark construction and routing method have limitations in scope and robustness. The benchmark uses single-image contexts, and the router relies on embedding-based heuristics.
7 Limitations
- The benchmarks may favor visually explicit cases and lack linguistic diversity.The datasets cover only single-image contexts, limiting generalization to settings with evolving or multiple visuals.Source: §7 · pp. 7–8
- The benchmark construction may select toward visually explicit cases, constraining the range of linguistic examples represented.Source: §7 · p. 7
- Router-Suggest's embedding-based heuristics may degrade under domain shift, and its routing choices lack interpretability.Source: §7 · p. 8
8 Ethical Considerations
The ethical discussion identifies potential risks in benchmark construction, user-study measurement, and router allocation. The related-work passage surveys multimodal dialog systems that generate responses using visual and textual history.
8 Ethical Considerations
- GPT-4V filtering and curated public corpora may introduce noisy labels, annotation biases, privacy concerns, and hallucination risks.Source: §8 · p. 8
- A small user pool and reliance primarily on TES may miss subtle misinformation or cultural and demographic mismatches, while sampling choices may limit generalizability.Source: §8 · p. 8
- Router invocation patterns may distribute latency, computational cost, or quality of experience unequally across input types or user groups.Source: §8 · p. 8
A Additional Related Work
- Related work includes systems for text, image, or mixed-modal response generation and approaches that use visual history or cross-modal attention for grounded responses.The cited examples include DIVTER, TIGER, BI-MDRG, MAGIC, and MATE.Source: A Additional Related Work · p. 10
B Benchmark Construction
The benchmarks are filtered for image-dialog relevance and formatted to preserve where visual content occurs in the dialog. The filtering retains only strongly relevant pairs, while the formatting inserts image features and a turn-level marker.
B.1 Relevance filtering using GPT-4V
- GPT-4V scores each image-dialog pair on a standardized five-point relevance scale, and Figure 5 shows the prompt template.The scale ranges from contradictory to critical for understanding.Source: §B.1 · p. 10, Figure 5 · p. 10
- Only samples scored 4 or 5 are retained, and the filtering removes over 66% of samples.Figure 6 illustrates examples judged highly image-relevant.Source: §B.1 · p. 10, Figure 6 · p. 11
B.2 Formatting interleaved inputs
- For models without native interleaved image-text support, image embeddings are prepended and an <IMAGE> token marks the image's dialog turn.This formatting enables attention to image features and their temporal alignment within the dialog.Source: §B.2 · pp. 10–11
C Additional Details for Experiments
The baseline suite includes text-only completion methods and vision-language models that process both text and images.
C.1 Baseline Models
- MPC builds a character-level trie from historical utterances, while MPC++ uses a suffix trie to improve coverage for unseen prefixes.Both are textual models and suggest completions based on historical user utterances.Source: §C.1 · p. 11
- QueryBlazer uses subword tokenization and n-gram language modeling to retrieve completions from historical logs and synthesize new predictions.Source: §C.1 · p. 11
- The evaluated VLMs combine visual encoders with language models to process images and text: MiniCPM-V, PaliGemma, and Qwen2-VL.Their described components include SigLIP encoders for MiniCPM-V and PaliGemma, and a ViT encoder for Qwen2-VL.Source: §C.1 · p. 11
C.2 Performance of MAC Benchmarks on Seen prefixes
Seen-prefix results show different patterns across MMDD and ImageChat, while Router-Suggest’s additional details describe its tuning search and accuracy–latency objective.
C.3 Additional Details of Router-Suggest
- Router-Suggest tuning used 50 random trials for each of five λ settings, totaling 250 experiments per dataset, with batch size 256 and dropout 0.2.The architecture and hyperparameter search combined network topology and training parameters, and each λ setting was tuned independently.Source: §C.3 · p. 11, Table 5 · p. 12
C.2 Performance of MAC Benchmarks on Seen prefixes
- Textual models dominate seen MMDD prefixes through memorization (MPC and MPC++: SM = 0.79, F1 = 0.81, TES = 0.72); ImageChat VLMs match or slightly surpass textual models at Partial-F1 ≈0.48.On MMDD, VLMs have F1 ≈0.27–0.30 and TR ≈0.99; the paper characterizes them as having greater contextual grounding.Source: §C.2 · p. 11, Table 4 · p. 12
C.3 Additional Details of Router-Suggest
- The tuning objective balanced accuracy and latency using a weighted score, with costs normalized by the maximum observed latency.The maximum cost was 2.0891 for MiniCPM-V; the formulation supported comparison across trade-off settings and selection of the highest-scoring model overall.Source: §C.3 · p. 12