Source-linked AI summary
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence
TL;DR
Foundation models trained on different domains store complementary knowledge, but adapting them to new multimodal tasks often relies on costly finetuning or task-specific data. Socratic Models compose pretrained models through multimodal-informed language prompting without training, achieving strong zero-shot results while supporting broader multimodal applications.
Problem
Foundation models have domain-specific capabilities, while adapting them to new multimodal tasks can require expensive finetuning, additional data collection, and access to stale or limited information.
Method
Socratic Models compose multiple pretrained models and external APIs through structured language-mediated prompting to perform downstream multimodal tasks without training.
Results
SMs achieve strong zero-shot performance across image captioning, contextual image description, and video-to-text retrieval, including new zero-shot state-of-the-art results.
Takeaways & Limitations
Language-based composition can reuse heterogeneous pretrained models to capture new multimodal functionalities without additional domain-specific training data.
Takeaways & Limitations
Caption detail is limited by the visual-language model, particularly for attributes, facial expressions, and spatial relationships.
Abstract
from arXiv · showhide
Large pretrained (e.g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with external APIs and databases (e.g., web search), and (iii) robot perception and planning.
1 Introduction
Foundation models trained on different domains develop complementary commonsense capabilities, and Socratic Models compose them through language to support new multimodal tasks without training. The paper introduces this framework and demonstrates strong performance and new applications.
- Visual-language models ground text in visual content, while language models support broader linguistic tasks from text-only data.
- Socratic Models compose heterogeneous pretrained models through structured language-based dialogue to generate joint predictions for multimodal tasks.
- The framework uses multimodal prompting and language-based world-state history for video understanding without requiring training.
- The paper demonstrates example systems for image captioning, video-to-text retrieval, egocentric perception, multimodal dialogue, and robot perception and planning.
2 Problem Setting, Background, and Related Work
The paper frames multimodal applications as transfer from pretrained surrogate-task knowledge to new target tasks, contrasting weight-based and joint-training paradigms with prompting-based composition. Existing approaches commonly rely on task-specific architectures, datasets, or finetuning, motivating alternatives that reuse models through language.
- Problem Setting: The problem setting applies knowledge from surrogate tasks such as text completion and image-text similarity to multimodal target tasks such as captioning and robot planning.
- Background: Pretraining-weight transfer initializes target-task parameters with pretrained weights that are either frozen or finetuned.
- Related Work: Multimodal learning commonly trains task-specific maps on large multimodal datasets, combining pretrained parameters with joint training.
- Related Work: Prior multimodal systems combine pretrained models for captioning, video understanding, visual question answering, and speech-text modeling, often with task-specific finetuning.
- Related Work: Prompting enables pretrained language models to perform new tasks without training, while chain-of-thought prompting can substantially affect target-task performance.
3 Socratic Models
Socratic Models compose pretrained multimodal models and external APIs through language-mediated prompting, allowing modules to exchange information in a task-specific computation graph without training. The framework targets limitations of finetuning, including cost, possible loss of robustness, stale knowledge, and limited access to dynamic or proprietary information.
- Socratic Models: An SM is a task-specific computation graph whose modules are pretrained models or external APIs connected by language-based communication.
- Socratic Models: Modules transform outputs into language that connected modules can use for further inference.
- Socratic Models: Multimodal prompting substitutes information from nonlanguage domains into language prompts that a language model can use for reasoning.
- Socratic Models: The prompting pipeline can involve repeated model calls and back-and-forth interactions that form closed-loop feedback between graph nodes.
- Socratic Models: SMs can use simple preprocessing and postprocessing to make models such as CLIP communicate through language without training.
- Motivation: The framework addresses finetuning costs, potential robustness loss, stale knowledge, and limited access to dynamic or proprietary data.
4 Evaluation: Methods and Results
The evaluation applies Socratic Models to image captioning, contextual image description, and video-to-text retrieval through zero-shot multimodal prompting. Across these tasks, SMs improve over zero-shot baselines and can approach or exceed task-specific methods under supported conditions, while visual detail remains limited by the VLM.
- Image captioning: SMs generate image captions by substituting VLM-detected scene information into LM prompts, sampling candidate captions, and ranking them with the VLM.The method uses place, object, image-type, and people-count predictions before LM generation and VLM-based selection.
- Image captioning: 18.0 →44.5 CIDEr improves over zero-shot ZeroCap, while three-caption few-shot prompting reaches 76.3 and exceeds MAGIC.SMs remain below directly finetuned ClipCap in the zero-shot comparison; the evaluation uses a random subset of 100 MS COCO test images.
- Limitations: Visual detail in captions is limited by the VLM’s capabilities, especially for attributes and spatial relationships.The paper suggests open-vocabulary detectors or combining multiple captioning models as future directions.
- Contextual image description: 11.3 →38.9 CIDEr for captions and 17.4 →22.6 for descriptions outperform Kreiss et al. on Concadia despite zero-shot evaluation.The evaluation covers 9,691 test images, while conditioning on image, article text, and ground-truth description reaches CIDEr 93.8.
- Video-to-text retrieval: SMs augment CLIP video retrieval with speech-to-text transcripts, LM-generated summaries, and RoBERTa similarity reweighting.When audio is absent or transcripts are too short, the system defaults to the CLIP-based baseline.
- Video-to-text retrieval: 40.3 →44.7 R@1 establishes a new zero-shot state of the art on MSR-VTT, with the improvement attributed to language-based multimodal reasoning.On videos with long transcripts, performance rises from 40.3 to 54.9, roughly matching the finetuned CLIP2Video result of 54.6 R@1.
5 Applications: Methods and Demonstrations
Socratic Models combine pretrained models through language to support egocentric video understanding, multimodal assistance, and robot perception and planning without task-specific training.
- SM applications integrate external modules, such as web search and robot policies, as participants in structured multimodal interactions.
- Egocentric Perception: Egocentric video is converted into a language-based world-state history by captioning key moments, enabling summarization, free-form questions, and event forecasting.The approach reframes video Q&A as short-story reading comprehension over extracted and captioned moments.
- Egocentric Perception: Without additional context, event-log completions are likely biased toward typical schedules represented in the language model’s Internet-scale data.
- Multimodal Assistive Dialogue: Multimodal assistive dialogue guides users through recipes, supports ingredient substitutions, and retrieves visual references through web search and video search.Prompted key phrases such as “(image)” initiate interactions that retrieve recipe videos and related frames.
- Robot Perception and Planning: Robot systems use visual object descriptions, language-model planning, and language-conditioned policies to execute free-form human instructions.Plans can be represented in natural language or pseudocode and extend tasks beyond the policy’s originally trained primitives.
6 Discussion
The discussion presents Socratic Models as a modular, zero-shot way to combine pretrained models for new multimodal capabilities while highlighting practical benefits and risks.
- Socratic Models use structured language dialogue between pretrained models to make joint predictions for new multimodal tasks without additional data collection or finetuning.
- The framework is demonstrated across image captioning, video-to-text retrieval, egocentric perception, multimodal dialogue, and robot perception and planning.
- Off-the-shelf models can provide an interpretable language-based view of system behavior and support new functionalities with minimal compute resources.
- The systems may reflect unwanted biases from Internet-scale training data, and their flexible unintended uses should be monitored and checked for correctness.
B Unsupervised Socratic Model Selection
Socratic Model selection can be evaluated without annotated ground truth by measuring how well candidate models compensate for weaknesses in complementary models.
- Complementary models enable an evaluation question focused on how well one model compensates for another’s weakness rather than on absolute performance alone.
- The unsupervised approach compares relative distances between outputs from weak and strong model combinations using semantic-distance measures.
- A new VLM can be evaluated by comparing its language-based world-state predictions with pseudo-ground-truth predictions from a baseline VLM paired with a strong LM.
- Table 5 reports higher-is-better unsupervised VLM evaluation using weak and strong LMs, with results suggesting that better ImageNet VLMs improve Socratic synergies.
- Sentence-embedding similarity from a distinct language model scores predictions, avoiding spurious correlations with the models under evaluation.
C Additional Notes on Experiments
The experiments prioritize publicly available models and use a reduced image-captioning evaluation subset to improve reproducibility and control API costs.
- Main experiments use publicly available models, including CLIP, ViLD, Wav2CLIP, and Google Cloud Speech-to-Text.
- MS COCO image-captioning experiments evaluate a random subset of 100 test images to reduce GPT-3 API runtime costs.The reported approximate cost is $150 USD per run with n = 20 generated candidate captions per image.
C.2 Contextual Image Captioning on Concadia
On Concadia, Socratic Models combine VLM and LM information for zero-shot contextual image captioning and description. LM-only prompting is surprisingly competitive, with opposite effects across the two tasks.
- SMs combine VLM information with LM prompting for zero-shot image captioning and description on Concadia.The evaluation uses the Concadia dataset and compares the SM combination with an LM-only variant.
- 2.0 CIDEr points lower image-description performance occurs when VLM information is removed.
- 1.2 CIDEr points higher captioning performance occurs when VLM information is removed.
- The results suggest caption and article-text distributions may overlap strongly enough for an LM alone to perform competitively.The authors note that overlap with the LM’s training data could also confound this interpretation.
C.3 Video-to-text Retrieval on MSR-VTT 1k-A
On MSR-VTT 1k-A video-to-text retrieval, the paper evaluates zero-shot and finetuned methods under the standard protocol with 20 captions per video. Correcting an earlier one-caption baseline raises its R@1 from 27.2 to 58.0, while SMs reach 60.7 R@1 zero-shot.
- The evaluation compares finetuned and zero-shot video-to-text retrieval methods on MSR-VTT’s 1k-A subset.Table 8 also distinguishes audio use and the CLIP encoder used.
- 27.2 R@1 was the original CLIP baseline under an evaluation using only one randomly sampled caption per video.
- 58.0 R@1 is obtained for the corrected CLIP baseline when each video is associated with 20 captions.
- 60.7 R@1 is achieved by SMs with LMs and ALMs zero-shot under the corrected protocol.
- Egocentric perception motivates broader SM applications because first-person videos differ from generic third-person Internet footage and remain difficult for standard data-driven models.
D.2 Additional Details on Language-Based World-State History from Video
The system converts egocentric video into a language-based world-state history by summarizing key frames through guided exchanges among perceptual models and language models. This record supports open-ended video reasoning, while denoising and audio integration introduce useful capabilities and notable limitations.
- Key video moments are captioned and recursively summarized into a timestamped language-based event log used as LM context for open-ended reasoning.
- Socratic image summaries answer where-am-I, what-do-I-see, and what-am-I-doing questions before producing one caption per frame.
- VLMs rank places and objects, LMs generate candidate activities from them, and VLMs re-rank those activities for the frame.The iterative process can continue by generating new entities conditioned on the current places, objects, and activities.
- The summaries can denoise perceptual outputs and produce contextually specific answers, such as inferring enjoyment of wine with a friend from scene descriptions.
- LM denoising may ignore notable rare events, motivating methods that index such events appropriately for downstream applications.
- Audio-informed summaries can improve a case involving footsteps, but Wav2CLIP is less robust than CLIP across varied egocentric background noise.
- Search-based history can omit events causally related to the queried event, such as campfire-making after wood-chopping.
D.3 Open-Ended Reasoning on Egocentric Video
Socratic Models use language-based world-state histories and multimodal prompting to support open-ended reasoning over egocentric video. Demonstrations cover recall, temporal, causal, subjective, forecasting, summarization, and multimodal video-search tasks, while performance depends strongly on the quality and relevance of the reconstructed event log.
- System and task scope: SMs condition open-ended text prompts on egocentric video through a language-based world-state history generated by multi-model discussion.Responses may include text, visual results, and audio clips.
- System and task scope: The demonstrated system unifies summarization, open-ended Q&A, forecasting, corrections, and visual or audio video search under language-based multimodal reasoning.These tasks have predominantly been studied in isolation.
- Outputs and limitations: The approach generates summaries from visual content, supports meaningful binary and contextual answers, and can return images or audio selected through language prompting.Answer accuracy remains conditioned on whether necessary information is present in the world-state history.
- Recall and temporal reasoning: SMs answer recall questions with positive or negative natural-language responses and explanations, extending beyond nearest-neighbor video-frame retrieval.Examples include identifying dinner, outdoor activity, or the absence of driving.
- Recall and temporal reasoning: Timestamped world-state histories support temporal retrieval and event counting, such as identifying the last coffee time or counting packages received.Each key moment is associated with a time of day.
- Causal and subjective reasoning: Causal answers require both cause and effect to appear in the world-state history, making results highly dependent on key-frame sampling.Uniform sampling is currently the best variant for this reasoning, while search-based sampling can miss relevant frames.
- Causal and subjective reasoning: Subjective questions may rely on language-model dataset biases without additional context, creating potential negative consequences that require safety and groundedness mechanisms.Full personalization depends on constructing better context from prior user behavior.
- Outputs and limitations: The Q&A bottleneck is the event log’s recall and precision, while audio retrieval remains limited by Wav2CLIP robustness and open-language coverage.The authors identify better detectors, captioning systems, and audio-language datasets as possible remedies.
F Additional Notes on Robot Experiments
The robot Socratic Model combines visual object descriptions, language planning, and a language-conditioned robot policy to execute multi-step tasks zero-shot. Demonstrations show sensitivity to task order, nuanced language, and changing detected objects.
- Robot system: The robot system uses a VLM to describe scene objects, a language model to generate multi-step plans, and a pretrained language-conditioned robot policy to execute them.The policy is CLIP-conditioned.
- Demonstrated capabilities: Zero-shot SMs perform multi-step robot tasks out-of-the-box, including rearrangements and correcting an earlier placement.The example moves colored blocks to a target, places one in a bowl, then undoes that placement.
- Demonstrated capabilities: The system reasons over action order and nuanced language such as clockwise versus counterclockwise, while adapting to different detected block colors.The robot receives few-shot prompts to generate pseudo-code actions.
G Socratic Deductive Reasoning
Socratic deductive reasoning treats multimodal observations as a language-based world-state history that a language model can connect into explanations. The examples illustrate multi-step inference, while the paper emphasizes better history construction and recursive reasoning as open directions.
- Deductive reasoning: SMs formulate video Q&A as reading comprehension, allowing a language model to connect commonsense relationships with knowledge learned from Internet-scale data.The system operates on a language-based world-state history.
- Deductive reasoning: The dog-sickness example links the dog’s kitchen visit, groceries left in the kitchen, and chocolate among the groceries to generate a possible explanation.The answer remains a hypothesis rather than a verified observation.
- Future directions: These results suggest research directions in assembling richer world-state histories and using chain-of-thought prompting to decompose multi-step problems.The proposed extension explores hypotheses through recursively higher-resolution downstream subprograms.
- Broader implication: Composing capable models with minimal training resources may help retarget large pretrained models to multimodal applications without considerable additional compute.This implication assumes that large foundational pretrained models are already available.