PaperlayerView exact revision on arXiv

Source-linked AI summary

From Images to Tasks: Characterizing Multimodal LLM Interactions in the Wild

Jinyi Ye, Scott Counts, Gaurav Verma, Kate Lytvynets, Weiwei Yang

arXiv:2610.00701v1cs.HC

TL;DR

Evidence about how people use multimodal inputs in natural settings, and how that use relates to text-only tasks and existing benchmark coverage, remains limited. The study analyzes image-upload conversations from Copilot and ChatGPT to characterize capabilities, task spaces, and benchmarks. It finds that image-upload tasks are often compositional, multimodal task space is broader than text-only task space, and common generation workflows are comparatively under-tested.

  • Problem

    Large-scale evidence about what users do with multimodal inputs in natural settings remains limited, including how multimodal tasks relate to text-only use and whether benchmarks cover capabilities users invoke.

    Source: §1 · p. 1

  • Method

    The study analyzes image-upload conversations from Copilot and ChatGPT, deriving a capability taxonomy and comparing multimodal and text-only task spaces and benchmark coverage.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Image-upload tasks are predominantly compositional, multimodal interactions occupy a broader task space than text-only ones, and common writing-, code-, and data-generation workflows are evaluated only incidentally.

    Source: §6 · pp. 10–11

  • Takeaways & Limitations

    The findings provide an empirical basis for benchmarks and interfaces that reflect observed user behavior, including compositional generation workflows and users’ offloading objectives.

    Source: §6 · p. 11

    The eyes-off analysis uses privacy-preserving summaries that may omit fine-grained visual details, and findings may not fully generalize to future systems as capabilities and interfaces change.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multimodal large language models (LLMs) increasingly integrate vision and text, yet how people use them in natural settings remains underexplored. We seek to answer the question: when users upload images, what tasks are they trying to accomplish? Analyzing over 40,000 de-identified image-upload conversations from Microsoft Copilot, we characterize real-world multimodal use through a hierarchical framework of ten capabilities, spanning perception, cognition, and generation. First, we characterize the distribution and composition of these capabilities, finding that the majority of image-upload tasks involve multiple capabilities. Second, we find that multimodal use spans a broader and more diverse task space than text-only interactions, with asymmetric coverage and task classes that rely on cross-modal grounding. Third, mapping observed capability demand onto 253 existing benchmarks reveals uneven alignment between benchmark coverage and real-world use: benchmarks concentrate on perception and reasoning toward fixed answers, while common workflows involving text, code, and data generation remain comparatively undertested. We validate our taxonomy and findings on an independent ChatGPT dataset. Our results provide a large-scale empirical characterization of what users seek to accomplish with multimodal LLMs and highlight opportunities for benchmark design grounded in observed user demand.

1 Introduction

The introduction identifies limited empirical evidence about how people use image-upload features, which capabilities their tasks combine, and how multimodal tasks relate to text-only work. It proposes analyzing real-world conversations to characterize capability use, compare task spaces, and assess benchmark coverage.

1 Introduction

  • Large-scale analyses of LLM use have largely focused on text, leaving how often people use multimodal inputs and what tasks they perform with them in the wild unclear.Multimodal workflows can involve plots, tables, screenshots, photos, and diagrams.Source: §1 · p. 1
  • The study asks which capabilities image-upload tasks require and how frequently users invoke individual capabilities and combinations.It frames perception, cognition, and generation as layers that real tasks may chain.Source: §1 · pp. 1–2
  • Whether multimodal use overlaps with or extends text-only task space remains an open question, including for tasks that depend on visual and textual cues together.The introduction contrasts tasks representable in either modality with inherently multimodal tasks.Source: §1 · p. 2
  • Existing multimodal benchmark coverage has not been examined against the capabilities and combinations users invoke in real-world use.The introduction notes that recent benchmarks cover varied capabilities, but their alignment with observed usage remains unknown.Source: §1 · p. 2
  • The study analyzes image-upload conversations from Copilot and ChatGPT to derive a capability taxonomy, compare multimodal and text-only task spaces, and assess benchmark coverage.The taxonomy is hierarchical and organizes capabilities into perception, cognition, and generation.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Works

Prior work has analyzed naturalistic LLM use and developed multimodal benchmarks, but comparatively little work characterizes real-world image-upload tasks from production assistants or aligns benchmark coverage with observed usage.

2 Related Works

  • Studies of LLM use in natural settings span public conversation corpora, on-platform analyses, occupational tasks, and use cases mined from public discussions.Work on multimodal interactions also includes crowdsourced evaluation platforms such as WildVision and VisionArena.Source: §2 · p. 2
  • This study complements crowdsourced multimodal evaluations by analyzing naturalistic image-upload interactions from general-purpose production assistants.Its focus is what users seek to accomplish with multimodal LLMs.Source: §2 · pp. 2–3
  • Multimodal benchmarks cover perception, reasoning, integrated capabilities, interleaved interactions, and long-context understanding, but their coverage remains largely top-down rather than grounded in observed user behavior.Even platforms motivated by real-world use are organized around curated categories, model comparison, or human preference.Source: §2 · p. 3

3 Datasets

The study analyzes image-upload sessions from a two-month Copilot sample and an independent ChatGPT dataset, using de-identified, privacy-preserving data.

3 Datasets

  • The primary dataset is a two-month random sample of anonymous Copilot conversations from August–September 2024, excluding enterprise, educational, and commercial accounts.Source: §3 · p. 3
  • An independent ChatGPT dataset spanning November 2024 and January 2025 is used to assess the reliability and generalizability of the taxonomy and findings.Source: §3 · p. 3
  • The analysis includes 42,617 Copilot and 23,413 ChatGPT sessions in which users uploaded at least one image, excluding multimodal outputs.A session is a complete conversation with one or more turns.Source: §3 · p. 3
  • All data are de-identified, summarized without reproducing original user text or accessing raw images, and processed in secure, access-controlled environments.Findings are reported in aggregate form.Source: §3 · p. 3

4 Developing a Taxonomy of Multimodal Capabilities

The study derives a hierarchical capability taxonomy from image-upload conversations, labels sessions at scale, and evaluates the resulting pipeline through robustness, annotation-agreement, and summary-fidelity checks.

4 Developing a Taxonomy of Multimodal Capabilities

  • The analysis treats each conversation as a task and labels sessions with the capabilities they invoke to characterize capability frequencies and combinations.Source: §4 · pp. 3–4
  • Each session receives task and image summaries of up to 50 words, and subsequent analyses use these summaries rather than raw images.The two summaries describe the user's overall objective and the image plus requested operation; Figure 2 gives an example.Source: §4 · p. 4, Figure 2 · p. 4
  • Taxonomy induction iteratively refines categories over 50 batches of 200 sessions, producing ten capability categories from a 10,000-session sample.Seed categories were adapted from prior multimodal benchmarks.Source: §4 · p. 4
  • The induced taxonomy assigns each session a primary capability and, when applicable, a secondary capability; refinement adds subcategories and groups capabilities into perception, cognition, and generation.The taxonomy also distinguishes supporting capabilities from terminal capabilities corresponding to intended user outputs.Source: §4 · p. 4
  • Evaluations find the taxonomy stable across prompt configurations and datasets, LLM annotations close to human judgments, and summary-based labels comparable to labels from raw conversations.Human annotation and eyes-on validation each use N = 200.Source: §4 · p. 4

5 Results

Real-world image-upload sessions commonly combine multiple capabilities, and their task space extends beyond text-only interactions. Benchmark coverage is strongest for image interpretation toward fixed answers, while common workflows that turn visual input into sustained writing, code, or data-driven decisions are less covered.

5.1 Capability Distribution Analysis

  • Recognition, Knowledge, and OCR & Text Extraction are the three most frequently invoked capabilities in image-upload sessions.Overall, 74.17% of sessions invoke multiple capabilities, commonly coupling perception with cognition or generation; Knowledge + Language & Text Generation is the most frequent combination.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2 Task Space Analysis Between Multimodal and Text-Only Interactions

  • Text-only-trained detectors flag more multimodal tasks as OOD: 76% vs. 43% under KNN and up to ∼30% vs. <1% under Isolation Forest.The asymmetry persists after modality-specific words such as “image” and “chart” are removed.Source: Table 3 · p. 7, §5.2 · p. 7
  • 65.55 vs. 76.61 average perplexity shows that multimodal-trained models better cover text-only summaries than text-only-trained models cover multimodal summaries (Mann–Whitney U, p < 0.001).The comparison uses per-sample perplexity from Llama-3.1 8B models fine-tuned separately on each modality.Source: Figure 4 · p. 7, §5.2 · p. 7
  • Multimodal embeddings overlap with text-only tasks but extend into distinct clusters, with Code Generation, Document Processing, and OCR & Text Extraction showing the highest OOD ratios.Frequent OOD combinations, including Recognition + Reasoning and Recognition + Code Generation, suggest expansion is associated with composing visual grounding with downstream reasoning and generation.Source: Figure 5 · p. 8, §5.2 · p. 8
  • An audit found that annotators judged 30% of sampled multimodal sessions (60/200) uniquely multimodal, meaning they could not reasonably be completed from text alone.The audit involved two annotators assessing 200 sessions.Source: §5.2 · p. 8

5.3 Multimodal Evaluation Benchmark Analysis

  • Recognition + Reasoning (S/D=3.12), Knowledge + Reasoning (1.92), Recognition + Language Generation (1.91), Knowledge + Recognition (1.26), and OCR + Recognition (2.00) have STRONG benchmark coverage.Coverage is smaller when images are starting points for usable outputs such as documents, programs, or decisions.Source: Table 4 · p. 9, §5.3 · p. 10
  • Knowledge + Language & Text Generation, the most frequent real-world combination at 8.4% of sessions, has only INCIDENTAL coverage (S/D=0.22).Matching benchmarks stop at short explanations or answers, while real requests often require sustained, structured writing; OCR + Language & Text Generation has LIMITED coverage (S/D=0.43).Source: Table 4 · p. 9, §5.3 · p. 10
  • Code Generation + Recognition (S/D=0.09) and Code Generation + Reasoning (S/D=0.11) together match only five tasks, while Data Analysis + Recognition has LIMITED coverage (S/D=0.39).The matched code tasks read or lightly edit existing code, and Data Analysis + Recognition benchmarks stop at reading charts rather than computing, comparing, or recommending actions.Source: Table 4 · p. 9, §5.3 · p. 10

6 Discussion and Conclusion

The study characterizes image-upload tasks as compositional workflows and finds that multimodal interactions occupy a broader task space than text-only interactions. It argues that benchmark coverage and interface design should better reflect these observed workflows, while noting limits to the study’s scope.

6 Discussion and Conclusion

  • Image-upload tasks are predominantly compositional, and multimodal interactions occupy a broader task space than text-only use, driven by cross-modal grounding rather than images alone.The study finds that benchmarks concentrate on perception and reasoning toward fixed answers, while common compositional generation workflows remain undertested.Source: §6 · p. 10
  • Current benchmarks cover reading an image and reasoning toward a fixed answer better than common writing-, code-, and data-generation workflows, especially Knowledge + Language & Text Generation.The authors propose evaluating complete visual-to-artifact workflows and scoring output fidelity, executability, and decision relevance.Source: §6 · pp. 10–11
  • Image uploads often let users share structured content instead of retyping it, and multimodal sessions skew toward higher-order activities such as Apply, Analyze, and Create.The authors suggest interfaces should recover code, tables, or layout as structured input and accept mixed image-and-text prompts.Source: §6 · p. 11
  • Privacy-preserving summaries may omit fine-grained visual details, findings may not generalize to future systems, and privacy constraints prevent release of original conversations.Eyes-on validation found limited loss of capability signal; the authors also call for longitudinal studies as usage evolves.Source: §6 · p. 11

Ethical Considerations

The study reports privacy safeguards for its Copilot and ChatGPT datasets and conducts its analyses on de-identified summaries rather than raw images or original user text.

Ethical Considerations

  • The primary dataset comprises de-identified Copilot conversations analyzed in secure, access-controlled research environments.The independent validation dataset came from a provider under data-protection safeguards and informed user consent.Source: Ethical Considerations · pp. 11–12
  • Automated processing removed personally identifiable information, and subsequent analyses used concise task and image summaries without access to raw images or original user text.The study reports no attempt to re-identify users or infer sensitive attributes, and presents results only in aggregate.Source: Ethical Considerations · p. 12
  • The study reports that it involved no interaction with human participants.Source: Ethical Considerations · p. 12

A.1 Additional Information about the Copilot Dataset

The appendix examines task distributions, semantic-space differences, and task complexity in Copilot sessions. It reports directional asymmetry between text-only and multimodal task spaces, including after removing modality-specific vocabulary.

A.1 Additional Information about the Copilot Dataset

  • An LLM-based taxonomy generator and classifier was used to analyze user intents and primary domains; observed distributions may reflect Copilot’s early visual-capability adoption phase.Figure 6 presents the intent-frequency and top-domain distributions.Source: §A.1 · p. 19, Figure 6 · p. 19
  • OOD analyses found directional asymmetry consistent with multimodal task representations covering a broader semantic space than text-only interactions.Table 5 reports the overall embedding-space detection results.Source: §A.1 · p. 19, Table 5 · p. 20
  • Code Generation, Document Processing, and OCR & Text Extraction had the highest per-capability OOD ratios, indicating the strongest contributions to semantic expansion beyond text-only interactions.Source: §A.1 · p. 20, Table 6 · p. 20
  • Text→Multimodal OOD rates remained higher than the reverse after modality tokens were removed: 15.24% vs. 0.16% for Isolation Forest and 77.09% vs. 41.46% for KNN.The authors interpret the persistent asymmetry as evidence that the broader multimodal task space is not merely a lexical artifact.Source: §A.1 · p. 20
  • OOD workflows commonly combined perception with downstream reasoning or generation, including Recognition + Reasoning and OCR & Text Extraction + Language & Text Generation.The authors attribute semantic expansion to cross-modal grounding and capability composition rather than image modality alone.Source: §A.1 · p. 20
  • 51.79% of multimodal sessions involved Apply, 12.05% Analyze, and 13.38% Create, compared with 36.13%, 5.01%, and 10.01% of text-only sessions, respectively.The difference in task-complexity distributions was significant (χ2(8) = 6087.13, p < .001).Source: §A.1 · p. 21, Figure 7 · p. 21

A.2 ChatGPT Dataset Validation Results

The appendix tests the Copilot-derived taxonomy and semantic-space findings on a ChatGPT dataset, while documenting constraints on dataset comparability and prompt templates used in the analyses.

A.2 ChatGPT Dataset Validation Results

  • The ChatGPT validation dataset spans ∼732,000 sessions from 7,676 users, including 23,413 sessions with at least one image, but covers US desktop interactions and does not distinguish image uploads from generations.The authors therefore treat it as a validation dataset rather than a primary analysis source.Source: §A.2 · p. 21
  • Image Editing & Generation, Language & Text Generation, and Recognition were the most frequent primary capabilities; Image Editing & Generation’s lead is partly due to pooling generated images with uploads.Only 62 sessions (0.26%) were labeled “Undefined”; the distribution supports cross-platform robustness of the taxonomy.Source: §A.2 · p. 21, Figure 9 · p. 22
  • ChatGPT embedding analyses showed the same directional OOD asymmetry across KNN and Isolation Forest, with multimodal embeddings extending beyond text-only regions and forming distinct clusters.Figure 8 visualizes task-summary embeddings, and the appendix reports OOD detection results in Table 7.Source: §A.2 · p. 22, Figure 8 · p. 22, Table 7 · p. 23

B Prompt Templates

  • The prompt templates provide capability categories with descriptions and require categories to be specific, distinct, noncontradictory, and faithful to the examples.The reference table includes categories such as Recognition, OCR, and Reasoning.Source: B Prompt Templates · p. 24

C.1 Taxonomy Robustness

Across prompt configurations and datasets, the induced taxonomies converged on the same ten canonical capabilities. Human labeling and privacy-preserving summaries also showed strong agreement with LLM labels and conversation-based labeling.

C.1 Taxonomy Robustness

  • Across configurations and datasets, the induced taxonomies converged on the same ten canonical capability categories.Researchers merged semantically equivalent labels and resolved disagreements through discussion; runs used 10- and 20-cluster limits on Copilot and a 10-cluster setting on ChatGPT.Source: §C.1 · pp. 27–28, Table 8 · p. 28

C.2 Human Validation of LLM Annotations

  • LLM–human agreement was F1 = 0.763–0.773, comparable to human–human F1 = 0.783 and hit rate = 0.960.Two annotators independently labeled 200 Copilot sessions, assigning up to two labels per session.Source: §C.2 · pp. 28–29

C.3 Eyes-on Validation of the Summarization Pipeline

  • Human agreement across conversations and summaries was F1 = 0.73 and hit rate = 0.91; LLM performance was comparable across the two conditions.For LLM labeling, summaries yielded F1/hit rate = 0.80/0.96 versus 0.85/0.95 for conversations; the authors conclude that summaries preserve capability information for taxonomy induction and annotation.Source: §C.3 · p. 29

C.4 Benchmark-Annotation Human Validation

A human audit found lower agreement with the benchmark-task adjudicator than between the two primary model annotators. The appendix also provides expanded descriptions of the ten capability categories.

C.4 Benchmark-Annotation Human Validation

  • Human–adjudicator agreement was macro κ = 0.579 and mean F1 = 0.761, versus macro κ = 0.679 and mean F1 = 0.802 for primary-model agreement.The audit sampled 71 of 500 adjudicated tasks, stratified by annotator disagreement, from 864 benchmark tasks identified as involving static images.Source: §C.4 · p. 30, Table 11 · p. 30

D Detailed Taxonomy Description

  • The appendix’s detailed taxonomy table gives descriptions of the ten capability categories, complementing the concise descriptions in the main text.The table descriptions extend to 200 words per category.Source: Table 12 · p. 31

E Capability Combination Examples

The listed task examples illustrate that image-upload interactions often combine capabilities rather than relying on isolated skills. Frequent combinations span visual recognition with analysis, text extraction, and generation.

E Capability Combination Examples

  • Multimodal interactions are compositional, coordinating perception, reasoning, and generation rather than using isolated capabilities.The examples include analysis of uploaded data, extracting text from images, and generating outputs based on visual inputs.Source: E Capability Combination Examples · p. 32, Table 13 · p. 32
  • Data Analysis + Recognition examples include computing financial metrics from tables, recommending charts, and deriving insights from spending distributions.Other examples include reconstructing project folders from screenshots and generating reports from presentation slides.Source: E Capability Combination Examples · p. 33
  • OCR & Text Extraction + Recognition examples include computing metrics from tables, analyzing charts, and extracting text before answering questions.The examples span structured data, resumes, scanned documents, and image-based alt text.Source: E Capability Combination Examples · p. 33
  • Other examples combine code generation with reasoning, including debugging applications, generating programs from pictured problem statements, and building software from UI designs.The OOD examples also feature combinations such as recognition with reasoning or code generation.Source: E Capability Combination Examples · p. 33
  • Recognition paired with language generation appears in tasks such as transcribing or summarizing document text and producing captions or formatted tables from images.Examples also include generating product descriptions and alt text from visual content.Source: E Capability Combination Examples · p. 35

Related papers

Loading 2610.00701v1…