LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing

• GPT-4V-based multimodal agents challenge open-source research because image inputs mainly contextualize text, outputs are text, and training and architecture details are undisclosed.Source: §1 · p. 1
• LLaVA-Interactive is an open-source prototype for multi-turn dialogue with multimodal inputs and responses, combining LLaVA, SEEM, and GLIGEN without additional training.Source: §1 · p. 1
• Visual prompts—including strokes, drag-and-drop, and bounding boxes—let users express intent for segmentation, generation, and editing; the system reportedly follows intent better than standalone GPT-4V or LLaVA.Source: §1 · p. 1
• The system and code base are publicly available to facilitate future community improvements.Source: §1 · p. 2

• Systems enabling image output either train models for image generation, editing, or segmentation, or prompt LLMs to activate expert vision models at inference time.Source: §2 · p. 2
• Unlike those approaches, LLaVA-Interactive combines three models without additional training or LLM prompt engineering and supports user-drawn visual prompts for segmentation and generation/editing.Source: §2 · p. 2
• Unified vision-language models remain exploratory, and LLaVA-Interactive illustrates their potential through cross-task work and a multimodal promptable interface.Source: §2 · p. 2

3 LLaVA-Interactive

• The interface separates the current image and visual prompts, natural-language chat, and a three-tab visual interaction panel.Source: §3 · p. 2, Figure 1 · p. 3
• Users can remove or change objects with segmentation masks, inpaint objects positioned by bounding boxes, or generate a new scene from spatial layouts and a caption.Source: §3 · pp. 2, 4
• The workflow starts with an uploaded or generated image, then supports chat, segmentation, or editing; chat suggestions can inform later object removal or addition.Source: §3 · p. 4

3.1 Behind the Scenes: Individual Models

• The demo connects LLaVA for visual chat, SEEM for multimodal-prompt segmentation, and GLIGEN for bounding-box-conditioned image generation and editing.Source: §3.1 · pp. 4–5

3.2 Development Challenges

• Implementation challenges included GLIGEN’s inability to fill background holes, limited Gradio drag-and-drop support, integrating complex models, and dependency conflicts; the system used LaMA, a custom image component, and separate services.Source: §3.2 · p. 5

4 Case Study: Multimodal Interactive Creation for Photographic Artists

• The assistant suggests removing a pier from the uploaded lake image; the user segments it with a stroke and generates an updated image without it.The user can drag the segmentation mask out before clicking Generate.Source: §4 · p. 5
• After suggesting removal of a small dock, the assistant’s recommendation is followed by text-prompt segmentation and generation that remove the dock.The dock’s segmentation mask is shown in magenta.Source: §4 · p. 7
• The user then adds a sunset to the sky and its reflection to the lake through text-prompt-based segmentation and replacement.The assistant recommends the sunset, and the user identifies the water reflection as a further improvement.Source: §4 · p. 7
• The assistant comments that the final version effectively conveys the beauty of nature.Source: §4 · p. 7

5.1 Iterative Co-Creation of Visual Scene and Description

• A generated outdoor scene may be imperfect, so users can revise it with visual prompts and ask the assistant for descriptions, promotion, bilingual poetry, or feedback on image plausibility.The example removes a gray shore and adds a white goose; the text description and response are often coherent with the edited images.Source: §5.1 · p. 7

5.2 Graphic Design for a Halloween Poster

• For a Halloween poster, users can choose an idea and refine the image by adding, replacing, or removing themed elements before requesting feedback.The example includes adding a bat and skeleton, replacing scarecrows with a ghost, and substituting pumpkins with a spider web; the assistant affirms that the design captures Halloween’s spirit.Source: §5.2 · p. 7

5.3 Fashion Design for Kid’s Clothing

• The assistant answers questions about the English text and picture on a Chinese child’s sweatshirt, then edits the picture at the user’s request by adding a blue hat and sunglasses.The example describes text recognition on clothes as useful where users have less knowledge of designed English text.Source: §5.3 · pp. 7–8
• After mentioning Denver and guessing it may be the child’s hometown or a place they enjoy visiting, the assistant supports a new scene with a lake, boat, tent, and snow mountains.The user removes the bear, leaves the background, and specifies the objects’ spatial layout.Source: §5.3 · p. 8

5.4 Food Preparation

• For a romantic dinner, the assistant suggests adding salads and candles, and users can compare virtual dinner options before asking for etiquette guidance.The user can submit options for assistant evaluation and seek advice tailored to the dinner.Source: §5.4 · p. 8
• When a user replaces butter with rice in a recipe query, the assistant provides updated ingredient and instruction lists with revised cooking tips.Source: §5.4 · p. 8

5.5 Visual Content Creation and Story Telling

• The assistant provides detailed image descriptions and a children’s story, then users can inpaint playful objects in chosen positions.Examples include a growing mushroom in front of the first character and an oversized drum for the fourth.Source: §5.5 · p. 8

5.6 Education

• Science-learning scenarios use familiar, visually appealing imagery to prompt questions about dinosaurs, robots, and scientific concepts.The example includes questions about a T-rex, why the sky is pink, and how its color changes during the day.Source: §5.6 · pp. 8–9
• In cartoon interpretation, removing the “PRESS” label can substantially change responses about a bound man and the pressures journalists face.The passage notes that without the label, interpretations may differ considerably.Source: §5.6 · p. 9

5.7 Interior Design

• Users can modernize a large living room by incorporating suggestions, then segment objects and adjust the design with stroke and text prompts.Suggested changes include minimalist elements, potted plants, and modern art.Source: §5.7 · p. 9
• In a small living room, users can select a sofa and table for style updates, add a plant by grounded inpainting, and modify other objects.The assistant also suggests possible changes to the TV position or ceiling lamp.Source: §5.7 · p. 9

5.8 Identification of Unusual and Risky Items

• Removing the person shifts the unusual element to a chair on a moving car, while replacing the taxi with flowers shifts it to a bicycle carrying a bouquet.The authors describe this analysis-by-synthesis approach as potentially effective for examining visual anomalies.Source: §5.8 · p. 9

6 Conclusions and Future Work

• The system combines pre-trained LLaVA, SEEM, and GLIGEN models through web services without additional model training.The paper describes it as a fully vision-language multimodal system, particularly distinctive for visual prompts in segmentation and generation/editing.Source: §6 · p. 9
• The initial assessment across real-world application scenarios demonstrated the system’s ability to perform new, complex tasks.The authors hope the work will inspire further research into multimodal foundation models.Source: §6 · pp. 9–10
• LLaVA-Interactive’s abilities are restricted by the performance limits of its pre-trained models.The authors suggest replacing modules or improving individual models; they also note that combining current skills does not create new skills through hidden-space interpolation.Source: §6 · p. 10