Source-linked AI summary
PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback
Xueqing Wu, Ashwin Balasubramanian, Bingxuan Li, Dawei Zhu, Kai-Wei Chang, Yale Song, Yiwen Song, Rui Meng, Tomas Pfister, Nanyun Peng
TL;DR
Single-turn scientific diagram generation often misses authors’ evolving visual and communicative preferences, motivating scalable multi-turn evaluation. The paper introduces MTPaperBananaBench and a simulator, then proposes PaperBanana-Interact, which improves quality and reduces forgetting relative to baselines.
Problem
Single-turn generation struggles to satisfy nuanced preferences, while multi-turn refinement remains underexplored despite users requesting revisions and rating refined diagrams more highly.
Method
The paper introduces MTPaperBananaBench with a user simulator and develops PaperBanana-Interact, a multi-agent system with critique-and-refine and history summarization.
Results
PaperBanana-Interact consistently mitigates quality drift and forgetting, improving quality scores by 11.9–18.6 percent over baselines and reducing forgetting by 3.7–6.2 percent.
Takeaways & Limitations
Multi-turn diagram refinement can be benchmarked with simulated feedback, and PaperBanana-Interact supports iterative refinement without the baseline pattern of declining quality.
Abstract
from arXiv · showhide
Recent efforts have aimed to automate scientific diagram generation from paper content (Lin et al., 2026; Zhu et al., 2026a). However, fully satisfying an author's visual and communicative preferences in a single turn is challenging: in our formative user study (N = 14), all participants requested further revisions after viewing an initial draft, and 86% of them rated the refined diagrams as more satisfactory. Despite the clear demand, the multi-turn workflow remains largely underexplored. To bridge this gap, we present MTPaperBananaBench, a benchmark for multi-turn diagram generation containing 292 images annotated with 3,518 user requirements. To reduce expensive human studies and enable scalable benchmarking, we construct a user simulator that, at each turn, identifies unsatisfied requirements and converts k of them into natural language feedback. Evaluating both requirement satisfaction and overall diagram quality reveals two key failure modes shared across baseline multiturn systems: (1) quality drift, where diagram quality progressively declines over turns, and (2) forgetting, where previously implemented features are lost in subsequent turns. To address these issues, we introduce PaperBanana-Interact, a multi-agent system that refines diagrams via an internal critique-and-refine loop. PaperBanana-Interact consistently improves rather than degrades diagram quality across turns, outperforming baselines by 11.9-18.6 points in quality score and reducing forgetting by 3.7-6.2 points.
1. Introduction
The paper establishes multi-turn scientific diagram refinement as an underexplored need, introduces a benchmark and user simulator, and proposes PaperBanana-Interact to address baseline failures.
- Motivation: All 14 formative-study participants requested revisions, and 86% rated refined diagrams more satisfactorily.The study motivates iterative interaction after users review an initial draft.
- Benchmark: MTPaperBananaBench benchmarks multi-turn diagram refinement using 292 diagrams annotated with 3,518 user requirements.The benchmark evaluates requirement satisfaction and overall diagram quality.
- Benchmark: The user simulator identifies unfulfilled preferences at each turn and converts k selected requirements into natural-language feedback.This design enables scalable and reproducible evaluation without collecting human interactions for every run.
- Failure Modes: Baseline systems exhibit quality drift, with quality declining across turns, and forgetting, where later refinements overwrite previously satisfied requests.Reported quality declines from 50.3 to 19.0 for NanoBananaPro and from 50.3 to 47.1 for PaperBanana-DirectRefine; forgetting affects 14.1–22.9% of such requests.
- Proposed System: PaperBanana-Interact uses a multi-agent critique-and-refine loop to mitigate quality drift and forgetting.Its multi-objective critic evaluates current and prior requests, source-context adherence, and presentation quality.
- Results: PaperBanana-Interact improves quality scores by 11.9–18.6 percent over baselines and reduces forgetting by 3.7–6.2 percent.These results support the system’s ability to improve rather than degrade diagram quality across turns.
2. Related Work
Related work spans generative image refinement and scientific diagram generation, including code-based rendering and newer generative-model pipelines.
- Generative Models: Generative models increasingly support image-conditioned and instruction-guided refinement beyond one-shot image generation.Recent work also explores multi-turn interaction in which users iteratively provide feedback and corrections.
- Scientific Diagram Generation: Scientific diagram generation requires factual accuracy and flexibility for nuanced author insights.Earlier systems commonly rendered diagrams by generating TikZ or PythonPPTX code, while newer work integrates generative models.
3. Task and Dataset
The task models iterative diagram updates from source context and user feedback, while MTPaperBananaBench combines expert-derived requirements, simulated interaction, and dual-axis evaluation.
- 3.1. Task Formulation: In the multi-turn setting, each updated diagram I_t is conditioned on prior diagrams and user inputs, including the source context x_0 and feedback x_t.The initial diagram I_0 is generated from source context, then refined iteratively.
- 3.2. Formative User Study: The formative study involved 14 experienced academic writers and scientific illustrators creating and refining diagrams for their own research.Participants reviewed an initial PaperBanana draft and refined it with PaperBanana-DirectRefine.
- 3.2. Formative User Study: Multi-turn refinement increased average satisfaction from 3.2 to 4.2, with 12 of 14 participants rating refined diagrams higher.Individual improvements ranged from 0.5 to 2 points on a 1–5 scale.
- 3.2. Formative User Study: Nine participants said the initial draft shaped or changed their presentation vision, and 64% of sessions produced requests only after viewing a draft.Interviews also reported reduced factual faithfulness, degraded aesthetic quality, and failure to preserve satisfied requests.
- 3.3. MTPaperBananaBench: MTPaperBananaBench contains 292 diagrams and 3,518 requirements, with annotations grounded in source context and human-created reference diagrams.Requirements are generated with LLM assistance and manually reviewed, removing 0.5% and revising 24.7% of candidates.
- 3.3. MTPaperBananaBench: The user simulator evaluates each requirement against the latest diagram, selects the first k failed requirements, and turns them into natural-language feedback.Interaction ends when no failures remain or the maximum turn budget is reached.
- 3.3. MTPaperBananaBench: Evaluation measures requirement satisfaction and diagram quality, the latter by comparison with a human-created reference across multiple dimensions.The requirement satisfaction rate averages binary satisfaction scores across requirements.
4. Methods
The methods frame diagram refinement as prompt generation followed by visualization, then extend it with shared memory and an internal critic–refiner loop for multi-turn constraint satisfaction.
- Generative Models as Refiners: Generative-model refiners update the latest diagram using all user utterances through the current turn.This establishes the basic refinement formulation before introducing the multi-agent design.
- Generative Models as Refiners: The baseline pipeline uses an agent A to generate a detailed image prompt P, which a visualizer V renders as the diagram I.The initial turn is defined as P_0 = A(x_0), followed by I_0 = V(P_0).
- PaperBanana-DirectRefine: In multi-turn refinement, A_r updates the previous prompt using the latest image and full user-utterance history before V renders the next image.The direct-refinement baseline limits image and prompt inputs to the most recent turn to control computational cost and long-context degradation.
- PaperBanana-Interact: PaperBanana-Interact summarizes multi-image interaction history into shared textual memory so agents receive visual context without repeatedly processing raw images.The memory records diagram content, changes from the previous revision, and alignment with that turn’s request.
- PaperBanana-Interact: At each turn, the system starts from the previous image and prompt, then iterates critic, refiner, and visualizer agents to generate revisions.The loop stops when no further edits are required or when the iteration budget τmax is reached.
- PaperBanana-Interact: The multi-objective critic evaluates current and prior requests, source faithfulness, and presentation quality with a structured critique for each constraint.These explicit constraint-level critiques guide subsequent updates and target multi-turn forgetting and quality drift.
5. Experiments
The experiments evaluate multi-turn diagram refinement with simulated feedback, measuring requirement satisfaction, diagram quality, per-turn satisfaction, and forgetting. Baselines exhibit quality drift and forgetting, whereas PaperBanana-Interact improves quality and mitigates these failures through iterative critique and refinement.
- Evaluated Systems: PaperBanana-Interact is compared with generative and agentic refiners using standardized Gemini-3.1-Pro and NanoBananaPro backbones.The evaluation includes NanoBananaPro, GPT-Image-2, PaperBanana-DirectRefine, and other single-turn generators.
- Evaluation Setup: The benchmark evaluates multi-turn refinement using k=1 and k=3 simulated feedback settings, up to five turns, and early stopping when requirements are satisfied.Evaluation reports quality, requirement satisfaction, per-turn requirement satisfaction, and forgetting rate.
- Main Results: Multi-turn interaction improves overall requirement satisfaction, while refiners achieve 50 to 80 per-turn requirement satisfaction across systems.The reported per-turn metric measures satisfaction of a request in the turn when it is raised.
- Quality Drift: All baseline refiners suffer quality drift, with larger declines for generative models; drift is amplified when each turn contains three requirements.PaperBanana-Interact instead mitigates drift and improves quality over single-turn outputs, although gains are more modest for k=3.
- Quality Drift: PaperBanana-Interact outputs were preferred by a human annotator in 81.3% of comparisons against NanoBananaPro and 76.7% against PaperBanana-DirectRefine.The pairwise evaluation covered 150 samples.
- Forgetting: Baseline forgetting rates range from 14 to 23%, whereas PaperBanana-Interact reduces forgetting to 10–13% but does not eliminate it.Smaller internal iteration budgets produce monotonically higher forgetting rates.
6. Conclusions
The paper establishes a benchmark and multi-agent refinement framework for iterative scientific diagram generation, while highlighting quality drift, forgetting, and visual hallucination risks.
- 6. Conclusions: PaperBanana-Interact uses a history summarizer and multi-objective critic to mitigate quality drift and requirement forgetting during multi-turn refinement.The framework is presented as a foundation for interaction-aware scientific diagram generation and refinement.
- 6. Conclusions: The authors warn that generative systems can produce plausible but factually incorrect scientific representations, requiring human oversight.They intend to position the tool as assistive and require disclosure of generated images in academic publications.
- 6. Conclusions: Figure 8 summarizes the distribution of requirements per diagram and across taxonomy categories in MTPaperBananaBench.
B. Data Annotation Details
Requirements are generated with LLM assistance and then manually reviewed for faithfulness, taxonomy accuracy, and balanced coverage across semantic categories and severity levels.
- B. Data Annotation Details: The annotation process first generates candidate requirements with Gemini-3.1-Pro, followed by independent human review that removes or revises unfaithful candidates.The final benchmark contains 3,518 requirements across 292 diagrams, averaging 12 requirements per diagram.
- B. Data Annotation Details: The taxonomy covers Content, Organization, and Visual representation, along with Critical, moderate, and minor impact severity levels.
- B. Data Annotation Details: Annotators are instructed to balance the semantic categories approximately evenly and cover all three severity levels.
- B. Data Annotation Details: Manual review uses source context, the human-drawn diagram, and model-generated descriptions to ensure faithfulness and correct taxonomy labeling.The human-drawn diagram is treated as the primary source for capturing authors’ design choices.
- B. Data Annotation Details: Three author-annotators trained jointly, then worked independently on overlapping batches to measure inter-annotator agreement.
- B. Data Annotation Details: Cohen’s κ was 0.767 on 207 potentially problematic requirements, indicating substantial agreement.
C. Details for Formative User Study
The formative study shows that experienced researchers use iterative refinement to improve scientific diagrams, while the process can also expose new design problems.
- C. Details for Formative User Study: All 14 participants had substantial academic writing and scientific illustration experience and selected papers to which they had made substantial intellectual contributions.
- C. Details for Formative User Study: Participants supplied methodology text and a figure caption for initial generation, then used natural-language feedback to refine diagrams and could stop when satisfied.
- C. Details for Formative User Study: Nine of 14 participants reported that viewing the initial diagram changed or refined their thinking about how the method should be presented.
- C. Details for Formative User Study: 79% of sessions reported at least one new problem after revision, including degraded aesthetics, reduced factual faithfulness, or failure to preserve satisfied requests.
D. Experimental Details
The experiments use specified model settings and examine iterative refinement trajectories, while acknowledging limited statistical reporting because most main-text results come from single runs.
- D. Experimental Details: NanoBananaPro uses gemini-3-pro-image at temperature 1.0, while Gemini-3.1-Pro uses gemini-3.1-pro-preview with temperature 1.0 and a 50,000-token maximum output.
- D. Experimental Details: Figure 13 reports the distribution of PaperBanana-Interact’s internal agentic iterations.
- D. Experimental Details: Most main-text experimental numbers come from a single run because computational constraints prevented full descriptive statistics for every experiment.
- D. Experimental Details: Across five reruns, diagram quality was 61.2 ± 1.8, requirement satisfaction was 58.0 ± 0.4, and forgetting rate was 12.6 ± 1.2.Per-turn requirement satisfaction was 77.2 ± 0.3; quality was less stable than the other reported metrics.
- D. Experimental Details: A complete five-turn PaperBanana-Interact trajectory and additional quality-drift examples are presented in Figures 14–17.
F. Evaluation Details
Diagram quality is evaluated by comparing generated and human-reference diagrams across four dimensions, with faithfulness and conciseness treated as primary.
- Quality evaluation compares a model-generated diagram I_T with a human reference diagram I_H across faithfulness, conciseness, readability, and aesthetics.Faithfulness and conciseness are primary dimensions; readability and aesthetics are secondary dimensions.
- Faithfulness measures alignment with the methodology description and diagram caption.
- Conciseness measures whether the diagram focuses on core information without visual clutter.
- Readability assesses intelligible layouts, legible text, and avoidance of excessive crossing lines, while aesthetics assesses conformity to academic manuscript style.
- Each dimension receives a win, tie, or loss against the reference, and a two-tier voting procedure aggregates these outcomes into an overall result.A decisive result on the primary dimensions determines the overall outcome; primary ties invoke the next tier.
F.2. Requirement Satisfaction
Requirement satisfaction is evaluated with structured VLM judgments, while examples and evaluator comparisons document how refinement systems behave across turns.
- Requirement satisfaction is assessed by prompting a VLM with the generated diagram and all associated requirements to produce a binary score and rationale for each requirement.The requirements are evaluated in a batched format to reduce computation.
- Figure 14 shows a five-turn PaperBanana-Interact refinement trajectory starting from PaperBanana output with k=1 simulated feedback.
- NanoBananaPro examples show sudden aesthetic drops and distorted artifacts during repeated image editing.
- PaperBanana-DirectRefine examples show unexpected styling drift during refinement.
- Cohen’s κ=0.663 indicates substantial agreement between Gemini-3.1-Pro and Claude-Opus-4.8 as requirement-satisfaction evaluators.The comparison used five-turn PaperBanana-DirectRefine outputs generated with the k=3 user simulator.
F.3. Human Evaluation
Human evaluations were used to validate both the requirement-satisfaction metric and the comparative quality advantage of PaperBanana-Interact.
- The human evaluations were conducted by the paper’s authors to assess experimental reliability.
- Cohen’s κ=0.812 indicates near-perfect agreement between human judgments and VLM requirement-satisfaction outputs.A human evaluator independently assessed 200 requirements on five-turn PaperBanana-Interact refinements using the k=3 simulator.
- Table 5 presents two examples of user-simulator inputs and outputs, using k=1 and k=3 failed requirements.The k=1 example corresponds to the second turn in Figure 14.
- Across 150 data points, human evaluators preferred PaperBanana-Interact over PaperBanana-DirectRefine in 76.7% of cases and over NanoBananaPro in 81.3%.Evaluators ranked anonymized outputs using faithfulness, conciseness, readability, and aesthetics.
H. PaperBanana-Interact Details
The refinement details specify structured prompts for simulation, summarization, and iterative critique, with separate checks for prior requests, source fidelity, and presentation.
- Refinement loop: The refinement loop implements critic and refiner turns consecutively within one conversation, using distinct prompts for the first and later iterations.
- Requirement evaluation: Requirement evaluation uses the user requirement as the primary expectation and the method section and caption for contextual grounding.
- Requirement evaluation: Requirements are judged with strict pass/fail criteria, while explicitly permitted alternatives must be executed consistently and flawlessly.
- Requirement evaluation: The evaluator outputs the requirement verbatim, a rationale, and a Satisfied or Unsatisfied verdict for every requirement.
- User simulator: Simulator feedback is written from a non-expert user perspective and describes problems or desired changes without prescribing exact design fixes.
- User simulator: The user simulator generates one short, casual feedback message from the supplied issues, raising exactly k issues without adding others.
- Critique procedure: The critique procedure emits separate entries for the latest request, each prior request, source content, and presentation rather than combining them into one critique.