VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It

• Selector rankings reverse with frame budget, Motion-score declines after its peak, stronger reading reminders lower accuracy, and lower crop resolution lowers field-value ANLS.The probes use non-test films and training-form crops, not the main test sets.Source: §1 · p. 2
• ViewCraft jointly refines what agents see and how they read, pairing an executable interface that selects, transforms, and presents evidence with a reading procedure.Failure traces, visual replay, and paired development evaluation guide evolution while the executor remains frozen; eligible visual edits from rejected joint proposals can be retested with the incumbent reader.Source: §1 · p. 2
• Across eight benchmarks, ViewCraft improves over Direct by 6.1 points on average and remains 5.2 points ahead at a matched input-token budget.Source: §1 · p. 2
• With frames replaced by another film, 12 of 64 previously correct questions remain correct (Wilson 95% CI [11%, 30%]).The four interface probes are independent of Table 2’s test sets.Source: Figure 2 · p. 3

2.1 Feedback-driven agent optimization

• Prior feedback-driven methods optimize prompts, compound LM programs, verbal experience, executable skills, workflows, or module configurations, with most retained state textual or structural.Source: §2.1 · p. 3

2.2 Multimodal skill evolution

• Multimodal skill work includes visual tools, prompts, and selection, while MPO and Dynamo jointly optimize prompts or co-evolve skills and visual tools; ViewCraft saves and replays an executable interface with its reader.Interface projection retests visual components of eligible rejected proposals with the incumbent reader.Source: §2.2 · p. 3

3 ViewCraft

• ViewCraft saves an executable evidence interface with its reader, and its evolution loop combines reuse, visual support, coupled editing, transformations, replay-screened evaluation, and interface-only retries.Source: §3 · p. 3, Table 1 · p. 4

3.1 Skill state and objective

• A skill w = (θ, ϕ, ψ) uses ϕ to select evidence, ψ to arrange views and formats, and θ as the frozen executor’s reading procedure.The objective maximizes expected task reward over ϕ, ψ, and θ for the frozen executor; text-only, interface-only, and joint search edit different coordinates.Source: §3.1 · p. 4

3.2 Complementary success factors

• Evolved views with the initial reader recover +2.3 of the +8.2-point joint gain under a fixed trajectory, while token-matched Direct recovers +0.9 points.Equation 2 separates evidence availability, successful reading, and correctness without required evidence; controls test whether delivery or input volume alone explains the gain.Source: §3.2 · pp. 4–5

3.3 Failure-guided proposals

• Failure traces record questions, sent views, responses, and scores; they guide revisions, while contracts specify the views, answer format, and available requests.External guards check execution budgets, valid requests, and support for question-derived evidence; replay filters visual edits that leave views unchanged.Source: §3.3 · p. 5

3.4 Paired evaluation

• The gate accepts a candidate when paired task score improves, or when task score is unchanged and evidence support improves; eligible visual edits from losing joint candidates may be retried with incumbent θ.A retry requires non-degraded evidence support and remaining submission and gate allowances; acceptance advances the skill, while rejection restores the incumbent. Table 2 pools three runs and averages tasks equally in its Mean column.Source: §3.4 · pp. 5–6, Table 2 · p. 6

4.1 Setting and comparisons

• Evaluation uses frozen, source-disjoint splits; three benchmarks center on one image or form, while five require evidence across sources.Main comparisons pool three repetitions, and the evolving methods share proposal and rollout budgets.Source: §4.1 · p. 6, Table 3 · p. 7

4.2 Task scores and coordinate effects

• All eight benchmarks favor joint evolution over Direct, with a median held-out gain of +5.9 points.Joint evolution leads on every benchmark; Figure 4 reports the per-benchmark gains over Direct.Source: §4.2 · p. 7, Figure 4 · p. 8
• All terminal interactions are positive, averaging +1.8 points (p = 0.004); across coordinate partitions, every task estimate favors ViewCraft (p = 0.008).Seven interaction bootstrap intervals exclude zero; SlideVQA’s includes it. Holm correction retains FunSD and STI-Bench as individually significant.Source: §4.2 · pp. 7–8

4.3 Search strategies at matched budget

• Joint search beats alternating search by +1.3 points and the strongest same-space baseline by +2.6; evaluation matching yields 0.691 versus 0.692 for full ViewCraft.Trace-guided proposals lead random same-space proposals by +1.5 points, and retention mechanisms add +1.6.Source: §4.3 · pp. 8–9

4.4 Mechanisms and evaluation cost

• Removing one retention stage costs 0.7–0.9 points, while removing all stages costs 1.6; every paired interval excludes zero.The tested stages are replay, interface projection, and the evidence tiebreak.Source: §4.4 · p. 9
• Asymmetric projection leads the retry designs; keeping the verified interface while reverting the reader gains +0.6 points over the reverse direction.The contract-frozen condition remains within 0.5 points of full ViewCraft on average.Source: §4.4 · p. 9

4.6 Case studies

• Case studies show view edits exposing usable relations, while evolved readers match objects across rows, read frames in sequence, and preserve compound form queries.Responses must retain field–value associations and the required format.Source: §4.6 · pp. 11–12

5 Conclusion

• ViewCraft makes evidence construction a persistent, editable part of a multimodal skill, evolving what the executor sees together with how it reads.The interface selects, transforms, and organizes observations; the reader interprets the resulting views.Source: §5 · p. 12
• The joint approach improves both evidence presentation and reading, outperforming single-coordinate search and same-space baselines while retaining an advantage at matched input budget.Learned selection sends 8%–15% fewer input tokens than Initial skill on all five multi-source tasks.Source: §5 · p. 12
• The reusable unit is an executable evidence interface paired with a reading procedure that uses its rendered evidence to answer new questions.Replay and paired evaluation make candidate views inspectable and tie retention to task outcomes.Source: §5 · p. 12

A Evaluation protocol

• Main-comparison scores pool three repetitions, with binary metrics using denominator 3n and FunSD, SlideVQA, and OpenEQA using continuous scores.The reported continuous metrics are field-value ANLS, ANLS, and mean LLM-Match, respectively.Source: A Evaluation protocol · pp. 17, 19
• Train, gate, and test sets use pairwise-disjoint source media, and the frozen test sample scales with benchmark size while preserving whole sources and category coverage.The full test set is used below 500 questions, half is sampled at 500–1000, and one quarter above 1000.Source: Table 6 · p. 17, A Evaluation protocol · p. 17
• Evidence-interface probes use a frozen executor on non-test VideoHolmes questions and FunSD training crops, varying frame budgets, prompts, and crop resolution.Replacing frames for 64 previously correct questions with another film yields 12 correct and 52 wrong answers, with no abstentions or mismatch flags.Source: A Evaluation protocol · p. 17
• FunSD is converted to field-value QA, split by form, and scored with continuous field-value ANLS on lowercased transcriptions.Of 200 constructed pairs, 192 link the correct field (96.0%); with gold labels, Direct rises from 0.306 to 0.312 and ViewCraft from 0.412 to 0.415.Source: A Evaluation protocol · pp. 17–18
• Development accounting covers 24 task–seed searches per arm and 250 submissions, with projection candidates reusing validated visual edits but still receiving paired evaluation.All evolving methods share four rounds and 96 training rollouts; capping joint evaluations at 200 yields mean score 0.691 versus 0.692 for full ViewCraft.Source: A Evaluation protocol · p. 18, Table 10 · p. 19
• The clean comparison separates reader-only and interface-only states from terminal states by restoring unused coordinates to Initial skill.The interface-only condition excludes reasoning and decision rules; the input-token count for each item includes requests across all its observations.Source: A Evaluation protocol · p. 19

B Uncertainty and contract controls

• ViewCraft’s gain over MPO-fullspace has CI [+1.3, +3.9], and every task favors ViewCraft.The reported exact sign-test p-value is 0.008.Source: B Uncertainty and contract controls · p. 19, Table 13 · p. 20
• +6.1 points is the across-task mean gain over Direct, with every task favoring ViewCraft and Holm-significant gains on FunSD and STI-Bench.The median gain is +5.9 points; the exact sign-test p-value is 0.008.Source: B Uncertainty and contract controls · p. 20, Table 14 · p. 21
• The contract-frozen control preserves most of ViewCraft’s gain, and format-normalized rescoring retains its advantage on FunSD.On FunSD, Direct gains 1.2 points and ViewCraft 0.9 points under format-normalized rescoring.Source: B Uncertainty and contract controls · p. 20, Table 16 · p. 21
• With the observation trajectory fixed, evidence and reading transplants each recover part of the joint gain; redrawing gate items shifts every arm by under half a point.The displayed-mean joint-over-alternating gap is preserved under gate redraws.Source: B Uncertainty and contract controls · p. 21, Table 17 · p. 21, Table 18 · p. 22

C Transfer and search diagnostics

• Every evaluated transferred skill beats its target executor’s Direct baseline by at least 1.5 points; three also exceed the target-evolved skill.The remaining transfers fall within 3.3 points of that skill.Source: C Transfer and search diagnostics · p. 22
• Acceptance declines across rounds, and the accepted cohort includes edits that both repair and regress items, with a net improvement selected by the paired gate.Among 204 accepted edits, 153 both repair and regress items, averaging 5.4 repairs and 2.9 regressions; frequently accepted co-edits revise both reader and view contract.Source: C Transfer and search diagnostics · p. 23
• A rejected FunSD proposal returned “None” on 89% of paired items after literal query-label matching excluded valid composite fields; the next proposal relaxed that requirement.The score fell from 0.273 to 0.075.Source: C Transfer and search diagnostics · p. 23

D Discrete proposal space

• The clean partition assigns inference and decision rules to the reader while the interface holds executable rendering, view metadata, and answer/action formats.Alternative clause-content and strict partitions move render-referencing clauses into the interface, changing the partition definition used for interaction comparisons.Source: D Discrete proposal space · p. 23

E Reading the recorded cases

• The figures pair recorded questions and responses with view changes and reading rules, presenting ten repairs and two regressions.Long video stacks are reflowed or excerpted for legibility while retaining source timestamps.Source: E Reading the recorded cases · p. 23
• In one regression, the response retains both numbers but drops field labels required by the compound-field scorer.The numbers remain visible, but the missing labels make the answer fail the scorer.Source: E Reading the recorded cases · p. 24, Figure 8 · p. 24

F Preserving query semantics and answer format

• The cases separate query semantics from answer format: both responses denote 6 cm, but only option C passes the recorded scorer.A numerically correct answer can therefore fail when it omits the required option letter.Source: F Preserving query semantics and… · p. 24, E Reading the recorded cases · p. 24, Figure 8 · p. 24

G Absolute input-token costs

• Multi-source task input counts fall approximately 8–15% from Initial skill, while counts rise 15–23% versus Direct on multi-source tasks and 67–108% on single-source tasks.Table 21 reports mean input tokens per item across all requests.Source: Table 21 · p. 24
• Figure 9 compares task-score gains with token changes relative to Initial skill and reports input tokens divided by mean score for continuous metrics.Negative horizontal values indicate token savings; circles mark multi-source tasks and diamonds single-source tasks.Source: G Absolute input-token costs · p. 24, Figure 9 · p. 25
• ViewCraft adds 7.6 seconds per item relative to Direct and generates more output tokens than Initial skill; search cost is amortized over reuse.Table 22 averages deployment cost equally over the eight tasks.Source: G Absolute input-token costs · p. 25, Table 22 · p. 25
• +5.2 points: ViewCraft leads Direct at an identical input-token budget, with positive gains on every task.The reported confidence interval is [+3.6, +6.9].Source: G Absolute input-token costs · p. 25, Table 23 · p. 26

H Gate definitions and round procedure

• Across mechanism ablations, gate evaluations number 208 for full ViewCraft, 238 without replay, 204 without projection, 208 without tiebreak, and 200 for gate-only.The full condition includes 21 projections, 30 inert proposals, and 12 guard failures filtered.Source: Table 24 · p. 26
• The verifier checks support for up to three question-derived atomic requirements against the actual sent views, and reference answers are withheld from extraction and verification.Two annotators audited 100 stratified judgments; the frozen verifier matched consensus on 89%, and removing the tiebreak lowers task scores.Source: H Gate definitions and round procedure · p. 26
• Replay checks rendered changes for visual proposals, while reader-only and answer-text-only proposals bypass the pixel filter; execution checks still apply to every candidate.Pixel-identical rendering edits are rejected before evaluation.Source: H Gate definitions and round procedure · pp. 26–27
• After a task-score loss with nonnegative coverage change, projection can restore the incumbent reader while keeping a candidate’s visual edits, subject to submission and gate allowances.The 21 accepted projection retries count among 204 accepted edits and 750 submitted candidates.Source: H Gate definitions and round procedure · pp. 26–27
• A proposal is accepted when task score improves, or when task score is unchanged and coverage improves; paired evaluations consume gate allowance.The round permits up to four submissions, and pre-evaluation rejections consume no gate units.Source: H Gate definitions and round procedure · p. 27

I A complete evolved skill

• The MMComp skill is stored in SKILL.md, VIEW.yaml, and LAYOUT.yaml, with reasoning clauses assigned to the reader and runtime bindings supplied by the domain configuration.Its reader grounds option nouns and builds a verdict table; the interface colors notes by option and explains their identities.Source: I A complete evolved skill · p. 27

Step 1 — Noun-grounding pre-pass (ALWAYS, before any other reasoning)

• The noun-grounding pre-pass identifies each option’s concrete nouns, checks whether they are visible, and records question polarity before the verdict table is completed.The procedure says this pre-pass prevents over-elimination and polarity flips.Source: Step 1 — Noun-grounding pre-pass… · p. 28

Step 1-V — MANDATORY VERDICT TABLE (for all multi-option questions)

• The verdict table marks each option APPLIES or ABSENT, selects by question polarity, and maps each option to a colored image note at the region found or sought.For all-absent cases, the most specifically false option is selected.Source: Step 1-V — MANDATORY VERDICT TABLE… · p. 28

Step 2 — Initial full read

• The initial full read records panel count, question type, text legibility, and whether relevant content appears inside an embedded screen or framed image.Answer immediately only when full-scale content is unambiguous and the grounding pre-pass resolves the question.Source: Step 2 — Initial full read · p. 29, Step 3 — Decide if one pass is enough · p. 29

Step 3 — Decide if one pass is enough

• When fine detail is needed, the procedure zooms relevant panels, regions, or embedded screens, reruns noun-grounding for an inset, and stops once the answer is clear.It warns against skipping needed zooms or using more crops than necessary.Source: Step 3 — Decide if one pass is enough · p. 29, Anti-patterns to avoid · p. 29

Rendering parameters

• Rendering parameters allow up to two crops per step, and canvas expressions color note rings by option ordinal to make letter-to-region mapping visually checkable.The view contract identifies the composite source image and maps rings to options A–D.Source: Rendering parameters · p. 30, Canvas expression: canvas_expr · p. 30, Request and history parameters · p. 31

Protocol contract: protocol_contract

• The request protocol uses a composite image and an answer contract requiring only the final uppercase option letter, while the protocol contract specifies zoom, note, view, compose, and answer actions.The answer contract keeps reasoning and the verdict table in the note field.Source: Request and history parameters · p. 31, Protocol contract: protocol_contract · p. 31

J Additional Related Work

• Additional related work covers visual personalization, retrieval and memory, perception, editing agents, contextual generation, and diffusion-model optimization.The cited examples include visually grounded agent skills and visual feedback or structured tool plans for editing.Source: J Additional Related Work · p. 31