Seeing the Options: Typed Decisions over Visual Candidates with Reusable Option Memory

1 Introduction

• Existing multimodal System One models accept images in requests but keep candidates as text in letter slots, leaving visual candidates unsupported by their interface.The paper identifies this as a gap for agents selecting screenshot regions, product photos, or charts.Source: §1 · p. 2
• Order, candidate count, null outputs, and reuse are key requirements because visual choices lack canonical order, candidate sets can be large, correct choices may be absent, and image encoding is costly.The paper argues these needs are especially consequential for visual candidates.Source: §1 · p. 2
• Iris extends Jeva to requests and candidates containing text and images, while leaving its decision module unchanged.The paper tests whether the four properties follow from the decision module when the candidate encoder changes.Source: §1 · pp. 2–3
• The paper formalizes typed decisions over visual candidates and presents a benchmark from Multimodal-Mind2Web with screenshot-crop candidates, K=16, and three generalization splits.The benchmark is deduplicated and order-shuffled so candidate position carries no information.Source: §1 · p. 3
• Iris preserves the four properties by construction, and its evidence includes zero answer flips, linear latency through 2,048 image candidates, and reuse speedups within 5% of Proposition 2.The paper also compares independent, joint, and state-conditioned encoding, and reports a systems optimization for the vision tower’s patch embedding.Source: §1 · p. 3

2 Typed decisions over visual candidates

• A typed decision takes a state, question, decision type, and candidate set, and returns a choice distribution, truth probability, or expected score.Iris changes Jeva’s text-only state and candidates to segments that can contain text and images.Source: §2 · p. 3
• Joint letter-slot encoding makes visual candidate sequences costly, order-dependent, and repeatedly re-encoded across requests.At K=16 and about 256 image tokens per candidate, candidates alone occupy 4k tokens.Source: §2 · pp. 3–4
• A bi-encoder permits candidate caching, but its request representation must anticipate options for competitive closed-set accuracy, and cosine scores provide no null probability.The section contrasts this bottleneck with Jeva’s decision block, which reads the candidate set through attention.Source: §2 · p. 4
• Runoff removes order dependence at inference time but re-encodes candidates with the state and has no null output; the cited letter-slot image reranker is capped at 16 candidates.The section notes that the reranker re-encodes candidates per request and does not measure order dependence.Source: §2 · pp. 4–5
• The four requirements are an explicit null output, order invariance, linear cost without a fixed candidate limit, and candidate computation cacheable across requests.The section states that Jeva’s construction satisfies these requirements for segments without changing its decision path.Source: Figure 1 · p. 4, §2 · p. 5

3.1 Encoder: the vision tower Jeva discards

• The vision-language backbone encodes mixed segments, with separate state and candidate templates and last-token pooling into vectors.The candidate template contains the decision type and candidate segment only, without the question, state, index, or other candidates.Source: §3.1 · p. 5

3.2 Decision module (unchanged)

• Jeva’s decision stack updates only the state as it attends over fixed candidate vectors, then combines candidate scores with a coverage gate to produce choice probabilities including a null option.For choice, the coverage gate assigns probability to none of the above; the other decision types read the conditional distribution or its expectation.Source: §3.2 · p. 6

3.3 Guarantees carry over

• For any per-candidate encoder, the decision stack preserves order invariance and exact reuse of previously computed candidate entries.The stated guarantee applies to arbitrary segments and fixed decision-stack weights.Source: §3.3 · p. 6

3.4 Candidate memory for images

• Candidate memory entries are keyed by rendered text and image-byte digests, allowing identical content to share entries across sources.The hybrid cache assembles a candidate set from cached entries and encodes only missing candidates.Source: §3.4 · p. 7

3.5 Two candidate encoders: independent and state-conditioned

• State-conditioned encoding lets each candidate see the request state but not other candidates, preserving order invariance while giving up cross-request candidate reuse.A forked state cache shares the state pass across candidates and questions about the same state.Source: §3.5 · pp. 7–8

3.6 Patch embedding as a matrix product

• Replacing the patch-embedding convolution with an equivalent matrix product reduces a 448×448 image pass from 5.2 s to 15 ms on an A800.The reported reduction is 350×, with a maximum absolute difference of 2×10^-6.Source: §3.6 · p. 8

3.7 Training

• Iris-4B training uses two stages: multimodal state training on OneJev-Data, followed by candidate-side training on 7.7k Mind2Web steps with crop candidates.Only the second stage trains with image candidates.Source: §3.7 · pp. 8–9

4 A visual-candidate benchmark from Mind2Web

• Mind2Web retains Jeva’s K=16 and three test splits, replacing each HTML candidate with a padded screenshot crop and short tag-and-attribute label.Crops use 6 px padding, a minimum side of 48 px, and a maximum side of 336 px.Source: §4 · p. 9
• Deduplication keeps positives among identical candidates, then per-record shuffling makes the gold position uniform; positives absent from cleaned HTML remain null targets.The resulting train / cross-task / cross-website / cross-domain sets contain 7,751 / 1,337 / 1,017 / 4,042 steps, respectively.Source: §4 · p. 9
• Full-page cross-task pools contain 658,639 candidates across 1,337 steps; 55% are previously shown elements, and within-task reuse reaches a median of 61%.The pool has a median of 353 and a maximum of 3,270 candidates per step; 298,328 crops are distinct.Source: §4 · p. 9
• The same-data comparison uses 94,707 OneJev training rows, holds out 2,000, and includes multimodal states, null detection, and noul/score types.The released rows comprise 34,622 noul, 45,507 choice, and 14,578 score examples; 65,643 contain media.Source: §4 · p. 10

5.1 Exact equivalence on text; 5.2 Zero-shot transfer to visual candidates

• Iris is bit-for-bit equivalent to Jeva on text; without image training, it reaches 31.11 / 25.07 / 32.73% on visual candidates and has zero flips across 6,396 records.The zero-shot results exceed the 6.25% chance level; OneJev changes answers on 51–54% of records under reordering.Source: §5.1 · p. 11, §5.2 · p. 11

5.3 Training on visual candidates

• Independent encoding is exactly stable, with zero flips across all 6,396 records; retrained OneJev-4B reaches 90.58 / 89.48 / 91.59% after 7,751 crop steps.Training shrinks OneJev’s flip rate from 50–54% to 7.6 / 10.1 / 8.0%.Source: §5.3 · p. 12

5.4 State-conditioned candidates

• 92.07 / 87.02 / 90.20% is the released state-conditioned model’s accuracy across the three Mind2Web splits, with zero flips and a 3.0- and 1.4-point gap to retrained OneJev on the generalization splits.Its abstention margin was selected on cross-task, so that split’s score is a selection-split result.Source: §5.4 · p. 14

5.5 What the remaining gap is not

• Set attention, token-level reads, and multi-vector memory do not improve positive-step accuracy beyond split noise; candidate-vector collapse leaves visually different crops nearly indistinguishable to pooled readouts.On wrong cross-website records, the gold and hardest-negative vectors have cosine 0.999; the authors identify encoder representation collapse, not missing interaction, as the remaining gap.Source: §5.5 · pp. 14–15

5.6 Typed decisions: JevBench and ImaJev-Bench

• Zero-shot Iris scores 100% on JevBench original and easy but trails larger models by 13–15 points on hard and ImaJev-Bench; Mind2Web-only candidate training also degrades text-side competence.Replaying 10,000 OneJev-Data rows partly restores performance on hard and ImaJev-Bench, while the JevBench text-only rows are not protected by that replay.Source: §5.6 · p. 16

5.7 Reranking retrieval candidates: MMEB-V2

• Retrieval tuning raises mean MMEB-V2 hit@1 to 40.3, but image-only or composed-query tasks remain at 4–6%, below the 6.25% uniform-guess level.The tuned model does not reach its own first-stage retriever on any task; independent candidate vectors cannot encode a comparison with an image query.Source: §5.7 · p. 17

5.8 The released checkpoints; 5.9 Null detection and typed outputs

• Conditioned models rank null steps well (AUROC 0.92–0.97), while null oversampling raises abstention to 77–79%; the released independent and conditioned checkpoints serve different candidate-reuse settings.On held-out typed outputs, the released mixture model returns part of stage 1’s performance on choice accuracy, noul MAE, and score MAE.Source: §5.8 · p. 17, §5.9 · pp. 18–19

5.10 Scaling and reusing image candidates

• At full candidate reuse, measured speedup grows from 3.6× at K=16 to 37× at K=256 and 191× at K=2,048; the sampled workload does not establish real-world per-site reuse rates.The crop-to-label encoding-cost ratio is 2.4–3.2 for these small crops, and practical speedup depends on the reuse ratio.Source: §5.10 · pp. 19–20

System One models

• ReFlex and AnyJev aggregate predictions across candidate orders, while Runoff [8] uses pointwise consideration sets; these approaches address order dependence with added costs or no null output.The discussion also notes that decision fine-tuning alone does not remove order dependence.Source: §6 · p. 20
• Iris inherits Jeva’s decision module and is presented as the first System One model to admit visual candidates.The related-work discussion contrasts this with multimodal System One models that keep candidates textual.Source: §6 · pp. 20–21

Multimodal System One models

• Existing multimodal System One models accept images or video in requests, but keep candidates textual; Liu et al. [9] rerank images through letter slots.MetaEncoder [4] is the closest comparison with image candidates, but uses cosine similarity and lists options in the request to recover closed-set competence.Source: §6 · p. 21

Retrieval and set functions

• Iris combines independently precomputed candidates with candidate interaction in the decision module, drawing on retrieval encoders and permutation-invariant set functions.The related-work section distinguishes this from standard dense retrieval, which precomputes candidates but does not add Jeva’s decision-block interaction.Source: §6 · p. 21

Web agents

• Unlike DOM-ranking or page-grounding approaches, Iris treats screenshot crops as candidates in a cacheable, order-invariant typed decision.The stated application is web-agent element selection.Source: §6 · p. 21

Modality and encoder

• Iris handles images only; video is reduced to stills, and audio is not addressed.Whether tuning the frozen vision tower helps is untested.Source: §7 · p. 21

Training and evaluation

• Mind2Web does not ablate stage 1, the crop’s text label, or pixel budget; its one pixel-budget comparison was uninformative because images were already below budget.The vision tower is frozen, so the reported gains come from language-model and decision-module adaptation to visual candidate vectors.Source: §7 · p. 21

Data and comparisons

• Mind2Web has few negative steps, so visual-candidate null detection is only coarsely measured; typed metrics use OneJev-Data.Jeva’s curriculum data are not public, and Iris is trained from the released Jeva-4B checkpoint; Jeva text-candidate results are not a controlled comparison because crops and HTML carry different information.Source: §7 · p. 21

8 Conclusion

• Iris preserves the decision module’s visual-candidate guarantees, with zero flips, linear latency through 2,000 candidates, and reuse speedups within 5% of the closed form.The conclusion contrasts zero flips with 8–54% for the joint-encoding alternative.Source: §8 · p. 21
• Independently encoded crops trail a same-step joint encoder by 12–17 points; encoding candidates after the request recovers all of the gap on one split and most on two others.The state-conditioned design preserves the guarantees except cross-request reuse, while pooled candidate-interaction readouts recover none of the remaining gap because the encoder collapses candidates to one point.Source: §8 · p. 22
• The conclusion recommends encoding candidates once when they exist before the request, and letting them read the request when each request brings its own candidates.It locates the design boundary at pooling a candidate to a vector, rather than at whether candidates can compare with one another.Source: §8 · p. 22

• The repository includes benchmark construction, deduplication, and shuffling scripts, plus training configurations and evaluation outputs.Source: Reproducibility · p. 22
• Iris uses Jeva’s decision module verbatim.Source: Reproducibility · p. 22
• The checkpoint initialization script reproduces the exact-equivalence check described in §5.1.Source: Reproducibility · p. 22

• Table 8 lists the settings; everything not listed uses Jeva-4B defaults.Source: Table 8 · p. 23

• Each crop uses the element’s bounding box, 6 px padding, a 48 px minimum side, and a 336 px maximum side after downscaling.Source: B Benchmark construction details · p. 23
• Labels combine the tag with the first non-empty listed attribute, truncated to 60 characters; negatives come from step-level candidates using a seeded random generator.Source: B Benchmark construction details · p. 23
• 109 candidates were removed as duplicates across 103 of 1,337 cross-task steps; cross-website and cross-domain totals were 89 across 79 of 1,017 and 400 across 330 of 4,042.The training set had 714 duplicates across 603 of 7,751 steps; byte-identical candidates receive identical probabilities, so duplicates are removed from saved distributions and the result renormalized.Source: B Benchmark construction details · p. 23
• Steps were dropped for missing screenshots or gold crops outside screenshot bounds, with one cross-task and ten cross-domain missing-screenshot cases.Between one and eight out-of-bounds gold crops were dropped per split.Source: B Benchmark construction details · p. 23
• 7.3 / 6.9 / 7.0% of positive cross-task / cross-website / cross-domain steps place the gold candidate first after seeded shuffling, against a 6.25% uniform expectation.Records with fewer than 16 candidates after deduplication raise the first-position rates slightly.Source: B Benchmark construction details · p. 24

C.1 Abstention margin

• The model predicts ⊥ when its probability is at least the largest candidate probability plus δ; Table 9 sweeps δ on cross-task and applies the selected value to other splits.The rule is applied identically in every candidate order, preserving order invariance.Source: §C.1 · p. 24, Table 9 · p. 24

C.2 Training trajectory of the state-conditioned model

• Every 50 optimizer steps, a checkpoint trained on 32-row batches was evaluated on cross-task in original candidate order; positive and null accuracy measure distinct step subsets.Positive accuracy is measured when the gold element is among candidates; null accuracy is the share of null steps correctly abstained on.Source: §C.2 · p. 25
• From 1.2 epochs onward, positive-step accuracy is stable, while total accuracy is driven by volatile abstention on 81 null steps.Figure 8 plots cross-task total accuracy, positive-step accuracy, and correct abstention on the null steps.Source: Figure 8 · p. 24, Table 10 · p. 25, §C.2 · p. 25