Neos: Learning to Interleave Search and Visual Reasoning

• Many systems fail to use visual evidence consistently during search, while training tasks can permit textual shortcuts or lack intermediate visual dependencies.The paper identifies ambiguous process credit and fixed evidence interfaces as additional limitations.Source: §1 · p. 2
• Neos alternates policy reinforcement learning with MEI supervised fine-tuning so evolving evidence needs shape goal-conditioned observations and subsequent policy learning.The lightweight Multimodal Evidence Interface transforms retrieved content into observations.Source: §1 · p. 2
• Evidence-dependent synthesis creates tasks in which newly acquired visual evidence is necessary for subsequent search decisions.Source: §1 · p. 2
• Reference-guided credit assignment uses verified successful trajectories to localize critical policy errors while preserving positive credit for productive decisions in failed searches.Source: §1 · p. 2
• Across ten benchmarks, Neos-9B and Neos-35B outperform their respective base workflows by 17.4 and 15.5 points on average, achieving leading open-source multimodal deep-search performance.Source: §1 · p. 2

• Visual interaction methods add inspection, manipulation, grounding, or computation to reasoning, while multimodal deep search must acquire and verify external evidence iteratively.Examples include image manipulation [Zheng et al., 2026], question-guided region selection [Zhang et al., 2025a, Huang et al., 2026c], and visual computation [Zhao et al., 2025a, Zhang et al., 2025b].Source: §2 · p. 3
• Multimodal deep-search agents use multi-turn evidence acquisition, process-oriented rewards, and scalable task or trajectory supervision [Wu et al., 2025, Geng et al., 2026b, Yan et al., 2026b, Chen et al., 2026].Data approaches include graph-based construction [Chu et al., 2026, Yao et al., 2026] and rollout-feedback adaptation [Huang et al., 2026a].Source: §2 · p. 3
• Prior agents vary in evidence handling, including on-demand image acquisition [Du et al., 2026], reusable visual state [Zhang et al., 2026b], and webpage summarization with thumbnail verification [Dai et al., 2026].Source: §2 · p. 3

3.1 Problem Setting and Agent Workflow

• Neos frames deep search as policy interaction with a tool environment over a question and initial images, updating history through action–observation pairs until a final answer.Source: §3.1 · p. 3

3.2 Multimodal Evidence Interface

• A lightweight, trainable MEI conditions retrieved material on the policy’s local goal, selecting and summarizing evidence as ordinary tool observations while the agent retains control of queries and visual inspection.Source: §3.2 · pp. 3–4, §3.1 · p. 3

3.3 Evidence-Dependent Task Synthesis

• Tasks require intermediate visual checkpoints for later relational inference; structural identifiability, blind grounding, rewriting, and four shortcut/difficulty probes filter questions before verified trajectories are curated.Source: §3.3 · pp. 4–5

3.4 Reference-Guided Credit Assignment

• Verified successful trajectories guide turn-level credit: critical errors and downstream effects are penalized, while productive or corrective decisions in failed searches retain positive credit.Credit is rescaled so each assessed failure remains negative in aggregate, while successful trajectories retain uniform advantages.Source: §3.4 · p. 6

3.5 Agent–Environment Co-evolution

• Neos alternates policy updates with MEI fine-tuning on retained online calls, so evolving evidence requests shape MEI training and revised observations reshape subsequent decisions.Correct trajectories contribute valid MEI calls; failed trajectories contribute calls labeled productive or corrective.Source: §3.5 · pp. 6–7

4.3 Ablation Studies

• Combining synthetic search, REDSearcher-MM, and VQA training sources reaches 60.4, 44.7, and 79.7 on BrowseComp-VL, MMSearch-Plus, and FVQA.The 1K synthetic-example setting scores 58.4, 43.2, and 78.3 on those benchmarks, exceeding the REDSearcher-MM variant by 1.5, 4.3, and 1.0 points.Source: Table 2 · p. 8, §4.3 · p. 8
• Alternating policy RL with MEI SFT improves over a fixed co-evolved MEI by 0.7, 0.7, and 1.0 points on MMSearch-Plus, BrowseComp-V3, and VDR-Bench.Frequent MEI updates with small K are unstable; larger K stabilizes performance, but gains are neither monotonic nor linear.Source: Table 3 · p. 8, §4.3 · pp. 9, 11, Figure 7 · p. 11

4.2 Main Results

• Neos leads specialized multimodal deep-search agents at both scales; Neos-35B scores 52.7 on MMSearchPlus, 48.3 on MM-BrowseComp, and 47.0 on BrowseComp-V3.These exceed the strongest prior specialized-agent results by 12.1, 17.8, and 12.0 points, respectively; Neos-9B and Neos-35B gain 17.4 and 15.5 points over their base workflows.Source: §4.2 · p. 8

4.3 Ablation Studies

• With the Neos-9B policy fixed, its co-evolved MEI beats the Qwen3.5-9B MEI baseline by 4.5, 5.1, and 6.6 points on MM-BrowseComp, BrowseComp-VL, and VisBrowse-Bench.It outperforms Qwen-3.7-Flash, GPT-5.6-Luna, and GPT-5-nano on all three benchmarks, remaining within 1.8 points of Gemini-3.7-Flash on each.Source: Table 4 · p. 9, §4.3 · p. 9
• The full reference-guided credit method scores 63.2, 39.5, and 83.3 on BrowseComp-VL, MM-BrowseComp, and FVQA, exceeding outcome-only GRPO by 3.6, 4.2, and 5.0 points.First-error penalties improve on productive-only credit, and downstream attribution and corrective credit then yield further improvements across all three benchmarks.Source: Table 5 · p. 9, §4.3 · p. 9
• Search horizons vary: MMSearch and VQA benchmarks concentrate in the first ten turns, while other benchmarks span broader horizons; reaching the 50-turn limit alone indicates neither success nor failure.At the limit, trajectories are predominantly correct on LiveVQA and FVQA but predominantly incorrect on MM-BrowseComp, BrowseComp-V3, and VDR-Bench.Source: Figure 5 · p. 10, §4.3 · p. 10
• Across RL update windows, TextSearch falls from 59.7% to 44.5% of calls as WebVisit rises from 21.2% to 32.1%, while calls per trajectory drop 32.0%.Removing any tool hurts all three tool-ablation benchmarks: ReverseImageSearch is most critical on SimpleVQA (71.7 to 58.0), while FetchImage causes the largest drops on BrowseComp-V3 and MM-BrowseComp (3.7 points each).Source: Figure 6 · p. 10, §4.3 · p. 10, Table 6 · p. 11
• Neos improves on REDSearcher-MM across all nine model–benchmark pairs, with average gains of 12.9, 9.0, and 7.8 points for GPT-5.6, Qwen3.5-35B-A3B, and Qwen3.5-9B.Qwen3.5-9B with Neos surpasses Qwen3.5-35B-A3B with REDSearcher-MM on MM-BrowseComp and MMSearch-Plus.Source: Table 7 · p. 11, §4.3 · p. 11

5 Conclusions

• Neos sustains an evidence–decision loop by learning evidence acquisition and evidence-conditioned decision making, with MEI co-adaptation and reference-guided credit assignment.The framework alternates policy RL with MEI SFT, using rollout outcomes to supervise both components.Source: §5 · p. 11

A.1 Details of Our Data Pipeline

• Synthetic questions require newly acquired visual evidence during later search, with intermediate images resolving entities and target images supplying requested visual attributes.The pipeline constructs relational reasoning graphs, derives and verifies programs with explicit visual dependencies, grounds them in online images, and curates successful trajectories.Source: §A.1 · p. 16
• Seed selection favors entity neighborhoods with diverse relations and image-associated neighbors, while downweighting highly connected hubs and generic or taxonomic neighborhoods.Successful visual grounding and resistance to textual shortcuts are assessed downstream.Source: §A.1 · p. 16
• Relational prerequisites are recursively expanded into a directed acyclic graph whose edges point from prerequisites to the entity they identify.Neighboring entities and relations are selected to distinguish each expanded entity within the knowledge-graph snapshot.Source: §A.1 · p. 16
• Visual checkpoints make a grounded entity feed a subsequent relation or make a retrieved target image provide the requested answer property.Programs are checked against the complete local knowledge-graph snapshot for a unique terminal entity conditional on planned visual bindings.Source: §A.1 · p. 16
• Grounding uses blind image descriptions, separate crop observations, and entity matching against candidates including distractors; unsupported or unresolved matches are rejected.For visual answers, both full-image and crop observations must support the requested attribute; the solver receives only the question and entry image and must acquire later images during search.Source: §A.1 · pp. 16–17
• Candidates are rejected if text-only answering, text-only search, entry-image VQA, or short multimodal search answers them correctly within four tool turns.Surviving questions receive long-horizon expert searches, and an independent judge selects successful trajectories for SFT and training-side success references.Source: §A.1 · p. 17

A.2 Training Data

• The pipeline produces approximately 2K synthetic search instances with verified successful trajectories and retains the top 60% of REDSearcher-MM SFT data by interaction-turn length.It also samples approximately 3.7K candidate questions from FVQA, LiveVQA, InfoSeek, and VisualProbe, generating new Neos-workflow trajectories and retaining successful ones through rejection sampling.Source: §A.2 · p. 17
• For FVQA and LiveVQA external questions, sampled instances come from official training splits and have 0% overlap with evaluation benchmarks after deduplication.The VQA mixture draws from FVQA, LiveVQA, InfoSeek, and VisualProbe.Source: §A.2 · p. 17, Table 8 · p. 18
• The SFT dataset contains 2,898 trajectories: 989 synthetic, 1,177 from REDSearcher-MM questions, and 732 from the VQA mixture.The remaining 973 synthetic instances are reserved for RL; the SFT data combines interleaved multimodal search demonstrations with broader visual question answering.Source: §A.2 · p. 17, Table 8 · p. 18
• The RL dataset contains 2,572 questions: 973 reserved synthetic instances plus 1,599 questions selected after cold-start policy rollouts.Selected questions were answered correctly in zero or one of three rollouts; the 1,599 comprise 599 synthetic, 500 REDSearcher-MM, and 500 VQA-mixture questions.Source: §A.2 · p. 17, Table 8 · p. 18

B Engineering Techniques in Agentic RL

• Turn-level asynchronous execution overlaps model generation with tool execution and judging while keeping policy and MEI versions fixed within each rollout round.Requests are scheduled at turn boundaries without binding a trajectory permanently to one replica.Source: B Engineering Techniques in Agentic RL · p. 18
• Adaptive mixed-group sampling replenishes groups with the fewest completed samples and prioritizes groups with homogeneous outcomes until mixed-outcome groups are available.It selects exactly BG trajectories with a nondegenerate group-relative learning signal, without enforcing a balanced success/failure ratio.Source: B Engineering Techniques in Agentic RL · p. 18
• Failure credit assessment overlaps remaining rollout work, but only assessments attached to selected trajectories affect optimization; malformed or timed-out assessments fall back to ordinary GRPO.Asynchrony remains within a rollout round rather than using stale trajectories across policy versions.Source: B Engineering Techniques in Agentic RL · pp. 18–19
• Neos-9B rollout time stays around 80–90 minutes per update, while policy optimization stabilizes near 6 minutes after the first update.Selected-batch reward, surrogate objective, gradient norm, and PPO likelihood-ratio proxy show noisy-but-bounded dynamics without divergence or instability.Source: Figure 8 · p. 19, B Engineering Techniques in Agentic RL · p. 19

C.1 Benchmarks and Evaluation Splits

• Three factual visual question-answering benchmarks use the same 300-example subsets released with Vision-DeepResearch.LiveVQA emphasizes up-to-date visual knowledge, while FVQA includes both search-required and search-free questions.Source: §C.1 · p. 19
• Multimodal search and browsing evaluation includes MMSearch, MMSearch-Plus, BrowseComp-VL, and MM-BrowseComp.The evaluation uses all 300 MMSearch queries, all 311 MMSearch-Plus questions, 399 BrowseComp-VL questions across both difficulty levels, and the expanded 400-question MM-BrowseComp release.Source: §C.1 · p. 19
• Visual-native, verifiable search uses complete evaluation sets for BrowseComp-V3 and VisBrowse-Bench, plus VDR-Bench’s official 500-example testmini subset.BrowseComp-V3 tests cross-modal, multi-hop browsing with verifiable reasoning; VisBrowse-Bench emphasizes inspecting newly discovered web images.Source: §C.1 · p. 20

C.2 Baselines and Evaluation Protocol

• Baselines comprise general-purpose MLLMs with and without tool access and specialized multimodal search agents.Direct-answer models receive the question and available images without external tools, testing supplied-input understanding and parametric factual reasoning.Source: §C.2 · p. 20
• In the agentic workflow, general-purpose models receive up to 50 interaction turns without additional search-specific training.Comparison with direct answering measures the benefit of external evidence acquisition, while comparison with Neos assesses gains from specialized training.Source: §C.2 · p. 20
• Specialized-agent baselines span model scales and multimodal search-training approaches, retaining prior-publication coverage and settings where results are reused.Unreported benchmark scores are marked with “–”; answer correctness is judged with Qwen3.7-Flash using the Vision-DeepResearch prompt across all ten benchmarks.Source: §C.2 · p. 20

C.3 Training and Implementation

• Search infrastructure combines Google text search and image retrieval, Google Lens reverse-image search, and Jina Reader webpage access.FetchImage and CropImage expose selected images and regions as pixels in an episode-local visual register; hierarchical caching reduces repeated external calls and transfers.Source: §C.3 · p. 21
• Qwen3.5-9B and Qwen3.5-35B-A3B are policy models, while MEI remains Qwen3.5-9B; vision encoders and multimodal mergers stay frozen during training.Cold-start policy SFT uses 2.9K examples for two epochs, and policy targets include reasoning, tool calls, and final answers.Source: §C.3 · p. 21, Table 11 · p. 21
• Co-evolution runs policy RL and online MEI SFT on separate GPU groups, with MEI updates activated between rollout batches.The setup uses 32 H100 GPUs for policy RL and 8 for MEI serving and SFT; MEI training retains valid interactions from correct trajectories and productive or corrective turns from selected failures.Source: §C.3 · p. 21, Table 12 · p. 22

D Supplementary Training Details

• The supervised objective trains the policy or MEI on selected target tokens, excluding conditioning content and padding from the loss.Policy targets are reasoning, tool calls, and final answers; MEI targets are structured evidence responses conditioned on local goals and retrieved material.Source: D Supplementary Training Details · p. 22
• For assessed failures with negative credit mass, the target margin is set from the baseline and token count, with amplification capped by M.Severity and confidence lie in [0, 1]; the method also specifies bounds for the other credit parameters.Source: D Supplementary Training Details · pp. 22–23
• Failure advantages are rescaled so their sum remains negative, with each advantage bounded; successful trajectories retain their original advantages.The target margin is enforced only when feasible, and the original advantages also remain when no valid critical-error assessment exists or negative credit mass is zero.Source: D Supplementary Training Details · p. 23
• The policy objective uses a dual-clipped surrogate, and judge assessments and adjusted advantages are detached from policy optimization.The clipping parameter is specified as greater than 1.Source: D Supplementary Training Details · p. 23

E Additional Ablation Studies.

• The training-stage ablation supports complementary roles for SFT initialization and RL refinement in learning the evidence–decision loop.The discussion attributes a structured starting behavior to cold-start SFT and on-policy refinement to RL; it presents reduced rollout exploration and training compute as a plausible mechanism, not a measured result.Source: E Additional Ablation Studies · p. 23, Table 13 · p. 23
• Neos-9B improves over the base Qwen3.5-9B workflow by an average of 11.1 points across four text-only benchmarks.The reported gains are 12.2 on BrowseComp, 9.7 on BrowseComp-ZH, 11.9 on HLE-Text, and 10.6 on DeepSearchQA; Table 14 covers these text-only benchmarks.Source: E Additional Ablation Studies · p. 24, Table 14 · p. 24
• During RL, corrective turns move earlier, while the share of credited failure turns attributed to downstream consequences declines from 66–70% to 56–59%.Figure 9 tracks credit-type median positions and composition in five-step windows; productive turns remain at turn 4 and corrective-turn share rises from 3.2% to 4.8%.Source: E Additional Ablation Studies · p. 24, Figure 9 · p. 24

F Prompts

• The policy prompt specifies interleaved retrieval and selective visual observation, with tools disabled on the final turn.The prompt materials include native tool schemas, a user-message template, and a 50-turn limit; discovered images become addressable through reusable identifiers.Source: Figure 10 · p. 25, F Prompts · p. 25, Figure 11 · p. 26
• MEI prompts turn webpage content, image-search candidates, and reverse-image comparisons into goal-conditioned evidence or visual-match judgments.WebVisit uses extracted text, a local goal, and page-image references; ImageSearch and ReverseImageSearch prompts also use candidate pixels or query images.Source: Figure 15 · p. 29, Figure 16 · p. 30, Figure 17 · p. 30

G Visualization of Interleaved Multimodal Search

• The examples depict visual observations guiding later retrieval, with action strips naming the search, visit, image, reverse-search, fetch, crop, and answer actions.Earlier evidence can also be revisited after several retrieval turns.Source: G Visualization of Interleaved… · pp. 33, 37
• One constructed question combines a newly retrieved card with two visual branches whose geographic and emblem clues converge on Yasuto Honda.The reference trajectory checks both biographical relations before answering.Source: G Visualization of Interleaved… · pp. 33–34, Figure 22 · p. 34
• A mural and its town-name crop redirect the search from Cedar Falls to Mount Vernon, guiding later searches that identify George E. Petrick.The initial path followed a journal’s publication place.Source: Figure 23 · p. 35, G Visualization of Interleaved… · p. 35
• Other examples inspect a concert photograph for recorder controls, check hotel interiors using a late crop, and compare umbrella and scarf regions in a campaign image.The examples span MMBC, MMSearchPlus, and VisBrowse.Source: Figure 24 · p. 36, Figure 25 · p. 37, Figure 26 · p. 38
• The 26-turn BC-V3 example combines tram and shipyard clues to target a painting, then revisits acquired pixels amid conflicting descriptions and failed downloads.The question asks for the direction of the horses in the central group.Source: Figure 27 · p. 39, G Visualization of Interleaved… · pp. 39–40, Figure 28 · p. 40
• The 28-turn VDR example reuses a foreground crop to distinguish similar species, then seeks conservation evidence after an uninformative page.A conservation source gives Calochortus pulchellus a California Rare Plant Rank of 1B.2 and identifies Alameda and Contra Costa counties as its range.Source: Figure 29 · p. 41, G Visualization of Interleaved… · pp. 41–42, Figure 30 · p. 42