PaperlayerView exact revision on arXiv

Source-linked AI summary

DLLM-Searcher: Adapting Diffusion Large Language Model for Search Agents

Jiahao Zhao, Shaoxuan Xu, Zhongxiang Sun, Fengqi Zhu, Jingyang Ou, Yuling Shi, Chongxuan Li, Xiao Zhang, Jun Xu

arXiv:2602.07035v1cs.AIcs.LG

TL;DR

Search agents face serial ReAct latency, while dLLMs lack the reasoning and tool-calling abilities needed for practical deployment. DLLM-Searcher combines Agentic SFT, Agentic VRPO, and P-ReAct to improve agent capability and prioritize tool-call decoding. It achieves comparable performance to mainstream LLM-based search agents and approximately 15% acceleration over ReAct.

  • Problem

    Serial ReAct execution creates a latency bottleneck, while dLLMs remain weak in complex reasoning and strict tool-call format following.

  • Method

    DLLM-Searcher combines Agentic SFT and Agentic VRPO with P-ReAct, which prioritizes tool-call decoding for parallel reasoning and action.

  • Results

    Approximately 15% inference acceleration over conventional ReAct is achieved while performance remains comparable to mainstream LLM-based search agents.

  • Takeaways & Limitations

    DLLM-Searcher enables dLLMs to keep thinking while waiting during external tool execution.

Abstract

from arXiv · show

Recently, Diffusion Large Language Models (dLLMs) have demonstrated unique efficiency advantages, enabled by their inherently parallel decoding mechanism and flexible generation paradigm. Meanwhile, despite the rapid advancement of Search Agents, their practical deployment is constrained by a fundamental limitation, termed as 1) Latency Challenge: the serial execution of multi-round reasoning, tool calling, and tool response waiting under the ReAct agent paradigm induces severe end-to-end latency. Intuitively, dLLMs can leverage their distinctive strengths to optimize the operational efficiency of agents under the ReAct agent paradigm. Practically, existing dLLM backbones face the 2) Agent Ability Challenge. That is, existing dLLMs exhibit remarkably weak reasoning and tool-calling capabilities, preventing these advantages from being effectively realized in practice. In this paper, we propose DLLM-Searcher, an optimization framework for dLLM-based Search Agents. To solve the Agent Ability Challenge, we design a two-stage post-training pipeline encompassing Agentic Supervised Fine-Tuning (Agentic SFT) and Agentic Variance-Reduced Preference Optimization Agentic VRPO, which enhances the backbone dLLM's information seeking and reasoning capabilities. To mitigate the Latency Challenge, we leverage the flexible generation mechanism of dLLMs and propose a novel agent paradigm termed Parallel-Reasoning and Acting P-ReAct. P-ReAct guides the model to prioritize decoding tool_call instructions, thereby allowing the model to keep thinking while waiting for the tool's return. Experimental results demonstrate that DLLM-Searcher achieves performance comparable to mainstream LLM-based search agents and P-ReAct delivers approximately 15% inference acceleration. Our code is available at https://anonymous.4open.science/r/DLLM-Searcher-553C

1 Introduction

DLLM-Searcher addresses the latency and agent-capability challenges of dLLM-based search agents with two-stage post-training and P-ReAct. It achieves comparable performance to mainstream LLM-based search agents while delivering approximately 15% acceleration over ReAct.

  • Serial ReAct execution accumulates delays from reasoning, tool-call generation, and waiting for tool responses.
  • Vanilla dLLMs often fail strict tool-calling formats and trail autoregressive models in reasoning-intensive agentic scenarios.
  • P-ReAct guides dLLMs to decode tool-call instructions first, enabling parallel reasoning while waiting for tool responses.Its inference process uses tool-call token pre-filling and confidence biasing.
  • Agentic SFT and Agentic VRPO form a two-stage post-training pipeline for improving dLLM information seeking, reasoning, and retrieval capabilities.Agentic SFT uses multi-hop trajectories, while Agentic VRPO refines reasoning and retrieval using filtered post-SFT rollouts.
  • Approximately 15% acceleration is achieved over the ReAct paradigm while performance remains comparable to mainstream LLM-based search agents.

2 Related Work

Prior work establishes dLLMs and search agents as promising foundations, but dLLMs remain limited in complex reasoning and agentic tasks while ReAct retains a serial latency bottleneck. DLLM-Searcher combines agentic post-training with parallel reasoning and action to address these limitations.

  • Diffusion Language Models: dLLMs use parallel decoding and flexible generation as alternatives to autoregressive left-to-right decoding.
  • Diffusion Language Models: Despite architectural advantages and competitive mathematical performance, dLLMs lag autoregressive models in complex reasoning and agentic tasks.
  • Search Agents: Search-agent research uses tool calling and post-training to improve retrieval-grounded reasoning and generation quality.
  • Search Agents: Serial ReAct execution makes latency a critical bottleneck by forcing reasoning, tool calling, and response waiting to occur sequentially.
  • Search Agents: DLLM-Searcher breaks ReAct’s serial mechanism by leveraging dLLMs’ flexible generation for parallel reasoning and action execution.

3 Preliminary

The preliminary section describes diffusion language modeling through forward corruption and reverse prediction, then formalizes ReAct search agents as iterative thought, action, and observation processes. These foundations define the generation and interaction mechanisms targeted by DLLM-Searcher.

  • 3.1 Diffusion Large Language Models: dLLMs progressively mask a clean sequence in a forward diffusion process and predict masked tokens in reverse.The reverse model uses p_θ(·|y_t,x) to estimate original token values.
  • 3.1 Diffusion Large Language Models: For block diffusion models, the input is partitioned into K continuous blocks, each of length B.
  • 3.2 Search Agents with ReAct: Under ReAct, an agent receives a system prompt and query, then iterates through thought, action, and observation.
  • 3.2 Search Agents with ReAct: At each iteration, the agent generates a thought, executes a parsable action, waits for an observation, and updates its context.
  • 3.2 Search Agents with ReAct: The complete agent trajectory consists of repeated interaction iterations until the agent selects a final answer action.

4.1 Overview

DLLM-Searcher consists of a two-stage post-training pipeline and the P-ReAct agent paradigm. The pipeline improves agentic capabilities, while P-ReAct prioritizes tool-call decoding to support parallel reasoning and action.

  • DLLM-Searcher combines a two-stage post-training pipeline with the P-ReAct agent paradigm.
  • Two-stage post-training pipeline: Agentic SFT uses teacher-generated trajectories to improve tool-call format following and combine information retrieval with reasoning.
  • P-ReAct agent paradigm: P-ReAct pre-fills tool-call boundary tokens and applies positional confidence bias between them during subsequent decoding.
  • P-ReAct agent paradigm: P-ReAct encourages tool-call instructions to be generated ahead of thinking with near-perfect controllability.

4.2 Agentic SFT

Agentic SFT adapts dLLM training to search-agent trajectories by using filtered teacher data, block-aware noising, and an Agentic ELBO focused on reasoning and tool calls.

  • Existing dLLMs lack the multi-step reasoning and tool-calling abilities required in Search Agent settings.
  • Teacher trajectories are filtered for correct answers, complete reasoning, and strictly valid tool-call formats before training.
  • Block Attention uses bidirectional attention within blocks and causal attention across blocks, while concatenating clean and noised trajectories for one forward computation.
  • Agentic Noising injects noise into think and tool_call components while masking tool_response tokens to prevent train–inference mismatch.
  • The training objective uses Agentic ELBO to estimate log π_θ(y|x), with loss contributions restricted to masked non-tool-response positions.

4.3 Agentic VRPO

Agentic VRPO extends the SFT model with preference optimization using P-ReAct rollouts and Agentic ELBO-based policy and reference comparisons.

  • Agentic VRPO further improves reasoning and information retrieval abilities while adapting the model to the P-ReAct paradigm.
  • The objective estimates policy and reference probabilities with Agentic ELBO, and ΔL measures the policy’s Agentic ELBO advantage over the reference model.
  • The method constructs preference pairs from two P-ReAct rollouts where both trajectories follow the tool-call format but only one produces a correct answer.
  • Agentic Noising is retained so tool responses do not disturb learning from preferred and rejected trajectories.

4.4 P-ReAct Agent Paradigm

P-ReAct imposes a tool-first decoding order by pre-filling tool-call boundaries and biasing confidence within the enclosed span, allowing tool dispatch while reasoning continues.

  • dLLMs can use global context from undecoded tokens, but their uncontrolled generation order motivates P-ReAct’s tool-first hierarchy.
  • Prioritizing tool_call decoding enables immediate search-parameter dispatch while the model continues generating the think component.
  • The standard dLLM process predicts logits only for masked positions, then unmasks high-confidence tokens and updates the remaining mask.
  • P-ReAct pre-fills <tool_call> and </tool_call> at designated positions, creating a structural span for valid tool content.
  • A positive confidence bias α is added to tokens inside the tool_call region so they rank above reasoning tokens during decoding.

5 Experiments

The experiments evaluate DLLM-Searcher on four multi-hop QA benchmarks against RAG methods, LLM-based agents, and dLLM baselines. Results report strong performance, generalization from limited training data, and efficient P-ReAct inference.

  • Experimental Setup: Experiments compare DLLM-Searcher with traditional RAG methods, LLM-based search agents, and dLLM-based agents across four multi-hop QA benchmarks.The evaluation uses HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle, with accuracy as the primary metric because search-agent outputs are often verbose.
  • Experimental Setup: The training data include 3977 filtered Agentic SFT trajectories and 2237 qualified queries paired with 4474 trajectories for Agentic VRPO.Agentic SFT trajectories come from sampled multi-hop queries, while VRPO uses contrasting correct and incorrect rollouts from the SFT model.
  • Overall Performance: DLLM-Searcher improves performance across all multi-hop QA benchmarks under both ACC_R and ACC_L metrics.It improves over traditional RAG, gains substantially over vanilla dLLMs, and performs comparably to ARM-based search agents, with a gap relative to R1Searcher on Musique.
  • Overall Performance: Despite training on only 8k samples, DLLM-Searcher performs strongly on in-domain datasets and generalizes to the out-of-domain Bamboogle dataset.The reported results associate this performance with integrating retrieval of relevant documents with internal reasoning during training.

5.3 Empirical Analysis

The empirical analyses show that post-training repairs vanilla dLLM tool-calling failures, while P-ReAct accelerates inference with minimal degradation. A case study illustrates tool-call prioritization before the thinking region is fully decoded.

  • RQ1: Effectiveness of Post-training: Vanilla SDAR fails to complete a successful interaction on 500 HotpotQA samples because all cases terminate during the first ReAct turn from formatting errors.Observed errors include empty output, missing tool calls, and think-format errors.
  • RQ1: Effectiveness of Post-training: Agentic SFT enables strict tool_call formatting, and Agentic VRPO adds gains exceeding 3 percentage points in both ACC_R and ACC_L across all benchmarks.The post-SFT model can execute multi-step reasoning, while VRPO consistently improves performance further.
  • RQ2: Inference Efficiency: P-ReAct reduces inference time by 14.77%, 21.00%, 22.08%, and 12.67% relative to ReAct across the four datasets with almost no performance degradation.It pre-fills tool-call boundary tokens and applies confidence bias to prioritize tool-call decoding, overlapping reasoning with tool-response waiting.
  • RQ3: Order-free Generation: ARMs can generate tool_call first only after prompt restructuring, whereas DLLM-Searcher preserves tool-call quality while decoding that region before the think region.The analysis attributes this contrast to bidirectional attention within diffusion blocks.
  • Case Study: In the case study, DLLM-Searcher produces the correct tool-call format while the vanilla model outputs an incorrect special token.At timestep 32, the tool_call region is fully decoded while the think region remains masked until the final decoding stage.

6 Conclusion

DLLM-Searcher combines dLLM-specific post-training with P-ReAct to address agent capability and latency challenges. Across four benchmarks, it achieves comparable performance to mainstream ARM-based search agents while delivering approximately 15% inference acceleration over conventional ReAct.

  • DLLM-Searcher addresses dLLM search-agent adoption by targeting both Agent Ability and Latency Challenges.
  • Its solution combines Agentic SFT and Agentic VRPO with the P-ReAct agent paradigm.
  • Approximately 15% inference acceleration over conventional ReAct is achieved while maintaining comparable performance to mainstream ARM-based search agents.The results are reported across four benchmarks.
Loading 2602.07035v1…