PaperlayerView exact revision on arXiv

Source-linked AI summary

SearchJev: A Fast and Calibrated System-1 Model for Search Agents

Congfeng Cao, Lipeng Zuo, Konstantinos Papakostas, Qiwei Xu, Songwei Xu, Lun Zhou, Zhaochun Ren, Yougang Lyu, Xiaohui Yan

arXiv:2610.05107v1cs.IRcs.CL

TL;DR

Search agents need frequent short decisions, but generative judgments add latency and often have unreliable confidence. SEARCHJEV directly predicts schema-defined options, learns calibrated probabilities, and delegates uncertain judgments to System 2. It improves decision and agent-level outcomes while reducing decision and search time.

  • Problem

    Search agents need a unified way to handle heterogeneous short decisions efficiently and with reliable confidence, rather than relying on separate task models or autoregressive judgments.

    Source: §1 · p. 1

  • Method

    SEARCHJEV scores legal options from a search state and decision schema, uses SLCD to learn calibrated probabilities from uncertain supervision, and delegates uncertain judgments to System 2.

    Source: §1 · p. 2

  • Results

    SEARCHJEV improves decision quality and average calibration over same-size Qwen3.5 AR (JSON) models with 5.2–5.3× faster decisions; BrowseComp-Plus accuracy rises from 45% to up to 54% versus System-2-only.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Calibrated confidence provides a basis for deciding when to invoke System 2 for a judgment.

    Source: §1 · p. 2

    Evaluation covers text-based search decisions in Chinese and English and BrowseComp-Plus agents; other languages, modalities, and domains require further evaluation.

    Source: H Limitations · p. 20

Abstract

from arXiv · show

Search agents repeatedly make short decisions about relevance, evidence sufficiency, and search actions. Using generative language models for these decisions introduces latency and unreliable confidence. We present SearchJev, a fast and calibrated System-1 model that separates search decisions from System-2 reasoning and generation. Given a search state and a decision schema, SearchJev directly scores legal options without autoregressive output generation. We propose Soft-Label Learning for Calibrated Decisions (SLCD) to learn decision probabilities from uncertain supervision and calibrate their confidence. In a dual-system search agent, SearchJev handles short decisions and delegates uncertain judgments to System 2, which retains planning, query generation, and answer composition. We also introduce SearchDecision-Bench, a benchmark unifying six types of search decisions for training and evaluation. On SearchDecision-Bench, SEARCHJEV improves decision quality over same-size Qwen3.5 autoregressive models, achieves 5.2-5.3 times faster decisions, and reduces average expected calibration error by 41-74%. On BrowseComp-Plus, the dual-system agents achieve a 3.7-4.7 times speedup in active search time while improving answer accuracy from 45% to up to 54%.

1 Introduction

Search agents make frequent short judgments that shape evidence use and search, but handling them through generation adds decoding overhead and can yield miscalibrated confidence. SEARCHJEV separates these decisions from System-2 reasoning and generation, improving decision and agent-level results.

1 Introduction

  • Search trajectories require short judgments about relevance, evidence sufficiency, and which search step to take next.These decisions shape which evidence is used and how search proceeds.Source: §1 · p. 1
  • SEARCHJEV is 5.2–5.3× faster than same-size Qwen3.5, while BrowseComp-Plus accuracy rises from 45% to up to 54% with 3.7–4.7× faster median active search.The BrowseComp-Plus comparison is against System-2-only agents.Source: Figure 1 · p. 1, §1 · p. 2
  • Task-specific models handle individual decisions, while prompted general LLMs generate short judgments autoregressively and often have miscalibrated confidence.Reliable confidence is needed to decide whether to accept a judgment or seek further reasoning.Source: §1 · pp. 1–2
  • SEARCHJEV predicts legal options from a decision schema and search state, while SLCD learns calibrated probabilities from uncertain supervision.The dual-system agent delegates uncertain judgments to System 2, which retains planning, query generation, and answer composition.Source: §1 · p. 2
  • SearchDecision-Bench unifies six decision types for training and evaluation, measuring decision quality, calibration, latency, and generalization.The paper reports improved decision quality and average calibration over same-size autoregressive models.Source: §1 · p. 2

2 SearchDecision-Bench

SearchDecision-Bench unifies heterogeneous search-decision resources as state-schema pairs for training and evaluation. It covers six decision families and three output types, with constructed supervision and held-out tests for in- and out-of-distribution assessment.

2 SearchDecision-Bench

  • SearchDecision-Bench unifies heterogeneous search resources as state-schema pairs, covering six decision types and three output types.It is intended to support training and evaluation of short decisions in agentic search.Source: §2 · p. 2

2.1 Task Definition and Taxonomy

  • The task predicts legal options for schema-defined decisions from a search state; the six families include routing, rewriting, relevance, sufficiency, navigation, and verification.Choice, Score, and Noul represent categorical selection, ordinal scoring, and binary judgments, respectively.Source: §2.1 · pp. 2–3

2.2 Data Construction and Splits

  • Samples combine serialized states, decision fields, and supervision drawn from original annotations, derived labels, and LLM teachers or votes.Sources include query-understanding, retrieval, multi-hop QA, agent demonstrations, and claim-evidence resources; soft targets reflect supervision-source confidence.Source: §2.2 · pp. 3–4
  • Held-out tests cover all six decision types in-distribution and four out-of-distribution, while BrowseComp-Plus is reserved for end-to-end agent evaluation.The construction process removes conflicting or unjudgeable samples and prevents test-state overlap in training and validation inputs.Source: §2.2 · p. 4

3 SEARCHJEV

SEARCHJEV separates schema-conditioned search decisions from System-2 reasoning and generation. It predicts legal options with calibrated probabilities and delegates uncertain judgments to System 2.

3.1 System-1 Search Decision Model

  • Given search state and schema, SEARCHJEV scores legal option labels from the LM’s first-output logits, avoiding autoregressive generation.The schema supplies each task’s question and legal options; option descriptions may span multiple tokens, but labels must be single tokens.Source: §3.1 · pp. 4–5

3.2 Soft-Label Learning for Calibrated Decisions

  • SLCD learns decision distributions from soft targets that represent supervision uncertainty, then calibrates probabilities with output-type temperatures.Its training combines cross-entropy and Brier losses, with weights that can downweight LLM-assigned labels, demonstrations, or constructed rewrites.Source: §3.2 · pp. 5–6

3.3 Dual-System Search Agent

  • SEARCHJEV handles schema-defined judgments, while System 2 retains planning, query generation, and answer composition.A confidence gate delegates judgments below threshold to System 2; the controller maps resolved decisions to search actions and updates state with retrieved evidence.Source: §3.3 · p. 6

4 Experiments

The experiments pose questions about decision effectiveness, end-to-end search, and generalization, then specify comparison systems, evaluation measures, and implementation settings.

4.1 Research Questions

  • The experiments ask whether SEARCHJEV improves decision effectiveness and efficiency, end-to-end answer quality and efficiency, and generalization across held-out datasets and decision families.Source: §4.1 · p. 6

4.2 Baselines

  • Comparisons use same-backbone Qwen3.5 autoregressive JSON models, Jev 1.13, and a System-2-only agent against dual-system agents using SEARCHJEV.The dual-system agents use SEARCHJEV-0.8B or SEARCHJEV-4B for short decisions.Source: §4.2 · pp. 6–7

4.3 Evaluation Metrics

  • Decision evaluation reports accuracy for five classification families, NDCG@10 for relevance, ECE for calibration, and median latency per decision.End-to-end evaluation measures judge-assessed answer accuracy and reports median active time per question, including decision inference, retrieval, and System-2 calls.Source: §4.3 · p. 7

4.4 Implementation Details

  • Models are initialized from Qwen3.5-0.8B/4B and trained with LoRA using SLCD; end-to-end tests sample 100 BrowseComp-Plus questions and use Qwen3.5-27B with BM25.Decision latency is measured at batch size 1; end-to-end evaluation considers relevance, sufficiency, and verification decisions.Source: §4.4 · p. 7

5 Experimental Results and Analysis

SEARCHJEV improves decision quality, calibration, and latency across SearchDecision-Bench evaluations, with further tests on held-out data and end-to-end BrowseComp-Plus search. The results show gains in both short decisions and agent performance, alongside task-dependent calibration and accuracy-efficiency trade-offs.

5.1 Decision Effectiveness and Efficiency (RQ1)

  • Both SEARCHJEV sizes outperform same-size Qwen3.5 AR (JSON) across all six in-distribution decision families, with the 4B model leading five.Against Jev 1.13, both sizes improve sufficiency, routing, rewriting, and verification, are comparable on relevance, and trail on navigation.Source: §5.1 · p. 7
  • SEARCHJEV-0.8B and -4B are 5.2–5.3× faster than same-size Qwen3.5 AR (JSON), and achieve a 7.2–9.6× speedup in median decision latency versus Jev 1.13.Jev retains the lowest mean ECE and is best calibrated on navigation and verification.Source: §5.1 · p. 8

5.2 End-to-End Agentic Search (RQ2)

  • Both dual-system agents outperform the System-2-only (27B) agent in answer accuracy; SEARCHJEV-4B gains 8.0 percentage points over Jev 1.13, while SEARCHJEV-0.8B matches it.Against the System-2-only agent, the dual-system variants reduce output tokens by 3.3–4.8× and median active time by 3.7–4.7×; the 4B variant uses more resources than Jev but achieves higher accuracy.Source: §5.2 · pp. 8–9, Table 4 · p. 8

5.3 Decision Generalization (RQ3)

  • On held-out datasets, both sizes outperform same-size Qwen3.5 AR (JSON) on routing, rewriting, and relevance, with routing gains of 11.9–12.1 percentage points.Jev remains stronger on routing and verification; verification improves over Qwen3.5 for 0.8B but declines for 4B.Source: Table 5 · p. 8, §5.3 · p. 9
  • Held-out calibration gains vary by task: SEARCHJEV-0.8B lowers ECE on rewriting and relevance, while 4B improves routing calibration; Jev is better calibrated on the other three families.Both sizes remain 5.3–5.5× faster than same-size Qwen3.5 AR (JSON) and 7.4–9.9× faster than Jev 1.13 on held-out datasets.Source: §5.3 · p. 9

6 Conclusion

The paper concludes that separating short search decisions from generative reasoning can support more accurate and efficient search agents. Its dual-system procedure accepts confident decisions and delegates uncertain ones while reserving planning and answer composition for System 2.

6 Conclusion

  • SEARCHJEV directly predicts schema-conditioned decisions, uses SLCD to learn and calibrate probabilities, and delegates uncertain judgments while System 2 handles planning, query generation, and answer composition.SearchDecision-Bench unifies training and evaluation across six search-decision types; the reported experiments improve decision quality and average calibration while reducing latency, and improve BrowseComp-Plus answer accuracy while reducing System-2 calls, output tokens, and active search time.Source: §6 · p. 9

B Dual-System Search Procedure

  • The core search step accepts a judgment when its calibrated confidence meets a threshold and sends uncertain relevance or sufficiency judgments to System 2.The procedure applies relevance retention or sufficiency-based search control, then executes the action or updates the search state; planning, query generation, and answer composition remain with System 2.Source: B Dual-System Search Procedure · p. 15
  • Verification occurs after answer composition and reports calibrated support probability without revising the answer or restarting search.The appendix describes benchmark and implementation sections, additional results, a calibration-utility guarantee, limitations, and related materials.Source: Appendix organization · p. 14, B Dual-System Search Procedure · p. 15

C SearchDecision-Bench Details

C.1 Data Construction and Splits

SearchDecision-Bench brings heterogeneous search-decision data into a common state-and-schema format, with soft targets that retain label confidence. The appendix documents source datasets, filtering and split rules, representative prompts, and the model’s direct option-scoring inference.

C.1 Data Construction and Splits

  • Routing, rewriting, relevance, sufficiency, navigation, and verification are assembled from varied source datasets, including multi-hop MuSiQue states for relevance, sufficiency, and navigation.Sources include BANKING77 and CLINC150 for routing, LCQMC and QReCC for rewriting, and VitaminC and CFEVER for verification.Source: §C.1 · p. 15
  • Each source is serialized as a search state with schema fields and soft targets whose confidence reflects label provenance and reliability.Labels include original annotations, derived labels, and LLM teacher or vote labels; LLMs participate in 33.7% of training rows.Source: §C.1 · p. 15
  • The construction pipeline deduplicates inputs, removes conflicting labels, masks undecidable fields, isolates splits, and excludes evaluation-source variants from training.Corpus counts distinguish rows, state-schema pairs, and decisions; reported validation and test evaluations are capped per dataset.Source: §C.1 · p. 16

C.2 Schema and Prompt Examples

  • Prompts express each decision as a state, a natural-language question, and legal options; sufficiency instructions are paraphrased, and FRAMES supplies a held-out schema.A MuSiQue episode example follows next-query selection with a later sufficiency judgment; FRAMES contributes no training data.Source: §C.2 · pp. 16–17

D.1 Decision-Model Inference

  • At inference, SEARCHJEV gathers logits for legal option labels, applies output-type temperature scaling, normalizes over the options, and maps probabilities back to them.Independent field prompts can be batched with left padding.Source: §D.1 · p. 17

D.2 Training and Calibration

SEARCHJEV trains on soft-label cross-entropy and Brier loss, with output-type temperatures fitted on validation decisions. The BrowseComp-Plus configuration specifies relevance and sufficiency controls, while the reported aggregate logs leave the confidence threshold’s numerical value unresolved.

D.2 Training and Calibration

  • Training minimizes cross-entropy to soft labels plus a weighted Brier score, and fits a temperature per output type on 5,000 validation decisions.The listed setup uses AdamW, a cosine schedule, bf16 precision, and LoRA over attention, Gated DeltaNet, and MLP projections.Source: Table 6 · p. 17, §D.2 · p. 17

D.3 BrowseComp-Plus Agent Configuration

  • BrowseComp-Plus retains documents with relevance probability at least 0.2, checks sufficiency at specified answer/search stages, and limits forced continuation to twice per question.The confidence gate threshold δ is distinct from the retention threshold τkeep; uncertain relevance judgments share a System-2 call, while uncertain sufficiency judgments each require one.Source: §D.3 · p. 17
  • Post-answer verification scores calibrated support probability but neither revises the answer nor triggers further search.Planning and answer generation use System 2.Source: §D.3 · p. 18
  • Aggregate logs do not uniquely recover δ, so zero explicit short-decision calls do not establish that the confidence gate was disabled.The reported runs still include other System-2 calls.Source: §D.3 · p. 18

E.1 Per-Dataset Decision Results (RQ1)

SEARCHJEV is more accurate than same-backbone autoregressive models on nearly all in-distribution datasets, while agent diagnostics report fewer System-2 reasoning tokens and less decision time per search step.

E.1 Per-Dataset Decision Results (RQ1)

  • SEARCHJEV is more accurate than same-backbone autoregressive models on every in-distribution dataset except 0.8B agent demonstrations, where all models reach at most 39.0%.Table 7 reports classification accuracy; its relevance rows are an additional diagnostic, not a decomposition of Table 3’s NDCG@10.Source: §E.1 · p. 18

E.2 Agent Resource Diagnostics (RQ2)

  • The SEARCHJEV agents record no explicit System-2 short-decision calls, fewer System-2 reasoning tokens overall, and less decision time per search step than the System-2-only agent.Table 8 separates calls dedicated to short decisions from planning, query-generation, and answer-generation calls in the end-to-end totals.Source: §E.2 · p. 18, Table 8 · p. 19

F Comparison with System-1 Models

F.1 Evaluation Protocol

  • The comparison uses JevBench peers’ reported datasets and scores SEARCHJEV on the same items where possible; otherwise systems use their own test samples.SEARCHJEV samples contain up to 1,000 items, or 10,000 for RAGTruth; its zero-shot row applies its readout to the backbone before training.Source: §F.1 · p. 19

F.2 Evaluation Datasets

  • The peer evaluation spans routing, rewriting, relevance, sufficiency, navigation, verification, and general JevBench datasets.It includes 860 Open-Jev out-of-distribution navigation questions; 59 of 1,500 bev-decision states overlap training, and excluding them changes scores by at most 0.7 points.Source: §F.2 · p. 19

F.3 Results

Across peer comparisons, SEARCHJEV-4B leads on some routing datasets but trails on several sufficiency, verification, and navigation tasks. A calibration proposition relates population calibration error to the utility gap between score-based and recalibrated decisions.

F.3 Results

  • SEARCHJEV-4B leads reported scores on BANKING77 (89.0) and CLINC150 (94.6), but trails peers on several sufficiency, verification, and Open-Jev control tasks.It exceeds other open systems on bev-decision (69.5 vs. at most 66.5), while scoring 62.2 on JevBench’s hard tier, below its zero-shot backbone (65.8).Source: §F.3 · p. 19, Table 9 · p. 20

G Calibration and Decision Utility

  • The proposition uses unbinned population ECE, so finite-sample ECE estimates in Tables 3 and 5 cannot be substituted directly into its bound.ECE(ρ) is defined as Eρ|g(ρ) − ρ|.Source: G Calibration and Decision Utility · p. 20
  • When population calibration error is zero, the score-based rule matches the best utility rule constructed from the same score; otherwise, the proposition bounds the expected utility gap.The comparison is against a rule using the model’s own scores, not one with additional information.Source: G Calibration and Decision Utility · p. 20

H Limitations

The evaluation covers text-based search decisions in Chinese and English, with end-to-end testing on BrowseComp-Plus; other languages, modalities, and domains remain to be evaluated. Training-label sources can still introduce biases that affect generalization.

H Limitations

  • SEARCHJEV’s evaluation is limited to text-based search decisions in Chinese and English, with end-to-end agent evaluation on BrowseComp-Plus.Its performance in other languages, modalities, and application domains requires further evaluation; legal options must be enumerable, while open-ended query and answer generation stays with System 2.Source: H Limitations · p. 20
  • Training data combine original annotations, derived labels, and LLM-assisted supervision, and training-data biases can still affect generalization.LLMs label about a third of training rows; derived labels depend on the filtering models and agents producing demonstrations.Source: H Limitations · pp. 20–21

I Ethics Statement

  • The evaluation focuses on decision quality, calibration, and efficiency; the work reports use of public benchmarks and synthesized environments without sensitive personal information.The authors state that they will open-source code and used LLMs only for language refinement.Source: I Ethics Statement · p. 21

K Data and Model Licenses

The appendix records licenses for the datasets and models used, with usage conditions noted for datasets lacking a stated license.

K Data and Model Licenses

  • Official-release licenses for datasets and models are recorded; datasets without a stated license are used only for non-commercial research.The query-assessment and query-document quality sets are constructed for SearchDecision-Bench, while source webpages remain subject to their sites’ terms.Source: K Data and Model Licenses · p. 21, Table 10 · p. 22

Related papers

Loading 2610.05107v1…