PaperlayerView exact revision on arXiv

Source-linked AI summary

Avoiding What Your Teacher Already Knows: Entity Familiarity for Deep-Search Distillation

Yifei Liu, Zhixiang Liang, Wentao Tan, Yicheng Wang, Jiaqi Wang, Nan Duan, Qiong Cao

arXiv:2610.16251v1cs.CLcs.AI

TL;DR

Structurally complex questions can still be easy for teachers familiar with their entities, reducing search demand and supervision value. The paper evaluates entity-familiarity signals and introduces SEF gating for graph synthesis; SEF-gated trajectories increase search demand and student accuracy, while reused graphs raise Qwen3.8-27B from 44.10% to 66.41% on a BrowseComp subset.

  • Problem

    Structurally complex questions may remain easy for teachers familiar with key entities, limiting the search behavior and information acquisition represented in distillation trajectories.

    Source: §1 · p. 2

  • Method

    The paper evaluates evidence-verified familiarity and uses a cheaper self-reported familiarity gate to admit entities into graph-based question–answer synthesis.

    Source: §1 · p. 2, §2.2 · pp. 3–4

  • Results

    Across three teachers, SEF-gated trajectories show greater observed search demand and higher student accuracy; with reused graphs, Qwen3.8-27B rises from 44.10% to 66.41% on a BrowseComp subset.

    Source: §1 · pp. 2–3, §7 · p. 10

  • Takeaways & Limitations

    Target-teacher SEF reassessment permits reuse of other teachers’ graphs to expand deep-search training data.

    Source: §1 · p. 2, §7 · p. 10

    EVF and SEF are behavioral signals rather than calibrated knowledge boundaries; validation mainly probes root entities, and empirical scope covers three teachers, two student backbones, one evaluation set, and graph-based entity QA.

    Source: §7 · p. 10

Abstract

from arXiv · show

Deep-search agents are commonly trained by distilling tool-use trajectories from stronger teacher models. However, structurally complex questions may remain relatively easy for teachers familiar with key entities in the underlying entity graph, resulting in lower search demand and weaker supervision. We refer to this teacher-specific variation in question difficulty as teacher-relative difficulty and investigate entity familiarity as an early predictor. We first introduce Evidence-Verified Entity Familiarity (EVF), which estimates familiarity by verifying a teacher-generated factual brief against online evidence. Across multiple teachers, EVF is significantly correlated with teacher-relative difficulty with AUCs of 0.60--0.70. However, its high cost makes it impractical for constructing entity graphs at scale. To address this limitation, we propose Self-Reported Entity Familiarity (SEF), a lightweight gating signal that directly asks a teacher whether it is familiar with an entity before admitting that entity into the graph. Across multiple teachers, trajectories from SEF-gated synthesis exhibit greater observed search demand and yield higher downstream student accuracy than those from ungated synthesis. Because familiarity is teacher specific, we further study cross-teacher reuse. We screen graphs collected for other teachers using Kimi-K3's SEF judgments and use the retained graphs to produce 5,411 additional trajectories. Combined with 2,200 trajectories from newly constructed, SEF-gated graphs, these data improve Qwen3.8-27B's accuracy from 44.10% to 66.41% on a BrowseComp subset, and achieving state-of-the-art performance among models of comparable size on BrowseComp, SEAL-0, HLE-text, and xbench. Overall, SEF provides a low-cost way to synthesize more informative supervision for deep-search distillation.

1 Introduction

The introduction argues that structural complexity alone does not ensure search demand when teachers already know key entities. It presents entity familiarity as an early predictor and SEF-gated synthesis and cross-teacher reuse as ways to improve deep-search supervision.

1 Introduction

  • Structurally complex questions can remain easy for a teacher familiar with key entities, so structural complexity alone may not produce meaningful search decisions or information acquisition.The introduction calls the teacher-specific external information need teacher-relative difficulty.Source: §1 · p. 2
  • EVF predicts closed-book correctness from seed-entity familiarity with AUCs of 0.60–0.70, but factual-brief verification makes repeated graph screening costly.Lower EVF scores correspond to questions less likely to be answered correctly without tools.Source: §1 · p. 2
  • SEF is a tool-free binary familiarity query applied at seed selection and next-hop expansion, admitting only entities the teacher reports as unfamiliar.Unlike EVF, it requires neither search nor evidence verification.Source: §1 · p. 2
  • SEF-gated trajectories contain more search turns, reasoning tokens, and tool calls, alongside lower tool-assisted solve accuracy, consistent with higher observed search demand.The comparison holds the other synthesis stages fixed.Source: §1 · p. 2
  • 44.10% to 66.41%: Qwen3.8-27B accuracy rose on the evaluated BrowseComp subset using 7,611 Kimi-K3-specific trajectories, including reused graphs.The 7,611 trajectories combine 5,411 from graphs collected for other teachers and 2,200 from Kimi-K3 SEF-gated synthesis.Source: §1 · p. 2
  • +8.20, +6.15, and +8.46 percentage points: equal-sized SEF-gated distillation comparisons improved student accuracy across three teacher settings.The trajectories also showed greater observed search demand than those from ungated synthesis.Source: §1 · pp. 2–3

2 Problem Setup and Study Design

This section defines deep-search distillation and teacher-relative difficulty, describes the graph-based synthesis pipeline and its SEF intervention points, and frames three research questions.

2.1 Deep-Search Distillation Setting

  • Teacher-relative difficulty concerns the external information a particular teacher must acquire; closed-book correctness and search behavior serve as observable proxies, and teacher trajectories supervise student fine-tuning.A trajectory includes reasoning, tool actions, observations, and a final answer.Source: §2.1 · p. 3

2.2 Graph-Based Synthesis Pipeline

  • Multi-hop web exploration collects entity facts and evidence-supported relations into a graph, which supports question generation and difficulty enhancement before tool-using teacher solving.SEF can screen both seed entities and candidate next-hop entities; the gated and ungated variants otherwise remain identical.Source: §2.2 · pp. 3–4

2.3 Study Design

  • The study asks whether entity familiarity predicts closed-book answerability, whether SEF gating increases search demand and student performance, and whether graphs can be reused across teachers.Source: §2.3 · p. 4

3 Entity Familiarity as a Signal of Teacher-Relative Difficulty

Section 3 tests whether teacher-specific entity familiarity signals closed-book answerability and can serve as a low-cost screening gate during entity-graph construction.

3.1 EVF Tracks Closed-Book Answerability

  • EVF positively tracked closed-book correctness across three teachers, with AUCs of .600–.706; low-EVF questions were less answerable without tools.EVF < 0.25 is used as the operational definition of low familiarity.Source: Table 1 · p. 4, §3.1 · p. 5

3.2 SEF Provides a High-Recall Low-Familiarity Gate

  • Questions with SEF-no seed entities had lower closed-book accuracy; screening retained nearly all low-EVF entities while excluding most high-EVF entities.The retained pool had a higher low-EVF share and a lower high-EVF share, although a substantial high-EVF fraction remained for Kimi-K3.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 SEF Enables Low-Cost Early Screening

  • Roughly 9–20× lower mean wall-clock time made repeated entity-admission screening practical; SEF screens seeds and next hops.The wall-clock comparison applies within measured deployment settings.Source: Table 3 · p. 5, §3.3 · p. 6, Figure 1 · p. 6

4 SEF-Gated Synthesis and Distillation

SEF gating screens entities at both seed selection and graph expansion while leaving the rest of the synthesis pipeline unchanged. The resulting trajectories show greater search demand and improve student accuracy over ungated synthesis in the evaluated settings.

4.1 SEF-Gated Entity Admission

  • SEF gating admits unfamiliar entities at seed selection and during next-hop expansion, while the other graph-based synthesis stages remain unchanged.If no unfamiliar candidate remains, that branch stops and its partial graph is preserved.Source: §4.1 · pp. 6–7

4.2 SEF Gating Increases Observed Search Demand

  • Across full teacher pools, all nine search-demand measures rose: turns by 2.3–34.6%, reasoning tokens by 4.8–18.3%, and tool calls by 2.9–39.0%.Tool-assisted solve accuracy fell by 12.02 and 5.50 points for GLM-5.1 and DeepSeek-V4-Flash, respectively; Kimi-K3 made 21.6% more turns and 19.4% more calls, with accuracy changing by −0.92 points.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 SEF-Gated Trajectories Yield Higher Student Accuracy

  • With equal-sized training sets, SEF-gated trajectories raised student accuracy by 8.20, 6.15, and 8.46 percentage points across the three teacher settings.Students were evaluated on the 130-question BrowseComp subset; each comparison held the number of trajectories fixed.Source: Table 5 · p. 7, §4.3 · pp. 7–8

5 Cross-Teacher Adaptation through Graph Reuse

The section tests whether evidence graphs collected for one teacher can support target-specific supervision for another. After target-specific SEF screening, reused graphs add trajectories and are associated with higher mean student accuracy, while cross-benchmark comparisons remain indicative rather than strictly controlled.

5.1 SEF Judgments Overlap but Require Target-Specific Reassessment

  • Pairwise SEF agreement across teachers is 84.4–89.9%, but 10.1–15.6% directional disagreement means the target teacher must reassess candidate paths.The agreement analysis uses entities sampled from three teacher-specific pools.Source: §5.1 · p. 8, Table 6 · p. 8

5.2 Reuse Evidence Graphs, Regenerate Target-Specific Supervision

  • Graph reuse retains entities, evidence-backed relations, retrieved evidence, and candidate paths, but familiarity screening, question synthesis, and trajectory generation remain target-specific.Accepted paths are resynthesized into question–answer pairs before the target teacher generates new search trajectories.Source: §5.2 · p. 8

5.3 Scaling Target-Specific Supervision with Reused Graphs

  • Adding 5,411 trajectories from reused graphs raised Qwen3.8-27B BrowseComp-subset accuracy from 63.59 ± 1.18% to 66.41 ± 2.35%, a 2.82-point gain.The comparison adds reused-graph trajectories to 2,200 native SEF-gated trajectories; the authors do not isolate graph reuse from the effect of additional training examples.Source: §5.3 · p. 8
  • Comparisons across five benchmarks are indicative rather than strictly controlled because student models and training procedures differ across methods.Some compared methods use mid-training and reinforcement learning rather than supervised fine-tuning alone.Source: §5.3 · p. 9, Table 7 · p. 9

6 Related Work

Prior work shapes synthetic question difficulty through structural design or solver feedback, and audits whether search agents truly search rather than rely on memory. This paper situates teacher familiarity as an entity-level data-construction rule and evaluates its relationship to teacher-trajectory search demand and downstream student gains.

Synthetic Deep-Search Data and Difficulty Control

  • Difficulty-control methods shape entity paths, constraints, evidence, or clues, or use solver feedback to revise or filter questions; this work adds teacher familiarity as a distinct, compatible dimension.The related-work discussion names TaskCraft, FORT-Searcher, OpenSeeker, and WebLeaper as examples of prior approaches.Source: §6 · p. 9

Are Search Agents Actually Searching?

  • Prior search audits test whether answers persist without usable evidence and examine retrieval of benchmark copies, while distillation work filters or aligns trajectories.The paper contrasts these evaluations and trajectory interventions with its entity-level data-construction rule.Source: §6 · p. 9

7 Conclusion

The conclusion presents teacher-specific entity familiarity as an early signal for constructing deep-search distillation data. The synthesis workflow builds evidence graphs, generates questions from them, and screens entity admissions with the target teacher’s self-report.

7 Conclusion

  • Across three teachers, SEF-gated synthesis yielded greater observed search demand and higher student accuracy than ungated synthesis; target-specific reassessment also supported cross-teacher graph reuse.The conclusion frames familiarity as a data-construction signal rather than relying on graph structure alone to anticipate useful difficulty.Source: §7 · p. 10
  • The empirical scope covers three teachers, two student backbones, and one fact-seeking evaluation set; EVF and SEF are behavioral signals, not ground-truth or calibrated knowledge boundaries.Validity tests mainly probe root entities, and adapting familiarity gating beyond entity-based graph QA synthesis remains future work.Source: §7 · p. 10

A.1 DeepForge Question Synthesis

  • The workflow records an entity transition path by exploring web evidence and relations, then passes the resulting evidence graph to a question generator.The recorded path describes exploration, not necessarily the reasoning needed to answer a generated question.Source: §A.1 · p. 13
  • SEF-gated synthesis screens both seed and next-hop admissions, stops if no candidate passes, and does not transfer a source teacher’s questions or trajectories when graphs are reused.The target teacher’s self-report screens entities before question generation and trajectory collection.Source: §A.1 · p. 13

A.2 Depth versus Closed-Book Answerability

The appendix contrasts recorded graph depth with closed-book answerability and specifies the SEF and EVF prompting and verification procedures. It also documents the verifier’s configured turn limit.

A.2 Depth versus Closed-Book Answerability

  • Depth-stratified correctness rises for GLM-5.1, is nonmonotonic for DeepSeek-V4-Flash, and shows no increasing trend for Kimi-K3, so greater exploration depth does not reliably lower closed-book answerability.One proposed explanation for GLM-5.1 is that longer paths may encounter familiar entities, but hop-level familiarity and connectivity were not measured.Source: §A.2 · p. 13, Table 8 · p. 13

B Exact EVF and SEF Prompt Templates

  • The prompt appendix states that the same probed teacher writes the EVF brief and verifies it, first without and then with browser access.The appendix also specifies temperature zero for calls and identifies Kimi-K3 as the final-answer judge.Source: B Exact EVF and SEF Prompt Templates · p. 13

B.1 SEF Self-Report

  • SEF is a tool-free YES/NO query about whether the teacher can recall multiple specific, verifiable facts about the exact entity; uncertainty or vague knowledge counts as NO.Judgments are cached by entity and teacher-model identifier during synthesis.Source: §B.1 · p. 14

B.2 EVF Factual Brief

  • For EVF, the teacher writes a factual brief without browser tools; if it provides no brief, EVF is set to zero without verification.The template requests confident, verifiable facts in a single paragraph under 150 words.Source: §B.2 · p. 14

B.3 EVF Evidence Verification

  • EVF is the fraction of extracted brief claims labeled true after browser verification, with zero assigned when no claims are extracted.The verifier extracts up to eight atomic claims and marks unconfirmed or imprecisely supported claims unknown.Source: §B.3 · pp. 14–15

B.4 Verifier Turn Limit

  • The verifier’s tool-loop limit is configurable; after forced finalization, results are not stored in the EVF cache.Cost measurements were made without cache reuse.Source: §B.4 · p. 15

C Evaluation Details for the Reported Tables

The evaluation details define how familiarity, search behavior, student accuracy, and cross-teacher SEF agreement are measured. They also state key sampling and comparison boundaries for the reported tables.

C Evaluation Details for the Reported Tables

  • EVF analyses pair teacher-specific questions with their graph’s seed entity, while SEF screening analyses join valid SEF and EVF outcomes by entity identity.AUC binarizes mean closed-book correctness at ≥.5; EVF< .25 defines the low-EVF region, and SEF labels receive no external judgment.Source: C Evaluation Details for the Reported… · p. 15
  • EVF and SEF cost measurements use uncached calls and report mean LLM calls, browser calls, and wall time per entity; wall times are comparable only within a deployment.EVF includes brief generation and verifier turns, whereas SEF consists of one tool-free call.Source: C Evaluation Details for the Reported… · p. 16
  • Ungated and SEF-gated pools use the same synthesis and solving stages, but generate separate graphs and questions, so their comparisons do not isolate entity screening’s effect.Search behavior is measured on every raw trajectory, including turns, reasoning tokens, tool calls, and solve accuracy; the token proxy excludes tool observations and final answers.Source: C Evaluation Details for the Reported… · p. 16
  • Student accuracy is evaluated on a QUEST-selected 130-question BrowseComp subset across three independent runs, with generation failures, empty answers, and missing judgments counted incorrect.Search results pointing to Hugging Face are filtered in every student run to reduce benchmark-answer leakage.Source: C Evaluation Details for the Reported… · p. 16
  • Cross-teacher SEF agreement is measured on 5,995 deduplicated entity names, using fresh judgments from each of the three teachers.The sampled entities come from each teacher’s historical SEF cache; the table reports shared unfamiliar judgments, directional disagreements, and overall agreement.Source: C Evaluation Details for the Reported… · p. 16

C.7 Graph Reuse and Student Evaluation

Reused evidence graphs are screened with the target teacher’s SEF judgments, then used to resynthesize questions and collect new trajectories. The expanded training set outperforms the native-only set on the reported BrowseComp subset.

C.7 Graph Reuse and Student Evaluation

  • The 7,611-example set combines 2,200 native trajectories with 5,411 trajectories collected from reused graphs after target-specific SEF screening and resynthesis.Previously collected graphs supply candidate paths, not source-teacher questions or trajectories.Source: §C.7 · p. 16
  • 66.41±2.35% accuracy with all 7,611 trajectories compares with 63.59±1.18% using 2,200 native SEF-gated trajectories.Each value is the mean and sample standard deviation over three independent runs on the QUEST-selected BrowseComp subset.Source: §C.7 · p. 16

D Student Training Details

Students are fine-tuned on serialized teacher search trajectories, while records that cannot support a complete student-format interaction are removed.

D Student Training Details

  • Students are trained with supervised fine-tuning on serialized teacher search trajectories, not on closed-book familiarity probes.Both students use an effective global batch size of 32 packed sequences and a cosine learning-rate schedule.Source: D Student Training Details · p. 17
  • Each student run uses six epochs of full-parameter bfloat16 updates, with student-specific learning rates of 5 × 10−5 and 5 × 10−6.Qwen3-30B-A3B-Thinking uses 5 × 10−5; Qwen3.8-27B uses 5 × 10−6.Source: D Student Training Details · p. 17
  • Training examples exclude records with no assistant messages or tool use, malformed tool calls, unfinished actions, or incompatible student-format rendering; incorrect answers remain included.Source: D Student Training Details · p. 17

Related papers

Loading 2610.16251v1…