PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparing Retrieval Methods for Academic Advisor Discovery: A Six-Method Study of 768 CS Faculty Profiles Across 9 US Universities

Biraj Subedi

arXiv:2609.03901v1cs.IRcs.LG

TL;DR

Applicants seeking academic advisors must manually compare faculty profiles, and advisor discovery lacks a dedicated relevance-judged benchmark. This study compares six retrieval methods on a new faculty-profile collection; Reranked has the highest mean NDCG@10 across five queries, while only TF-IDF is significantly worse than four alternatives after correction.

  • Problem

    Advisor discovery is an under-studied retrieval task, and applicants must manually compare fragmented faculty information across potentially hundreds of profiles.

    Source: §1.1 · pp. 1–2

  • Method

    The study compares six retrieval methods for ranking faculty profiles, including a reranked combination of BM25, TF-IDF, and semantic scores.

    Source: §11 · p. 14

  • Results

    Across five queries, Reranked leads mean NDCG@10 (0.477); after Bonferroni correction, TF-IDF is significantly worse than BM25, Semantic, Hybrid, and Reranked.

    Source: §11 · p. 14

  • Takeaways & Limitations

    BM25 is the safest single-method choice when the query domain is unknown, and biography alone outperforms the full model combining biography with research-area tags.

    Source: §11 · pp. 14–15

    The convenience sample covers nine US public universities, excludes top programs and non-US universities, and may not generalize to institutions with different profile-maintenance practices.

    Source: §10.6 · p. 14

Abstract

from arXiv · show

We present a comparative evaluation of six information retrieval methods for the task of academic advisor discovery: ranking CS faculty members by relevance to a graduate applicant's research interest statement. The methods span sparse lexical matching (Jaccard overlap, TF-IDF, BM25), dense semantic retrieval (all-MiniLM-L6-v2 sentence embeddings), hybrid score fusion, and learning-to-rank. Evaluation uses a new domain-specific collection: 768 faculty profiles scraped from 9 US CS departments, with 162 graded relevance judgments (grade 0/1/2) across 5 queries representing distinct graduate student research profiles. Across all five queries, Reranked achieves the highest mean NDCG@10 (0.477, std 0.138), followed by Semantic (0.450), Hybrid (0.421), BM25 (0.406), Jaccard (0.303), and TF-IDF (0.246). After Bonferroni correction across all 15 pairwise comparisons, TF-IDF is significantly worse than BM25, Semantic, Hybrid, and Reranked; no other pairwise difference survives correction at 5 queries. A field ablation reveals that biography alone (NDCG 0.634) outperforms the full model combining biography with research area tags (0.593). A controlled experiment shows that concatenating arXiv paper abstracts reduces NDCG@10 by 0.176, motivating a late-fusion architecture. All code, scrapers, and relevance labels are released openly.

1. Introduction

Academic advisor discovery is framed as an information-retrieval task, but applicants lack a dedicated benchmark and must manually compare fragmented faculty information. The study builds a judged collection, compares six retrieval methods, and reports cross-query results and profile-field findings.

1.1 The Problem

  • Applicants must manually inspect potentially hundreds of faculty profiles, while vocabulary mismatch, absent rankings, and fragmented information impede systematic comparison.No prior work had benchmarked faculty advisor discovery specifically or constructed a relevance-judged collection for it.Source: §1.1 · pp. 1–2

1.2 Our Approach and Contributions

  • The benchmark contains 768 faculty profiles from 9 US CS departments and 162 graded relevance judgments across five queries, with profiles, labels, and evaluation scripts released openly.The six-method comparison spans lexical matching, sentence embeddings, hybrid fusion, and learning-to-rank.Source: §1.2 · p. 2
  • The contributions include field ablation and controlled arXiv concatenation experiments, identifying biography as more informative than research-area tags and naive abstract concatenation as harmful.The observed greater score drops for relevant professors are based on n = 3 and are described as suggestive rather than confirmatory.Source: §1.2 · p. 2

1.3 Research Questions

  • The study asks whether semantic retrieval beats keyword methods, which profile fields matter, whether small-label LTR helps, and whether rankings vary across research areas.These questions organize the benchmark and its field-ablation, learning-signal, and cross-query comparisons.Source: §1.3 · p. 2

1.4 Key Findings

  • Across five queries, Reranked leads mean NDCG@10 (0.477), followed by Semantic (0.450), Hybrid (0.421), BM25 (0.406), Jaccard (0.303), and TF-IDF (0.246).Only TF-IDF is significantly worse than BM25, Semantic, Hybrid, and Reranked after Bonferroni correction; BM25 is most consistent (std = 0.090).Source: §1.4 · p. 3
  • Biography alone (NDCG 0.634) outperforms the full model (0.593); arXiv concatenation reduces NDCG@10 by 0.176, with larger drops for relevant professors suggestive rather than confirmatory.The arXiv comparison includes only three relevant professors among the largest score changes.Source: §1.4 · p. 3
  • On Q1, LTR has higher MAP@10 than Semantic (0.480 vs. 0.388) but lower NDCG@10 (0.568 vs. 0.593); neither difference is statistically significant.The two metrics therefore give different comparisons for this query.Source: §1.4 · p. 3

2. Related Work

Related work situates advisor discovery within expertise retrieval, academic search, and dense retrieval, while motivating a domain-specific benchmark and comparisons of hybrid and learning-to-rank methods. It also identifies evaluation choices and limits relevant to interpreting results.

2.1 Expertise Retrieval

  • Expertise retrieval ranks people by topic expertise; the paper’s aggregated profile text is a dense analogue of a language-model approach that scores candidates from their associated documents.No standard faculty-advisor benchmark exists, so this study constructs a collection with 162 graded judgments across five queries.Source: §2.1 · p. 3

2.2 Academic Expert Finding

  • WISER [14] and AMiner support academic expert finding at larger scale, but are not designed for PhD applicants searching targeted institutions; prior DBLP learning-to-rank work motivates an LTR baseline.This evaluation finds LTR underperforms an unsupervised semantic baseline at small label sizes practical for a new domain.Source: §2.2 · pp. 3–4

2.3 Semantic and Dense Retrieval

  • Dense retrieval uses shared query-document vector spaces: sentence-BERT [2] provides transferable semantic representations, while BEIR found inconsistent domain generalization; here zero-shot dense retrieval beats tuned keyword baselines on faculty profiles.The semantic method uses the speed-optimized all-MiniLM-L6-v2 variant.Source: §2.3 · p. 4

2.4 Hybrid and Multi-Field Retrieval

  • Sparse BM25 and dense embedding signals complement one another, motivating a priori linear score fusion; the field ablation finds biography contributes more than structured research-area tags.Reciprocal Rank Fusion is noted as a weight-free alternative left for future work.Source: §2.4 · p. 4

2.5 Learning to Rank

  • Pointwise LTR, including the classifier used here, needs fewer labels than listwise LambdaMART but is described as the weaker paradigm; with 67 labels, the reported LTR result is a lower bound.LambdaMART directly optimizes NDCG using a listwise loss.Source: §2.5 · p. 4

2.6 Scholarly Search and Paper Recommendation

  • Paper recommendation systems rank papers rather than people against user interests; SPECTER [22] uses citation-graph-informed paper embeddings, motivating separate paper embeddings and late fusion after naive abstract concatenation degrades retrieval.The separate-embedding approach is proposed as a next step, not as an experiment reported here.Source: §2.6 · p. 4

2.7 Retrieval Evaluation Methodology

  • NDCG and graded relevance motivate the three-point judgment scale; prior work calls for at least 25 topics for stable comparisons and documents bias from incomplete pooled judgments.The paper relates these concerns to its five-query evaluation and disclosure of its candidate-labeling pool.Source: §2.7 · p. 5

3. Dataset

The dataset comprises faculty profiles collected from nine US computer science departments, with custom scraping needed to handle differing website structures and protections.

3.1 Data Collection

  • Faculty profiles were scraped from 9 US computer science departments between April and May 2026.Each university required a custom scraper.Source: §3.1 · p. 5
  • Inconsistent HTML, JavaScript rendering, and anti-bot measures required university-specific scrapers.Source: §3.1 · p. 5
  • Table 1 presents the university breakdown for the 768 professors; its caption identifies the table’s scope but does not state the individual counts.Source: Table 1 · p. 5, §3.1 · p. 5

4. Retrieval Methods

The retrieval methods score each faculty profile against a query, using lexical, semantic, fused, or learned ranking signals. The section also documents implementation choices that constrain score comparisons and the LTR feature set.

4. Retrieval Methods

  • All six methods rank 768 professors by descending query score, using research text concatenated with the first 500 biography characters as embedding text.Source: §4 · p. 5
  • Jaccard uses stopword-filtered token overlap without term importance, while TF-IDF uses cosine similarity; its two implementations differ, so absolute NDCG values across Q1 and Q2–Q5 are not directly comparable.The ranking conclusions—that TF-IDF is weakest and BM25 outperforms it—were verified under both implementations.Source: §4 · p. 5
  • BM25 uses k1 = 1.5 and b = 0.75, with term-frequency saturation and document-length normalization for profiles of varying length.Source: §4 · p. 6
  • Semantic retrieval encodes each profile with all-MiniLM-L6-v2 and ranks by cosine similarity, capturing conceptual similarity even with zero token overlap.The profile embeddings have 384 dimensions and are L2-normalized.Source: §4 · p. 6
  • Hybrid combines min-max-normalized TF-IDF and semantic scores with a priori weights of 0.35 and 0.65; Reranked combines normalized BM25, TF-IDF, and semantic scores with weights 0.15, 0.20, and 0.65.The hybrid weights were set before relevance labels were collected.Source: §4 · p. 6
  • LTR uses a pointwise GradientBoostingClassifier with five-fold cross-validation on 67 Q1 labels; the authors call its result a lower bound because this label count is insufficient for reliable listwise training.Its nine features include six retrieval scores and three profile-related features.Source: §4 · p. 6
  • The full-corpus LTR model effectively has eight informative features, not nine; its zero arXiv-flag importance reflects a structural limitation, not evidence that arXiv coverage is uninformative.Source: §4 · p. 6

5. Results

Across five queries, Reranked has the highest mean NDCG@10, while TF-IDF is significantly worse than the other methods. On Q1, Semantic leads NDCG@10, and the section also reports distinct rankings and learning-to-rank results.

5. Results

  • Reranked leads five-query mean NDCG@10 at 0.477, while TF-IDF is significantly worse than all other methods (pBonf < 0.001).Figure 1 summarizes the five-query mean and per-query NDCG@10 alongside the Q1 results.Source: Figure 1 · p. 7
  • Semantic achieves the highest Q1 NDCG@10 (0.593), ahead of Reranked (0.558) and BM25, the best pure keyword method (0.454).LTR has the highest Q1 MAP@10 (0.480) but lower NDCG@10 (0.568) than Semantic; neither difference is statistically significant at this label set size.Source: §5 · pp. 6–7

5.1 Top-5 Results by Method

  • Only 4 of 10 top results overlap between TF-IDF and Semantic (Spearman ρ = 0.624), indicating they surface meaningfully different professors.Source: §5.1 · p. 7

5.2 Rank Correlations

  • BM25 and TF-IDF rankings are nearly identical (ρ = 0.954) despite a 4.8× NDCG difference, attributed to score calibration rather than rank reordering.Source: Table 4 · p. 7, §5.2 · p. 8

5.3 Learning-to-Rank

  • LTR achieves NDCG@10 = 0.568, with reranked score the most important feature (25.7%), followed by TF-IDF (17.6%) and Semantic (15.8%).The has arxiv flag has 0.0% importance; the passage attributes this to the binary flag providing insufficient signal, noting that paper content matters more than its presence.Source: §5.3 · p. 8

6. Ablation Study

The ablations show that biography alone outperforms the full profile text, while adding arXiv abstracts reduces retrieval quality on a controlled subpopulation. The findings motivate using biography alone and exploring late fusion for paper content.

6.1 Field Contribution (Profile-Only)

  • NDCG 0.634 for biography alone exceeds 0.593 for the full model combining biography and research area tags.The full model uses research text plus biography[:500] across 768 professors; the comparison is reported in Table 5.Source: Table 5 · p. 8, §6.1 · p. 8
  • Research area tags retain some independent signal but act as noise when prepended to biography, and biography-only embedding is the stated deployment implication.The authors suggest generic, comma-separated tags may dilute distinctions captured by richer biography prose.Source: §6.1 · p. 8

6.2 arXiv Concatenation Experiment (Controlled)

  • NDCG@10 falls by 0.176 when arXiv abstracts are concatenated on the controlled subpopulation.Both conditions use the same 296 professors, query, and model; three relevant professors are displaced from the top 10.Source: Table 6 · p. 9, §6.2 · p. 9
  • The controlled arXiv comparison isolates content change alone, and its result motivates embedding profiles and papers separately before combining retrieval scores.The proposed late-fusion design is intended to prevent abstract vocabulary from diluting the profile signal.Source: Table 6 · p. 9, §6.2 · p. 9
  • The jargon-dilution explanation is suggestive rather than confirmatory: the score-change comparison includes only 3 relevant professors.The authors note that a systematic comparison across all 24 relevant professors is needed before treating H1 as established.Source: §6.2 · p. 9

7. Statistical Significance

The section tests method differences with paired bootstrap resampling and Bonferroni correction, while emphasizing uncertainty from the limited label set. The current evaluation cannot establish that the methods differ statistically.

Method and correction

  • The paired permutation test resamples 67 labeled professors with replacement, and Bonferroni correction sets the per-test threshold at 0.0033 across 15 pairs.Unlabeled professors are held fixed and contribute grade 0 to NDCG.Source: §7 · p. 9

Baselines

  • Random rankings have expected NDCG@10 of 0.027 (±0.055), while the oracle ranking reaches 1.000.The authors report that all six retrieval methods substantially outperform chance.Source: §7 · pp. 9–10

Confidence intervals

  • The 95% bootstrap confidence intervals are wide, reflecting genuine uncertainty at the 67-label size.Table 7 reports bootstrap means from 10,000 resamples of 67 labeled professors.Source: Table 7 · p. 10, §7 · p. 10

Pairwise tests

  • No pairwise comparison is statistically significant after Bonferroni correction with the current 67-judgment label set.The raw Semantic-versus-TF-IDF and Semantic-versus-Jaccard p-values are below 0.05 but exceed the corrected per-test threshold of 0.0033; this does not mean their point estimates are equal.Source: §7 · p. 10

8. Multi-Query Evaluation

Across five distinct CS graduate-student research-profile queries, Reranked has the highest mean NDCG@10, but only TF-IDF is significantly worse than four methods after correction. Method rankings vary with query topic and vocabulary.

Findings

  • Reranked leads overall at mean NDCG@10 = 0.477, followed by Semantic (0.450), Hybrid (0.421), BM25 (0.406), Jaccard (0.303), and TF-IDF (0.246).The Reranked advantage is not statistically significant at this query-set size, though its direction is consistent across 4 of 5 queries.Source: Table 9 · p. 11, §8 · p. 11
  • TF-IDF is significantly worse than BM25, Semantic, Hybrid, and Reranked after Bonferroni correction; no other pairwise difference survives.The correction covers 15 comparisons, with α = 0.05/15 = 0.0033.Source: §8 · p. 11
  • BM25 is the most consistent method across query types, with standard deviation 0.090, below Hybrid (0.096), Reranked (0.138), Semantic (0.146), and Jaccard (0.244).The query set contains five distinct CS graduate-student research profiles.Source: §8 · pp. 10–11
  • No method leads every query: Semantic leads Q1, Jaccard Q2, Reranked Q3 and Q5, and BM25 Q4.Jaccard’s Q2 lead (NDCG 0.716) coincides with exact query-token overlap, while its Q5 Comp. Biology score is 0.066; relative performance varies with query vocabulary.Source: §8 · pp. 11–12

9. Research Landscape Clustering

The clustering visualizes a 10-cluster research landscape from faculty sentence embeddings and marks the query in the ML/NLP/Data Science cluster. The authors caution that this placement is not independent validation of semantic ranking.

9. Research Landscape Clustering

  • The user’s query falls in the ML/NLP/Data Science cluster, which contains 79 professors; its position is visually consistent with NLP and data-systems researchers in the 2D projection.Figure 2 shows UMAP + KMeans clustering of 768 CS faculty in 10 clusters; cluster labels are interpretive summaries, not validated category labels.Source: §9 · p. 12, Figure 2 · p. 12
  • The plot’s query placement is visually consistent with NLP and data-systems researchers, but is not an independent validation because both derive from the same embedding vectors.The largest clusters are AI/ML/Robotics (108 professors), Scientific Computing/AI/Graphics (89), and Security/Privacy (86).Source: §9 · p. 12

10. Discussion

The discussion explains query-dependent retrieval behavior and why profile text quality matters, while identifying limits in the evaluation and future integration work.

10.1 Why Semantic Embedding Wins

  • Semantic embeddings capture conceptual similarity despite low token overlap, addressing vocabulary mismatch between student queries and faculty descriptions.The discussion says keyword-search applicants may miss closely aligned professors whose work uses different terminology.Source: §10.1 · p. 13

10.2 Why arXiv Concatenation Hurts Performance

  • NDCG@10 fell from 0.772 to 0.596 (∆= −0.176) when arXiv abstracts were concatenated in the controlled 296-professor experiment.The authors hypothesize that technical language dilutes profile signals and uneven text lengths bias similarity; they propose separately scoring profiles and papers for late fusion.Source: §10.2 · p. 13

10.3 Why LTR Does Not Improve Over Semantic

  • With 67 labels, LTR underperforms semantic retrieval (0.568 vs. 0.593), and the authors characterize this result as a lower bound at this label size.The model uses nine features; its reranked score receives the greatest feature importance.Source: §10.3 · p. 13

10.4 Why Biography Matters More Than Research Areas

  • Biography alone (NDCG 0.634) outperforms the full model (NDCG 0.593), and the authors recommend embedding biography alone.They attribute the drop to generic tags diluting the biography signal; the reported gain is 0.040 NDCG at zero additional data-collection cost.Source: §10.4 · pp. 13–14

10.5 Cross-Query Consistency (RQ4)

  • Jaccard ranges from NDCG@10 0.716 on Q2 to 0.066 on Q5, while BM25 has std = 0.090 across five queries.The authors say the variation shows retrieval difficulty is query-dependent and single-query evaluation cannot characterize method behavior.Source: §10.5 · p. 14

10.6 Limitations

  • The evaluation uses five queries, one annotator, about 25 labelled candidates per query, and nine convenience-sampled US public universities; late fusion remains unimplemented.Semantic, hybrid, and reranked multi-query comparisons still lack significance; the authors note that findings may not generalize to institutions with different profile-maintenance practices.Source: §10.6 · p. 14

11. Conclusion

The conclusion reports the strongest overall retrieval method and key profile-text findings, alongside an openly released evaluation collection and system.

11. Conclusion

  • The study compares methods on 768 faculty profiles from nine US universities, and releases its system, scrapers, and relevance labels openly.The release is reproducible without proprietary data or API keys.Source: §11 · pp. 14–15
  • Reranked achieves the highest mean NDCG@10 across five queries (0.477); after Bonferroni correction, TF-IDF is significantly worse than BM25, Semantic, Hybrid, and Reranked.BM25 is the most consistent method (std = 0.090) and is presented as the safest single-method choice for unknown query domains.Source: §11 · p. 14
  • Biography alone (NDCG 0.634) outperforms the full model (0.593); arXiv concatenation reduces NDCG@10 by 0.176, and LTR with 67 labels does not conclusively outperform semantic retrieval.The conclusion identifies biography-only embedding as the highest-impact improvement without additional data collection and late fusion as the next step for arXiv content.Source: §11 · p. 15
Loading 2609.03901v1…