Source-linked AI summary
Overview of the TalentCLEF 2026: Skill and Job Title Intelligence for Human Capital Management
Luis Gasco, Hermenegildo Fabregat, Laura García-Sardiña, Paula Estrella, Warre Veys, Casimiro Pío Carrino, Matthias De Lange, Daniel Deniz Cerpa, Álvaro Rodrigo, Jens-Joris Decorte, Rabih Zbib
TL;DR
As labor markets change how organizations describe jobs, identify talent, and support careers, Human Capital Management increasingly requires NLP that can process and connect information. TalentCLEF 2026 provides a shared evaluation framework, and its strongest systems used hybrid, modular approaches across both tasks.
Problem
As labor markets change how organizations describe jobs, identify talent, and support careers, Human Capital Management increasingly requires NLP that can process and connect information.
Source: §1 · p. 1Method
The paper surveys TalentCLEF 2026, a shared evaluation campaign with realistic job and skill scenarios, datasets, common protocols, and public benchmarks.
Source: §1 · p. 2Results
The strongest systems in both tasks used hybrid, modular approaches; in Task B, classum led with NDCG(binary) 0.8340 and NDCG(graded) 0.8068.
Source: §4.2 · p. 15, §5 · p. 17Takeaways & Limitations
TalentCLEF provides shared datasets, protocols, and public benchmarks to support reproducible HCM research and future system comparison.
Source: §1 · p. 2Most participants did not explicitly address bias-oriented evaluation, despite systems showing different degrees of robustness.
Source: §5 · p. 17
Abstract
from arXiv · showhide
This paper presents an overview of the second edition of the TalentCLEF challenge, organized as a Lab at the Conference and Labs of the Evaluation Forum (CLEF) 2026. TalentCLEF is an initiative aimed at advancing Natural Language Processing research in Human Capital Management. The second edition of the challenge consisted of two tasks: Task A, contextualized job-person matching, focuses on identifying and ranking the most suitable candidates represented by their resumes for a given job vacancy in English and Spanish. Task B, job-skill matching with skill type classification, addresses retrieving the most relevant skills for a given job title in English and distinguishing between core and contextual skills. TalentCLEF attracted 113 registered teams and received more than 400 submissions in the two tasks, reflecting the growing interest of the research community in shared evaluation benchmarks for Human Capital Management. This paper describes the motivation and organization of the challenge, summarizes the datasets and evaluation settings, and reports the main results obtained by the participating teams.
1 Introduction
Human Capital Management increasingly needs NLP to connect information about jobs, people, and skills, but fragmented research settings make progress difficult to compare. TalentCLEF provides shared benchmarks and evaluation protocols, and this paper overviews its 2026 challenge and participation.
1 Introduction
- HCM increasingly requires NLP systems that process and connect information about jobs, people, and skills.These systems are relevant to recruitment, talent acquisition, workforce development, career guidance, skill-gap analysis, internal mobility, upskilling, and reskilling.Source: §1 · pp. 1–2
- Labor-market information is expressed in text, which NLP can extract, normalize, match, and link to support structured representations and downstream job- and skill-intelligence applications.Relevant sources include job ads, titles, professional profiles, curricula, learning resources, and skill or occupation taxonomies.Source: §1 · p. 2
- Research on NLP for HR spans established tasks such as skill extraction and matching as well as newer work on career paths, LLM comprehension, and recruitment fairness.The paper points to venues including NLP4HR and RecSys in HR.Source: §1 · p. 2
- Fragmented datasets, languages, task definitions, annotations, and evaluation protocols make it difficult to compare systems or assess progress in HCM NLP.The paper notes that this matters in domains such as recruitment, where systems can influence real-world decisions.Source: §1 · p. 2
- TalentCLEF offers a shared HCM evaluation framework with datasets, common protocols, and public benchmarks to support reproducible research and system comparison.The initiative is organized around competitive campaigns grounded in realistic job and skill intelligence scenarios.Source: §1 · p. 2
- The paper overviews TalentCLEF 2026, reporting 113 registered teams, more than 400 submissions, and methodological trends including hybrid retrieval, reranking, generative AI, and structured knowledge sources.Detailed task descriptions, dataset construction, annotation, and evaluation settings are covered in the task overview articles.Source: §1 · pp. 2–3
2 Overview of the Tasks
TalentCLEF 2026 evaluates candidate matching and job-skill matching, with task settings spanning multilingual retrieval, gender-bias analysis, and skill-type-aware ranking. The overview describes each task’s data and evaluation design.
2 Overview of the Tasks
- The challenge targets two HCM tasks: finding candidates for a vacancy and identifying professional skills relevant to job positions.Task A addresses candidate search; Task B addresses job-skill identification.Source: §2 · p. 3
2.1 Task A - Contextualized Job-Person Matching
- Task A expands prior job-title matching into contextualized job-person matching, ranking suitable candidates for vacancies using a manually annotated synthetic corpus.The corpus covers English and Spanish, has development and test partitions but no training data, and permits external resources; the synthetic records were guided by real-world co-occurrence evidence and reviewed by expert annotators.Source: §2.1 · pp. 3–4
Evaluation
- Task A evaluates same-language and English-vacancy/Spanish-resume rankings with MAP, and assesses gender-related ranking consistency with RBO.The competition uses a common Codabench environment and remains available as an open benchmark for continued evaluation.Source: Evaluation · p. 5
2.2 Task B - Job-Skill Matching with Skill Type Classification
- Task B extends skill retrieval by requiring systems to distinguish essential core skills from contextual skills that depend on industry, organization, or work environment.Contextual skills may be complementary or optional depending on context.Source: §2.2 · p. 6
Data
- Task B provides an English corpus with training, development, and test splits; training skills are linked to ESCO occupations and marked essential or optional.The development and test queries match the previous edition, while corpus items were expanded and revised using semantic techniques, human validation, and LLM-as-a-judge procedures.Source: Data · pp. 6–7
- Task B uses NDCG for ranking, with binary relevance treating relevant skills equally and graded relevance assigning core skills 2 and contextual skills 1.The graded setting rewards ranking core skills higher while still crediting relevant contextual skills; the dataset statistics include queries, corpus elements, and average relevant items per query.Source: Data · p. 7
3 Participants
Participants in both tasks mainly used retrieval pipelines, with Task A emphasizing multi-stage ranking and Task B placing greater emphasis on representation learning. Teams also used reranking, rank fusion, LLM methods, and supervised retriever adaptation.
3.1 Task A
- Nine out of the 12 teams explicitly reranked candidates within Task A’s predominantly multi-stage information-retrieval architecture.Task A attracted 91 registered participants; 21 teams submitted runs, producing 384 submissions, and 12 teams sent working notes.Source: §3.1 · p. 8
- Task A systems fused lexical, dense, cross-encoder, and LLM reranking signals to combine complementary strengths, which the paper suggests often yields better results than a single model.Source: §3.1 · p. 10
- Task A’s more extensive LLM use included listwise, pointwise, and tournament reranking, as well as query expansion, data augmentation, and retrieval-stage judging.Source: §3.1 · p. 10
- QTPride and classum extracted entities before retrieval; QTPride also used ESCO graph information, while teams applied LoRA, curriculum learning, distillation, and query expansion.Source: §3.1 · p. 10
3.2 Task B
- Task B combined hybrid retrieval with stronger representation-learning emphasis, and several systems added reranking, rank fusion, or LLM-based refinement.Fifteen teams submitted at least one run, producing 43 submissions.Source: §3.2 · p. 11
- Encoder-based models dominated Task B retrieval, while BM25 appeared less frequently than in Task A.Teams used general-purpose and JobBERT-family embeddings.Source: §3.2 · pp. 11–12
- Task B teams adapted dense retrievers to supervision with contrastive or metric-learning losses, curriculum learning, domain adaptation, external data, and LoRA.Source: §3.2 · p. 12
- Task B’s LLM uses included reranking, data augmentation, and query expansion, which the paper describes as useful for short or lexically different inputs.Source: §3.2 · p. 12
4 Results
Task A leaders topped both monolingual and English-to-Spanish matching, while rankings were highly stable under the tested gendered variants. In Task B, classum led graded and binary NDCG; the strongest systems generally combined retrieval with enrichment and reranking.
Overall Task A Performance
- 0.7140 average MAP gave classum the best monolingual Task A result, with 0.7122 in English–English and 0.7157 in Spanish–Spanish.QTPride ranked second overall at 0.6632, and bipboopbipboop third at 0.6532. The top systems used enriched job and resume representations, structured views, and combinations of retrieval, fusion, and reranking.Source: Overall Task A Performance · pp. 12–13, Table 5 · p. 13
Cross-Lingual Performance
- 0.7044 MAP made classum the top English-query/Spanish-corpus system, followed by bipboopbipboop at 0.6438.The top systems' cross-lingual and multilingual results differed relatively little, suggesting generalization to English–Spanish matching; classum combined cross-lingual dense representations with enriched job and resume views.Source: Cross-Lingual Performance · pp. 13–14, Table 6 · p. 14
Bias Evaluation
- 0.9904 average RBO made classum's rankings the most stable under the tested gendered variants, with the highest score in all three language scenarios.The evaluation compared rankings for paired masculine and feminine variants; the authors caution that this is not a comprehensive demonstration of fairness or bias mitigation, and top systems reported no dedicated mitigation component.Source: Bias Evaluation · pp. 14–15, Table 7 · p. 15
4.2 Task B Main Results
- 0.8068 graded NDCG and 0.8340 binary NDCG made classum the Task B leader; NightSun ranked second at 0.7913 and 0.8246, respectively.The leading approaches combined fine-tuned dense retrieval, reranking, and enriched representations; with limited textual context, embedding fine-tuning and data augmentation seem to have played an important role.Source: §4.2 · pp. 15–16, Table 8 · p. 16
5 Conclusions
TalentCLEF 2026 drew substantial participation and showed reuse of resources across editions. Across both tasks, the strongest systems commonly combined complementary retrieval signals and reranking, while bias-oriented evaluation remained an area for greater attention.
5 Conclusions
- 113 registered participants, 29 teams submitting official benchmark runs, and 17 teams contributing system papers marked substantial participation in TalentCLEF 2026.More than 400 submissions arrived across the two tasks; reuse of first-edition resources and the creation of new resources indicate that the initiative is beginning to support research in this domain.Source: §5 · p. 17
- In Task A, strong systems combined job and resume semantic representations with structured views; in Task B, fine-tuned encoders were paired with expansion, enrichment, fusion, and reranking.Task B systems used these methods to distinguish core and contextual skills.Source: §5 · p. 17
- Across both tasks, dense encoders, lexical or domain-specific models, graph resources, and extracted entity views supplied complementary signals, while rerankers refined retrieved candidates.Both LLM-based and recent neural rerankers were used in this refinement.Source: §5 · p. 17
- Systems performed strongly across languages and showed differing robustness in bias-oriented evaluation, although most participants did not explicitly address that aspect.The next challenge edition will place greater emphasis on extending this type of evaluation.Source: §5 · p. 17