Source-linked AI summary
AI Hiring with LLMs: A Context-Aware and Explainable Multi-Agent Framework for Resume Screening
Frank P. -W. Lo, Jianing Qiu, Zeyu Wang, Haibao Yu, Yeming Chen, Gao Zhang, Benny Lo
TL;DR
High-volume resume screening is difficult to conduct efficiently, and existing approaches can be hard to adapt and interpret. The paper proposes a modular, RAG-enabled multi-agent framework and reports performance comparable to HR evaluators on online resume datasets. Its current text-based scope leaves soft skills and communication abilities less assessed.
Problem
High application volumes challenge manual review, while traditional and single-LLM screening can lack adaptability and transparency.
Source: §1 · pp. 1–2Method
A four-agent framework extracts and evaluates resumes, summarizes assessments, and formats scores, with RAG retrieving company-specific criteria for the evaluator.
Source: §1 · p. 2Results
The framework achieved comparable performance to human evaluators on online resume datasets.
Source: §8 · p. 8Takeaways & Limitations
The modular RAG-LLM framework offers a context-aware approach to resume screening that can adapt to company-specific criteria without retraining or fine-tuning.
Source: §1 · p. 2, §8 · p. 8Text-based resume evaluation limits assessment of soft skills and communication abilities.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Resume screening is a critical yet time-intensive process in talent acquisition, requiring recruiters to analyze vast volume of job applications while remaining objective, accurate, and fair. With the advancements in Large Language Models (LLMs), their reasoning capabilities and extensive knowledge bases demonstrate new opportunities to streamline and automate recruitment workflows. In this work, we propose a multi-agent framework for resume screening using LLMs to systematically process and evaluate resumes. The framework consists of four core agents, including a resume extractor, an evaluator, a summarizer, and a score formatter. To enhance the contextual relevance of candidate assessments, we integrate Retrieval-Augmented Generation (RAG) within the resume evaluator, allowing incorporation of external knowledge sources, such as industry-specific expertise, professional certifications, university rankings, and company-specific hiring criteria. This dynamic adaptation enables personalized recruitment, bridging the gap between AI automation and talent acquisition. We assess the effectiveness of our approach by comparing AI-generated scores with ratings provided by HR professionals on a dataset of anonymized online resumes. The findings highlight the potential of multi-agent RAG-LLM systems in automating resume screening, enabling more efficient and scalable hiring workflows.
1. Introduction
Traditional resume screening struggles with high application volumes, nuanced language, changing job requirements, and limited transparency. The paper proposes a modular, RAG-enabled framework that retrieves company-specific criteria so recruiters can adapt evaluation without fine-tuning.
1. Introduction
- High application volumes make manual resume review difficult, while rule-based and keyword-matching methods can miss job-specific requirements and lack adaptability.Traditional methods also have difficulty interpreting nuanced wording and unstructured language.Source: §1 · p. 1
- Single-LLM screening couples extraction, evaluation, and feedback, making scoring changes harder to adapt and decisions harder for recruiters to interpret.Adapting criteria may require retraining or fine-tuning the entire model, such as with LoRA.Source: §1 · p. 2
- The framework separates screening into extraction, evaluation, summarization, and score formatting, with RAG enabling real-time updates to company-specific criteria without fine-tuning.Recruiters can upload job-requirement documents, and changes to scoring criteria do not interfere with extraction or feedback generation.Source: §1 · p. 2
- The paper frames modularity and dynamic retrieval as ways to improve transparency and tailor screening across roles, industries, and hiring criteria.Examples include prioritizing particular university rankings, certifications, or domain expertise.Source: §1 · p. 2
2. Related Work
AI-driven hiring has moved from keyword-based machine learning through deep learning to LLMs, which support more contextual resume assessment. The related work identifies limited LLM screening research and limited use of RAG for resume screening.
2.1. AI-driven hiring
- AI-driven hiring spans resume screening, classification, ranking, interview evaluation, salary prediction, and bias mitigation, with methods shifting toward context-aware decisions.Figure 2 groups the evolution into traditional machine learning (2010–2016), deep learning (2016–2022), and LLMs (2022–present).Source: §2.1 · p. 2, Figure 2 · p. 3
2.2. Resume screening systems
- Resume screening progressed from keyword-based machine learning to deep learning and LLMs, which can infer transferable skills and implicit qualifications.Earlier methods lacked semantic understanding, while deep-learning approaches required large labeled datasets and LLM screening remains relatively little studied.Source: §2.2 · pp. 2–3
3. Problem Definition
The problem definition frames resume screening as role-specific scoring: the same resume can receive different scores for different jobs. The framework uses evaluation criteria and weights while interpreting them in the context of the applied position.
3. Problem Definition
- The score vector covers self-evaluation, skills and specialties, work experience, basic information, and education, with a fixed weight assigned to each criterion across roles.The passage identifies the criterion scores as components of the role-aware scoring framework.Source: §3 · pp. 3–4
- A candidate's score should depend on the applied job role, so the same resume may receive different scores for an HR intern and an HR director.The framework is described as generating job-specific evaluation scores based on the applied position.Source: §3 · p. 4
- The framework computes a final candidate score after evaluating the resume, then formats the evaluation output as a structured numerical result.The proposed multi-agent framework includes a score formatter for the structured output.Source: §3 · p. 4, Figure 3 · p. 4
4. Detailed Information and Methodology
The methodology organizes resume screening into specialized agents. The extractor uses LLM reasoning to turn unstructured resume text into structured candidate information, including implicit skills.
4. Detailed Information and Methodology
- Four agents divide the workflow into resume extraction, evaluation, summarization, and score formatting.The evaluator scores job fit, the summarizer reports strengths and improvement areas, and the formatter standardizes numerical output.Source: §4 · p. 4
4.1. Resume extractor agent
- The extractor uses LLM contextual understanding to structure resume details and infer missing information or implicit skills.Extracted fields include position and level, self-evaluation, skills, work experience, basic information, and education.Source: §4.1 · p. 5
4.2. Resume evaluator agent
The resume evaluator scores candidates against job-specific criteria, using retrieved external knowledge and structured prompts to contextualize its assessment.
4.2. Resume evaluator agent
- The evaluator scores five categories and uses RAG to retrieve company-specific hiring criteria, job descriptions, and other relevant information.The categories are self-evaluation (0–1), skills and specialties (0–2), work experience (0–4), basic information (0–1), and education (0–2).Source: §4.2 · p. 5
4.2.1. Vector embedding
- The embedding function converts the original job query and external-source document chunks into dense vector representations.The query and chunk texts are denoted qtext and d_i,text; the vectors have dimension D.Source: §4.2.1 · p. 5
4.2.2. Cosine similarity computation
- Cosine similarity measures document-chunk relevance to the query, and chunks below the 0.3 relevance threshold are filtered out.Higher similarity values indicate greater relevance.Source: §4.2.2 · p. 5
4.2.3. Contextual prompt construction
- The evaluator receives a structured prompt combining the query, applied job position, and retrieved chunks to assess candidates against job-specific criteria.The query assesses skills, work experience, and education in relation to the job.Source: §4.2.3 · p. 5, Figure 4 · p. 5
4.2.4. Specific requirements from external sources
- External requirements draw on certifications, university rankings, outstanding candidates’ past resumes, and current skill demands to refine job-specific criteria.LLM summarization extracts qualifications, skills, and experience patterns from past hires to establish a dynamic baseline.Source: §4.2.4 · p. 5
4.3. Resume summarizer agent
The summarizer’s CEO, CTO, and HR sub-agents discuss evaluator scores to produce structured candidate feedback; the supplied passages also describe two evaluation tables and their metrics.
4.3. Resume summarizer agent
- The CEO, CTO, and HR sub-agents discuss evaluator scores, respectively assessing leadership, technical expertise, and soft skills and cultural fit.Their discussion refines personalized feedback into structured recommendations.Source: §4.3 · p. 5
- Table 1 compares single LLMs with multi-agent RAG-LLMs using different backbones, including DeepSeek-V3 (DS-V3).The evaluation metrics include PC and SC at the 20%, 15%, and 10% thresholds.Source: Table 1 · p. 6, §4.3 · p. 6
- PC and SC are higher-is-better correlations, while MAE is lower-is-better; PC10 evaluates candidates in the top and bottom 10% of ground-truth scores.The supplied table header lists PC20, SC20, PC15, SC15, PC10, and SC10.Source: §4.3 · p. 6
- Table 2 is an ablation study comparing multi-agent RAG-LLMs with and without the resume extraction agent.The supplied passage identifies the table’s scope but does not report its values.Source: Table 2 · p. 6
4.4. Score formatter agent
The score formatter standardizes evaluation-agent scores as a numerical array for downstream use.
4.4. Score formatter agent
- The score formatter converts raw scores across assessment components into a uniform numerical array for ranking models and decision-making pipelines.An example output is [1.0, 1.5, 3.5, 0.8, 1.5].Source: §4.4 · p. 6
5. Experimental Results
The experiments compare the framework with single LLMs and HR ratings, test structured extraction, and assess the dataset, scoring metrics, and generated feedback.
5. Experimental Results
- The implementation coordinates agents with CrewAI, uses ChromaDB for semantic retrieval and OpenAI embeddings, and accesses models including DeepSeek-V3 and GPT-4o via OpenRouter.Ollama is noted as an option for local execution when privacy concerns require it.Source: §5 · p. 6
5.1. Dataset
- The evaluation used 105 anonymized online resumes labeled by HR across five aspects and roles ranging from junior to leadership, primarily in human resources.Personally identifying names and company names were removed.Source: §5.1 · p. 6
5.2. Evaluation metrics
- The study used Pearson correlation, Spearman correlation, and MAE to assess score relationships, ranking relationships, and average absolute error against HR ratings.Spearman captures rank-based monotonic relationships, while Pearson measures linear relationships.Source: §5.2 · p. 6
5.3.1. Comparison with single model approaches
- PC10 = 0.84 and SC10 = 0.74 (both p-value < 0.001), with MAE 0.90, were the strongest reported DeepSeek-V3 results against single LLM baselines.Similar trends held at the 15% and 20% thresholds.Source: §5.3.1 · p. 7
5.3.2. Evaluating the impact of extraction agent
- Structured extraction improved Pearson and Spearman correlations across all thresholds, and DeepSeek-V3 performed best with the extraction module.The authors identify high-quality structured parsing as important to candidate evaluation.Source: §5.3.2 · p. 7
5.3.3. Comparison of AI and human resume screening
- HR and DeepSeek-V3 mean scores were 7.68 and 7.76, respectively, and their candidate score-range histograms showed similar patterns.Figure 6 compares score distributions across self-evaluation, skills and specialties, work experience, basic information, and education.Source: §5.3.3 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
5.4. Qualitative analysis of feedback system
- The summarizer consolidates multiple sub-agent insights into traceable, role-adapted feedback that highlights strengths and missing competencies for recruiter comparison.Feedback can be presented as bullets summarizing strengths and weaknesses.Source: §5.4 · p. 7
6. Discussion
The discussion presents modular multi-agent screening as potentially more explainable and reliable than single-model approaches, while emphasizing extraction quality and model reasoning.
6. Discussion
- The discussion reports advantages over traditional deep learning and single-LLM screening in explainability, decision efficiency, and evaluation reliability.Figure 7 depicts targeted recommendations following discussion among CEO, CTO, and HR sub-agents.Source: §6 · pp. 7–8, Figure 7 · p. 8
- The modular architecture potentially improves transparency and accountability by letting recruiters track how extracted resume information contributes to assessments.Single-model systems are contrasted as returning a final score without insight into the reasoning process.Source: §6 · p. 8
- Extraction quality affects assessment quality, and its effectiveness depends on a model’s reasoning ability to determine what information to extract.The authors report that models with stronger reasoning capabilities and more parameters tend to perform better at extraction.Source: §6 · p. 8
7. Future Work
Future work proposes multimodal hiring assessment and identifies bias and privacy as continuing concerns for AI-driven hiring.
7. Future Work
- Video interview analysis and generated aptitude or attitude tests are proposed to broaden text-based assessment of soft skills, communication, and claimed capabilities.The paper notes current systems mainly focus on textual resume evaluation.Source: §7 · p. 8
- Imbalanced training data may leave demographic groups underrepresented, motivating bias-aware retrieval and ranking strategies.The paper presents RAG as a potential way to retrieve diverse, up-to-date hiring criteria and reduce reliance on static, historically biased datasets.Source: §7 · p. 8
- External inference APIs may expose sensitive data to third parties, while local deployment is an option for companies with sufficient computational resources.The authors identify privacy-preserving architectures and encrypted inference as future research directions.Source: §7 · p. 8
8. Conclusion
The conclusion presents a four-agent RAG-LLM framework that evaluates resumes against company-specific criteria and reports performance comparable to HR evaluators.
8. Conclusion
- The framework uses agents to extract resume information, evaluate candidates against predefined criteria, summarize assessments, and format structured outputs.RAG supports tailored assessment against company-specific scoring criteria without model retraining or fine-tuning.Source: §8 · p. 8