Source-linked AI summary
Building Interpretable Feature Representations for Resume-Vacancy Matching by Distilling Production LLM Signals
Ilya Chekin, Vyacheslav Malyugin, Vladimir Chirkov, Mikhail Yurushkin
TL;DR
A single relevance score does not show recruiters why a candidate fits a vacancy. The paper uses a feedback-refined LLM labeler offline to supervise a feature bi-encoder for online service. The deployed student agreed with 888 of 927 recruiter-recorded values, in a selected, non-blinded production-feedback subset.
Problem
A single relevance score does not identify which matching dimensions explain why a candidate fits a vacancy.
Source: §1 · p. 1Method
A feedback-refined LLM labeler supplies supervision for a distilled feature bi-encoder that serves named matching dimensions.
Source: §1 · p. 1Results
888 of 927 recruiter-recorded values (95.79%) matched the deployed student’s predictions in the production-feedback subset.
Source: §10 · p. 7Takeaways & Limitations
The deployed approach surfaces named per-dimension values rather than only an opaque score, with the bi-encoder serving online on CPU without per-comparison API calls.
Source: §1 · p. 1, §6 · p. 5The feedback check is sparse, selected, and non-blinded, and the paper does not measure downstream hiring outcomes or claim causal business impact.
Source: Limitations · pp. 7–8
Abstract
from arXiv · showhide
Matching candidates to vacancies is central to recruitment, and a recruiter needs to see why a candidate fits, not only a single opaque relevance score. We provide this evidence as named, interpretable matching dimensions recruiters can act on - eight in our current deployment. We propose a two-part approach. The first is an LLM-based labeler whose prompts and feature definitions were refined from recruiter feedback while it served as an earlier production matching stage. In the current architecture, it is used only for offline labeling and is not called on online requests. The second is a feature bi-encoder distilled from it: a LoRA-adapted embedding backbone with compact per-dimension heads that runs on CPU and serves all online requests. Both parts keep improving: prompts are revised as feedback arrives, and the bi-encoder is retrained on the updated labels. The model is trained on 168,772 labeled vacancy-resume pairs (17,921 vacancies and 180,030 resumes). Recruiters using the service can confirm or revise surfaced feature predictions. On 927 recruiter-recorded values from this selected production-feedback subset, the deployed student agrees with the recorded decisions in 888 cases (95.79%). This is operational, non-blinded agreement rather than an independent human evaluation.
1 Introduction
The paper frames vacancy matching as a need for named, actionable dimensions rather than a single relevance score. It introduces a feedback-refined labeling and distillation approach designed for production constraints, informed by recruiter feedback and a corpus of real vacancy–resume pairs.
1 Introduction
- Recruiters need named matching dimensions because a single relevance score orders candidates but does not explain why they fit.The paper identifies dimensions such as job-title alignment, skills, experience recency, seniority, and certifications.Source: §1 · p. 1
2 Related Work
- Earlier person–job-fit and retrieval systems return an order or score, while this work targets per-dimension reasons for fit.The paper situates its approach alongside LLM-derived signals and distillation systems, with detailed comparison deferred to §8.Source: §2 · p. 2
3.1 System Evolution: Feedback-Driven Design
- Recruiter feedback refined prompts and feature definitions during production use, including splitting skill matching into distinct dimensions and adding certification matching.During a three-week pilot, three manual prompt updates covered 247 reviewed feature values, using a fixed regression set and newly observed error cases.Source: §3.1 · p. 2
3.2 Task and Production Constraints
- The system models fit as eight matching dimensions and must also meet low-latency, low-cost, scalability, interpretability, and document-format robustness requirements.Seven dimensions are ordinal and certification is binary.Source: §3.2 · pp. 2–3
3.3 Data
- The training corpus contains 168,772 labeled vacancy–resume pairs across 17,921 vacancies and 180,030 resumes.The data comes from a commercial recruitment setting with recurring users and heterogeneous domains and document formats.Source: §3.3 · p. 3
4 LLM-Distilled Feature Bi-Encoder
The approach turns a production-refined LLM labeler into offline supervision for an online feature bi-encoder. Independent document encodings and per-dimension heads support cached representations, while a vacancy-only applicability head gates certification output.
4.1 Training Data Construction
- The LLM now labels comparisons offline with production-refined prompts, and new feedback can trigger prompt revision, relabeling, and student retraining.The labels are treated as supervision signals rather than perfect ground truth.Source: §4.1 · p. 3
4.2 Encoder and Pair Representation
- The LoRA-adapted encoder independently represents vacancies and resumes, whose pair representation combines their embeddings, difference, and element-wise product.Resume embeddings can be precomputed and cached; the difference captures dimensional misalignment and the product captures agreement.Source: §4.2 · p. 3
4.3 Task-Specific Heads
- Independent three-layer MLP heads expose each matching dimension separately, with two outputs for ordinal dimensions and one for the binary dimension.Separate heads preserve per-dimension interpretability and allow individual evaluation, updating, or ablation.Source: §4.3 · pp. 3–4
4.4 Query-Only Applicability Heads
- A vacancy-only binary applicability head excludes certification output when the vacancy does not require certification.This avoids penalizing candidates for an unstated requirement and keeps vacancy-side applicability separate from pairwise matching.Source: §4.4 · p. 4
5 Training Procedure
The training procedure uses a cross-encoder as an offline source of soft per-class targets, then trains the bi-encoder in two stages with a hybrid loss. The cross-encoder more closely reproduces the LLM labels, but the student is the model used for online serving.
5.1 Cross-Encoder Teacher
- The cross-encoder reaches 0.934 accuracy and 0.897 macro-F1 on held-out pairs, above the bi-encoder student’s 0.886 accuracy and 0.849 macro-F1.Its soft outputs carry information beyond the hard labels and it serves only as an offline teacher because joint long-context GPU inference misses serving-cost targets.Source: §5.1 · p. 4
5.2 Two-Stage Training
- Training first freezes the encoder while fitting the heads, then unfreezes LoRA for joint encoder-and-head fine-tuning.The first stage provides stable initialization; the second adapts the representation space to the matching-signal distribution.Source: §5.2 · p. 4
5.3 Hybrid Loss
- The hybrid loss combines teacher distillation, smoothed hard-label supervision, ordinal-consistency regularization, and a label-distance-scaled margin loss.Distillation matches student and teacher cumulative probabilities using BCE with temperature.Source: §5.3 · p. 4
6 Feature Extraction and Interpretability
The feature bi-encoder returns named, per-dimension verdicts for recruiters rather than only a fused similarity score. Its architecture uses shared vacancy and resume encoding, while its training pipeline distills offline LLM labels into an online student.
6 Feature Extraction and Interpretability
- Each candidate receives per-feature classes across eight dimensions, giving recruiters named evidence rather than only a similarity number.The system focuses evaluation on the roughly 50 candidates a recruiter typically reviews per vacancy.Source: §6 · pp. 4–5
- A shared Qwen3-Embedding-0.6B + LoRA encoder represents vacancy and resume independently, and the pair representation feeds eight feature-specific MLP heads.A vacancy-only certification applicability head masks that dimension when the vacancy does not require it.Source: Figure 1 · p. 5
- Offline LLM labels train a cross-encoder teacher whose soft targets distill into the feature bi-encoder through a hybrid loss and two training stages.The deployed student removes online LLM calls.Source: Figure 2 · p. 5
- Feature values are directly supervised targets with fixed label spaces, so each dimension can be evaluated, corrected, retrained, or masked independently.Source: §6 · p. 5
7 Experiments
The experiments assess student fidelity to LLM labels, agreement with recruiter-recorded decisions, and serving efficiency. The deployed student matches 888 of 927 recorded values, while adapting the encoder raises macro-F1 from 0.669 to 0.849.
7.1 Evaluation Setup
- The evaluation separates student-to-teacher fidelity on held-out vacancy-split data, operational agreement with recruiter-recorded decisions, and serving efficiency.Recruiters saw the deployed prediction before confirming or revising a value; only selected cases and features were recorded.Source: §7.1 · p. 5
7.2 Student-to-Teacher Fidelity and Ablations
- Macro-F1 rises from 0.669 to 0.849 when the original frozen encoder is replaced by the adapted representation and heads.The cross-encoder has the highest fidelity but requires joint long-context GPU inference and cannot cache resume representations.Source: §7.2 · p. 5, Table 2 · p. 6
- Seniority Alignment has the lowest F1 (0.795), plausibly reflecting recruiter-specific meanings of seniority and motivating possible future decomposition.The reported fidelity, operational recruiter agreement, and efficiency evaluations use different targets and settings; the operational values come from a selected, non-blinded subset.Source: §7.2 · p. 6, §7.3 · p. 6
- Knowledge distillation provides the largest individual gain over hard-label cross-entropy; ordinal consistency and margin regularization add smaller complementary improvements.The absolute-difference term contributes more individually than the element-wise product, while the product adds complementary signal.Source: §7.2 · p. 6
7.3 Operational Agreement with Recruiter-Recorded Decisions
- 888/927 (95.79%) deployed-student matches to recruiter-recorded values were observed on the selected feedback subset, not independent blinded human accuracy.The 39 disagreements comprise 34 Seniority Alignment, three Certifications, and two Job Title Match cases.Source: §7.3 · p. 6
7.4 Efficiency
- About 5,500 resumes/s on a single CPU and roughly $4.7 × 10^-5 per resume are reported for serving, including offline precomputation.Resume embeddings are precomputed on a T4 and cached; online serving uses only lightweight MLP heads, with no GPU or external API access.Source: §7.4 · p. 6
8 Positioning Relative to Closest Work
The paper distinguishes its eight named dimensions from prior systems that produce a single score or free-text rationale. Its comparisons are qualitative against prior work and quantitative only through controlled internal ablations and recorded recruiter feedback.
8 Positioning Relative to Closest Work
- Closest systems distill LLM matching signals into a single score or free-text rationale, while this approach exposes eight independently supervised, named dimensions.The cited systems include Jouanneau et al. (2026), Fu et al. (2025), and Yu et al. (2026).Source: §8 · pp. 6–7
- The full representation and full objective perform best in the controlled macro-F1 ablations.The table caption identifies the ablation scope and result, but supplies no specific scores.Source: Table 3 · p. 7
- Table 4 concerns agreement between the deployed student and explicitly recorded recruiter decisions, which are non-blinded because recruiters saw the prediction first.Its Wilson intervals quantify uncertainty in observed proportions, not population accuracy.Source: Table 4 · p. 7
- Published F1 and accuracy values from the closest systems are not directly comparable because their datasets and prediction targets differ.The paper therefore reports controlled internal baselines rather than a direct quantitative benchmark against those systems.Source: §8 · p. 7
9 Production Lessons
The production lessons favor feature-level outputs and a division of labor: recruiter feedback refines LLM supervision, while a distilled model serves requests online. Applicability gating avoids scoring requirements a vacancy did not raise.
9 Production Lessons
- Per-dimension outputs localize mismatches in skills, seniority, recency, or certification instead of collapsing them into a fused score.Query-activated applicability heads score a feature only when the vacancy presents it, avoiding penalties for requirements it never raised.Source: §9 · p. 7
- Recruiter feedback on the live system refines LLM supervision, while the LLM bootstraps matching logic without manual labels.The refined LLM is used offline as a labeling engine; a distilled model serves online requests.Source: §9 · p. 7
10 Conclusion
The conclusion presents an eight-dimension matching system with a feedback-refined LLM labeler and a CPU-served distilled bi-encoder; recorded production feedback provides a non-blinded operational check. The authors qualify that evidence and the system’s scope with data, evaluation, and fairness limitations.
10 Conclusion
- 888/927 recruiter-recorded values (95.79%) agree with the deployed student on the selected, non-blinded production-feedback subset.This is operational agreement, not an independent human evaluation.Source: §10 · p. 7
- The current deployment predicts eight named matching dimensions using a feedback-refined LLM labeler and a CPU-served feature bi-encoder.The bi-encoder serves without per-comparison API calls, surfacing per-dimension values rather than an opaque score.Source: §10 · p. 7
Limitations
- Feedback is sparse, selected, and non-blinded, and single-recruiter records prevent reporting inter-annotator agreement.Multiply judged values are identified as future work.Source: Limitations · p. 7
- The proprietary vacancy and resume corpus cannot be publicly released, limiting external reproducibility.The corpus spans multiple domains and document formats, but evaluation on entirely new labor-market segments is still needed.Source: Limitations · p. 7
- No comparable external quantitative benchmark or downstream hiring-outcome measurement is available, so the authors make no causal business-impact claim.Education and language dimensions are omitted for insufficient labeled data, and evaluation is limited to English-language vacancies and resumes.Source: Limitations · p. 8
Ethical Considerations
- The study did not conduct a dedicated empirical bias audit, and anonymization does not rule out indirect textual proxies or bias transferred from the labeling LLM.Counterfactual testing and group-disparity analysis remain future work; feature explanations are model predictions, not objective assessments of ability.Source: Ethical Considerations · p. 8
A Comparison with Closest Work
The appendix contrasts the approach with closest work, defines its eight feature-label schemes, and documents the production prompt templates and their refinement protocol.
A Comparison with Closest Work
- Table 5 qualitatively contrasts the approach with closest LLM-for-person–job-fit systems discussed in the paper.The prior systems target a single score or free-text rationale, whereas this approach exposes eight independently supervised, named dimensions.Source: A Comparison with Closest Work · p. 9, Table 5 · p. 10
B Feature Definitions and Label Schemes
- The bi-encoder predicts eight matching dimensions: seven use three ordered classes, while certifications is binary after vacancy-only applicability gating.Class 2 represents the strongest match; certification gating removes the not_required case.Source: B Feature Definitions and Label Schemes · p. 9, Table 6 · p. 10
C LLM Prompt Templates
- The production prompt templates evaluate four skill-related features jointly and use analogous single-feature prompts for the remaining features, with vacancy-only gating before pairwise evaluation.Templates were used for offline labeling of 168,772 training pairs, and a shared system context inserts the vacancy and resume into each call.Source: C LLM Prompt Templates · pp. 9–10, Figure 3 · p. 11
C.1 Prompt Refinement Protocol
- During a three-week pilot, the ML team manually reviewed recruiter corrections and released three prompt updates after testing them against new errors and a fixed regression set.The 247 reviewed values supported iterative prompt development and were not an independent test set; revisions had to fix and generalize the target pattern without regression.Source: §C.1 · p. 10
D Stability and Agreement Protocols
The section measures label-generation stability across repeated runs and reports agreement intervals that account for clustering in the selected recruiter-feedback subset.
D Stability and Agreement Protocols
- 86.7%–98.7% of labels were unanimous across five runs, while first-run agreement with the five-run majority ranged from 94.8% to 99.7%.These measurements describe run-to-run stability, not calibrated probabilistic uncertainty.Source: D Stability and Agreement Protocols · p. 11
- Six dimensions use one run at scale; Industry Experience and Seniority Alignment use additional runs after disagreement or ambiguity.The three cases without a stable majority included two corrupted or nearly empty resumes excluded from analysis and one manually reviewed ambiguous case.Source: D Stability and Agreement Protocols · p. 11
- Candidate-cluster and vacancy-cluster percentile intervals for pooled exact agreement were 94.67–96.89% and 94.11–97.37%, respectively.The intervals account for clustering in the selected feedback subset but do not make it representative of all surfaced candidates.Source: D Stability and Agreement Protocols · p. 11
E Per-Feature Evaluation Results
Table 7 reports per-feature macro-F1 on a held-out vacancy-disjoint split across experimental conditions. The accompanying implementation details describe the feature prompts, model, training stages, and CPU serving setup.
E Per-Feature Evaluation Results
- Table 7 compares per-feature macro-F1 across experimental conditions on the same held-out, vacancy-disjoint validation split.The conditions include a majority baseline, an MLP on original encoder embeddings, and the full model; bold marks each feature’s best result.Source: Table 7 · p. 11, E Per-Feature Evaluation Results · p. 11
F Training and Implementation Details
- The encoder uses Qwen3-Embedding-0.6B adapted with LoRA, with 4096-token truncation and last-token representations.LoRA is applied to attention and MLP projections; BF16 mixed precision is used.Source: F Training and Implementation Details · p. 11
- Eight task-specific MLP heads process the 4096-dimensional pair representation, while the certification applicability head uses a separate two-layer trunk.The task-specific heads use intermediate dimensions 1024 and 512; the certification trunk uses 1024→768→384.Source: F Training and Implementation Details · p. 11
- Training uses a two-stage schedule: first freeze LoRA and train the heads, then unfreeze LoRA and fine-tune jointly.The setup uses AdamW for five epochs, with KD, monotonicity, and margin-loss hyperparameters specified.Source: F Training and Implementation Details · pp. 11–12
- A Qwen3-4B reranker is trained on LLM hard labels, and its per-pair logits are cached before bi-encoder training.Source: F Training and Implementation Details · p. 12
- The deployed model serves on autoscaled commodity x86 CPU instances; resume embeddings are precomputed offline and serving uses no GPU.Experiments ran on one A100-80GB GPU, while embedding precomputation uses a single T4 instance.Source: F Training and Implementation Details · p. 12
- One LLM call evaluates four correlated skill dimensions jointly against a shared required-skill cluster; separate prompts cover four other dimensions.Research and academic skill contexts are merged because annotated examples are limited and both represent non-production skill use.Source: Figure 4 · p. 13