PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Modeling Approaches for Experience-Level Classification in Job Description

Celia Liang, Eddie Wu, Shiqi Wang, Yonah You

arXiv:2610.04304v1cs.CL

TL;DR

Experience-level prediction must draw on structured recruitment text because job titles are ambiguous and relevant clues appear across sections. The paper proposes Section-Aware BERT alongside baseline and LLM evaluations, and reports 82.5% accuracy for its final model, with systematic errors across experience levels.

  • Problem

    Recruitment descriptions contain distinct sections with unequal experience clues, while job titles can be ambiguous across industries and companies.

    Source: §1 · p. 1

  • Method

    The paper proposes Section-Aware BERT, which encodes recruitment text by structured paragraphs for feature fusion, and evaluates LLMs with few-shot and fine-tuning supervision.

    Source: §1 · pp. 1–2

  • Results

    82.5% accuracy: final Section-Aware BERT achieves the best performance among non-LLM models, while errors significantly confuse Entry and Senior and leave Mid-level roles unstable.

    Source: §1 · p. 2, §9.1 · p. 11

  • Takeaways & Limitations

    The study offers a structured modeling approach and analytical framework as a reference for future recruitment-text research.

    Source: §1 · p. 2

    TF-IDF’s bag-of-words representation cannot disambiguate context-dependent uses of polysemous terms such as “lead.”

    Source: §7.2 · p. 9

Abstract

from arXiv · show

This paper investigates the task of predicting job experience levels in recruitment texts, aiming to automatically identify the qualifications required for positions. Unlike traditional text classification, recruitment texts typically possess explicit internal structures, with different paragraphs playing disproportionate roles in conveying experience clues. To address this, we propose a structure-aware Section-Aware BERT approach that segments and encodes key paragraphs (titles, responsibilities, requirements) for integrated modeling, building upon rule-based systems and classical baselines TF-IDF. Simultaneously, we evaluate large language models under both few-shot and fine-tuning settings on the same dataset to compare the capability boundaries of different modeling paradigms. Experimental results demonstrate that explicitly leveraging text structure significantly improves experience level prediction performance, particularly in scenarios with ambiguous job titles. Further error analysis reveals systemic challenges in this task, including confusion between Entry and Senior levels and the blurred boundaries of Mid-level positions. This research provides an effective modeling approach and analytical framework for understanding structured recruitment texts.

1 Introduction

Experience-level classification must combine clues across structured recruitment sections because titles are ambiguous and paragraphs contribute different kinds of evidence. The paper proposes Section-Aware BERT and compares it with traditional models and LLMs, while analyzing systematic classification errors.

1 Introduction

  • Recruitment sections convey experience clues from different dimensions, so classification requires combining information across the description rather than relying on job titles alone.Titles can map to different responsibilities and qualification levels across industries and companies.Source: §1 · p. 1
  • Section-Aware BERT encodes structured paragraphs and fuses their features, while the study also evaluates LLMs under few-shot and fine-tuning settings against traditional models.The comparisons aim to explore the performance ceilings and applicable scenarios of different approaches.Source: §1 · p. 2
  • Errors are systematic: Entry and Senior positions are significantly confused, while Mid-level roles remain unstable because their semantic boundaries are ambiguous.The analysis includes confusion matrices, keywords from misclassified samples, and model behavior comparisons.Source: §1 · p. 2
  • The study examines experience-level classification under relatively limited data conditions using baselines, structured modeling, and LLM experiments.It presents the model design and analytical framework as a reference for future recruitment-text research.Source: §1 · p. 2

2 Related Work

Recruitment-text research has moved from handcrafted features and keyword rules toward pretrained language models, while prior work has also explored document structure and experience inference. These studies motivate modeling more than job titles or whole-document text.

2 Related Work

  • Early recruitment-text classification relied on manually designed features and traditional models, including word n-grams and TF-IDF, while later work adopted Transformers to capture richer semantics.Malherbe et al. (2015) mapped advertisements to standardized job classifications; Senger et al. (2024) reviewed the shift toward pretrained language models.Source: §2 · p. 2
  • Li et al. (2020) used document segmentation and multi-head attention for experience-level prediction, reporting approximately 73% accuracy on manually annotated data.Their work focused primarily on candidate resumes rather than job advertisements.Source: §2 · pp. 2–3
  • Laosaengpha et al. (2024) found that job titles alone do not capture job semantics consistently, supporting the use of responsibilities, requirements, and skills.Their title representation method aggregated job descriptions to address title inconsistencies across companies and industries.Source: §2 · p. 3
  • Gnehm and Clematide (2020) combined text zoning with hierarchical classification and then used BERT on zoned text, demonstrating structured processing in recruitment-text understanding.Their framework used BiLSTM-CRF for paragraph identification.Source: §2 · p. 3

3 Dataset and Preprocessing

The study prepares LinkedIn job postings for experience-level classification by standardizing labels, cleaning records, extracting text sections, and creating stratified train, development, and test sets.

3.1 Dataset Description

  • The dataset contains approximately 124,000 LinkedIn postings with recruiter-provided experience labels, and cleaning and label mapping yielded 80,717 valid annotated postings.The final set includes 36,110 Senior, 31,453 Entry, 8,719 Mid, and 4,435 Executive postings.Source: §3.1 · p. 3, §3.4 · p. 4

3.2 Experience Level Mapping

  • The six original experience categories were mapped to four standardized labels: Entry, Mid, Senior, and Executive.Entry-level and Internship map to Entry; Associate to Mid; Mid-Senior to Senior; Director and Executive remain Executive-level.Source: §3.2 · p. 3

3.3 Data Cleaning

  • Cleaning retained relevant fields, removed missing or short descriptions and duplicate texts, replaced missing skills descriptions with empty strings, and standardized text formatting.Descriptions shorter than 30 characters were filtered; samples missing the essential experience-level label were discarded.Source: §3.3 · pp. 3–4

3.4 Section Segmentation

  • Paragraph analysis extracted Responsibilities, Requirements, and Other sections; extraction succeeded for 97.9% of Responsibilities and 91.1% of Requirements, with Other available in all samples.Alternative text segmentation supplemented postings whose headings could not be clearly identified.Source: §3.4 · p. 4

3.5 Dataset Splits

  • Stratified sampling split the data into 56,501 training, 12,108 development, and 12,108 test instances to preserve class representation across subsets.The strategy was intended to ensure minority categories such as Executive were represented and mitigate assessment bias from underrepresentation.Source: §3.5 · p. 4

4 Baseline Methods

The study compares a rule-based regular-expression system with a TF-IDF and logistic-regression baseline. The former applies ordered explicit-pattern checks, while the latter learns statistical patterns from concatenated posting text.

4.1 Rule-Based BaselineRegex System

  • The regex baseline checks title patterns in a hierarchy—Executive, Entry, Mid, then Senior—and examines description phrases if no title pattern matches.The system checks for “associate” as Mid unless “senior associate” appears, and defaults to Senior when no patterns match.Source: §4.1 · p. 4

4.2 Classical ML Baseline: TF-IDF + Logistic Regression

  • The TF-IDF baseline concatenates each job title and description, then uses logistic regression to learn statistical patterns rather than handcrafted rules.Vectorization uses up to 5,000 unigram and bigram features, with document-frequency filtering and English stop-word removal.Source: §4.2 · p. 5
  • Development-set tuning tested logistic-regression C values of 0.01, 0.1, 1, 10, and 100, selecting C = 10 for the final model.The classifier uses L2 regularization.Source: §4.2 · p. 5

5 Advanced Models

The section compares a single-sequence BERT baseline, a proposed structure-aware BERT model, and GPT-4o-mini under few-shot and fine-tuning settings. Section-Aware BERT encodes job-description sections separately and fuses their representations, while the LLM experiments compare supervision settings on the task.

5.1 Baseline BERT

  • Baseline BERT fine-tunes BERTbase-uncased on the concatenated title and description as one sequence for four-class prediction.The [CLS] representation feeds a dropout layer and linear classifier.Source: §5.1 · p. 5

5.2 Section-Aware BERT

  • Section-Aware BERT separately encodes Title, Responsibilities, Requirements, and Other before fusing their representations with multi-head attention.The sections provide distinct signals, including seniority markers, task complexity, and explicit qualification indicators.Source: §5.2 · p. 5

5.2.1 Model Architecture

  • A shared BERT encoder produces one 768-dimensional [CLS] vector per section, and the vectors are stacked for self-attention fusion.After attention, residual connection, and layer normalization, the model mean-pools the section vectors and classifies the result.Source: §5.2.1 · p. 6

5.2.2 Training

  • All four sections share the encoder and the baseline training configuration, while Section-Aware BERT uses four BERT passes per posting.Its section sequences are shorter—64–256 tokens each versus 512 tokens—partially offsetting the extra passes.Source: §5.2.2 · p. 6

5.3 LLM Baseline

  • GPT-4o-mini is evaluated in few-shot and fine-tuning settings to compare a general-purpose LLM with task-specific structural modeling.The comparison uses the same dataset, with the LLM experiments serving as ceiling baselines under different supervision settings.Source: §5.3 · p. 6

5.3.1 Few-Shot LLM

  • Few-shot prompting supplies task instructions, label definitions, and 12 examples, then classifies postings truncated to 1,500 characters.The evaluation uses a stratified sample of 400 test examples because of API cost constraints.Source: §5.3.1 · p. 6

5.3.2 Fine-Tuned LLM

  • Fine-tuned GPT-4o-mini uses 1,000 stratified training examples and 200 validation examples, then is evaluated on the few-shot setting’s test sample.The training set is 1.8% of the BERT training data, limiting direct comparability; API costs prevented scaling to the full training set.Source: §5.3.2 · pp. 6–7

6 Results

Section-Aware BERT leads the reported overall comparison, while performance varies substantially by experience class. Mid-level remains challenging, and the LLM results differ between few-shot and fine-tuned settings.

6.1 Overall Performance Comparison

  • 50.35% accuracy: the rule-based regex system relies on handcrafted patterns and struggles with implicit experience indicators.Its Entry and Mid F1 scores are 0.18 and 0.13, respectively, while Executive F1 is 0.75.Source: §6.1 · p. 7, §6.2 · p. 7
  • 73.41% accuracy: TF-IDF + Logistic Regression improves by 23 percentage points over regex, but Mid F1 remains 0.36.Entry and Senior F1 are 0.78 and 0.77; Executive recall is 0.63.Source: §6.1 · p. 7, §6.2 · p. 8
  • Section-Aware BERT achieves the strongest performance, with 82.56% accuracy and 76.00% macro F1, a 9.15 percentage point improvement over TF-IDF, by encoding postings into structured sections and learning their relative importance.Source: §6.1 · p. 7

6.2 Per-Class Performance Analysis

  • Section-Aware BERT has the strongest F1 across all categories, including Entry (0.88), Mid (0.54), Senior (0.85), and Executive.Mid-level remains its most challenging category despite gains over the baselines.Source: §6.2 · p. 8, Table 3 · p. 8
  • GPT-4o-mini few-shot reaches 0.85 F1 on Executive but only 0.18 on Mid; fine-tuning raises Entry and Senior performance while Mid recall falls to 0.11.The paper attributes the fine-tuned model’s Mid-level collapse to conservative classification for an ambiguous category with limited training samples.Source: §6.1 · p. 7, §6.2 · p. 8

7 Error Analysis

The error analysis traces performance and failure patterns across rule-based, TF-IDF, and Transformer models, finding that section-aware modeling improves classification while ambiguity and information loss remain important challenges.

7.1 Rule-Based System Limitations

  • The rule-based regex system achieves only 50.4% accuracy; explicit experience-level keywords occur in 24% of test samples, with coverage of 4.7% for Entry-level and 70.9% for Executive positions.Source: §7.1 · p. 8
  • Without keywords, regex defaults 76% of samples to Senior, misclassifying 4,047 Entry positions; polysemous titles such as coordinator also defeat surface-pattern matching.“Coordinator” has a 38.6% error rate, and the same title term can occur at different experience levels.Source: §7.1 · p. 8

7.2 TF-IDF Vocabulary Limitations

  • TF-IDF reaches 73.5% accuracy, 23 percentage points above regex, but its unordered representation loses word order, making opposite-meaning sentences indistinguishable.Table 4 illustrates pairs that imply different seniority levels but yield identical bag-of-words vectors.Source: §7.2 · p. 8, Table 4 · p. 8
  • TF-IDF’s global word weights cannot resolve polysemy: coordinator, analyst, and specialist errors are 38.6%, 37.6%, and 35.2%, while Entry↔Senior errors dominate and Mid-level F1 is 36.7%.Entry→Senior accounts for 830 errors and Senior→Entry for 807; Mid-level vocabulary overlaps both neighboring categories.Source: §7.2 · p. 9, Table 5 · p. 9, Figure 3 · p. 9

7.3.1 Baseline BERT

  • Baseline BERT reaches 78.5% accuracy, five percentage points above TF-IDF, and contextual embeddings disambiguate polysemous terms, though long descriptions may dilute title signals.It encodes the concatenated title and description as one sequence with a 512-token maximum.Source: §7.3.1 · p. 9

7.3.2 Section-Aware BERT

  • 82.6% accuracy, a 4.1-percentage-point gain over Baseline BERT, makes Section-Aware BERT the strongest model; all classes benefit, with Executive gaining most (+8.1%).The recall comparison is summarized in Table 6.Source: Table 6 · p. 9, §7.3.2 · p. 9
  • Executive samples where Section-Aware wins show 81.2% leadership-verb and 40.0% strategic-scope prevalence, versus 60.0% and 15.0% for Entry, aligning with the largest gain.Title receives the highest average attention (36.5%), while Responsibilities and Requirements together contribute 40.3%.Source: §7.3.2 · pp. 9–10, Table 8 · p. 10, Figure 4 · p. 10
  • Section-Aware fixes 1,048 Baseline BERT errors, chiefly bidirectional Entry↔Senior confusion in cases where ambiguous titles require section-level qualification or task-scope clues.Dedicated section token budgets also preserve Requirements and Responsibilities signals that may be truncated in Baseline BERT.Source: Table 9 · p. 10, §7.3.2 · p. 10

8 Discussion and Limitations

The discussion reports Section-Aware BERT’s comparative strength and identifies limits involving label quality, section segmentation, the LLM evaluation subset, and persistent Mid-level ambiguity.

8.1 Strengths

  • Section-Aware BERT achieves 82.6% accuracy, outperforming TF-IDF (73.4%) and fine-tuned LLMs (71.9%); titles contribute the most attention (36%).Source: §8.1 · p. 10

8.2 Limitations

  • Employer-provided LinkedIn experience labels can be noisy across industries and companies, bounding achievable performance.Source: §8.2 · p. 10
  • Rule-based section segmentation may introduce boundary noise when headings are missing or unconventional.Source: §8.2 · p. 10
  • API costs restricted LLM evaluations to a stratified subset rather than the full test set.Source: §8.2 · p. 10
  • Mid-level remains challenging across all models, with a best F1 of 0.54, reflecting ambiguity in this transitional tier.Source: §8.2 · p. 11

9 Conclusion and Future Work

The conclusion reports stronger performance from section-aware modeling and identifies remaining challenges in title modeling, segmentation, and inconsistent labels. It presents these as directions for further work.

9.1 Conclusion

  • Traditional baselines retain some reference value, but TF–IDF substantially confuses Entry–Senior and Mid-level categories when distinguishing fine-grained experience levels.The paper attributes this difficulty to lexical ambiguity, contextual gaps, and the limits of bag-of-words representations.Source: §9.1 · p. 11
  • 82.5% accuracy: final Section-Aware BERT achieves the best performance among non-LLM models and outperforms traditional methods in overall performance and category balance.Section-based modeling is especially effective when job titles are ambiguous, drawing on experience clues across key paragraphs.Source: §9.1 · p. 11
  • Standard BERT performs well with explicit experience indicators, while section-aware BERT better extracts and integrates implicit cues dispersed across paragraphs.Source: §9.1 · p. 11

9.2 Future Work

  • Future title modeling could use explicit interactions between titles and body paragraphs to balance title cues with implicit experience signals.The study co-models titles with responsibility paragraphs, but title cues still differ across industries and positions.Source: §9.2 · p. 11
  • Rule-based segmentation scales to large datasets but may introduce boundary noise; learning-based segmentation or dynamic attention could reduce reliance on fixed rules.Source: §9.2 · p. 11
  • Future work could explore larger or domain-adaptive models, multi-task learning, and cross-dataset transfer to mitigate noise from inconsistent platform label definitions.Source: §9.2 · p. 11
Loading 2610.04304v1…