Bug Severity Classification with Fine-Tuned Open LLMs: A Paired Comparison with TF-IDF and CodeBERT Baselines on Eclipse Reports

1. Introduction

• Severity is a fixed Bugzilla field triagers use to prioritize reports, but the Eclipse collection has many default-normal labels and duplicates can carry conflicting severities.The passage reports that two thirds of Eclipse reports are labelled normal and cites evidence of inconsistent severities among duplicates.Source: §1 · p. 1
• Published comparisons rarely establish whether 7–8B open models beat a much smaller fine-tuned encoder because datasets, labels, splits, and metrics differ.The introduction identifies scoring every model on the same reports with uncertainty as uncommon.Source: §1 · p. 1
• The study compares eleven configurations of TF-IDF classifiers, CodeBERT, and zero-shot or QLoRA-fine-tuned Llama 3 and Qwen 2.5 on one task and split.All models are evaluated on the same 2,000 test reports; the paper also releases split identifiers, per-report predictions, and the analysis script.Source: §1 · p. 2
• The work audits a public dump, measures LLM comparisons and class trade-offs, and releases CyberTriage to serve a fine-tuned classifier locally.The rationale written by CyberTriage’s second LLM cannot change the stored label.Source: §1 · p. 2

2.1. Severity prediction from report text

• Prior severity predictors range from rule learners, Naive Bayes, BM25 neighbours, and topic models to neural networks using report text and other features.Studies used different label schemes, including five Bugzilla levels and a binary severe/non-severe split.Source: §2.1 · p. 2
• Recorded Bugzilla severities are noisy: about 51% of duplicate reports had inconsistent labels, making agreement with the recorded label an imperfect yardstick.Many studies also drop the default normal label.Source: §2.1 · p. 2
• Selected studies span different data and label schemes, so their reported scores cannot be compared across table rows.A dash indicates information the authors could not confirm from the published abstract or record.Source: Table 1 · p. 3

2.2. Pre-trained encoders and generative models

• Later work applied fine-tuned encoders and generative models, including prompted or fine-tuned LLMs, but differences in data, labels, and splits prevent direct score comparisons.The reviewed approaches include GPT-2-based predictors and Llama 2 on a severe/non-severe Eclipse and Mozilla task.Source: §2.2 · p. 3
• QLoRA keeps frozen 7–8B model weights in 4-bit NormalFloat so they can be tuned on one GPU; the paper uses it as an off-the-shelf method [13].LoRA trains low-rank updates alongside frozen weights [24].Source: §2.2 · p. 3

3.1. Source and provenance

• The public dataset contains 361,006 unique Eclipse reports after removing 60,000 repeated rows; the dump also includes descriptions that may contain post-filing notes.All reports are resolved, closed, or verified, and the severity field has no history showing whether labels were revised.Source: §3.1 · p. 3

3.2. Labels and exclusions

• After excluding normal and enhancement and merging blocker with critical, the four-class pool contains 72,337 unique reports, with Major the largest class at 48.6%.The pool includes 48.6% Major, 26.3% Critical, 18.5% Minor, and 6.6% Trivial reports.Source: §3.2 · p. 4

3.3. Sampling, split and what the split contains

• The sampler drew 5,000 rows per class and split the 20,000-row sample 80/20, stratified by class, into 16,000 training and 4,000 test reports.Rerunning the preparation script reproduces the split files exactly.Source: §3.3 · p. 4
• Row-based sampling yielded 19,038 distinct reports, and 319 test reports (8.0%) duplicate a training description; 41 descriptions also have conflicting severities.The authors did not manually inspect reports and make no claim about the overall noise rate.Source: §3.3 · p. 4
• Only 64.4% of sampled reports were FIXED, while 6.4% exceeded the 4,000-character limit used for LLM inputs.The balanced test set has no natural-prevalence counterpart; the paper estimates shifts under the pool’s class mix.Source: §3.3 · p. 4

4. Models and evaluation protocol

• The study design compares eleven model configurations, with all models evaluated on a shared 2,000-report subset.The TF-IDF classifiers were fit on 12,800 of 16,000 training reports; other models were re-scored on the shared subset from stored files.Source: §4 · p. 4, Figure 1 · p. 5

4.1. TF-IDF classifiers

• TF-IDF word 1–3-gram features feed six classifiers, with hyperparameters selected by 10-fold grid search on macro-F1.The vectorizer was fit before cross-validation, so folds shared its vocabulary and inverse document frequencies; this affected hyperparameter choice, not the test split.Source: §4.1 · pp. 4–5

4.2. CodeBERT

• CodeBERT is fine-tuned end-to-end with a four-way classification head, and its checkpoint was selected using test macro-F1 after each epoch.Inputs were truncated at 512 tokens, cutting 13.2% of test reports; the final checkpoint was also the one a fixed three-epoch schedule would produce.Source: §4.2 · p. 5

4.3. Zero-shot LLMs

• Zero-shot Llama 3 8B and Qwen 2.5 7B use instruction-tuned variants, with no examples or output constraint in the prompt.Decoding used temperature 0.1 and at most 150 new tokens; model digests and Ollama version were not recorded.Source: §4.3 · p. 5

4.4. QLoRA fine-tuning

• QLoRA fine-tuning keeps the LLM base weights frozen and trains adapters on all seven linear projections in every transformer block.Training loss covers all record tokens, including report text, although the answer is only 6 or 7 tokens; exported models were quantised and served locally.Source: §4.4 · pp. 6–7

4.5. Evaluation protocol

• The four LLM configurations were scored on a seeded 2,000-report subset, and the other models were re-scored on those same reports.Uncertainty uses paired bootstrap resamples; comparisons also use McNemar tests with Holm correction. Outputs pass through a shared parser, while reports are treated as independent and each model was trained once.Source: §4.5 · pp. 7–9, Table 4 · p. 9, Figure 3 · p. 9

5.1. RQ1: fine-tuned LLMs against TF-IDF and CodeBERT

• Macro-F1 was 0.526 for fine-tuned Llama 3 and 0.518 for CodeBERT, while zero-shot LLMs (0.305, 0.321) trailed TF-IDF (0.430–0.489).Fine-tuned Llama 3 and CodeBERT showed no detectable difference (+0.007 macro-F1; 95% interval −0.017 to +0.032); fine-tuned Qwen 2.5 was 0.033 below CodeBERT on macro-F1.Source: §5.1 · pp. 9–10, Table 5 · p. 10

5.2. RQ2: zero-shot versus fine-tuned, per class

• Zero-shot models overpredicted Minor; after tuning, Llama 3 recalled 0.358 of Critical reports, while Qwen 2.5 recalled 0.691 Critical but only 0.214 Minor.The zero-shot and fine-tuned runs also differed in checkpoint, runtime, decoding, and output cap, so the score gap cannot be attributed to fine-tuning alone.Source: §5.2 · pp. 10–11

5.3. RQ3: duplicates and class prevalence

• Removing 159 duplicate test reports lowered TF-IDF scores by 0.005–0.031, while fine-tuned LLM scores moved by 0.007 or less.Reweighting to the pool’s class mix was an estimate: it put fine-tuned Llama 3 at 0.517 and CodeBERT at 0.462, but the interval did not cover the reweighting assumption.Source: §5.3 · pp. 11–12

5.4. Inference and training cost

• CodeBERT needed 49 ms per report and fine-tuned Llama 3 1.31 s, about 27 times longer, with no detectable macro-F1 difference on balanced data.QLoRA training took 3 hours for Llama 3 and 2 hours 40 minutes for Qwen 2.5, compared with 23 minutes for CodeBERT; the authors do not attribute runtime differences to fine-tuning because serving loops differed.Source: §5.4 · p. 12, Table 7 · p. 12

6. CyberTriage: serving the classifier

• CyberTriage lets users select an evaluated model, submit reports by typing, file upload, or browser speech recognition, and export analyses as Jira import files.The application has a React front end and FastAPI back end; supported uploads include PDF, DOCX, TXT, LOG, and JSON.Source: §6 · p. 13
• Stage 1 supplies the stored label; Stage 2 generates a justification, summary, and description, but its severity output is discarded.The guarantee is enforced in code and does not ensure correctness: Stage 1 was wrong on 47% of Llama 3 and 48% of Qwen 2.5 shared-test reports, and Stage 2 explanations were not evaluated or timed.Source: §6 · p. 13

7.1. What fine-tuning bought, and what it did not

• The zero-shot and fine-tuned gap cannot be attributed to task learning alone because checkpoint, prompt, serving, quantisation, and decoding also changed.Without raw responses, format failures cannot be separated from wrong answers, and the zero-shot Minor bias may partly reflect the parser default.Source: §7.1 · p. 13
• Fine-tuned Llama 3 did not detectably outperform CodeBERT, and its higher per-report cost and untested prevalence assumption weaken the practical case on the balanced set.The interval allows a difference of up to 0.032 in favour of Llama 3 or 0.017 in favour of CodeBERT; the LLM also required three hours of A100 training time.Source: §7.1 · p. 13
• The fine-tuned models showed opposite class biases, but one training run per model leaves their seed stability unknown.The authors do not attribute the bias difference to architecture.Source: §7.1 · pp. 13–14

7.2. Implications for triage

• Critical recall was 0.36–0.69 across fine-tuned Llama 3, Qwen 2.5, and CodeBERT, supporting human review rather than unreviewed routing.Reviewing reports predicted Major or above catches 95% of Critical reports for Llama 3 while requiring review of 57% of the balanced set; at the pool’s class mix, that queue would be about 74%.Source: §7.2 · p. 14

7.3. Threats to validity

• The target label is noisy Bugzilla severity; 41 identical descriptions have conflicting labels, and the study did not hand-audit a sample.Merging blocker into Critical and dropping normal narrows the task, while macro-F1 on a balanced set does not match real-world error costs.Source: §7.3 · p. 14
• Internal comparisons are limited by unequal training-set sizes, test-split checkpoint selection for CodeBERT, one seed per model, and unmeasured run-to-run variation.The zero-shot and fine-tuned conditions also differ in more than training, and the LLM output parser’s fallback firing rate is unknown.Source: §7.3 · p. 14
• External validity is limited to resolved Eclipse reports and a balanced, repeated sample, with no natural-prevalence, temporal, or cross-project test.The prevalence estimate is a re-weighting that ignores normal reports, which would dominate a real stream.Source: §7.3 · p. 14
• The conclusion-validity analysis covers only 2,000 of 4,000 test reports, and differences below about 0.03 macro-F1 are unresolved.The bootstrap treats reports as independent despite repeated bug IDs within the test split.Source: §7.3 · p. 14

8. Conclusion

• Zero-shot runs scored below every TF-IDF classifier, while fine-tuned Llama 3 showed no detectable difference from CodeBERT and Qwen 2.5 fell below it on macro-F1.The fine-tuned models differed most in how they traded one class against another, and none found Critical reports reliably enough to work without review.Source: §8 · p. 14
• The authors call for natural-prevalence and temporal tests, deduplication before sampling, validation splits for tuning, multiple seeds, and a second encoder baseline.They also propose comparing prompting and fine-tuning under constrained output.Source: §8 · pp. 14–15
• The released materials include split identifiers, per-report predictions, and the script that produces the paper’s tables and figures.These materials are intended as a starting point for the proposed extensions.Source: §8 · p. 15

A. Re-scored baselines on the full test split

• Table 8 reports TF-IDF classifiers and CodeBERT on all 4,000 test reports, with macro-F1 and wall-clock time alongside the most recent logged evaluation run.The re-scored values match the logged values.Source: A. Re-scored baselines on the full… · p. 15, Table 8 · p. 15
• Full-split scores were 0.002 to 0.016 higher than on the 2,000-report subset, making the subset a slightly harder draw.The comparison uses the full 4,000-report test split.Source: Table 8 · p. 15, A. Re-scored baselines on the full… · p. 15

CRediT authorship contribution statement

• Asset Kanatov is credited with conceptualization, methodology, software, investigation, data curation, and original-draft writing; Bayan and Boranbayev are credited with supervision and review and editing.Bayan is also credited with methodology, validation, and formal analysis.Source: CRediT authorship contribution statement · p. 15

• GitHub Copilot and Cursor were used during development of the CyberTriage software.Source: Declaration of generative AI and… · p. 15
• Claude was used to help rerun verification analysis and edit the manuscript text.Source: Declaration of generative AI and… · p. 15
• The authors reviewed and edited all content and take full responsibility for the publication.Source: Declaration of generative AI and… · p. 15

• The source data are public, and the repository provides code, training notebooks and logs, evaluation metrics, split identifiers, per-report predictions, and the analysis script.Source: Data availability · p. 15
• The analysis script rebuilds the split from source data and recomputes every table and figure from the released files.Source: Data availability · p. 15
• Trained model files are not redistributed, and raw LLM responses were not stored.Source: Data availability · p. 15