PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

Darin Keng, Zhewei Sun

arXiv:2609.13556v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how LLMs encode domain-specific jargon, an area with limited evidence despite its relevance to specialized-text processing. It builds medical jargon benchmarks and applies component-level analysis to compare general-purpose and medically fine-tuned Llama-3.1 models. The general-purpose model outperforms the fine-tuned model on both benchmarks, while the analysis links the latter’s miscalibration to greater emphasis on jargon-sensitive components.

  • Problem

    Little is known about how LLMs internally represent and acquire parametric knowledge of domain-specific jargon, despite its importance for interpreting specialized terminology.

    Source: §1 · p. 1

  • Method

    The paper creates two medical jargon benchmarks and compares general-purpose and medically fine-tuned Llama-3.1 models using component-level analysis.

    Source: §1 · p. 2

  • Results

    The general-purpose model outperforms UltraMedical on both benchmarks, while UltraMedical shows greater emphasis on jargon-favoring components and increased miscalibration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    This case study shows that medical fine-tuning does not necessarily improve jargon comprehension over a general-purpose counterpart.

    Source: §1 · p. 2, §6 · p. 9

    The comparison covers one Llama-3.1-8B general-purpose model and one medically fine-tuned variant, so training-data and procedure confounds prevent isolating fine-tuning’s causal effect.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized technical domains. Moreover, little is known about how parametric knowledge of domain-specific terms is encoded within these models. We address this gap by contributing two novel medical jargon evaluation benchmarks and evaluate a general-purpose Llama-3.1 model against a variant fine-tuned on medical-domain data. Surprisingly, the general-purpose model outperforms the medically fine-tuned model on both tasks. Using mechanistic interpretability tools, we find systematic patterns of miscalibration for the medically fine-tuned model. Instead of reorganizing parametric knowledge, the fine-tuned model places greater emphasis on a small subset of model components associated with jargon-favoring predictions. We find that applying component reweighting strategies against the benchmark tasks successfully suppresses these components and closes the gap with the general-purpose baseline. We also observe that some jargon-sensitive components transfer knowledge to the same tasks involving materials science jargon, suggesting they encode a partially domain-agnostic notion of specialized terminology. Our results provide a case study in which a medically fine-tuned checkpoint does not improve jargon comprehension over its general-purpose counterpart, highlighting that domain adaptation should not be assumed to yield better performance on specialized terminology.

1 Introduction

The paper examines how LLMs represent domain-specific jargon, a topic with limited prior attention despite jargon’s communication barriers. It introduces two complementary benchmarks and compares general-purpose and medically fine-tuned Llama-3.1 models, finding the general-purpose model performs better on both.

1 Introduction

  • The study addresses limited evidence about how LLMs internally represent and acquire knowledge about domain-specific jargon.Jargon can be specialized terminology or a familiar word with a different technical meaning.Source: §1 · p. 1
  • Misinterpreting medical jargon in patient-facing EHR summaries may mislead patients about their health data.The paper frames this as a reason to measure performance and understand why models succeed or fail.Source: §1 · pp. 1–2
  • The paper constructs Jargon Understanding and Jargon Identification benchmarks and uses component-level analysis to investigate how models encode jargon knowledge.JU tests interpretation of domain-specific meanings, while JI tests whether a term is jargon in context.Source: §1 · p. 2
  • The general-purpose Llama-3.1-8B-Instruct model outperforms medically fine-tuned UltraMedical on both jargon benchmarks.The paper also reports that UltraMedical emphasizes components associated with jargon-favoring predictions and increased miscalibration.Source: §1 · p. 2

2 Related Work

Prior LLM work has addressed jargon detection and explanation, but internal mechanisms behind jargon processing remain less understood. This paper pairs complementary jargon tasks with mechanistic interpretability to examine those mechanisms.

2.1 Processing jargon with LLMs.

  • Existing LLM jargon work mainly studies detection and definition generation, with additional work on scholarly jargon and jargon-aware prompting.The paper identifies internal mechanisms behind parametric jargon knowledge as a less-studied question.Source: §2.1 · p. 2

2.2 Mechanistic interpretability of LLMs.

  • Mechanistic interpretability offers tools for examining internal model behavior, and this paper uses component decomposition to study jargon processing.The adopted framework decomposes outputs into direct contributions from MLP blocks and individual attention heads.Source: §2.2 · pp. 2–3

3 Jargon evaluation benchmark

The benchmark suite draws medical jargon examples from README and builds tasks for definition understanding and contextual identification. A materials-science identification benchmark extends the component analysis across domains.

3.1 Data sources

  • Medical benchmark construction uses README-exp_good, expert-annotated EHR instances, and filters intended to retain specialized terminology.After deduplication and filtering, the datasets contain 5,977 jargon entries and 2,095 non-jargon entries.Source: §3.1 · p. 3

3.4 Cross-domain benchmark construction

  • The cross-domain benchmark uses MatScholar for materials-science jargon identification, while BoolQ provides a non-jargon control; Table 1 reports accuracy on both medical tasks.MatScholar has span annotations but no definitions, so it supports JI rather than JU. Table 1’s caption specifies 95% Wilson confidence intervals.Source: §3.1 · p. 3, §3.4 · p. 4, Table 1 · p. 4

3.2 Jargon Understanding (JU) task

  • The JU task presents an EHR context and jargon term, asking the model to select the correct definition from five options.It contains 4,183 training and 1,495 test examples; distractors are drawn from training-set definitions.Source: §3.2 · p. 3

3.3 Jargon Identification (JI) task

  • The JI task asks whether a term functions as jargon in context, using balanced positive and negative examples.Its test set contains 2,990 examples, and its training set contains 1,200 examples for component reweighting.Source: §3.3 · p. 4

4 Methodology

The method decomposes model logits into direct contributions from attention heads and MLP blocks, then uses component weights to assess accuracy and reweight predictions.

4 Methodology

  • The framework expresses output logits as a linear sum of contributions from individual attention heads and MLP blocks.For a transformer with L layers, the final hidden state sums per-layer attention and MLP outputs written to the residual stream.Source: §4 · p. 4
  • A component’s direct logit contribution is computed from its post-layernorm activation and the output embeddings for task labels.This enables per-component accuracy measurements, but captures direct contributions rather than indirect effects on later residual-stream computations.Source: §4 · p. 4
  • Component weights are optimized on training data with cross-entropy and L1 regularization while the model’s other parameters remain frozen.Because the decomposition is linear, reweighting can be applied without retraining the model.Source: §4 · p. 4

5 Results

The general-purpose Llama-3.1 Instruct model outperforms medically fine-tuned UltraMedical on both jargon benchmarks. Component analyses indicate shared jargon-sensitive locations with differing weights, and identify some jargon signal that transfers to materials science.

5.1 Comparative analysis

  • Across both tasks, Llama-3.1 Instruct outperforms UltraMedical, with the performance gap most pronounced on JI.This comparison motivates the subsequent calibration and error analyses.Source: §5.1 · p. 5
  • UltraMedical is overconfident in higher-confidence bins, while Instruct tracks the perfect-calibration diagonal more closely on both tasks.The figure marks perfect calibration with dashed diagonal lines.Source: §5.1 · p. 5, Figure 2 · p. 5
  • On JI, Instruct correctly predicts 368 examples UltraMedical misses, versus 197 in the reverse direction; the difference is significant (p < 0.001).UltraMedical tends to over-classify terms in clinical contexts, while Instruct sometimes misses context-dependent specialized meanings.Source: §5.1 · p. 5

5.2 Localization of domain-specific knowledge

  • On JU, both models’ top four components are L17H24, L26H3, MLP-29, and L31H1; weighted rank agreement is τ = 0.72 on JU and τ = 0.59 on JI.The models’ component rankings are similar, suggesting fine-tuning modulates contributions rather than relocating direct jargon signal.Source: §5.2 · pp. 6–7, Figure 3 · p. 6
  • On JU, ablating top-ranked components reaches chance by k = 20, while JI declines shallowly under both targeted and random ablation.Cross-ablation profiles also track the corresponding models’ own top-component ablations.Source: Figure 4 · p. 6, §5.2 · p. 7

5.3 Component reweighting

  • 3.0% accuracy gain on JI for UltraMedical after reweighting was significant (p = 0.01), while Instruct fell 1.0% on JI and JU gains were ≤0.2%.The gains suggest UltraMedical’s component weighting is not optimal for JI; Instruct’s errors are balanced, so suppression shifts errors toward false negatives without offsetting false positives.Source: §5.3 · pp. 7–8, Table 3 · p. 7
  • Downweighting L30H27 flips the JI predictions for addiction and tablet to the correct non-jargon label, although that component also contributes strongly to JU definition selection.The interpretation is limited to direct component contributions to output logits; indirect residual-stream effects are not ruled out.Source: §5.3 · pp. 7–8

5.4 Cross-domain generalization

  • Nine components rank in the top 50 on both Med-JI and Mat-JI but outside BoolQ’s top 50; L20H10 and L18H26 show the strongest cross-domain jargon signal.Those two components achieve near-full-model accuracy on Mat-JI and perform near chance on BoolQ.Source: §5.4 · pp. 8–9, Table 5 · p. 8

6 Conclusion

The medically fine-tuned model underperforms its general-purpose counterpart on both jargon benchmarks, with the difference linked to excessive emphasis on jargon-sensitive components rather than a reorganization of knowledge. The findings also suggest that some jargon-sensitive components transfer across domains.

6 Conclusion

  • UltraMedical underperforms the general-purpose model on both jargon benchmarks, with greater overconfidence on domain-specific lexical items.The conclusion frames this as a case study, not evidence that domain-specific fine-tuning never helps.Source: §6 · p. 9
  • The models retain similar patterns of jargon-sensitive components, but UltraMedical places excessive emphasis on specific components rather than reorganizing internal knowledge.This re-modulation of existing knowledge is described as not optimal for related downstream tasks.Source: §6 · p. 9
  • Some jargon-sensitive components transfer to materials science tasks, suggesting pretrained models hold partially domain-agnostic jargon knowledge.The conclusion uses this observation to question the necessity of domain-specific fine-tuning for specialized terminology.Source: §6 · p. 9

Limitations

The benchmark construction prioritizes precision for clearly specialized terms, while leaving ambiguous jargon cases for finer-grained future analysis. The study is also limited to one Llama-3.1-8B model pairing and English jargon, with confounds that prevent isolating the causal effect of fine-tuning.

Limitations

  • The SBERT-based pipeline may not always produce optimal results, and the benchmark retains terms whose medical definitions are sufficiently distant from conventional meanings.This prioritizes precision on the positive class and focuses claims on confidently specialized terminology.Source: Limitations · p. 9
  • Human annotations with confidence scores could enable finer-grained analysis of more ambiguous jargon cases.The authors identify this as a possible benchmark extension.Source: Limitations · p. 9
  • The study compares one general-purpose and one medically fine-tuned Llama-3.1-8B model, and training-data and procedure differences prevent isolating fine-tuning’s causal effect.Computational constraints limited the study; future work is proposed with additional families, scales, and controlled fine-tuning.Source: Limitations · p. 9
  • The evaluation covers English jargon only; multilingual extensions are left to future work.Source: Limitations · p. 9

Ethics Statement

The appendices describe the data sources, filtering and evaluation procedures, implementation settings, and prompt variants used in the study. The ethics statement limits the derived benchmarks to research on model behavior, not clinical deployment.

Ethics Statement

  • The derived benchmarks inherit their source datasets’ access conditions and are intended for research on model behavior, not clinical deployment.The authors report no additional identification of individuals and no newly collected human-subject data.Source: Ethics Statement · p. 9

A Computational setup

  • SBERT components use all-mpnet-basev2; model experiments use inference-only Llama-3.1-8B checkpoints, with decomposition projections cached and reweighting optimized on CPU.The SBERT processing runs on one RTX A6000; decomposition uses two RTX A6000 GPUs.Source: A Computational setup · p. 12

B Decomposition setup

  • The JU and JI tasks use zero-shot evaluation with one canonical prompt template per task; the seed setting does not affect deterministic evaluation.The listed setup uses n_shots 0, format 1, and seed_list [0].Source: B Decomposition setup · p. 12

C Details of README

  • README-exp_good supplies jargon terms with UMLS-retrieved definitions that passed README’s quality filtering.The authors use this source to associate evaluated jargon terms with definitions from an authoritative medical knowledge base.Source: C Details of README · p. 13

D Wiktionary-based definition filtering

  • The Wiktionary filter retains terms whose medical definitions have cosine similarity below 0.5 to conventional senses, using a manually chosen threshold.Conventional definitions are those whose Wiktionary tags do not include the selected technical-use tags.Source: D Wiktionary-based definition filtering · p. 13

E Data Statistics

  • After normalization, deduplication, and semantic filtering, the medical source is reduced to 8,072 entries.Within-term definitions are greedily retained when their pairwise SBERT cosine similarity is below the manually chosen threshold of 0.92.Source: E Data Statistics · pp. 13–14

F Example prompt for JU; G Example prompt for JI; H Prompt Sensitivity Analysis; H.2 Prompt Variants.

H.2 Prompt Variants.

H.1 Results

Across the tested prompt variants, Llama-3.1-8B-Instruct maintains higher accuracy than UltraMedical on both JI and JU, though absolute accuracy varies with wording. The Mat-JI construction uses materials-science entity spans as positives and technically perceived non-entities as negatives.

H.1 Results

  • Instruct outperforms UltraMedical on JI across all tested prompt variants, although absolute accuracy varies with prompt wording.Source: §H.1 · p. 14
  • Instruct also maintains higher JU accuracy than UltraMedical across all tested variants.Source: §H.1 · p. 14, Table 10 · p. 15
  • The prompt tests support a qualitative ordering robust to the variations considered, not invariance to arbitrary prompting strategies.The authors state the main comparison is not specific to the canonical wording used in the main experiments.Source: §H.1 · p. 15

I Construction of Mat-JI

  • Mat-JI treats contiguous BIO-tagged materials-science entity spans as jargon and mines technically perceived O-tagged tokens as challenging negatives.The construction avoids easy negatives such as stop words and basic academic verbs.Source: I Construction of Mat-JI · pp. 15–16
  • Mat-JI samples negatives across judge ratings 3–5 and combines them with positive spans at a 1:1 ratio in a balanced 2,990-item evaluation set.The set matches the size of the medical JI test set.Source: I Construction of Mat-JI · p. 16

J Examples of UltraMedical’s errors

UltraMedical’s false positives fall into two categories: everyday words and familiar medical terms treated as jargon in EHR contexts. The examples suggest its decisions lean more on clinical context than on terms’ intrinsic semantics.

J Examples of UltraMedical’s errors

  • UltraMedical misclassifies non-jargon terms as jargon in EHR contexts, including everyday words and lay-accessible medical terms.The examples include “elevated” and “consumption,” as well as “vaccination” and “hospitalizations.”Source: J Examples of UltraMedical’s errors · p. 16
  • UltraMedical treats clinical context as evidence of jargon status even when the term itself is familiar outside clinical settings.The passage suggests its decision boundary is more anchored to context than to intrinsic semantics.Source: J Examples of UltraMedical’s errors · pp. 16–17
  • Medical-text fine-tuning plausibly strengthens the contextual association at the expense of term-level discrimination.Source: J Examples of UltraMedical’s errors · p. 17

K Examples of Instruct’s errors

The section contrasts task-specific component rankings and error patterns across the models, then describes the ranking metric and ablation procedure used to examine those components.

K Examples of Instruct’s errors

  • Instruct’s errors reverse UltraMedical’s tendency: familiar surface forms can receive non-jargon labels despite specialized clinical senses.Examples include “dressings” used for wound coverings and “compression” used for a therapeutic protocol; the logit margins are narrow.Source: K Examples of Instruct’s errors · p. 17, Table 19 · p. 18

L Top performing components on both tasks for both models

M Weighted Kendall’s τ for component-rank agreement

  • Weighted Kendall’s τw emphasizes agreement among highly ranked components, unlike standard τ, which weights ranking pairs equally.The analysis negates accuracy vectors so higher-accuracy components receive lower ranks under the implementation’s ascending default.Source: M Weighted Kendall’s τ for… · p. 18

N Ablation Procedure

  • Ablation zeroes component outputs before they enter the residual stream, leaving learned weights unchanged and removing downstream effects as well as direct contributions.The random-ablation control samples outside the top-20 ranked components and applies cumulative ablation.Source: N Ablation Procedure · p. 19

O Individual Component Ablation

Individual ablations distinguish a single influential JU component from the more distributed JI contributions; the accompanying setup and statistical procedures specify how reweighting and comparisons were evaluated.

O Individual Component Ablation

  • 6.49% for Instruct and 7.29% for UltraMedical: ablating L17H24 alone sharply reduces JU accuracy, while other top components each cause drops below 0.7%.The L17H24 drops exceed those from other top-five components and lie well outside the random-ablation range.Source: O Individual Component Ablation · p. 19, Table 22 · p. 19
  • On JI, no single component has comparable impact; the largest top-component drops are 0.43% for Instruct and 0.13% for UltraMedical.UltraMedical’s maximum random-ablation drop, −1.10, exceeds its largest top-component drop, consistent with JI relying on aggregated contributions.Source: O Individual Component Ablation · p. 19, Table 22 · p. 19

P Experiment setup for component reweighting

  • Reweighting weights are trained for 1,000 epochs with SGD (lr = 0.01, λ = 0.01) and early stopping (patience = 3) on JI training loss, then evaluated on the JI test set.JI has 1,200 training and 2,990 test examples; the same procedure is applied to JU with 4,183 training and 1,495 test examples.Source: P Experiment setup for component… · p. 19

Q Details for statistical tests

  • The study uses exact two-sided McNemar tests for paired JI predictions and paired bootstrap intervals for weighted Kendall’s τw component-rank agreement.The bootstrap uses 10,000 replicates and reports the 2.5th and 97.5th percentiles as the 95% confidence interval.Source: Q Details for statistical tests · p. 20
Loading 2609.13556v1…