Domain Adapted Large Language Models for Additive Manufacturing

1 Introduction

• Domain-aware LLMs support specialized additive-manufacturing tasks such as knowledge retrieval, parameter selection, process optimization, and intelligent automation.Local or on-premise deployment can be necessary for national security, patient data, or environmental constraints.Source: §1 · p. 2
• Domain-adaptive pretraining suits largely static domain knowledge because it enables accurate responses without the additional context consumption required by retrieval-augmented generation.RAG remains useful for dynamic information, while DAPT targets static sources such as journal articles, figures, and textbooks.Source: §1 · p. 2
• The study creates multimodal domain-adapted models based on Gemma 3 (12B), Qwen 3 (8B), and Gemma 4 (31B), trained with text and image data from open-access additive-manufacturing articles.The models use domain-adaptive pretraining and visual instruction tuning with a curated public dataset.Source: §1 · pp. 2–3
• The Additive-Manufacturing-Benchmark evaluates capabilities including melt-pool dimensional prediction, anomaly identification, and general additive-manufacturing knowledge.The benchmark measures whether the models provide enhanced knowledge within the additive-manufacturing domain.Source: §1 · p. 3

Research gap

• No prior work investigated additive-manufacturing model adaptation across language and vision through continual pre-training and instruction tuning, including within agentic systems.The paper identifies this as the gap addressed by its domain-adapted approach.Source: §2 · pp. 3, 6

Fine-tuning

• 82% accuracy was achieved by fine-tuned DistilBERT for classifying laser powder bed fusion defect regimes from natural-language process parameters.22The comparison included DistilBERT (66M), SciBERT (110M), T5-Small (60M), and Llama 3.2 (1B), using MeltpoolNet and FLOW-3D data.Source: §2 · p. 3

In-context learning

• 96% classification accuracy was achieved by Fang et al.’s in-context method for distinguishing normal and defective vat-photopolymerization build layers using images, examples, and text descriptions.42GPT-4o received camera images of resin deposition alongside positive and negative samples and their textual descriptions within the conversation context.Source: §2 · p. 5

Retrieval augmented generation

• RAG-based systems have been reported to improve model-prediction accuracy, while Khanghah et al.21 extend RAG into vision space for multimodal detection and classification of anomalies in laser powder.The described multimodal approach uses models such as Qwen2-VL-2B and GPT-4o-mini.Source: §2 · p. 5

Agentic systems

• The agentic system enabled clear improvements in mechanical performance through compression testing.24Both works relied on off-the-shelf large language models, specifically GPT-4o, Claude, and Gemini.23,24Source: §2 · p. 6

3.1 Large Language Models

• Large language models use transformer-based architectures trained through next-token prediction and can be extended to interpret images, videos, and 3D models.They commonly improve with larger parameter counts, compute budgets, and datasets.Source: §3.1 · p. 6

3.1.1 Transformer Architecture

• Transformers use attention to model long-range dependencies, with encoder stacks attending bidirectionally and decoder stacks generating from previous output tokens.Decoder-only stacks such as GPT have become dominant for generative and reasoning tasks because they scale better.Source: §3.1.1 · p. 7

3.1.2 Multi-modal Input Representation

• Multimodal transformers tokenize images, videos, and 3D models while preserving spatial or temporal structure through patches, positional embeddings, and point-based subdivisions.Naively flattening multidimensional data can lose spatial and temporal information, while Point-BERT uses point-based patches to preserve 3D structure.Source: §3.1.2 · pp. 7–8

3.1.3 Model Scaling

• Model scaling produces emergent behaviors such as reasoning, with performance depending strongly on parameter count, dataset size, and training compute.Kaplan et al.65 established a power-law relationship involving parameter sizes from 768 to 1.5B and dataset sizes from 22M to 23B.Source: §3.1.3 · p. 8

3.2.1 Chain-of-Thought

• Chain-of-Thought prompting elicits step-by-step reasoning by decomposing problems into multiple steps, supporting arithmetic fidelity, interpretability, and reward-hacking monitoring.Excessive optimization can conceal intent within CoT and create avenues for hallucination.38,69Source: §3.2.1 · p. 9

3.2.3 ReAct

• ReAct combines reasoning with actions and observations, updating the model’s context and policy dynamically through feedback across decomposed task steps.Each step includes a “Thought,” an “Action” such as a tool call, and an “Observation” returned by that action.70Source: §3.2.3 · p. 10
• The ReAct framework repeats Thought, Action, and Observation steps until finish[answer] is produced or an iteration limit is reached, exposing chain-of-thought reasoning throughout.70Actions are parsed into tool calls such as search[entity] and lookup[string], whose observations are appended to the context.70Source: §3.2.3 · p. 11

3.3 Domain Adaptation

• Domain adaptation specializes pretrained general knowledge for downstream applications through DAPT, LoRA, and supervised instruction tuning.44,64,75LoRA addresses the prohibitive cost of updating every parameter in increasingly large models by training a smaller set of effective parameters.75Source: §3.3 · p. 11

3.3.1 Domain-Adaptive Pretraining

• DAPT improves responses over base RoBERTa across BioMed, CS, News, and Reviews, especially in BioMed, CS, and Reviews when source and target domains differ.26DAPT continues next-token prediction on a smaller, focused dataset such as research papers or scientific text.26,76–78Source: §3.3.1 · pp. 11–12

3.3.2 Low-Rank Adaptation

• LoRA reduces trainable parameters by up to 10,000-fold while requiring only one-third of the training memory, by optimizing low-rank adapter weights over frozen pretrained weights.75The adapter represents updates as ∆W = BA, and pretrained and adapter weights can be merged for inference without added latency.75Source: §3.3.2 · pp. 12–13

3.3.3 Instruction Tuning

• Instruction tuning is supervised fine-tuning with labeled datasets that aligns language models toward better question-answering responses; its gains require sufficient parameter size, while models under 8B may suffer hindered task performance from limited capacity.44,64Source: §3.3.3 · p. 14
• 92.53% accuracy was achieved by LLaVA on Science QA, demonstrating the potential of multimodal instruction tuning.44LLaVA uses image captions and object localization to construct Conversation, Detailed Description, and Complex Reasoning instruction formats.44Source: §3.3.3 · p. 14

4 Methodology

• The study adapts Gemma 3, Qwen 3, and Gemma 4 using domain-adaptive pretraining and instruction tuning, evaluating development with the Additive-Manufacturing-Benchmark.The benchmark includes additive-manufacturing knowledge questions, visual identification tasks, and data-based prediction tasks.Source: §4 · p. 15

4.1 Models

• Three base models were selected: Gemma 3 12B in pretrained and instruction-tuned forms, Gemma 4 31B instruction tuned, and Qwen 3 8B instruction tuned.Gemma models were chosen for their balance of functionality and memory consumption, while multimodal inputs were required for additive-manufacturing tasks.Source: §4.1 · p. 16

4.1.1 Gemma 3

• Gemma 3 provides multimodal text-and-image inference through a decoder-only transformer paired with a frozen 400M-parameter SigLIP vision encoder.The 12B instruction-tuned variant performs near the 27B instruction-tuned model and better than the 4B instruction-tuned variant on several general reasoning and understanding benchmarks.Source: §4.1.1 · p. 16

4.1.2 Qwen3

• Qwen 3 was included to examine domain adaptation in a model with strong multimodal and reasoning capabilities, using its 8B dense 36-layer variant with a 128K-token context length.Qwen3-8B surpassed Gemma-3-12B in MMLU, GPQA, and other reasoning benchmarks.Source: §4.1.2 · p. 17

4.1.3 Gemma 4

• Gemma 4 31B was selected to explore domain adaptation with fewer parameter limitations, supporting text and image modalities and a 256K-token context window.Only the instruction-tuned variant was used because building from the pretrained base did not reach the capability of the base instruction-tuned variant (Appendix D).Source: §4.1.3 · p. 18

4.2 Dataset

• The dataset uses all 1,704 open-access articles published in four peer-reviewed additive-manufacturing journals, with text, images, and visual instruction tuning examples compiled; it contains over 24,000 extracted images.Source: §4.2 · pp. 18–19
• Around 57 million tokens—45 million text and 11 million image—were assembled from open-access additive-manufacturing articles, including approximately 20,000 visual-instruction-tuning examples.Figure captions were converted into question-answer conversations and detailed descriptions using GPT-OSS (120B), while improperly extracted figure-caption pairs were excluded.Source: §4.2 · pp. 19–20

4.3 Training

• Training proceeded through text domain-adaptive pretraining, image domain-adaptive pretraining with language weights frozen, and visual instruction tuning on generated conversations and descriptions.LoRA adapted the relevant attention weights, with the final stage training both language and vision weights using supervised fine-tuning.Source: §4.3 · pp. 20–21

4.4 Benchmarking

• The Additive-Manufacturing-Benchmark evaluates general process knowledge, melt-pool prediction, FDM defect recognition, machine identification, and LPBF anomaly identification across language and visual modalities.Its tasks draw on domain-expert course materials and published datasets, using short-answer or multiple-choice questions, RMSE, binary classification, and F1-based evaluation as appropriate.Source: §4.4 · pp. 23, 25, Figure 6 · p. 24

5 Results and Discussion

• Additional text pretraining and instruction tuning improved domain specialization, while image pretraining reduced defect and anomaly-detection performance; the base model was rarely best except Gemma 4 31B IT.The image-training decrease suggests potential loss of parametric visual information.Source: §5 · p. 26

5.1 Benchmark Results

• The Gemma-3-12b-it visual instruction-tuned model performed best in most image tasks, while DAPT text closely followed the base model elsewhere.Figure 7 reports this overall pattern across the benchmark.Source: §5.1 · p. 26, Figure 7 · p. 27

5.2 Domain Specialization Performance

• With about 45M text-training tokens, the final Gemma-3-12b-it vision-instruction-tuned model improved general-knowledge short answers, anomaly and defect detection, and machine recognition over its base model.The authors attribute this efficiency partly to additive manufacturing’s narrower domain and note that broader datasets remain worth exploring.Source: §5.2 · p. 29

6 Conclusion

• The domain-adapted models outperform their respective base counterparts after domain-adaptive pretraining and visual instruction tuning.Each model was trained on open-access additive manufacturing articles.Source: §6 · p. 29
• Around 45 million tokens trained for three epochs per stage establish that domain adaptation is possible with a relatively small dataset.Source: §6 · p. 29
• The Gemma-3-12b-it visual instruction-tuned variant achieves the best performance on vision and language tasks, with general-knowledge accuracy above 90%.Source: §6 · p. 30
• Gemma-4-31b-it shows greater capability in general-domain knowledge for short-answer and multiple-choice tasks but performs similarly on visual tasks.Source: §6 · p. 30
• Domain-adaptive pretraining combined with instruction tuning offers an accessible method for specializing large language models to additive manufacturing.Source: §6 · p. 30

Agentic systems

• Future agentic systems would evaluate efficient tool calls using adapted domain knowledge and act on in-situ process images to address potential issues.The proposed evaluation would use a VLM component similarly to LLM-3D Print24.Source: §7 · p. 30

Enhanced datasets

• A larger and more comprehensive additive-manufacturing dataset is expected to produce greater performance gains than the sufficient dataset used here.The current dataset already produced gains across many benchmarking tasks.Source: §7 · p. 30

Task specialization

• Further specialization for tasks such as laser powder bed fusion anomaly detection is expected to improve task performance but may reduce performance on other tasks.This trade-off motivates investigating continual learning approaches during domain adaptation.Source: §7 · p. 30

Continual learning

• Continual learning will be applied across training and fine-tuning cycles to retain trained abilities and alleviate catastrophic forgetting,109.The longer-term framework would integrate an agentic system’s action results into the language model through fine-tuning.Source: §7 · pp. 30–31

• Tokenization converts character strings into token representations that are embedded into vectors for large language models.110–112Source: Appendix A · p. 32
• It provides a more efficient data representation than raw text, which requires a large vocabulary and treats words as distinct units.110Source: Appendix A · p. 32
• Tokenization methods divide strings into word or subword units and index frequent sequences using Byte Pair Encoding.110Source: Appendix A · p. 32
• Sinusoidal positional encoding or Rotary Position Embeddings are added to token embeddings, preserving positional data for semantic and sequential next-token prediction.51,113Source: Appendix A · p. 32

Appendix A.1

• Subword neural machine translation segments text into subword units, enabling out-of-vocabulary words to be represented during training.The method supports combinations of word and subword tokens rather than requiring every word to appear in the dictionary.Source: Subword Neural Machine Translation · pp. 32–33
• The approach uses an adapted Byte Pair Encoding method proposed by Sennrich et al.110 to construct a table of frequently occurring character sequences.Unlike original byte-level BPE, it merges characters for word segmentation.Source: Subword Neural Machine Translation · p. 32
• A tunable number-of-operations limit controls when compression stops, balancing complete-word and subword entries in the dictionary.This differs from original BPE, which repeats until no bytes remain to merge.Source: Subword Neural Machine Translation · p. 32

Byte Pair Encoding

• Byte Pair Encoding was introduced by Philip Gage114 as a data-compression method that repeatedly replaces frequent adjacent byte pairs with unused bytes.The original process continues until no frequent pairs or unused bytes remain.Source: Byte Pair Encoding · p. 33
• The original BPE expansion routine processes the input in one pass, emitting literals directly while expanding byte pairs through a stack.Stack contents provide the next input byte when available; otherwise bytes are read from the input file.Source: Byte Pair Encoding · p. 33

Appendix B Chain of Thought Prompting

• Figure 10 presents a hypothetical comparison between standard prompting and chain-of-thought reasoning outputs.The supplied passage identifies the comparison but does not provide quantitative results or a reported winner.Source: Figure 10 · p. 34

Appendix C Additional Dataset Information

• Figure 11 shows that Additive Manufacturing Letters has the highest vocabulary similarity across journals, while Rapid Prototyping Journal and Journal of Additive Manufacturing have the lowest.The comparison covers open-access articles categorized by their respective journals in the dataset.Source: Figure 11 · p. 35

Appendix D Gemma 3 Pre-Trained Variant Results

• LPBF anomaly identification was the exception, with performance decreasing as additional training stages were applied.The decline occurred despite visual instruction tuning generally improving performance elsewhere.Source: Appendix D · p. 37
• Performance noticeably decreased at the DAPT image-training stage for both pretraining and instruction-tuned variants, especially on general-knowledge and some image-based tasks.Possible contributors include frozen language-attention weights, missing image-caption pairs, and catastrophic forgetting.Source: Appendix D · p. 37
• Visual instruction-tuned Gemma 3 models achieved the best performance across tasks, while pretraining-only variants were comparatively worse than instruction-tuned variants.Figure 14 summarizes this comparison across the adapted models.Source: Appendix D · pp. 35, 37, Figure 14 · p. 38