Source-linked AI summary
Domain Adapted Large Language Models for Additive Manufacturing
Peter Pak, Amir Barati Farimani
TL;DR
The paper addresses limited evidence on adapting large language models specifically to additive manufacturing and on using such adapted models in related systems. It trains multimodal open-weight models with domain adaptive pretraining and visual instruction tuning on open-access articles, then evaluates them with an additive manufacturing benchmark. The study reports that additional text pretraining and instruction tuning specialize domain knowledge, while image pretraining decreases performance on some defect and anomaly tasks.
Problem
Prior work had not established how domain-adapted fine-tuned models would perform for additive manufacturing within agentic systems.
Method
The study trains models on selected open-access additive manufacturing articles using domain adaptive pretraining and instruction tuning, then evaluates them on additive manufacturing benchmark tasks.
Results
Additional text pretraining and instruction tuning enabled further domain knowledge specialization, while image pretraining decreased performance on specific defect and anomaly detection tasks across models.
Takeaways & Limitations
The paper presents domain adaptive pretraining with instruction tuning as an accessible method for specializing language models to additive manufacturing.
Takeaways & Limitations
The authors identify the need to study whether a more comprehensive additive manufacturing dataset improves performance across many benchmarking tasks.
Abstract
from arXiv · showhide
This work presents a collection of multi-modal domain adapted large language models built upon the instruction tuned variants of open weight models (Gemma 3, Qwen 3, Gemma 4) using a relatively small dataset of around 50 million tokens. The dataset consists of open-access additive manufacturing journal articles with data extracted for the domain adaptive pretraining and visual instruction tuning processes. Various stages of the developed model are evaluated with the Additive-Manufacturing-Benchmark which consists of additive manufacturing domain specific tasks compiled published resources. Domain adapted and instruction tuned models exhibit proficiency in both language and vision based tasks, achieving accuracies upwards of 90% in general additive manufacturing knowledge. This domain adaptive pretraining and instruction tuning strategy outline an accessible specialization method for large language models to a domain such as additive manufacturing.
1 Introduction
The paper develops multimodal additive-manufacturing-specific language models through domain adaptation and visual instruction tuning, then evaluates them on domain-specific benchmark tasks. This approach targets efficient, locally deployable domain knowledge without the repeated context costs of retrieval.
- Motivation: Domain adaptive pretraining enables accurate domain responses without the additional context consumption associated with retrieval-augmented generation.The paper contrasts static journal-based domain knowledge with dynamic data, for which retrieval remains useful.
- Contribution: The work creates multimodal open-weight additive manufacturing models based on Gemma 3 (12B), Qwen 3 (8B), and Gemma 4 (31B).Domain adaptive pretraining and visual instruction tuning use text and image data from open-access additive manufacturing journal articles.
- Evaluation: The Additive-Manufacturing-Benchmark measures capabilities including melt pool dimensional prediction, anomaly identification, and general additive manufacturing knowledge.The benchmark is introduced as a domain-specific evaluation resource compiled from published materials.
- Implication: The training and architecture demonstrate an efficient strategy for tailoring large language models to provide enhanced knowledge within a selected field.The motivation includes use cases requiring edge or on-premise deployment because of security, patient-data, or environmental constraints.
2 Related Work
Prior additive manufacturing studies use fine-tuning, in-context learning, retrieval, and agentic systems for specialized tasks. The paper identifies limited exploration of domain-adapted models within additive-manufacturing agentic systems and introduces a broader training and evaluation process.
- Fine-tuning: Earlier studies fine-tuned models from 60 million to 1 billion parameters for laser powder bed fusion defect-regime prediction, reaching 82% accuracy with DistilBERT.The task classified keyholing, lack of fusion, balling, or no defect from natural-language process parameters.
- Present work: Figure 1 depicts domain adaptive pretraining in text and vision, visual instruction tuning from extracted captions, and evaluation on the Additive-Manufacturing-Benchmark.GPT-OSS 120B assists in generating description and conversation prompts for visual instruction tuning.
- In-context learning: In-context learning with GPT-4o classified normal and defective vat-photopolymerization layers with 96% accuracy using positive and negative image examples.The approach supplied images and text descriptions within the conversation context.
- Retrieval: Retrieval-augmented systems have been applied to additive manufacturing language and vision tasks, with one study reporting an accuracy improvement of around 12%.These systems retrieve relevant passages or multimodal information and provide it with the query to a language model.
- Research gap: Existing additive-manufacturing agentic studies rely on off-the-shelf models, while domain-adapted fine-tuned models in such systems remain unexplored.Prior agents addressed alloy evaluation and FDM defect detection, with reported improvements in mechanical performance for the latter.
3 Background
The background introduces transformer-based language and multimodal models, representation challenges for visual and three-dimensional inputs, and prompting methods for reasoning and action. It also reviews scaling, instruction tuning, domain adaptation, and reasoning-related limitations.
- Large language models: Large language models commonly use transformer architectures trained for next-token prediction and can be extended to interpret images, videos, and 3D models.Their performance is associated with parameter count, dataset size, and compute under scaling-law analyses.
- Transformer architectures: Transformers model long-range dependencies through attention and support encoder-only, decoder-only, and encoder-decoder configurations.Decoder-only stacks generate outputs by attending to previous tokens, whereas encoder-only models support contextual embeddings for retrieval.
- Multimodal representation: Naively flattening image or 3D data can lose spatial or temporal information, motivating patch-based representations with positional structure.Point-BERT partitions 3D models into point-based patches, while vision transformers use fixed image patches and positional embeddings.
- Reasoning: Chain-of-Thought prompting decomposes problems into multiple steps and can allocate additional computation to individual reasoning steps.The method formats intermediate reasoning before the final answer and is described as useful for multi-step arithmetic.
- Reasoning limitations: Verbose Chain-of-Thought can provide monitoring opportunities, but excessive optimization may conceal intent and create avenues for hallucination.The background describes monitoring as useful for preventing reward-hacking behavior while noting these risks.
- Agentic reasoning: ReAct combines reasoning traces with environmental actions so models can use context and observations to update subsequent choices.The approach decomposes tasks into smaller steps involving language-space reasoning and environmental actions.
4 Methodology
The study adapts Gemma 3, Qwen 3, and Gemma 4 models to additive manufacturing using multimodal open-access journal data, then evaluates them with a domain-specific benchmark. The methodology combines domain-adaptive pretraining, vision adaptation, visual instruction tuning, and parameter-efficient training across multiple model architectures.
- Base models: Three model families—Gemma 3, Qwen 3, and Gemma 4—are selected as bases for additive-manufacturing domain adaptation.The evaluated configurations include Gemma 3 12B pretrained and instruction-tuned variants, Gemma 4 31B instruction-tuned, and Qwen 3 8B instruction-tuned models.
- Training pipeline: The training pipeline uses extracted article text for domain-adaptive pretraining, figures and captions for vision-tower adaptation, and question-answer pairs with image descriptions for visual instruction tuning.The three stages separately update language-model attention weights, vision-tower attention weights with language weights frozen, and then all attention weights.
- Dataset: The adaptation dataset contains 1,704 open-access articles from four additive-manufacturing journals, covering text, images, and visual-instruction-tuning examples.The journals include Journal of Additive Manufacturing, Additive Manufacturing Letters, Journal of Manufacturing Processes, and Rapid Prototyping Journal.
- Dataset: The complete dataset contains around 57 million tokens, including 45 million text tokens and 11 million image tokens.The data configurations and extracted text and images are hosted publicly on HuggingFace.
- Benchmarking: The Additive-Manufacturing-Benchmark evaluates language and image understanding through knowledge questions, melt-pool prediction, defect identification, machine identification, and LPBF anomaly identification.The benchmark includes both short-answer and multiple-choice general-knowledge formats.
5 Results and Discussion
Across benchmark stages, domain adaptation and instruction tuning generally improved additive-manufacturing specialization, while image-focused adaptation sometimes reduced defect and anomaly detection performance. Gemma 3 visual instruction tuning performed strongly across visual tasks, Qwen3 underperformed on several vision tasks, and Gemma 4 led general-knowledge evaluation but was comparable or worse on image-based tasks.
- Benchmark setup: The benchmark evaluated Gemma- and Qwen-series models after DAPT text, DAPT images, and visual instruction tuning against their base models.Evaluation used tasks from the Additive-Manufacturing-Benchmark over five trials.
- Overall trends: Additional text pretraining and instruction tuning enabled further domain knowledge specialization, with base models rarely selected as best for specific tasks.The base model was not selected as best in any case except Gemma 4 31B IT.
- Overall trends: Image domain-adaptive pretraining noticeably decreased performance on defect and anomaly detection tasks across the investigated models.The decrease occurred specifically in tasks such as defect and anomaly detection.
- Gemma 3: Gemma 3 12B instruction tuning achieved the best results in most visual tasks, except FDM defect identification, while improving over its base model in several listed tasks.Reported improvements included general-knowledge short answer, LPBF anomaly detection, FDM defect detection, and machine recognition.
- Qwen3: Qwen3-vl-8b-it performed comparably on general-knowledge tasks but underperformed on vision tasks including FDM defect and powder-bed anomaly detection.The text attributes the decreased additive-manufacturing performance relative to Gemma 3 partly to Qwen3's smaller parameter count.
6 Conclusion
The domain-adapted models outperform their respective base counterparts after domain-adaptive pretraining and visual instruction tuning, with strong performance across language and vision tasks. The results indicate that additive-manufacturing specialization is possible using a relatively small dataset.
- Domain-adapted models outperform their respective base counterparts after domain-adaptive pretraining and visual instruction tuning.
- Domain adaptation is demonstrated using around 45 million tokens over 3 epochs for each training stage.
- The visual instruction-tuned Gemma-3-12b-it variant achieves the best performance in vision-based and language tasks.
- Accuracy in general additive-manufacturing knowledge reaches upwards of 90%.
- The larger domain-adapted Gemma-4-31b-it shows greater general-domain knowledge capability in short-answer and multiple-choice formats, while visual tasks are similar.
- Domain-adaptive pretraining combined with instruction tuning offers an accessible specialization method for additive-manufacturing language models.
7 Future Work
Future work targets agentic deployment, broader datasets, task-specific specialization, and continual learning to retain abilities across subsequent training cycles.
- Future work will apply the domain-adapted models in agentic systems and evaluate their tool-calling efficiency.
- An agentic system could evaluate in-situ process images with its vision-language component and take appropriate corrective actions.
- A larger and more encompassing additive-manufacturing dataset is expected to produce greater performance gains.
- Specialization for tasks such as anomaly detection in laser powder bed fusion is expected to improve task performance but may affect other tasks.
- Continual learning will be investigated to retain trained abilities during subsequent training and fine-tuning cycles.
Appendix A Tokenization
Tokenization converts human-readable text into token representations and embedding vectors that provide models with more efficient input representations and positional information for next-token prediction.
- Tokenization converts human-readable character strings into token representations that are embedded into vectors for language models.
- Tokenization provides a more efficient representation than treating words as distinct units in a large vocabulary.
- Word and subword units, including Byte Pair Encoding, index frequently occurring sequences in the training data.
- Rotary positional embeddings add positional data to token embedding vectors.
- The resulting representations encode semantic and sequential patterns used for next-token prediction.
Appendix A.1
The appendix describes subword-based preprocessing and Byte Pair Encoding variants that improve token coverage, including for out-of-vocabulary words, while limiting compression operations.
- Subword Neural Machine Translation: Subword Neural Machine Translation segments text into subword units, making it useful for encoding out-of-vocabulary words.
- Byte Pair Encoding: The adapted Byte Pair Encoding procedure builds a pair table from frequent character sequences and merges characters rather than bytes.
- Byte Pair Encoding: The compression routine stops after a specified number of operations instead of continuing until no bytes remain.
- Subword Neural Machine Translation: The method uses a tunable number-of-operations parameter to balance complete words and subwords in the dictionary.
- Subword Neural Machine Translation: Subword segmentation improves token coverage and allows out-of-vocabulary words to be represented as combinations of word and subword tokens.
- Byte Pair Encoding: Original Byte Pair Encoding replaces frequent adjacent byte pairs with previously unused bytes during iterative compression.
- Byte Pair Encoding: During expansion, literals pass directly to the output buffer while byte pairs are pushed onto a stack for iterative processing.
Appendix C Additional Dataset Information
Vocabulary similarity varies across the journals represented in the dataset. Additive Manufacturing Letters is most similar overall, while Rapid Prototyping Journal and Journal of Additive Manufacturing are least similar.
- Additive Manufacturing Letters exhibits the highest vocabulary similarity across all journals.
- Rapid Prototyping Journal and Journal of Additive Manufacturing show the lowest vocabulary similarity.
- Figure 11 compares vocabulary similarity among open-access articles grouped by journal.
Appendix D Gemma 3 Pre-Trained Variant Results
The Gemma 3 pre-trained variants generally perform worse than instruction-tuned base variants, while visual instruction tuning usually produces the strongest results. LPBF anomaly identification is the main exception, with performance decreasing across additional training stages.
- Visual instruction tuned models achieve the best performance in all tasks except LPBF anomaly identification.
- LPBF anomaly identification performance decreases with additional training stages.
- Instruction tuning improves an LLM’s general usability, while visual instruction tuning can affect performance in a specific domain.
- Performance noticeably decreases at the DAPT image training stage for both pre-trained and instruction-tuned base-model variants.
- Possible contributors to weaker image-stage results include frozen language attention weights, missing image-caption pairs, and catastrophic forgetting.
- Gemma-3-12b-pt adaptations generally perform worse than their Gemma-3-12b-it counterparts across evaluated tasks.