AMGPT: a Large Language Model for Contextual Querying in Additive Manufacturing

Introduction

• Fine-tuning can reduce hallucinations and improve task performance, but does not eliminate hallucinations and entails specialized-data, context-capacity, and computational constraints.The introduction cites the need for large specialized datasets, limited context tokens, and substantial processing power and GPU memory.Source: §1 · pp. 3–4
• AM production involves complex material behavior, process parameters, and stage interactions, while reliable journal-paper referencing by LLMs remains underexplored.The introduction says 3DGPT is proprietary and lacks specific considerations to reduce hallucinations.Source: §1 · p. 4
• RAG retrieves query-relevant text to augment an LLM with specialized knowledge; access to data from the training set can improve performance.For AM, the introduction anticipates that combining RAG with procedural guidelines will support more reliable outputs.Source: §1 · pp. 4–5
• AMGPT is an open-source, 7-billion-parameter model using RAG and chat memory to produce cohesive, evidence-based responses for additive manufacturing queries.The introduction frames its practical implementation as addressing challenges specific to the field.Source: §1 · p. 5
• Its stated contributions are to characterize RAG effectiveness on pretrained models, create an AM-paper-based subject-matter-expert LLM, and develop a chatbot deployment pipeline.The pipeline contribution is described as beginning with acquisition of a pretrained model.Source: §1 · p. 5

• The implementation spans loading models from Hugging Face through building a Streamlit-powered user interface.Source: §1 · p. 5

2.1. Deep Learning for Additive Manufacturing

• AMGPT extends task-specific AM deep-learning applications with LLM-RAG assistance for contextual support, decision-making, and knowledge management across specific and general AM tasks.Earlier work includes topology optimization with manufacturing constraints and in-situ defect or porosity prediction; AMGPT is presented as a broader language-model application.Source: §2.1 · pp. 6–7

2.2. Medium Language Models

• An OpenAI study reports that adding parameters does not directly boost response-generation performance, and human ratings found accuracy may not increase with parameter count; the authors select models with 1-10B parameters, such as Llama.The discussion adds that larger models tend to hallucinate, while additional parameters can improve output-text precision.Source: §2.2 · p. 7

2.3. Quantization

• Quantization lowers model-weight precision to reduce memory use, with FP8 or FP4 requiring half or a quarter of default memory at some performance cost.For their system, the authors use 7B models in FP16 and FP8 and do not use mixed-integer quantization.Source: §2.3 · pp. 7–8

2.4. Fine-tuning

• Because fine-tuning LLMs requires substantial compute and updating all weights for more epochs is unfeasible, efficient methods such as LoRA interweave trainable rank-decomposition matrices into each Transformer layer.Source: §2.4 · p. 8

2.5. Retrieval-Augmented Generation

• RAG supplies concrete source evidence without retraining model parameters or changing architecture, and a cited study reports GPT4.0 accuracy rising from 80.1% to 91.4% for preoperative instructions [43].The evaluation used an expert panel and healthcare guidelines [43].Source: §2.5 · p. 9
• RAG uses nonparametric memory to boost accuracy in specific domains at lower expense than costly tuning of parametric memory, and the authors consider it sufficient for parsing domain documents with a pre-trained medium model.The review also describes RAG as enabling factual, precise, and varied responses when a pre-trained generator and retriever are bootstrapped to a base LLM.Source: §2.5 · p. 9
• Zero-shot reasoners receive no prior examples or requisite knowledge with the query.Source: §2.5 · p. 10
• Appending “Let’s think step by step” is described as boosting LLM performance, while chain-of-thought prompting generates novel, accurate responses [49].Source: §2.5 · p. 10
• Retrieval-Augmented Thought (RAT) calls the RAG model at every thinking step, dynamically expanding the number of relevant tokens the LLM can access.Source: §2.5 · p. 10

3.1. Model

• The RAG system employs LLaMA2-7b [40] for its capabilities in understanding and generating natural language, and the model serves as the foundation for the generation process.Source: §3.1 · p. 10
• The authors cite LLaMA2’s balance between computational efficiency and performance across diverse NLP tasks as a reason for its selection.Source: §3.1 · p. 10
• Hugging Face Transformers provides access to pretrained models and supports implementation of custom workflows.Source: §3.1 · p. 10

3.2. Retrieval Mechanism

• Figure 1 depicts database querying with an embedding model to provide context to an LLM, which then generates an answer.Source: Figure 1 · p. 10
• The query encoder maps the prompt to a vector, while the document encoder maps corpus documents into the same semantic space.Source: §3.2 · p. 11
• The dual-encoder setup retrieves documents relevant to the input prompt using cosine similarity [50] between query and document vectors.Source: §3.2 · p. 11

3.3. Integration with Hugging Face

• The implementation uses Hugging Face’s transformers and datasets libraries.Source: §3.3 · p. 11
• The transformers library provides tools for LLaMA2, including pre-trained model weights and tokenizer configurations.Source: §3.3 · p. 11
• The datasets library manages the retrieval corpus, supporting the indexing and querying needed for RAG’s real-time information retrieval.The Hugging Face ecosystem is used to streamline training, fine-tuning, and deployment.Source: §3.3 · p. 11

3.4. Embedding model

• Text embeddings map strings into high-dimensional vectors, and embeddings can vectorize elements across data types, including images and audio.For this paper, text embeddings are the primary focus.Source: §3.4 · p. 12
• The embedding model is not quantized, although the language model can be reparameterized to lower-precision floats or integers.Source: §3.4 · p. 12
• Text embeddings represent semantic meaning, including context, tone, and similarity between words or phrases.Source: Figure 2 · p. 12
• Semantic search uses embeddings to focus on the intention of a user’s query, unlike standard keyword search.Source: §3.4 · p. 12
• The embedding model, sentence-transformers/all-mpnet-base-v2, encodes text as vectors for the paper’s focus on text embeddings.SentenceTransformers provides BERT-style transformer models fine-tuned for semantic search.Source: §3.4 · p. 12

3.5. Mechanisms and Concepts of Text Embeddings

• ELMo and BERT generate representations that use sentence or passage context to capture a word’s meaning.Source: §3.5 · pp. 12–13
• Vector-space models map text to high-dimensional vectors whose geometric relationships reflect semantic relationships.Source: §3.5 · p. 13
• Dimensionality reduction compresses high-dimensional vectors while preserving key relationships, aiding semantic and syntactic similarity capture and efficient computation.Source: §3.5 · p. 13
• Modern embeddings use neural networks ranging from shallow word-embedding models to transformer architectures such as BERT and GPT for deeper context understanding.Source: §3.5 · p. 13

3.6. Devices and Codebase

• The implementation uses the LLAMA2-7b chat model and sentence-transformers/all-mpnet-base-v2 embedding model on a local NVIDIA RTX A6000 GPU.Source: §3.6 · p. 13
• The code is made available for public benefit and further research.Source: §3.6 · p. 13
• Table 1 is identified as listing components and tools for RAG.The supplied caption gives the table’s scope but not its component entries.Source: Table 1 · p. 13

3.7. Operational Workflow

• PDF documents are converted into LaTeX with Mathpix to facilitate extracting relevant mathematical expressions from the corpus.The resulting vectors are stored in LlamaIndex’s VectorStoreIndex, which serves as the retrieval corpus.Source: §3.7 · p. 14
• The query encoder embeds each prompt as a vector to identify the most relevant documents in the retrieval corpus.The query vector is used to select relevant documents based on similarity scoring.Source: §3.7 · p. 14
• Retrieved information is passed to LLaMA2-7B with the original prompt to generate contextual responses.Figure 3 depicts document chunks being embedded, stored, retrieved, and supplied to the language model.Source: §3.7 · p. 14, Figure 3 · p. 14

3.8. Chat Interface

• Streamlit is an open-source Python library for creating interactive web applications without extensive web-development knowledge.Source: §3.8 · p. 15
• Streamlit was used to build a chat interface through which users interact with the model and receive responses to their input.The interface is described as user-friendly and is shown in Figure 4.Source: §3.8 · p. 15, Figure 4 · p. 15
• The interface records chat memory for future prompting to enhance continuity and relevance across interactions.Streamlit is identified as the tool used to build the interface.Source: §3.8 · p. 15

4.1. Influence of sampling temperature

• At temperature 1.5, strange language appeared; the 0.4 response meandered, while the 0.7 response highlighted oxidation and contamination.Vague or hallucinatory terms began appearing at temperatures of 1.5 or higher; the effects are illustrated in Table 2.Source: §4.1 · pp. 16, 18, 19, 20, Table 2 · p. 17

4.2. Influence of system prompt

• AMGPT followed system prompts, shifting between brief expert-style answers and simplified explanations that avoid complex terminology for a general audience.The expert-style response included the uncommon term “acuity”; the lay explanation was described as very simplified.Source: §4.2 · p. 21

4.3. Influence of max token length

• A max token length of 1,024 or more led to hallucinations after relevant information ran out, while the reported responses used 768 tokens.Lengths below 200 tokens constrain responses to a few brief sentences, whereas lengths above 256 support longer descriptive answers; straightforward one-line factual answers were unaffected.Source: §4.3 · p. 25

4.4. top_k Retrieval Methodology

• For the Llama2-RAG LLM, top_k=3 produced the most balanced response, providing more examples than top_k=2 without digressing when top_k was 4 or 6.Source: §4.4 · p. 26
• A lower-probability retrieved response digressed into etchants irrelevant to the nickel-superalloy query, and evaluators judged its quality worse.Source: §4.4 · p. 28

4.5. Evaluation

• Responses were assessed through fact-checking against known literature [59] and human researchers’ rankings of relevance and accuracy, with remarks on merits and shortcomings.Source: §4.5 · p. 29
• The human evaluation was blind: researchers were not told whether responses came from a standard LLM or a retrieval-augmented LLM.Source: §4.5 · p. 29
• 80% of prompts received factual, non-hallucinated responses from Llama2-RAG, compared with 86.7% from GPT-4.These results are reported across queries in Table 6.Source: §4.5 · p. 29
• A human expert assessed GPT-4 as giving a vague response to one query, whereas the RAG system maintained a high level of specificity.Source: §4.5 · p. 29

5. Conclusion and Future Work

• AMGPT is presented as a metal additive manufacturing expert LLM that users can query through a user interface.Source: §5 · p. 30
• Minimizing top_k and temperature yielded the most relevant results for the corpus-referencing task, which constrains answers to an external factual base.Source: §5 · p. 30
• Less than 0.5% of GPT-4’s parameters, AMGPT’s RAG system is reported to maintain high fidelity and accuracy of answers and enhance LLM question-answering capabilities.Source: §5 · p. 30
• Fine-tuning was excluded from AMGPT’s scope because of timeline, hardware, and external-subscription constraints.The authors identify fine-tuning as a possible benefit for future exploration.Source: §5 · p. 30
• Future work includes improving chat memory so the system better records and recalls prior interactions over prolonged use.Source: §5 · p. 30
• Future research will improve descriptions of images and diagrams in journal papers using image-recognition and natural-language-processing techniques.The stated aims are more precise descriptions and clearer, more accessible visual data.Source: §5 · pp. 30–31

Appendix

• The RAG response correctly identified multiple NDT methods, while GPT-4 incorrectly classified thermal imaging as NDT and attributed its results to thermal-conductivity variations.The RAG examples include X-ray CT, ultrasonic testing, magnetic particle inspection, eddy-current testing, and liquid penetrant testing.Source: §6 · p. 41
• GPT-4 was judged more accurate on keyhole defects; the RAG answer incorrectly linked high conductivity to intensified heat concentration and treated powder porosity as a material property.The supplied comparison describes high reflectivity and thermal conductivity as relevant to keyhole formation.Source: §6 · p. 42
• Phase-composition, titanium, high-strength-aluminum, and Rosenthal answers were judged factual or core-correct, though detail and emphasis varied, and the RAG response misstated the Eagar-Tsai equation’s origin.The comparison says Eagar-Tsai is derived by solving the heat-conduction equation with a different heat-source formulation, not from the Rosenthal equation.Source: §6 · pp. 44–47
• Both models gave reasonable DDC explanations and comparable creep answers; DDC susceptibility depends on localized strain, reduced ductility, alloy composition, cooling rates, and thermal gradients.The RAG answer also lists preheating, a high P/v ratio, and atmosphere control as mitigation strategies; creep factors include microstructure, process parameters, post-processing, and design.Source: §6 · pp. 48, 52
• The RAG laser-types response was more detailed, but GPT-4 was more up-to-date about CO2 lasers and the RAG response alone mentioned diode lasers.The supplied comparison describes fiber lasers as prevalent in modern LPBF systems and CO2 lasers as less common today.Source: §6 · p. 50
• Both density explanations were judged accurate, with the RAG response more specific; tapped density exceeds bulk density because tapping reduces void spaces, and their ratio can indicate flow behavior.Bulk density includes interparticle space without external force, while tapped density uses the settled volume after tapping or vibration.Source: §6 · p. 51
• Five metal-powder production methods are described: atomization, electrolysis, chemical reduction, mechanical alloying, and vapor deposition.The passage associates these methods with different metals and desired powder characteristics, including purity and microstructure.Source: §6 · p. 53
• A response on powder production is called fairly accurate but is criticized for substituting “electrochemical atomization” and “Vapor deposition” for the specified process terms.The suggested terms are “electrode induction melting gas atomization” and “Vapor phase condensation,” respectively.Source: §6 · p. 53
• AM enables FGM production through layerwise control of composition and structure, material versatility, design flexibility, digital control, and small-batch cost-effectiveness.These capabilities support tailored gradients and complex geometries; the responses are described as factually accurate and comparable.Source: §6 · p. 54