Source-linked AI summary
Beyond Maintenance Manual Multimodal RAG: Suggesting What Tool
Seongjun Ha, Md Rashedul Islam
TL;DR
Aircraft technicians must search difficult-to-use maintenance documents, while MRAG retrieval does not identify all hand and special tools needed for a task. MRAG-SWAT extends MRAG with IPC-linked tool mapping and hierarchical resolution, and its eight-query demonstration returned correct tool suggestions for six queries, with documented scope and mapping limitations.
Problem
Maintenance manuals omit hand-tool requirements and may reveal special tools only at the relevant procedure step, requiring additional IPC-based inference and preparation.
Method
MRAG-SWAT passes an MRAG-generated response to SWAT, which uses an IPC-linked tool mapping table and hierarchical tool resolution to suggest hand and special tools.
Results
The suggested tools matched the ground truth in six of eight queries, while queries #5 and #7 returned incomplete tool sets.
Takeaways & Limitations
By presenting tool suggestions with retrieved procedures, MRAG-SWAT may help technicians obtain the tools needed to perform maintenance tasks.
Takeaways & Limitations
SWAT does not account for accessibility-dependent extensions, adapters, or offset drives, and ambiguous hardware specifications require expert review before deployment.
Abstract
from arXiv · showhide
Aircraft technicians are required to consult the maintenance manual (MM) for nearly every task, and locating the relevant procedure across hundreds of pages remains time-consuming. Multimodal retrieval augmented generation (MRAG) has been proposed to address this, allowing technicians to retrieve procedures, together with the accompanying figures, through natural-language queries. However, retrieval alone does not tell the technicians which tools the task requires. The MM identifies special tools only when the corresponding step is reached, and it does not state hand tool requirements at all; to select hand tools, technicians are required to find the hardware dimension from the illustrated parts catalog (IPC) and infer the right tool from it. We therefore propose MRAG-SWAT, an extension of the MRAG pipeline that returns the required hand tools and special tools alongside the retrieved procedure. The framework was implemented for the Lycoming IO-360-N1A engine and demonstrated on eight test queries. By presenting the correct tools together with the procedure, MRAG-SWAT may help reduce repeated trips to the tool crib, prevent damage to aircraft caused by improper tool selection, and thereby avoid additional maintenance tasks and support continued airworthiness.
I. INTRODUCTION
Aircraft maintenance requires searching multiple, difficult-to-navigate documents, while existing MRAG retrieves manual content without fully supporting technicians’ tool-preparation workflow. MRAG-SWAT addresses this gap by suggesting hand and special tools alongside retrieved procedures.
- Document-search problem: Technicians consult maintenance manuals, illustrated parts catalogs, service bulletins, and airworthiness directives, but locating relevant information is time-consuming.The large volume and poor usability of these documents make maintenance tasks frustrating.
- Limits of existing MRAG: Existing MRAG systems retrieve multimodal maintenance-manual information but cannot provide information absent from the manual’s vector database.MRAG was developed to incorporate visual information because technicians rely on figures as well as text.
- Tool-preparation gap: Maintenance manuals do not clearly specify required hand tools, and special-tool requirements may appear only when technicians reach the relevant procedure step.Hardware dimensions needed to infer hand tools are instead found in the IPC.
- Tool-preparation gap: These omissions can require technicians to estimate hardware sizes, rely on judgment, and make repeated trips to the tool crib during a task.The problem is especially likely when technicians do not review the full procedure in advance.
- Proposed response: MRAG-SWAT is proposed as an advanced MRAG framework that suggests the hand and special tools required for maintenance tasks.The proposal is motivated by limitations in both maintenance-manual design and existing retrieval systems.
II. RESEARCH OBJECTIVE
The research objective is to extend MRAG with a tool-suggestion process that combines generated maintenance responses with IPC-linked mappings. SWAT uses hierarchical resolution to return tools at either chapter-section or task level.
- Research objective: MRAG-SWAT aims to retrieve maintenance information while simultaneously suggesting the tools required by the generated response.The framework is designed to provide the procedure and associated tools together.
- MRAG pipeline: A technician’s query passes through ColPali retrieval and a vision-language model before the generated response is sent to SWAT.The MRAG pipeline retrieves image-based document pages, which the VLM uses to generate the response.
- SWAT: SWAT uses a tool mapping table and hierarchical tool resolution to return suggested tools alongside the task procedure.HTR determines whether the response should be resolved at chapter-section level or at the more specific task level.
- Tool mapping table: The tool mapping table links special tools directly from MM entries and resolves hand tools through hardware drive type and dimension.Hardware dimensions are obtained through IPC lookup and converted into tool categories and sizes using published standards.
- Hierarchical Tool Resolution: HTR first checks for a mapped chapter-section and then checks whether the response identifies a specific task.If no chapter-section is identified, the response is returned without modification; otherwise, the resolution level depends on response specificity.
IV. CASE STUDY: LYCOMING IO-360-N1A
The case study applies MRAG-SWAT to the Lycoming IO-360-N1A engine and uses its maintenance manual and illustrated parts catalog to support tool mapping.
- Case-study documents: The Lycoming IO-360-N1A case study uses the engine maintenance manual and illustrated parts catalog as its two maintenance documents.The MM describes inspection, cleaning, repair, and assembly/disassembly, while the IPC provides component views and dimension-bearing part descriptions.
A. MRAG Pipeline for Lycoming IO-360-N1A
MRAG-SWAT extends multimodal retrieval for the IO-360-N1A MM by mapping retrieved procedures and IPC hardware information to hand-tool and special-tool suggestions. Its HTR algorithm selects suggestions at either chapter-section or task granularity based on the generated response.
- MRAG Pipeline: The IO-360-N1A implementation embedded the MM and IPC and used ColQwen2 retrieval with GPT-4.1 to generate responses from retrieved document images.The combined documents contained 279 PDF pages.
- SWAT Output: The resulting framework passes the MRAG-generated response to SWAT, which returns tool suggestions from the mapping table.Special tools are drawn from the MM, while hand tools are inferred from hardware identified through the IPC.
- MM Hierarchy: The MM organizes information hierarchically by chapter-section, task, and procedure, providing the structure used for tool resolution.The chapter-section format follows ATA numbering, while tasks contain the procedural steps and tool-related hardware.
- Tool Mapping Table: The tool mapping table links MM tasks to special tools and to IPC hardware dimensions used to infer corresponding hand tools.Hardware dimensions were converted to tool sizes using published sizing tables for bolts, screws, and hex-socket fittings.
- Hierarchical Tool Resolution: HTR first checks whether the response identifies a mapped chapter-section and then checks whether it identifies a specific task.Chapter-section matches return section-level tools, whereas task-specific matches return tools mapped to that task with the procedure.
C. Test Queries and Assessment
The evaluation used eight queries crossing expected tool-output category with HTR suggestion level. Each generated tool set was qualitatively compared with ground truth derived from the MM and IPC.
- Test Design: The eight-query test set crossed four expected outputs—no tools, special tools only, hand tools only, or both—with chapter-section and task-level resolution.Each output category was tested once at each HTR level.
- HTR Levels: At chapter-section level, SWAT returns all tools mapped to the section, while task level returns only tools mapped to the specific task with its procedure.The level reflects whether the query is general or task-specific.
- Assessment: Each query was processed through MRAG-SWAT, and its suggested tool set was assessed qualitatively against MM-and-IPC-derived ground truth.The validation scenarios are listed in Table 6.
- Assessment: The no-tool scenario was not applicable for tool-suggestion assessment because it required no tools.This scenario was explicitly excluded from tool-suggestion applicability.
V. RESULTS AND DISCUSSION
Across eight queries, MRAG-SWAT generally matched expected tool outputs, but errors arose when generated responses did not reliably identify the intended hierarchy or when retrieved procedures spanned pages.
- HTR Classification: Six of eight queries received correct HTR chapter-section classifications, while queries #5 and #7 were classified at the task level instead of chapter-section level.These mismatches prevented tools associated with the broader chapter-section from being returned.
- Prompting Limitation: The authors attribute unreliable level matching to the absence of a structured prompt template that consistently states the chapter-section or task.They identify prompt-template design and its connection to SWAT as areas for further study.
- Tool Suggestions: Six of eight queries produced tool suggestions matching ground truth, while queries #5 and #7 returned incomplete tool sets.Queries #1–#2 returned no mapped tools, #3–#4 special tools, #6 hand tools, and #8 both categories.
- Retrieval Limitation: ColPali treats pages as self-contained semantic units, so procedures continuing onto the next page may not be retrieved completely.In query #6, SWAT still suggested hand tools because it resolved the query level from the generated response and mapping table.
- Document Linking: Because the MM and IPC lacked a common contents structure and matching figure names, hardware cues were matched by chapter and component instead.Part-number resolution therefore relied on manual interpretation.
VI. LIMITATIONS
The study’s limitations concern tool-selection context, ambiguous hardware specifications, and the scope of its claims about maintenance support.
- Scope of Tool Mapping: SWAT does not account for extensions, adapters, offset drives, accessibility constraints, or other physical task conditions that can change tool requirements.Its deterministic dimension-to-tool mapping covers hardware dimensions but not the physical situation of the task.
- Expert Review: Some hardware specifications remain ambiguous, so the tool mapping table requires review by aircraft maintenance experts before deployment.Fillister head slotted screws are given as an example because their descriptions specify head style but not slot width.
- Supported Scope: The conclusion describes MRAG-SWAT as adding IPC-linked tool suggestions to MRAG responses for helping technicians perform maintenance with correct tools.The supplied conclusion does not establish deployment-level effectiveness beyond that stated support.
Appendix A: MRAG-SWAT Outputs and Ground Truth for the Eight Test Queries
The eight test queries pair retrieved maintenance procedures with tool suggestions, including hand tools inferred from hardware and special tools named in the MM. Outputs show both correct matches and query-level omissions when retrieval is classified at an unsuitable level.
- The oil pressure relief valve inspection procedure and the unscheduled corrective-maintenance chapter-section were both reported with no hand or special tools.
- Propeller flange bushing removal and installation used the ST-115 removal/installation tool and required no hand tool.The removal procedure directs technicians to label bushings, consult Service Instruction No. SI-1098, and use ST-115.
- The cleaning-guidelines query returned no hand or special tools at the task level, while chapter-section ground truth listed a 3/16 hex key and 5/32 hex key.The chapter-section output contained the two hand tools, but the task-level output reported none.
- The cylinder-removal procedure required a 3/4 socket or wrench, 3/8 blade slotted screwdriver, and 9/16 socket or wrench, with no special tool.
- Crankcase disassembly required nine listed hand-tool sizes and three special tools: ST-271, ST-317, and ST-389.The procedure identifies ST-271 and ST-317 for removing crankcase thru-studs, while the output also lists ST-389 for crankcase separation.
- Crankcase dimensional inspection required a 3/4 socket or wrench and the ST-222 torque hold-down plate.