PaperlayerView exact revision on arXiv

Source-linked AI summary

Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports

Parsa Bakhtiari, Hassan Bashiri, Alireza Khalilipour, Masoud Nasiripour, Moharram Challenger

arXiv:2608.22817v1cs.CL

TL;DR

Industrial technical reports combine heterogeneous, scattered information that is difficult to retrieve and reason over, while general-purpose models lack the depth and practical communication style specialists need. The study constructs industrial-document datasets through an automated extraction and generation framework, and finds that Claude produces cleaner data and larger fine-tuning gains than Qwen at much higher cost, while Qwen remains cost-effective.

  • Problem

    Industrial reports contain heterogeneous, scattered information, and general-purpose models often lack the deep, precise, concise, and actionable knowledge needed in industrial settings.

  • Method

    The study builds industrial-document training datasets through an automated pipeline that extracts text and tables, removes images, and generates data using open-weight and closed API models.

  • Results

    Claude-Opus-4.6 produced a cleaner raw dataset and larger downstream fine-tuning gains than Qwen3-30B-A3B-Instruct, but cost roughly two orders of magnitude more.

  • Takeaways & Limitations

    Using the same pipeline for open-weight and frontier models enables direct comparison across data quality, fine-tuning gains, cost, and general-knowledge retention.

  • Takeaways & Limitations

    Fine-tuning did not improve AccPerIBM beyond 0% for Qwen3-4B-Instruct, indicating persistent brittleness to question rephrasing in that backbone.

Abstract

from arXiv · show

Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.

1 Introduction

Industrial reports contain valuable, specialized knowledge, but heterogeneous formats and scattered information make them difficult to retrieve and use with general-purpose models. Industrial-Instruction addresses this gap by releasing two Panasonic QA datasets and a reproducible construction pipeline for training and evaluating small language models.

  • Motivation: Industrial reports support applications such as condition-based maintenance and FMEA but combine technical information, tables, charts, and diagrams across dispersed sections.This heterogeneous organization complicates categorization, retrieval, and extraction of relationships among related facts.
  • Motivation: Tables are important because they complement surrounding text, support decision-making, and encode relationships among variables such as failure modes and sensor patterns.Large language models may analyze both table structure and content to identify conceptual relationships and dependencies.
  • Motivation: 29% accuracy was achieved on industrial benchmarks by open-weight models fine-tuned on general datasets such as HotpotQA, while GPT-4 and Llama averaged 53.5% on FailureSensorIQ.These results illustrate the difficulty of industrial question answering for models trained without sufficient domain-specific evidence.
  • Motivation: Industrial language models face gaps in deep knowledge, specialist communication style, and domain-appropriate evaluation criteria.Industrial users require precise, concise, actionable guidance rather than general explanations, while specialized datasets and benchmarks remain necessary.
  • Contributions: Industrial-Instruction contributes two Panasonic QA datasets and an end-to-end pipeline combining layout-aware extraction, semantic indexing, automated filtering, and multi-scenario retrieval-augmented generation.The parallel datasets use Qwen3-30B-A3B-Instruct or Claude-Opus-4.6 and include instruction-tuning data with held-out benchmark splits for models under 10B parameters.
  • Contributions: The same pipeline enables comparison of open-weight and closed API data generators across quality, fine-tuning gains, cost, and general-knowledge retention.The raw corpus contains 906 Panasonic documents spanning 7,525 pages, and the code, pipeline, scripts, and datasets are publicly released.

2 State of the Art

Prior industrial language-model research highlights difficult failure-mode and sensor reasoning tasks and the importance of representing tables effectively. Existing studies compare table-to-text methods across fine-tuning and retrieval-augmented-generation settings.

  • Industrial benchmarks: FailureSensorIQ evaluates industrial reasoning with 8,296 expert-selected multiple-choice questions spanning 10 assets, including electric motors, steam turbines, and power converters.Its tasks cover identifying sensors for a failure and inferring failure modes from abnormal sensor data.
  • Industrial benchmarks: 53% average accuracy was achieved by the best models on FailureSensorIQ, while question stress and variation could reduce accuracy to 12%.The benchmark targets robustness to noisy industrial data and changes in problem phrasing.
  • Table representation: Table-to-text research compares Markdown templates, hand-designed serialization, traditional pretrained models such as BART, and large-language-model-based descriptions.These methods are evaluated in domain-specific fine-tuning and retrieval-augmented-generation paradigms.
  • Table representation: Table-to-text conversion choices produced metric differences ranging from 2.8% to 16%, while domain-specific fine-tuning methods generated more specialized terminology and diverse verbs.The findings emphasize that representation choices affect industrial language-model performance.

3 Methodology

The methodology combines learning-based PDF extraction, semantic retrieval, and scenario-based multiple-choice QA generation to construct industrial datasets from technical documents. It filters generated samples to retain balanced, structured QA data while comparing Qwen- and Claude-generated outputs.

  • Dataset construction: The study collects Panasonic documents and targets industrial training and evaluation datasets containing text, tables, and associated source documents.The dataset is designed for information retrieval-augmented generation and industrial multiple-choice QA.
  • Information extraction: Learning-based extraction methods are preferred because industrial PDFs contain complex layouts, tables, images, and often lack reliable digital structure.Vision-language models can jointly recognize layout, text, and relationships among visual document elements.
  • Information extraction: Dots.OCR converts PDF pages into Markdown, JPG, and structured JSON outputs, supporting inspection of extracted regions and semantic document structure.The model is used for text extraction despite risks including hallucinations and inaccurate localization in dense document areas.
  • Knowledge-base structuring: After extraction, the pipeline retains text and Markdown tables, removes images and low-content or duplicate pages, and indexes content with EmbeddingGemma and FAISS.This design focuses evaluation on text-based language models while enabling efficient semantic retrieval.
  • Dataset creation: The retrieval system generates QA examples across scenarios ranging from irrelevant documents to single- and multi-document support, requiring evidence integration or direct answer extraction.The scenarios model different relationships between a query and retrieved documents, including multi-step reasoning from scattered clues.
  • Question-and-answer preprocessing: 23,910 Q&As are initially generated, then filtering yields balanced per-scenario data, with Claude outputs requiring less filtering than Qwen outputs.The resulting distributions contain 5,215–5,273 samples per scenario; Claude generation costs approximately $330 versus roughly $3.2 for Qwen.

4 Experimental Setup

The experiments evaluate fine-tuned small language models on Panasonic and FailureSensorIQ benchmarks using task-specific metrics, while comparing LoRA and full fine-tuning. Full fine-tuning improves Panasonic performance, whereas the reported LoRA configurations do not.

  • Evaluation framework: The evaluation framework tests models on held-out Panasonic data and FailureSensorIQ to measure dataset impact.Models are first evaluated, then trained on the remaining split and evaluated again.
  • Evaluation metrics: Set-Match-Accuracy, F1-Score, and Jaccard Similarity evaluate unordered, set-based Panasonic answers rather than exact formatting.Set-Match-Accuracy checks set equality, F1 balances precision and recall, and Jaccard measures set overlap.
  • Evaluation metrics: FailureSensorIQ comparisons use AccOrgIBM, AccPerIBM, F1-Macro, and F1-Micro, while Panasonic comparisons use Set-Match-Accuracy, F1-Score, and Jaccard Similarity.AccPerIBM evaluates robustness to rephrasing, and the F1 metrics are macro- and micro-averaged across question categories.
  • Model scope: The study focuses on small models because they require fewer hardware resources and may be practical for small companies and research teams.Qwen-4B-Instruct is used as a base model in the training and evaluation process.
  • Fine-tuning: LoRA fine-tuning produced no significant changes, so the experiments proceeded to full fine-tuning of all model parameters.Changing LoRA-related parameters had no impact on the error rate or model improvement.
  • Fine-tuning: Full fine-tuning increased Set-Match Accuracy from 28.5% to 42%, with F1-Score reaching 63.48% and Jaccard-Similarity 57.95%.These results exceeded the baseline results reported for the Panasonic benchmark.

5 Results and Discussion

Fine-tuning on either generated dataset substantially improves Panasonic performance, but effects on FailureSensorIQ and general knowledge differ by generator. Claude-generated data preserves general knowledge more closely, while perturbation robustness remains an unresolved limitation.

  • General knowledge: Claude fine-tuning retains essentially all base-model MMLU knowledge with a 0.05-point drop, whereas Qwen fine-tuning loses 1.26 points overall.Claude remains on par with or marginally better than the base model across categories, while Qwen forgetting concentrates in Humanities.
  • FailureSensorIQ benchmark: Qwen fine-tuning raises FailureSensorIQ F1-Macro from 40% to 43% and F1-Micro from 66% to 74%, but lowers AccOrgIBM from 34% to 27%.AccPerIBM remains 0% for the Qwen variants, while RAG-Instruct-Llama3-8B scores 33% on that metric.
  • FailureSensorIQ benchmark: Claude fine-tuning raises FailureSensorIQ AccOrgIBM from 34.0% to 49.6% but lowers F1-Macro from 40.0% to 33.5% and F1-Micro from 66.0% to 50.3%.AccPerIBM remains 0% for both the base and Claude-fine-tuned Qwen models.
  • Robustness limitation: Across comparisons, AccPerIBM remains 0% for the base and both fine-tuned Qwen models after question perturbation.RAG-Instruct-Llama3-8B scores 33%, indicating that domain fine-tuning did not resolve the Qwen3-4B backbone’s rephrasing brittleness.

6 Conclusion and Future Directions

The research delivers a framework and datasets for training and evaluating small models on industrial documents, while identifying cost-quality trade-offs and robustness gaps for future work.

  • Contributions: The study constructs an industrial-document dataset for training and RAG, using extracted Panasonic product information and tables to generate and evaluate QA data.The pipeline uses Dots.OCR, Qwen-30B-Instruct-2507, Claude-Opus-4.6, Gemma3-300m embeddings, and FAISS retrieval.
  • Contributions: Models with fewer than 10 billion parameters improve on evaluation metrics after training on the constructed dataset.The study evaluates Qwen-4B-Instruct, Phi-3-mini-4k, and Llama3-based RAG-Instruct-8B.
  • Generator trade-offs: $3.2 in local compute versus roughly $330 in API usage makes Claude-Opus-4.6 generation about two orders of magnitude more expensive, despite cleaner data and larger fine-tuning gains.The reported improvement from Claude was not proportional to the cost difference, supporting open-weight generation as a cost-effective option.
  • Future directions: Future work will broaden model, document, RAG, and multimodal coverage to build a larger industrial benchmark.Planned extensions include diverse industrial sources, advanced RAG architectures, and documents combining text with images, diagrams, and structured visual content.
  • Future directions: All evaluated models failed every perturbed FailureSensorIQ question, revealing that the current construction pipeline does not include rephrased or adversarial variants.The authors identify generating paraphrased or adversarially rephrased QA pairs as a promising robustness direction.

A Dataset Creation Process

Dataset construction was completed in 1 hour and 43 minutes on a Pro 6000 WS system.

  • 1 hour and 43 minutes was required to complete the dataset construction process on a Pro 6000 WS system.

B Model Training Settings and Time

Training took just over 12 hours on two NVIDIA RTX 5090 GPUs, using gradient accumulation to achieve the target effective batch size under memory constraints.

  • Training duration and batch size: 12 hours, 3 minutes, and 3 seconds of training used two NVIDIA RTX 5090 GPUs with an effective batch size of 64.GPU memory limits required a per-device batch size of 2 and 32 gradient accumulation steps.

C Additional MMLU Detail

Additional MMLU analysis compares subject-level changes across fine-tuned models and identifies both shared shifts and a large Qwen-specific regression requiring verification.

  • Subject-level changes: Tables 16 and 17 report the largest subject-level MMLU changes relative to the base model for the Qwen and Claude fine-tunes.Full 57-subject scores for all three models are available in the released evaluation logs.
  • Shared subject shifts: Public relations and anatomy regress under both fine-tunes, while college mathematics, machine learning, and moral disputes improve under both.The passage characterizes these shared changes as potentially related to fine-tuning on the base model and dataset-independent generalization, respectively.
  • Qwen-specific regression: A -10.7-point swing in moral_scenarios for finetuned_v1_qwen warrants checking for answer-format drift rather than assuming genuine reasoning loss.

D Prompt Templates for the Five RAG Scenarios

The five scenario prompts share a common structure for generating questions, answers, and multiple-choice options, while varying document count and the query–document requirement. Listing 5 gives the r0 template, and the remaining templates and source code are publicly available.

  • Shared prompt structure: All five prompts instruct Qwen3-30B-A3B-Instruct to generate q*, a*, and, for multiple-choice tasks, five A–E options from supplied documents.The prompts share document placeholders, instruction simulation, and output formatting.
  • Scenario differences: The prompts differ only in the number of supplied documents and the requirement defining the query–document relationship.Table 18 summarizes these scenario-specific differences.
  • Representative r0 template: Listing 5 presents the complete prompt for the Useless Document r0 scenario, including its document input and simulated-instruction fields.The r0 prompt requires a related question whose document cannot provide useful answering information.
  • Multiple-choice formatting: When the simulated instruction is multiple-choice, generated questions must contain exactly five A–E options and answers must identify the correct option list.The output follows the JSON format {"q*": ..., "a*": ..., "options*": ...}.
  • Template reuse and release: The remaining four prompts retain the same structure as Listing 5, changing only document count and the first requirement, while full templates and the pipeline are publicly released.These differences are listed in Table 18 and supported by the public repository.
Loading 2608.22817v1…