PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine-Readable Operator Manuals over the ISO 11783 Network for Offline, Multilingual LLM Assistance

Timo Oksanen, Julius Gun

arXiv:2610.31863v1cs.CLcs.AIcs.IR

TL;DR

Agricultural machinery lacks a way to provide an implement’s technical manual to a local tractor-side assistant, limiting offline access to documentation. This paper proposes manual exchange over ISO 11783 and evaluates transfer and locally run models on cross-lingual questions. Complete manuals transferred in 31 to 89 seconds; English answers were stronger than answers on other language editions, where semantic retrieval was essential.

  • Problem

    The ISO 11783 series provides no mechanism for an implement to supply technical documentation to a tractor-side assistant, despite the need for offline access to manuals.

    Source: §1 · p. 1

  • Method

    The study transfers manuals over ISO 11783 and evaluates locally run open-weight LLMs answering questions with retrieval from the transferred documentation.

    Source: §1 · pp. 1–2, §4.3 · p. 4

  • Results

    31 to 89 s was required to transfer a complete manual; English-model F1 scores were 0.77 to 0.83, while scores on the other editions were 0.12 to 0.30 lower, and semantic retrieval was essential across languages.

    Source: §8 · p. 12

  • Takeaways & Limitations

    The exchange uses a new parameter group with existing ISO 11783 request and transport mechanisms, making it a candidate for standardization.

    Source: §1 · p. 1

    The proof of concept used one fertilizer-spreader manual and author-written questions, and native-language queries and embedded-hardware deployment were not tested.

    Source: §1 · p. 1, §7.7 · p. 12

Abstract

from arXiv · show

If agricultural machinery is to support intelligent, offline assistance, then the system must access technical documentation dynamically. In this paper, the Semantic Synchronization of operator manuals over ISO 11783-based CAN buses is analyzed, defined as the process that brings the local knowledge base of a tractor-side assistant into agreement with the operator manual of the connected implement. Data transfer performance was benchmarked using Markdown, JSON, and XML; various open-weight Large Language Models (LLMs) were also evaluated in cross-lingual query tasks. The automatic evaluation of the answers was validated against four human raters. A complete manual was transferred over a physical ISO 11783 network in 31 to 89 s, depending on format and transport settings, which is comparable to the object pool upload that operators already wait for at the first coupling of an implement. Markdown produced the smallest files, while the question-answering performance was comparable across the three formats for most models. With retrieval-augmented generation, locally run models answered most questions on the English manual correctly, with no advantage of the largest model over 7B and 8B models, but performed worse on the five other language editions. Keyword retrieval failed across languages, whereas semantic retrieval provided the cross-lingual capability. The exchange requires only a new parameter group that uses the existing request and transport mechanisms of ISO 11783, which makes it a candidate for standardization. The study is a proof of concept with inference on PC-grade hardware; deployment on embedded agricultural hardware remains future work.

1. Introduction

The paper addresses the lack of a mechanism for an implement to provide technical documentation to a tractor-side assistant over the standardized vehicle network. It proposes exchanging machine-readable manuals over ISO 11783 and evaluates transfer, formats, and cross-lingual question answering.

1. Introduction

  • The proposed system accesses digital operator manuals stored in an implement ECU from a tractor-cabin terminal, supporting use where internet connections are unreliable.The manual stays with the machine throughout its lifecycle; the proof of concept evaluates transfer on a physical network and question answering on PC-grade hardware.Source: §1 · p. 1
  • The ISO 11783 series provides no mechanism for an implement to supply technical documentation to a tractor-side assistant, while existing RAG systems rely on centrally maintained document collections.The paper proposes adding an application-layer parameter group that uses existing network management, request, and extended transport mechanisms.Source: §1 · p. 1
  • Semantic Synchronization is defined as bringing the tractor-side assistant’s local knowledge base into agreement with the connected implement’s manual.The study evaluates transfer and indexing; version checking and several first-coupling design elements remain proposed rather than evaluated.Source: §1 · pp. 1–2
  • The study evaluates whether a complete manual can be transferred over ISO 11783 within practical first-coupling time and compares Markdown, JSON, and XML across size, transfer time, token count, and question-answering performance.It also asks whether locally run open-weight models can answer questions from transferred manuals across languages and what retrieval components contribute.Source: §1 · p. 2

2. Motivation

The proposed self-describing architecture sends an implement’s structured documentation over the ISOBUS network to the tractor’s local LLM, allowing conversational access to safety information regardless of cellular coverage.

2. Motivation

  • The implement stores its documentation in a structured format and transmits it over the ISOBUS network to the tractor’s local LLM.The described data flow allows the operator to maintain conversational access to safety information regardless of cellular coverage.Source: §2 · p. 2

3. State of the Art

The related work spans LLM applications, ISO 11783 transport, cross-lingual capability, retrieval-augmented generation, and long-context processing. The paper distinguishes its focus on machinery-network exchange and highlights retrieval as a way to supply relevant context to models.

3.1. Applications of LLMs

  • Prior automotive, heavy-equipment, and agricultural work applies LLMs to interaction, decision-making, machinery management, and agricultural question answering.The paper distinguishes its work by enabling LLM knowledge access directly over ISO 11783, where hot plug-and-play functionality constrains system design.Source: §3.1 · p. 2

3.2. ISO 11783 Transfer Mechanism

  • Extended Transport Protocol supports messages up to approximately 117 million bytes, whereas classic Transport Protocol is limited to 1785 bytes.ETP uses connection-management and data-transfer parameter groups; the receiver can request retransmission if it detects packet loss.Source: §3.2 · p. 2

3.3. Cross-lingual Capability in Machinery

  • Prior multilingual agricultural studies report gains from multilingual data and fine-tuning, while the cited cross-lingual study reports difficulty with deep cross-lingual knowledge transfer.The related work suggests that cross-lingual capability could let a model answer in one language using documentation stored in another.Source: §3.3 · pp. 2–3

3.4. Retrieval-Augmented Generation (RAG)

  • Agricultural RAG systems retrieve relevant information from scientific documents, knowledge graphs, or vector databases before generating answers.The cited work describes multilingual query interfaces and reports accuracy gains over non-RAG baselines across agricultural subdomains.Source: §3.4 · p. 3

3.5. Large Context Window

  • Despite context windows reaching 128k tokens, models can miss information in the middle of long contexts, and prior work found small models perform better when RAG supplies a small relevant context.The cited work presents this approach as avoiding degradation associated with processing massive documents directly.Source: §3.5 · p. 3

4. Materials

The materials combine a regulated fertilizer-spreader manual with a curated question set for evaluating retrieval and answer behavior. The manual editions share a layout, while the questions test both answerable and unanswerable cases.

4.1. Operator Manual Structure

  • Regulatory requirements make operator-manual chapters predictable, although document organization is not unified, and the study selected a fertilizer-spreader manual with a shared structure.The selected machine’s spreader type has its own safety and health standard, ISO 4254-8.Source: §4.1 · pp. 3–4

4.2. Selected Operator Manual

  • The test case is the 165-page Kverneland Exacta-TLX Geospread GS3 manual, with six language editions verified to cover the same machine and software version.The editions have identical layouts; their Markdown versions contain approximately 61k English tokens and 83k to 89k tokens in the other languages.Source: §4.2 · p. 4

4.3. QA Test Dataset

  • The 108-question dataset balances 54 answerable and 54 unanswerable questions to test retrieval accuracy and whether models decline when manual content lacks an answer.All questions were posed in English regardless of document language to isolate cross-lingual retrieval capability.Source: §4.3 · p. 4

5. Methods

The methods test manual transfer over ISO 11783 and local question answering with converted manuals, multiple retrieval configurations, and locally run open-weight models. They compare three text formats and test English queries against manuals in six language editions.

5. Methods

  • Manuals were converted page by page from PDF into Markdown, JSON, and XML, preserving text, lists, tables, and recognized picture captions without manual editing.Page headers and footers were omitted; files were stored as UTF-8 with Windows line endings.Source: §5 · p. 4

5.1. Semantic Synchronization

  • Transfer tests used ISO 11783 ETP, a temporary proprietary parameter group, and the standard request message on a physical CAN bus running other control functions.The bus background load was below 2%, representing first implement coupling; the design proposes version checks to avoid repeat transfers unless the manual changes.Source: §5.1 · p. 5

5.2. RAG Configuration

  • Hybrid RAG combined BM25 keyword search and Qwen3-Embedding-8B semantic retrieval, fused rankings, and passed the top three chunks to the model.Documents were split into 200-token chunks with 100-token overlap; each retriever returned ten candidates before fusion.Source: §5.2 · p. 5

5.3. Models

  • Locally run open-weight models were evaluated at temperature 0, with question-level bootstrap confidence intervals and paired bootstrap comparisons; four human raters validated the automatic judge.The model set ranged from 1.5B to 120B parameters and used 4-bit quantization; the judge agreed with human-rater majority on 93.5% of items (Cohen’s κ=0.87).Source: §5.3 · p. 5, Table 1 · p. 6

5.4. Formatting Analysis

  • Markdown, JSON, and XML were selected as self-describing text formats; unstructured text, HTML, office files, and binary serialization were excluded for hierarchy, security, startup-CPU, or schema-coupling concerns.Markdown was chosen as a stream-based format with minimal structural syntax and human-verifiable content.Source: §5.4 · p. 6

5.5. Cross-lingual Tests

  • Cross-lingual tests posed English questions to German, French, Dutch, Italian, and Spanish manuals to isolate retrieval across language differences.The manuals were tested in their official editions, with all six editions verified as referring to the same machine and software version.Source: §5.5 · p. 6

6. Results

Transfer time and token use varied by format, while question-answering performance was generally similar across formats. Locally run models performed better on English manuals than on other language editions, where semantic retrieval substantially outperformed keyword retrieval.

6.1. Semantic Synchronization

  • Markdown produced the smallest files and transferred the English manual in 47 s at CTS 32; XML added approximately 5% overhead and indented JSON increased overhead by 75%.Without indentation, the English JSON file would be 351166 bytes rather than 556063.Source: §6.1 · p. 6, Figure 3 · p. 7

6.2. Formatting Efficiency

  • Markdown was the most token-efficient format; English required approximately 30% fewer tokens than the other European languages, while JSON used approximately 95% more tokens than Markdown in English.Token efficiency varies by model architecture and tokenizer implementation.Source: §6.2 · p. 7, Figure 4 · p. 7

6.3. Performance in LLM Tasks

  • For English, format differences were at most 0.05 for six of eight models and up to 0.12 for two DeepSeek-R1 models, with none significant; on other editions, Markdown exceeded JSON for two models.The non-English differences were +0.09 for Llama 3.1 8B and +0.08 for Qwen 2.5 7B; other format differences were not significant.Source: §6.3 · p. 7, Figure 5 · p. 8
  • With English Markdown manuals, F1 ranged from 0.54 for DeepSeek-R1 1.5B to 0.83 for Qwen 2.5 7B; GPT-OSS 120B scored 0.81 and did not exceed 7B and 8B models.The seven models with at least 3B parameters scored 0.77–0.83 with overlapping confidence intervals.Source: §6.3 · p. 7

6.4. Feasibility for Cross-lingual Tasks

  • With Markdown, F1 fell by 0.12–0.30 from English to the other editions pooled; GPT-OSS 120B and Qwen 3 8B scored highest on those editions, at 0.68 and 0.66.Llama 3.2 3B showed the largest drop, from 0.77 to 0.47.Source: §6.4 · pp. 7–8, Figure 6 · p. 8
  • On non-English editions, semantic retrieval placed the gold page among the top three chunks for 80–93% of questions, versus 9–24% for BM25; oracle context still left an English-performance gap of 0.11–0.23 F1.BM25 F1 was 0.35–0.47 lower than semantic and hybrid retrieval; oracle-context F1 was 0.06–0.11 higher than hybrid retrieval.Source: §6.4 · p. 8

7. Discussion

The discussion finds Markdown effective for transferring manuals over CAN, while emphasizing that initialization, parsing, hardware, and verification constraints shape deployment. Cross-lingual performance remains imperfect, and the study’s scope and test conditions limit how broadly its findings can be applied.

7.1. Protocol and Formatting

  • Markdown transferred manuals in 47 s versus 83 s for JSON, with 47–55 s at CTS 32 comparable to typical object-pool uploads.Markdown manuals were 317 to 372 kB; Figure 7 compares English F1 with pooled results for the five other editions by model and format.Source: §7.1 · pp. 8–9, Figure 7 · p. 9

7.2. Initialization and Resource Management

  • Initialization separates network-bound manual transfer from compute-bound embedding, enabling low-priority image transfer to overlap embedding.The proposed transfer occurs in a safe-state initialization phase; embedding is computationally intensive on embedded hardware, so the tractor caches the vector store for daily use.Source: §7.2 · pp. 9–10

7.3. Context Preservation

  • Markdown’s linear text stream supports locating retrieved passages and displaying their surrounding document context for operator verification.The terminal can match a retrieved chunk in the source Markdown and render, scroll to, and highlight that location.Source: §7.3 · p. 10

7.4. Hardware Outlook

  • The tested concept is not feasible on legacy tractor-side ECUs, though next-generation compute platforms may make deployment viable.Inference was tested on PC-grade hardware, not embedded tractor hardware.Source: §7.4 · p. 10, §7.6 · p. 11

7.5. Safety and Verification

  • The proposed verification interface pairs generated answers with source text, but incorrect highlighting and automation bias remain risks.The interface is proposed rather than evaluated with operators; its aim is to reduce, not remove, automation bias.Source: §7.5 · pp. 10–11, Figure 8 · p. 11

7.6. Model Capabilities

  • The 120B-parameter model offered no advantage over 7B and 8B models, while cross-lingual performance remained below English results.Semantic retrieval was essential for the other editions because keyword retrieval failed on translated text; even with gold-page context, an English-edition gap of 0.11 to 0.23 remained.Source: §7.6 · pp. 11–12

7.7. Limitations

  • Evaluation used one fertilizer-spreader manual and author-written English questions, which do not establish performance across manuals or native-language queries.The six language editions are translations of the same manual, not independent validation sets.Source: §7.7 · p. 12
  • The study did not test ECU inference, operator use of the verification interface, or security threats such as manipulated manuals and prompt injection.Answer evaluation was automatic, and each configuration was run once; transfer measurements used a laboratory bus under low background load.Source: §7.7 · p. 12

8. Conclusions

A complete operator manual was transferred over ISO 11783 in 31 to 89 s, while Markdown produced the smallest files and answer quality was comparable across formats for most models. Semantic retrieval supported cross-language use where keyword retrieval failed; the paper also proposes, but does not evaluate, first-coupling and image-transfer designs.

8. Conclusions

  • 31 to 89 s was sufficient to transfer a complete operator manual over ISO 11783, within the time of a typical first-coupling object pool upload.Markdown produced the smallest files, shortest transfer times, and fewest tokens, while answer quality was comparable across formats for most models.Source: §8 · p. 12
  • Semantic retrieval was essential across languages because keyword retrieval failed on translated editions, while hybrid retrieval added no advantage over semantic retrieval alone.Even with the gold page, some performance drop remained, partly because models answered in the language of the manual.Source: §8 · p. 12
  • The paper proposes, but does not evaluate, a version-checking first-coupling protocol and on-demand or background image transfer.A single compressed image transferred in 5.8 s.Source: §8 · p. 12

Glossary

The glossary expands abbreviations and identifies the paper's principal network and retrieval terms.

Glossary

A.1. Answerable Question Samples

The appendix provides examples of answerable and unanswerable manual questions, along with the prompts used to answer and judge them.

A.1. Answerable Question Samples

  • Answerable examples cover maintenance, controls, setup, and operating specifications, including 50 Nm for vane lock nuts and 35 mm grease level below the filler opening.Source: §A.1 · p. 14

A.2. Unanswerable Question Samples

  • Unanswerable examples ask about information absent from the manual, such as extra diesel consumption and recommended tire pressure, and mark the response “Not found in context.”Source: §A.2 · pp. 14–15

B.1. Question prompts

  • The question prompt asks for a precise answer grounded in the context, in English, and instructs the model to respond “Unknown” or “Not found in context” when unsure.Source: §B.1 · p. 15

B.2. Evaluation Prompt

  • The evaluation prompt judges whether model and expected answers convey the same technical meaning, allowing insignificant differences and requiring a yes-or-no response.Source: §B.2 · p. 15

C. Appendix: Detailed Test Results

The retrieval-only evaluation found strong semantic retrieval on translated manuals, while BM25 and hybrid fusion retrieved the gold page less often. The appendices also detail model-answer outcomes and physical-bus transfer measurements.

C. Appendix: Detailed Test Results

  • The detailed-results appendix reports F1 by model and format for English and pooled non-English editions, and breaks answer outcomes down by model.Of all answers, 884 (5.7%) were in the manual edition's language; the judge accepted 355, and rescoring them would raise pooled non-English F1 by 0.01 to 0.10 depending on model.Source: C. Appendix: Detailed Test Results · p. 15, Table 3 · p. 16, Table 4 · p. 16

D.1. Data Transfer Performance Data

  • Increasing CTS from 16 to 128 packets per burst reduced Dutch-manual transfer time from 66.6 s to 30.8 s; indented JSON took longest among the formats.The physical-bus tests used two desktop PCs and a bus carrying Virtual Terminal, Task Controller, and file servers.Source: §D.1 · p. 15, Table 5 · p. 17

E. Appendix: Retrieval Ablation

  • The retrieval appendix describes a retrieval-only evaluation and an end-to-end ablation; Figure 9 reports converted-manual token and character counts by format and language edition.The earlier study's best-performing hybrid configuration was retained for the main experiments.Source: E. Appendix: Retrieval Ablation · pp. 15–16, Figure 9 · p. 16

E.1. Retrieval-only evaluation

  • 80 to 93% of questions in the five non-English editions had their gold page retrieved by semantic search, versus 9 to 24% with BM25 and 74 to 85% with hybrid fusion.BM25-ranked chunks displaced semantic hits in the fused ranking; the three methods retrieved the English gold page for 80 to 94% of questions.Source: §E.1 · p. 17, Table 6 · p. 17

E.2. End-to-end Ablation

Hybrid retrieval substantially improved F1 over no context across English and other-language manuals. For other editions, semantic and hybrid retrieval outperformed BM25, while oracle context improved on hybrid but did not eliminate the language gap.

E.2. End-to-end Ablation

  • The comparison tested five context conditions across four models on Markdown editions of six languages, scoring F1 with question-level bootstrap intervals and Holm-corrected paired contrasts.Conditions were no context, BM25, semantic retrieval, hybrid retrieval, and oracle gold-page context.Source: §E.2 · p. 17
  • Hybrid retrieval raised F1 by 0.65 to 0.81 for the English edition and by 0.34 to 0.64 for the other editions.Without context, three models declined every question; Llama 3.2 3B answered 11% of answerable questions correctly.Source: §E.2 · p. 17, Table 7 · p. 18
  • For other editions, semantic and hybrid retrieval exceeded BM25 by 0.35 to 0.47, while English retrieval methods did not differ significantly.Hybrid and semantic retrieval also did not differ significantly for the other editions.Source: §E.2 · p. 17
  • Gold-page context raised F1 above hybrid retrieval by 0.06 to 0.11 for the other editions, significantly for Gemma 3 12B and Qwen 2.5 7B.The English–other-edition gap narrowed from 0.13–0.26 with hybrid retrieval to 0.11–0.23 with oracle context, leaving some gap when retrieval was not a factor.Source: §E.2 · pp. 17–18

F. Appendix: Validation of the LLM Judge

Four non-author raters substantially agreed, and the GPT-OSS 20B judge agreed with their majority on most sampled items. Disagreements were concentrated in judge rejections of answers that raters accepted, making the reported pooled F1 conservative.

F. Appendix: Validation of the LLM Judge

  • Four independent non-author raters assessed a stratified 210-item sample; answers in the manual’s language or mixed English-language answers were excluded from sampling.Raters had basic machinery knowledge but were not experts on this machine or manual; they saw the question, expected answer, and model answer without model, edition, format, or judge verdict labels.Source: F. Appendix: Validation of the LLM Judge · p. 18
  • The judge matched the raters’ majority on 188 of 201 items (93.5%), with weighted accuracy of 0.96.Rater agreement was Fleiss’ κ=0.83; judge-to-majority agreement was Cohen’s κ=0.87.Source: F. Appendix: Validation of the LLM Judge · p. 18, Table 8 · p. 18
  • Most disagreements were judge rejections of answers accepted by raters: 10 of 13 disagreements.Among answerable questions, the judge was wrong on 15.6% of negative verdicts and 4.3% of positive verdicts; for unanswerable questions, the corresponding rates were 2.1% and 0%.Source: F. Appendix: Validation of the LLM Judge · p. 18
  • Correcting judge errors would raise pooled F1 for English and language-neutral answers from 0.64 to about 0.67, so reported scores are conservative.For items judged with low reasoning effort, the judge agreed with the majority on 12 of 15 items.Source: F. Appendix: Validation of the LLM Judge · p. 18

Related papers

Loading 2610.31863v1…