PaperlayerView exact revision on arXiv

Source-linked AI summary

INVAR-Agri: Detecting Discrepancies in Multilingual Operator Manuals of Agricultural Implements with a Fine-Tuned LLM

Timo Oksanen

arXiv:2610.35423v1cs.CL

TL;DR

Reviewing every language version of an agricultural manual requires competent reviewers, making checking a bottleneck. This proof of concept compares back-translation and direct checking with a fine-tuned LLM; the back-translator concealed planted errors while the checker detected most. The findings indicate possible support for review, but remain bounded by the study’s data and setting.

  • Problem

    Review of every translated manual by competent persons is a bottleneck, and the paper tests whether automatic checks can keep discrepancies visible or detect them.

    Source: §1.1 · p. 1

  • Method

    The study compares two workflows on agricultural manuals: a back-translator learns from translation pairs, while a checker learns from correct pairs and synthetic errors.

    Source: §4.3 · p. 6

  • Results

    The trained back-translator concealed planted errors, whereas the trained checker detected most, though it remained weak for some unit changes.

    Source: §6.2 · p. 12, §7 · p. 14

  • Takeaways & Limitations

    Within this proof-of-concept setting, a trained checker may support competent-person review, but the study tests a precondition for support rather than the support itself.

    Source: §6.2 · p. 13, §7 · p. 14

    The tests use planted errors, one manufacturer, one model, English as the sole reference, and manuals for machine types represented in training.

    Source: §6.6 · p. 13

Abstract

from arXiv · show

Operator manuals of agricultural implements are delivered in many languages, and the standards require that competent persons review every translation, which makes this review the bottleneck. This paper presents a proof of concept that compares two training workflows on the same LLM, EuroLLM-1.7B-Instruct, trained on operator manuals of one manufacturer. In approach A, a back-translator translates each language version back into English, and an English reader compares it with the English version. In approach B, a checker reads the English version and the translation together, where the checker reports differences. The test sets contain errors planted in numbers, units, opposite words, obligations, negations, and conditions in 24 languages, and they include unmodified text and meaning-preserving edits for alerts and false alarms. The trained back-translator concealed many planted errors, most of all weakened obligations, while translation-quality scores rose. In contrast, the separately trained checker detected most planted errors, although it was weak for units and for number changes unlike those in training. A check by the author confirmed the concealment. In addition, a simple rule check caught the number and unit changes, but it alerted on harmless rewrites. A check of alerts on real Finnish translations showed that most alerts were false, caused by language-specific wording, so the precision on real text remains a limitation. Within this setting, a trained checker might support, not replace, the review by competent persons, and firmer conclusions need more data.

1. INTRODUCTION

Multilingual agricultural manuals require competent review in every language, making translation checking a bottleneck. The paper compares back-translation with direct discrepancy checking as a proof of concept for possible LLM support.

1.1 Translated operator manuals of agricultural implements

  • Translation can change safety-relevant instructions, while checking every language requires reviewers who understand both the language and machine; checking, not translation, is the bottleneck.The paper notes that errors in numbers, units, or words can create risks of injury or machine damage.Source: §1.1 · p. 1

1.2 Two ways to let one expert check all languages

  • Approach A translates each language version back into English for comparison, whereas approach B reads the English and translated versions together and reports differences.The paper tests whether training changes whether planted discrepancies remain visible or are detected, as well as alerts on unmodified text and harmless edits.Source: §1.2 · pp. 1–2

1.3 Contribution

  • The trained back-translator concealed planted errors, while the trained checker detected most; this proof of concept makes no claims beyond agricultural machinery manuals.The paper identifies checker support as a possible precondition for review, not the review itself.Source: §1.3 · p. 2

2. STATE OF THE ART

Prior work examines round-trip translation, critical-error detection, multilingual fine-tuning, and machinery documentation standards. The paper situates its study where automatic checking of translations remains unaddressed by the cited documentation standards.

2.1 Back-translation and round-trip checks

  • Back-translation can make an incorrect forward translation appear correct, and its sentence-level semantic scores have been weak despite stronger system-level correlations with human ratings.A study with 52 users found higher confidence but not better translation quality; physician testing found back-translation caught a critical error more often than quality estimation.Source: §2.1 · p. 2

2.2 Quality estimation and critical error detection

  • Critical-error detection covers meaning deviations with safety or other consequences; the best WMT 2021 system reached a Matthews correlation of 0.55 for English to German.Synthetic-error training and rule-based number and unit detectors provide related approaches.Source: §2.2 · p. 2

2.3 Fine-tuning LLMs on multi-parallel domain data

  • Prior work links multi-parallel training with English-side overfitting and target memorization [24]; EuroLLM studies covered at most eight language pairs and did not test manual discrepancy hiding.The cited EuroLLM studies fine-tuned English–Spanish, English–Japanese, and eight English-source pairs.Source: §2.3 · p. 3

2.4 Technical documentation of machinery

  • Technical-documentation standards assign translation review to competent persons, but the cited documents do not address machine translation or automatic translation checks.ISO 20607 also asks for translatable text and instructions without weakening or negative words.Source: §2.4 · p. 3

3. MATERIALS

The materials combine public multilingual operator manuals, model and helper-model workflows, and test sets with planted translation errors. The study aligns manual text across languages and evaluates both corpus preparation and error-checking materials.

3.1 Corpus of operator manuals

  • The corpus contains 12 manuals from one manufacturer across 10 implement types, with aligned English and non-English units; page matching covered at least 98.3% of English content pages.A median of 96.1% of English blocks aligned, and author review removed 3,735 misaligned pairs; the resulting corpus has 19,272 English blocks and 354,390 aligned pairs.Source: §3.1 · p. 3
  • Training used 198,030 pairs from 10 manuals, while tests used two manuals whose text units overlapped training text by 69% and 42%, respectively.Tests cover sentences and paragraphs, not headings, tables, captions, or figure text; the source language and translation path are unknown.Source: §3.1 · p. 3

3.2 Model and helper models

  • Both approaches fine-tuned EuroLLM-1.7B-Instruct; helper models planted or validated errors, and full-weight tuning fit the available 128 GB unified memory.A different error planter was used for checker training to reduce the chance that it learned one planter’s habits.Source: §3.2 · pp. 3–4
  • 0.983 accuracy on 60 author-reviewed Finnish pairs for Clef, versus 0.917 for Gemma 4 and 0.850 for gpt-oss-120b, supported alignment judging.Clef processed about 63,000 pairs at about 0.6 seconds per pair without failures.Source: §3.2 · p. 4

3.3 Test sets with planted errors

  • INVAR-Agri contains 5,332 planted errors in 24 languages across numbers, units, polarity, obligations, negations, and conditions; pooled detection rates do not weight error types by risk.The test items retain the English unit unchanged and plant exactly one error in the translation.Source: §3.3 · p. 4
  • Meaning-preserving edits and unmodified translations measure false alarms and alerts, while planting checks found 185 of 192 sampled errors in 16 added languages valid.The comparison uses 2,016 unmodified units per system and 432 meaning-preserving edits.Source: §3.3 · p. 4

4. METHODS

The study compares back-translation review with direct checking using the same LLM, then fine-tunes both workflows and evaluates their performance and checks selected items manually.

4.1 Approach A: back-translation review

  • Approach A back-translates each language version into English for an English reviewer, with element rules flagging planted values that appear in the back-translation.A planted error is counted as detected only when carried through; hidden or garbled errors do not count.Source: §4.1 · pp. 4–5

4.2 Approach B: direct checking

  • Approach B reads the English original and translation together and returns “Same meaning.” or “Check:” with a description of the difference.Its training uses correct pairs and synthetic errors, with test errors generated differently to assess generalization to unseen changes.Source: §4.2 · pp. 5–6
  • A rule baseline flags differing numbers or number-unit pairs, but its limited normalization raises alerts on some meaning-preserving edits.It does not convert units, and the evaluation excludes valid unit conversions.Source: §4.2 · p. 6

4.3 Fine-tuning

  • Both workflows fully fine-tune EuroLLM-1.7B-Instruct across three seeds, but use different training data, supervision, filtering, and development measures.Approach A uses 198,030 pairs to reproduce English; approach B uses 17,295 examples to predict a verdict and explanation. Pass selection uses chrF for A and balanced accuracy for B.Source: §4.3 · p. 6

4.4 Measures

  • Detection is the share of planted errors flagged; alerts on unmodified units and flags on meaning-preserving edits are measured separately.The study reports seed variation and bootstrap uncertainty, and preregistered a hypothesis about whether fine-tuning hides more errors in text close to training data.Source: §4.4 · p. 7

4.5 Checks by the author and by LLM

  • The author checked 117 back-translations blind to system and automatic verdict, and separately checked Finnish plantings and unmodified units.Three of 120 questions were excluded as nonsensical; seven items could not be judged.Source: §4.5 · pp. 7–8

5. RESULTS

Fine-tuning sharply improved the direct checker’s detection of planted errors but made back-translation conceal more errors, especially weakened obligations. The results also show limits in transfer to unseen number and unit changes and in the precision of alerts on real Finnish text.

5.1 Fine-tuning and testing in the standard way

  • Fine-tuning raised back-translation COMET from 80.0 to 84.9 and chrF from 50.9 to 70.7 on training-close text, but new-text COMET rose only from 76.9 to 77.6.For new text, Clef judged 67.8 % of fine-tuned back-translations to say the same as the English, versus 80.5 % for untuned; word-for-word copies rose from 1.3 % to 21.1 %.Source: §5.1 · p. 8, Table 4 · p. 8
  • On unmodified test text, the fine-tuned checker reduced alerts from 18.5 % to 3.4 % and brought answers in the required form from 47.0 % to every answer.Approach A alerts on unmodified text fell from 24 % to 17 %, while approach B alerts fell from 19 % to 3 %.Source: §5.1 · p. 8, §5.2 · p. 8

5.2 Planted errors: untuned versus fine-tuned

  • Without conditions, detection rose from 22 % to 90 % for approach B but fell from 87 % to 67 % for approach A; with conditions, approach B rose from 19 % to 91 %.For approach A, weakened-obligation detection fell from 94 % to 30 %; approach B’s four word-level types reached 93 % to 98 %.Source: Table 5 · p. 8, §5.2 · p. 8, Figure 3 · p. 9

5.3 Approach A: what fine-tuning hides

  • Fine-tuning concealed weakened obligations in approach A: carried results fell from 94 % to 30 %, while hidden results rose from 3 % to 62 %.The author saw the planted error in 9 of 9 untuned back-translations and 0 of 9 fine-tuned ones; the author did not interpret approach A’s condition result.Source: §5.3 · p. 9

5.4 Approach B: what the checker learns

  • The untuned checker detected 19 % of planted errors without examples, but flagged 99 % of items with three examples; the fine-tuned checker detected 93 % to 98 % of word-level errors.Transfer was uneven: ten-times-smaller numbers were detected in 4 % to 55 % of cases, and “Nm” changed to “N” was never detected.Source: §5.4 · pp. 9–10
  • The rule check detected every planted number and unit error, with alerts on 0.8 % of unmodified units but false alarms on 38 % of meaning-preserving edits.The checker alone detected 90 % of errors without conditions with 3 % alerts and 13 % false alarms; combining methods raised detection to 97 % but false alarms to 39 %.Source: §5.4 · p. 10, Table 6 · p. 10

5.5 Copied and new text

  • The fine-tuned checker detected 88 % to 93 % of errors without conditions across similarity groups, while approach A lost detection in every group.For weakened obligations, approach A fell from 94 % in exact copies to 26 %, and from 94 % to 42 % in new text.Source: §5.5 · p. 10, Table 7 · p. 10

5.6 Languages

  • The opposite directions appeared in all 24 languages: approach A lost 10 to 85 points on weakened obligations, while approach B gained in almost every language and error type.The fine-tuned checker’s word-level detection reached 81 % to 100 % in every language apart from Bulgarian obligation; units were its weakest type, at 47 % to 92 %.Source: §5.6 · pp. 10–11, Figure 6 · p. 11

6. DISCUSSION

The discussion attributes the workflows’ opposite directions partly to their different training targets and explains how their use could support, but not replace, competent review. It also bounds the findings by the English-reference setting, corpus preparation, and test design.

6.1 Why the two training workflows act in opposite directions

  • Approach A’s fine-tuned back-translator restored expected English wording, including “must” where translations said “should,” and hiding also occurred on new text.The training pairs often lacked an obligation in the other-language version, while the English target contained “must”; hiding increased with training intensity, though the study did not isolate the proposed roles of memorized sentences and learned conventions.Source: §6.1 · p. 11
  • Approach B targets reporting differences, while approach A targets restoring English; their training data and selection also differ, so the design does not isolate one cause of their opposite directions.The checker generalized to word-level errors from another planter but transferred numbers and units only partly.Source: §6.1 · p. 11

6.2 What this means for checking manuals

  • A checker-plus-translator workflow remains proposed, not tested; unflagged units still need review, and false alerts can consume review time.The rule check detected planted number and unit changes but also alerted on harmless rewrites, while the checker can miss weak error types.Source: §6.2 · p. 12

6.3 The English version as reference

  • English is a practical reference for one English-reading expert, but it may not be the source language, and translations may follow varied paths.The study assumes there is no single source language across its manuals; a difference from English may reflect the translation path rather than an error.Source: §6.3 · p. 13

6.4 Languages in tuning

  • The study fine-tuned one model across 24 language versions, where the opposite workflow directions appeared in every language.Earlier cited EuroLLM fine-tuning studies covered at most eight language pairs; none compared 24 European languages.Source: §6.4 · p. 13

6.5 Building the corpus

  • Corpus construction required a separate page map for each language, and alignment validation still left errors.Some translations inserted or omitted pages, one manual reflowed its text and was excluded, and an alignment error appeared in the Finnish check.Source: §6.5 · p. 13

6.6 Limitations

  • Negation plantings must be plausible in each language; the test sets also limit conclusions to planted errors, one manufacturer and model, English-reference checks, and overlapping machine types.The checks did not measure reviewer reliance, author validation covered only Finnish and English, and headings, lists, tables, captions, and figure text were not tested.Source: §6.6 · pp. 13–14

7. CONCLUSION

In this proof of concept, the trained checker detected most planted discrepancies, while the back-translator concealed them. False alerts on real Finnish translations leave practical usefulness unestablished; the checker might support review, but firmer conclusions require more data and a decision about model size.

7. CONCLUSION

  • The trained checker detected most planted errors but remained weak for some unit changes.The back-translator trained on the manufacturer’s manuals concealed planted errors, especially in text close to the training text.Source: §7 · p. 14
  • Most alerts on real Finnish translations were false, leaving practical usefulness for naturally written translations unestablished.The paper notes that firmer conclusions need a far more comprehensive corpus with much more data and an answer about proper model size.Source: §7 · p. 14
  • The back-translator concealed planted errors, especially in text close to its training text, while the checker was trained with synthetic errors.The conclusion says the back-translator finding was in line with earlier observations that back-translation can repair errors on the way back.Source: §7 · p. 14
  • Within this proof of concept, a fine-tuned checker might support competent persons’ review, but does not establish that support in practice.Firmer conclusions require more data and an answer to the proper model size.Source: §7 · p. 14

Related papers

Loading 2610.35423v1…