PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Training Is Not Enough: Conditioned Cross-Granularity Training for Multimodal Document Understanding

Chengguang Gan, Yunhao Liang, Hanjun Wei, Qinghao Zhang, Shiwen Ni

arXiv:2609.00756v1cs.CL

TL;DR

The paper asks whether fine-grained field extraction and coarse document-level classification mutually reinforce each other, rather than assuming joint training provides that benefit. It constructs Doc-MRE and compares single-task, joint, and training-only conditioned prompts across three corpora. Mixed training reinforces on none, while conditioned training reinforces on two corpora and trades on one, with controls showing that gains can depend on prompt structure or cross-granularity content.

  • Problem

    The study addresses limited evidence about whether fine and coarse document-understanding tasks reinforce each other and which training arrangement produces that effect.

  • Method

    The paper builds Doc-MRE and compares single-task, joint, and gold cross-granularity conditioned training across two receipt corpora and one scanned-forms corpus.

  • Results

    Mixed joint training reinforces on no corpus, whereas conditioned training reinforces on two of three corpora and trades on the third.

  • Takeaways & Limitations

    Mutual reinforcement is not automatic under joint training and appears more consistently when cross-granularity conditioning is explicitly trained.

  • Takeaways & Limitations

    The evidence is limited to two receipt corpora, one scanned-forms corpus, one VLM family with LoRA, and small FUNSD splits with wide intervals.

Abstract

from arXiv · show

The Mutual Reinforcement Effect (MRE) asks whether a fine, span-level and a coarse, document-level task help each other when one model handles both. We test it in multimodal document understanding on three corpora, two of receipts and one of scanned business forms, comparing single-task, joint and conditioned training, which puts one granularity's gold output in the other's prompt during training only. We build Doc-MRE, an annotation layer pairing gold field extraction (point) with four document-level facets (line), from a three-judge LLM committee under a pre-registration, validated by blind re-annotation. One predicate, fixed in advance: at a shared recipe, a regime reinforces if it beats the matched single-task model on both granularities. Mixed joint training, the arrangement prior MRE work assumes, reinforces on no corpus at the main scale: it is below both single-task models on CORD and trades one granularity for the other on the two others, as single-task tuning does. Conditioned training reinforces on two of the three, CORD (+0.5 point, +4.8 line) and the forms corpus (+7.2 point, +11.0 line), resolvably on the coarse side and directionally on the fine one, and trades on WildReceipt; at that recipe no alternative measurably beats it on either side anywhere. Two byte-identical-prompt controls separate content from format: shuffled conditioning destroys the coarse-side skill but costs the fine side far less, and a neutral-content control reproduces the whole fine-side gain on WildReceipt, which is therefore prompt structure but buys nothing resolvable on the other two. On the forms corpus conditioning buys collapse avoidance: mixed training and the neutral control both assign the majority semantic label to all 50 test documents; only conditioning recovers the gold distribution. Probes find the information decodable under every regime with no resolvable increase under conditioning.

1 Introduction

This study tests whether fine-grained field extraction and coarse document-level classification mutually reinforce each other under controlled training arrangements. Mixed joint training fails the fixed reinforcement criterion, whereas conditioned training succeeds on two corpora, with controls separating content from prompt format.

  • Research question: The study fixes reinforcement as beating the matched single-task model on both point and line tasks at one shared recipe.It evaluates three corpora: two receipt datasets and one scanned business-forms dataset.
  • Main finding: Mixed joint training reinforces on no corpus, while single-task tuning trades one granularity against the other.Mixed training is below both single-task models on CORD and trades across the other two corpora.
  • Main finding: +0.5 point and +4.8 line on CORD, while +7.2 point and +11.0 line on FUNSD, are the conditioned-training gains over matched single-task models.The coarse-side gains are resolvable in both cases, whereas the fine-side gains are directional.
  • Controls: Shuffled conditioning damages coarse-side performance far more than fine-side performance, showing that incorrect cross-granularity content is especially harmful to the coarse task.On CORD, the shuffled condition costs 27.5 line points but 3.3 fine-task points relative to conditioning.
  • Controls: Neutral conditioning reproduces the entire fine-side gain on WildReceipt but provides no resolvable gain on CORD or FUNSD.Thus, the WildReceipt fine-side improvement is attributable to prompt structure within the reported controls, whereas content contributes on the other corpora.
  • Mechanistic analysis: The analysis finds cross-granularity information decodable under every regime, with no resolvable increase under conditioning.Input interventions and gradient attribution are consistent with a readout-level account but do not establish that mechanism.

2 Related Work

The paper places multimodal document understanding within work on mutual reinforcement, privileged information, negative transfer, and probing. It tests whether explicitly conditioning cross-granularity tasks changes the assumption that joint modeling automatically produces transfer.

  • MRE proposes bidirectional gains from jointly modeling fine- and coarse-grained information-extraction tasks, but transfer between granularities is not automatic.
  • The paper moves MRE to document understanding, where both granularities share a page and gold evidence links can be annotated.
  • Mixed training produces no reinforcement on CORD, while reinforcement is larger and more consistent when trained explicitly through conditioning.
  • Conditioned training instantiates learning using privileged information by exposing training-time signals unavailable at test time.
  • The study adds a document-understanding diagnosis of negative transfer and a remedy that changes only the training data.
  • Using fixed-backbone comparisons, the paper adds facet labels and evidence links to existing document-understanding benchmarks and applies linear probing.

3 The Doc-MRE Benchmark

Doc-MRE augments document field extraction with four document-level facets and directional evidence links. The benchmark combines existing gold fields with committee-generated annotations that receive strong agreement and blind human validation.

  • Doc-MRE retains source-dataset gold field annotations as point labels and adds four document-level facet labels as line supervision.The facets are store_type, payment_method, has_discount, and has_surcharge.
  • A symbolic oracle mapping gold fields to structural facets reaches .811 line accuracy on CORD and .690 on WildReceipt, below zero-shot and nearly all trained 8B regimes.
  • The line task predicts macro accuracy over the four facets, while the point task predicts a JSON field list scored by exact-match multiset F1.
  • A three-judge LLM committee creates facet labels and directional evidence links by majority vote, retaining only links named by at least two judges.Samples lacking a facet majority are dropped, while evidence links are used for analysis rather than training targets.
  • Committee agreement reaches 99–100% per facet, with Fleiss κ ranging from .846–.978 on CORD and .830–.948 on WildReceipt.Evidence-link pairwise Jaccard ranges from .49–.98.
  • Blind human re-annotation agrees with committee labels at 95.8% on CORD, 92.2% on the second receipt corpus, and 91.0% on the forms corpus.CORD validation covers 400 facet decisions, with disagreements concentrated in store_type boundary cases.

4 Training Regimes

The training comparison holds the backbone, supervision, and optimization settings fixed while varying how point and line tasks are arranged. Conditioning supplies gold cross-granularity context only during training, with controls separating content from prompt format.

  • All regimes fine-tune the same backbone with identical LoRA and optimization settings on the same 794 CORD training receipts.
  • JOINT-MIXED places point and line tasks in one mixed dataset, whereas JOINT-ONEPASS requests both outputs in a single pass.
  • CONDITIONED places the other granularity’s gold output in each task’s prompt during training only; evaluation uses identical plain prompts without gold context.
  • Shuffled and neutral controls preserve the conditioning format while supplying wrong cross-document content or uninformative placeholders.The neutral control is byte-identical to conditioning and bounds format-level contributions.
  • The PIPELINE row conditions each task on the other task’s predicted stage-1 output at both training and test time, unlike the other regimes.
  • Doubling single-task epochs changes point F1 by −0.1 and line accuracy by −0.5 points, ruling out the supervision-budget difference as the explanation.

5 Main Results

Conditioned training is the only tested arrangement that consistently avoids cross-granularity damage and can reinforce both tasks, although its gains and mechanism vary by corpus.

  • Training regimes: −5.3 line and −7.6 point: single-task tuning damages the untrained granularity, while mixed training falls below both single-task models on CORD.Mixed training is below POINT-ONLY on point (.778 vs .804) and LINE-ONLY on line (.894 vs .899).
  • Training regimes: +4.8 line and +0.5 point: CONDITIONED beats LINE-ONLY on line while matching POINT-ONLY on point on CORD.Three full retraining seeds reproduce this pattern.
  • Training regimes: On each dataset, one conditioning direction supplies the gain: fields-to-facets on CORD and facets-to-fields on WildReceipt.COND.-FIELDS reaches line .944 on CORD, while COND.-HYPO reaches point .596 on WildReceipt; each is indistinguishable from full conditioning on the gaining side.
  • Controls: 27.5 line points: shuffled conditioning collapses coarse-side performance on CORD, while fine-side performance drops only 3.3 points.The shuffled prompt preserves format but uses another receipt’s content, showing that correct context matters more for the coarse task.
  • Controls: On CORD, CONDITIONED exceeds neutral conditioning by +4.3 line and +2.2 point, whereas on WildReceipt neutral conditioning reproduces the fine-side gain.Thus content contributes on CORD, but prompt structure accounts for the WildReceipt fine-side improvement.
  • Third domain: On FUNSD, mixed training and content-free controls assign the majority doc_type label to all 50 documents, while CONDITIONED recovers the gold distribution.At the shared recipe, only conditioning and its fields-to-facets variant reach zero-shot on the coarse side; rate changes alter mixed-training performance but not conditioning.

6 What the Analysis Battery Shows

The analysis battery finds cross-granularity information decodable under every regime, but conditioning adds no resolvable information and does not establish the favored readout account.

  • Encoding: Probes decode the other task’s labels under every regime, including zero-shot, while conditioning adds no resolvable information over zero-shot.The result makes an injection account unnecessary rather than impossible.
  • Use: Gold or corrupted hypotheses make several fine-tuned regimes sensitive, but neutral conditioning responds more strongly despite uninformative slots.This instrument therefore measures format exposure rather than learned use of cross-granularity content.
  • Attribution and modality: Conditioning concentrates attribution on annotated evidence and exceeds neutral conditioning on two facets, although it is not separable from mixed training.On store_type, attribution rises from 1.88 to 2.47, with a paired median difference of +0.59 over zero-shot.
  • Interpretation: The favored readout interpretation remains unestablished because patching is inconclusive and prompt-format confounds affect three analysis instruments.The battery is descriptive and bounds injection below its resolution rather than ruling it out.

7 Conclusion

Mutual reinforcement does not follow automatically from joint training: mixed co-training fails to clear both single-task models across the three corpora, whereas conditioning succeeds on two and trades on one.

  • Mixed co-training clears both matched single-task models on none of three corpora at the main scale, while conditioned training does so on two and trades on the third.This is the paper’s fixed-predicate comparison of reinforcement across granularities.

Limitations

The evidence is bounded to three datasets, one VLM family and two scales using LoRA, with further genres and full fine-tuning left open.

  • The study covers two receipt corpora and one scanned business-forms corpus, using one VLM family at two scales and LoRA fine-tuning.Full fine-tuning and further genres remain untested.
  • FUNSD has only 149 training and 50 test documents, yielding correspondingly wide intervals and limited seed coverage outside the main arms.Three seeds were run only for mixed, conditioned and neutral control.
  • The mechanism claim is deliberately scoped because three of four analysis instruments share a prompt-format confound with the measured effect.The battery is reported as descriptive, leaving a representation-level account for future work.
  • Line labels come from an LLM committee, with blind human re-annotation bounding noise at roughly 4% on CORD, 8% on WildReceipt and 9% on FUNSD.Identical labels across regimes mean this noise shifts absolute levels rather than reported contrasts.
  • Conditioned training presumes the other granularity’s gold labels exist during training, so producing them adds annotation cost.

Ethics Statement

The paper uses public research datasets under their research terms and redistributes no images, while Doc-MRE remains keyed to original file identifiers.

  • Human validation used three project-affiliated annotators working independently from a written guide, with no crowd workers employed.The receipt schema was reproduced in Appendix K.
  • Doc-MRE is redistributed as an annotation layer keyed to original file identifiers rather than as redistributed images.

A Training and Evaluation Details

The study uses a fixed Qwen3-VL-8B recipe, with additional controls and replications testing whether reported training effects depend on optimization, parsing, or evaluation choices.

  • Training recipe: Qwen3-VL-8B training uses LoRA rank 64, α 128, with 174.6M of 8.94B parameters trainable.Runs use bf16, learning rate 10^-4, onecycle scheduling, batch size 1 with accumulation 8, two epochs, and answer-token loss.
  • Training recipe: The main study uses 794 examples for POINT-ONLY, LINE-ONLY, JOINT-MIXED, and CONDITIONED regimes.Shuffled and neutral-content controls were added after the pre-registered run as post-hoc robustness checks.
  • Recipe controls: At 2×10^-5, mixed training remains below matched single-task models, while CONDITIONED reaches point .761 and line .932 but trades at that rate.At 10^-4, CONDITIONED leads the best-tuned mixed arm on line by +5.6 [3.0, 8.1].
  • Replication: On WildReceipt’s 467-image test split, CONDITIONED reaches point F1 .592, versus .536 for JOINT-MIXED and .435 for POINT-ONLY.The corresponding contrasts are CONDITIONED−POINT-ONLY +15.6 [12.8, 18.4] and JOINT-MIXED−POINT-ONLY +10.1 [7.7, 12.5].

B Additional Result Detail

Additional analyses test robustness, alternative transfer arrangements, oracle explanations, parsing effects, and facet-level behavior beyond the headline comparisons.

  • Robustness: +4.8 [2.3, 7.3] is CONDITIONED−LINE-ONLY line accuracy on CORD under paired bootstrap evaluation.The corrected estimate becomes +6.0 [2.5, 9.4] when training seeds are also resampled.
  • Robustness: The largest contrasts remain far from interval boundaries, while smaller differences such as +3.0, +3.5, and +6.4 are treated as supporting evidence.This calibration distinguishes load-bearing damage and CORD line gains from closer secondary comparisons.
  • Alternative arrangements: CONDITIONED−PIPELINE is −1.5 [−3.5, +0.3] on CORD line and −1.5 [−4.5, +1.5] on WildReceipt line, while its WildReceipt point contrast is +19.2 [13.6, 25.0].PIPELINE was added post hoc and leads CONDITIONED on line in both corpora within the reported intervals.
  • Alternative arrangements: Sequential transfer falls behind CONDITIONED by 5.3 and 5.1 line points for the two transfer orders.The two orders are sequential line-to-point and point-to-line continuation of an existing single-task adapter.
  • Facet decomposition: The symbolic oracle reaches line accuracy .811 on CORD and .690 on WildReceipt, below zero-shot in both cases.The oracle derives structural facets from gold point fields but cannot exceed the store_type majority class.
  • Parsing effects: On CORD, LINE-ONLY’s line damage changes from −7.6 [−11.0, −4.4] to −5.5 [−8.3, −2.9] after restricting to jointly parsed receipts.All CORD trained arms parse 99 of 99 on the relevant comparison, so gains have no format component.
  • Probe controls: For has_discount, every regime remains decodable above chance, while CONDITIONED is not above ZERO-SHOT.The facet’s skewed majority makes the permuted control uninformative, so selectivity is reported instead.

D Attribution and Modality

Attribution and modality analyses show that conditioning changes input use and evidence attribution, but probes do not show newly decodable cross-granularity information.

  • Probing: Layer-wise probes find line information decodable in point-task states under every regime, with no resolvable conditioning increase.Figure 4 compares zero-shot and CONDITIONED probe accuracy against majority baselines of .42 for store_type and .64 for payment_method.
  • Attribution: Evidence attribution exceeds the count-matched null for every regime, with CONDITIONED highest on the two facets surviving the lexical-overlap null.The attribution lift is defined as annotated-evidence density relative to all fields, with lift = 1 as the null.
  • Attribution: COND.-NEUTRAL reproduces 86% of CONDITIONED’s store_type gain over zero-shot but only 29% of its has_surcharge gain.CONDITIONED still exceeds the neutral control by +0.139 on store_type and +0.222 on has_surcharge.
  • Attribution: Attribution sharpness does not separate CONDITIONED from JOINT-MIXED, which is level on store_type and higher on has_discount.Because evidence links come from LLM judges, the paper treats alignment as convergent validity rather than validation of either side.
  • Modality: CONDITIONED is the only regime where field text alone exceeds image-only line accuracy, while fusion is superadditive for every model.For CONDITIONED, field text scores .957 versus .947 for image-only, a +1.0 paired difference whose interval spans zero.

E Third Domain: FUNSD

FUNSD extends the comparison to scanned business documents, where conditioning avoids a semantic-label collapse and its coarse-side advantage survives seed and recipe analyses.

  • Dataset and tasks: FUNSD contains 199 scanned 1990s business documents, using the official 149/50 train-test split and four corpus-specific document-level facets.The fine task reuses gold entity annotations, while the coarse task includes doc_type, has_checkbox, has_cc_field, and has_unfilled_field.
  • Main result: Conditioning recovers the gold distribution on FUNSD, whereas mixed training and COND.-NEUTRAL assign the majority semantic label to all 50 test documents.The declared headroom-based direction prediction is therefore refuted: the gain appears on the coarse side rather than the fine side.
  • Robustness: Seed replications separate the central arms on FUNSD’s coarse side, with pooled CONDITIONED−JOINT-MIXED line +15.3 [10.7, 20.3].The corresponding doc_type contrast is +46.0 [32.0, 60.0], while fine-side separation is inconsistent across seeds.
  • Recipe sweep: Oversampling the coarse task leaves doc_type at the majority label for all 50 documents, so task proportion does not explain the collapse.Repeating the line rows two or four times raises macro coarse accuracy from .770 to .805 but leaves doc_type at +0.0 against the balanced arm.
  • Main result: At the shared recipe, CONDITIONED avoids the collapse and mixed training leads it on the fine side only at the lower rate by +9.1 [2.2, 16.3].This is the study’s only arm that resolvably beats CONDITIONED on either side of any corpus.
  • Validation: The FUNSD annotation committee achieves 91.0% agreement with blind human re-annotation, with facet κ values from .57 to .96.The shuffled control is weaker on this corpus because it coincides with the document’s own labels for 8.7% of training examples.
Loading 2609.00756v1…