PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-Grounded Batik Captioning under Scarce Text Annotation: Label Retention and Source-Specific Error Propagation

Novanto Yudistira, Muhammad Dzaka Aufa Fadhillah, Irawati Nurmala Sari

arXiv:2610.30661v1cs.CV

TL;DR

Batik archives often provide category labels but little descriptive text, while captions must connect visual appearance with collection-specific terminology. This study evaluates a DINOv2–Qwen2.5-VL cascade with predicted-class conditioning; all injected conditions achieve the same class-label mention rate, while caption metrics rank models differently.

  • Problem

    Batik archives may provide category labels but sparse descriptions of visible colors, ornaments, and arrangement, and class conditioning can connect prediction errors to erroneous identities in captions.

    Source: §1 · pp. 1–2

  • Method

    The study crosses three captioner conditions with two inference prompts in a DINOv2–Qwen2.5-VL cascade, examining predicted-class conditioning and caption outcomes.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    All three injected variants achieve MMR of 0.9067, while PFT-NCI leads reference-based measures but has the lowest image–text compatibility score.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Class-label retention and factual grounding of visual descriptions are distinct evaluation targets, and the classifier–captioner interface is a key point for assessing cultural attribution.

    Source: §7 · p. 12, §8 · p. 13

    Independently constructed classifier and captioner partitions leave possible classifier-training overlap with caption-evaluation images unquantified, limiting conclusions about complete-cascade generalization.

    Source: §7 · p. 12

Abstract

from arXiv · show

Class labels offer compact supervision for describing cultural-heritage images when expert text annotations are scarce. We investigate a batik-captioning cascade that integrates a LoRA-adapted DINOv2 classifier with Qwen2.5-VL-7B, adapted using LoRA and loaded in 4-bit NF4. A $3\times2$ evaluation combines three captioner conditions with two inference prompts to examine adaptation and predicted-class injection. Across 150 caption-evaluation images, all three injected conditions achieve a ground-truth class-label mention rate of 0.9067. The classifier's separate test yields 96.10% pooled accuracy, with 98.33% on 300 public-source images and 75.76% on 33 local images. Supported-class macro-F1 and balanced accuracy on the local subset are 0.8493 and 0.8302, respectively. The classifier and captioner use separate partitions, and their training--evaluation overlap remains unquantified. Differences in label semantics and training support also affect interpretation of the source-specific accuracy gap. Reference-based caption metrics and image--text compatibility scores produce different model rankings. Qualitative examples show both retention of correctly predicted identities and propagation of classification errors into fluent descriptions. These findings characterize how class conditioning preserves label information and transfers upstream errors into batik captions under scarce text annotation.

1 Introduction

Batik archives often pair category labels with sparse descriptive text, so useful captions must connect visual appearance with collection-specific terminology. The paper studies a two-stage classifier–captioner cascade, focusing on label retention and the propagation of classification errors.

  • The study evaluates how a visual classifier’s cultural-class prediction conditions a captioner, linking correct predictions to identity inclusion and incorrect predictions to fluent but erroneous attribution.The evaluation distinguishes label retention, reference-text agreement, visual compatibility, and factual correctness; recurring reference templates can reward shared wording across different images.Source: §1 · pp. 2–3
  • A six-condition evaluation crosses three captioner conditions with two inference prompts to examine adaptation and predicted-class injection under scarce text annotation.The study also analyzes source- and class-specific classifier performance, label retention, caption-metric disagreement, and downstream errors.Source: §1 · p. 3
  • The study focuses on class-conditioned caption generation with independently constructed classifier and captioner partitions, while implications for complete-cascade generalization are discussed separately.The extent of classifier-training overlap with caption-evaluation images is unquantified.Source: §1 · p. 3

2 Related Work

Prior work spans batik recognition and synthesis, culturally informed and classifier-guided captioning, and image–text evaluation. This study examines how cultural-class recognition connects to generated batik descriptions.

  • Batik research has addressed recognition and motif synthesis, while Indonesian image captioning has used ViT–IndoBERT on general image–text data.The study instead examines the connection between cultural-class recognition and generated batik descriptions.Source: §2 · p. 3
  • Culturally aware captioning uses external context, and classifier-guided captioning passes compact domain information to a language-generation stage.The study uses this interface to analyze retention of predicted identities and propagation of classification errors.Source: §2 · p. 3
  • DINOv2 and Qwen2.5-VL provide the visual and multimodal model components, while LoRA and quantized adaptation support separate adaptation of the cascade stages.The related work also notes transfer learning for batik recognition and diffusion- and GAN-based motif synthesis.Source: §2 · p. 3
  • Reference-based metrics measure agreement with target captions, CLIPScore measures image–text compatibility, and hallucination research motivates checking visual assertions directly.These perspectives distinguish cultural identity from visible attributes in caption content.Source: §2 · p. 3

3 Data and Task

The study combines batik collections with different class inventories and metadata-based caption targets, then compares captions generated with and without predicted-class conditioning.

3.1 Image collections and cultural labels

  • Dataset A contains 346 locally curated East-Java images across four regional classes, with metadata describing class, colors, theme, and pattern shape.Reference captions use a recurring template built from selected metadata fields, excluding historical interpretation, symbolism, fabric type, dyes, and production techniques.Source: §3.1 · p. 4
  • Dataset B has 3,000 images across 20 classes, but its repository identifier, creator, release, and license are unspecified, limiting reconstruction of the collection.Source: §3.1 · p. 4
  • The combined inventory has 24 classes: Dataset A uses regional labels, while Dataset B primarily uses motif names, so source comparisons also reflect differences in label semantics and sample support.Source: §3.1 · p. 4

3.2 Class-conditioned caption generation

  • For each image, the classifier predicts a class that can be supplied to the captioner; paired generic and class-conditioned prompts use the same image.This comparison characterizes the effect of predicted-class conditioning on generated text.Source: §3.2 · p. 5, Figure 2 · p. 5

4 Methodology

The captioner is adapted using pseudo-captions generated from labeled images and manually reviewed. The methodology separately adapts the classifier and captioner, then evaluates three captioner conditions with two inference prompts.

4.1 Pseudo-caption generation

  • For Dataset B, Qwen2.5-VL-7B-Instruct generates pseudo-captions from images and class labels, with prompts requesting visible colors, elements, arrangement, and isen-isen while excluding unsupported historical or symbolic interpretations.The prompt requires the class label in the opening sentence; manual review removes or regenerates unsuitable captions, yielding 3,000 pairs split into 2,700 training and 300 validation pairs.Source: §4.1 · p. 5

4.2 Classifier and captioner adaptation

  • The DINOv2 backbone is frozen apart from LoRA modules and a trained 24-way classification head, and training uses a weighted focal-style loss with γ = 2.The stated training setup includes AdamW, a 3 × 10^-4 learning rate, 10 epochs, cosine scheduling, 10% warmup, batch size 16, and accumulation over two batches.Source: §4.2 · pp. 5–6
  • Qwen2.5-VL-7B-Instruct is loaded with NF4 and double quantization and adapted with LoRA; instruction and image tokens are masked from the answer-token cross-entropy objective.Adaptation uses 10 epochs, while dynamic-resolution processing has a 262,144-pixel ceiling.Source: §4.2 · p. 6
  • The six-condition evaluation crosses Baseline, PFT, and CIFT captioners with no-class-injection and class-injection inference prompts.PFT training prompts omit class labels, while CIFT training prompts include them.Source: §4.2 · p. 6

5 Experimental Setup and Evaluation Protocol

The protocol evaluates caption generation under independently constructed classifier and captioner partitions, using classification, reference-agreement, label-mention, and image–text compatibility measures. Runtime variability is described across images, while paired uncertainty and training-run variability remain open evaluation requirements.

5.1 Data partitions and evaluation scope

  • The 150-image caption evaluation uses independently constructed classifier and captioner partitions, leaving possible classifier training exposure to evaluation images unquantified.Interpreting it as complete-cascade generalization would require a shared partition that excludes evaluation images from both components, with exact- and near-duplicate grouping before assignment.Source: §5.1 · p. 6

5.2 Evaluation metrics

  • Supported-class macro-F1 averages per-class F1 over labels with positive ground-truth support, while balanced accuracy averages recall over those supported labels.Predictions into labels outside the supported set remain errors; class supports and the averaging set accompany aggregate metrics.Source: §5.2 · p. 6
  • BLEU-1/4, ROUGE-L, METEOR, SPICE, and BERTScore measure reference agreement, while MMR counts captions containing the ground-truth class string.Because reference captions reuse a template, reference metrics are sensitive to shared syntax and repeated nouns; MMR reflects both upstream label correctness and retention, not factual accuracy of remaining content.Source: §5.2 · pp. 6–7
  • CLIPScore serves as an image–text compatibility diagnostic, but rare local class names may be weakly represented in its embedding space.Its encoder checkpoint and score scaling are unspecified, so interpretation considers disagreement alongside direct caption inspection.Source: §5.2 · p. 7

5.3 Computing environment and variability

  • Runtime means and standard deviations are measured across 30 images, but paired uncertainty and training-run variability remain open evaluation requirements.The standard deviations describe image-level runtime variability; complete per-image outputs and repeated-training summaries are unavailable.Source: §5.3 · p. 7

6 Results

The results characterize classifier performance, captioner behavior, qualitative error propagation, training dynamics, and resource use across the batik-captioning cascade.

6.1 Classifier performance across sources

  • 96.10% pooled classifier accuracy masks source-specific results of 98.33% on Dataset B’s 300 images and 75.76% on Dataset A’s 33 images.On Dataset A, supported-class macro-F1 is 0.8493 and balanced accuracy is 0.8302; including four zero-support labels yields macro-F1 of 0.4246.Source: §6.1 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8

6.2 Captioner adaptation and class-label retention

  • All three CI conditions achieve MMR of 0.9067, while PFT-NCI leads all six reference-based caption measures despite the lowest CLIPScore.The disagreement shows that model rankings depend on the evaluation criterion.Source: §6.2 · pp. 8–9, Table 4 · p. 8, Table 5 · p. 9

6.3 Qualitative analysis of caption content and error propagation

  • Tasik Madu’s correct Malang identity appears with bee and honeycomb descriptions under CIFT-CI, while two incorrect Trenggalek predictions enter other CI captions.The selected cases illustrate identity retention and upstream-error propagation, but do not establish how frequently these behaviors occur across the evaluation set.Source: §6.3 · pp. 10–11, Figure 3 · p. 10

6.4 Training dynamics with pseudo-caption supervision

  • Training loss falls from approximately 2.10 to 0.35 on Dataset A alone and from 0.92 to 0.12 on A+B, but corpus differences limit interpretation as evidence of better held-out caption quality.The loss comparison does not resolve the contribution of pseudo-caption supervision to held-out generation.Source: §6.4 · p. 11

6.5 Resource measurements

  • NF4 model allocation is 5.4946 GB versus 9.6551 GB for BF16/hybrid, while the latter encounters a NaN failure that prevents caption-quality comparison.The NF4 cascade averages 22.6069 seconds per image, with the captioner accounting for approximately 96.23% of runtime.Source: §6.5 · p. 11, Table 6 · p. 11

7 Discussion and Limitations

The discussion distinguishes class-label retention from visual factuality and notes that metric rankings, partition overlap, and training-data differences constrain interpretation.

  • All three CI variants attain MMR of 0.9067, but this measures class-string presence rather than the factuality of colors, ornaments, or spatial descriptions.Selected examples show both a correct Malang identity and incorrect Trenggalek identities entering captions; their selection does not establish error prevalence across the set.Source: §7 · p. 12
  • PFT-NCI leads reference-based measures but has the lowest CLIPScore, reflecting distinct evaluation properties and the influence of recurring reference templates and cultural-term representation.Reference agreement and image–text compatibility therefore provide complementary, not interchangeable, views of generated descriptions.Source: §7 · p. 12
  • Independent classifier and captioner partitions leave possible classifier-training overlap with caption-evaluation images unquantified, limiting conclusions about complete-cascade generalization.Source accuracy comparisons also combine different taxonomies, class supports, and acquisition conditions; Dataset A’s test has 33 images, including one Tulungagung image.Source: §7 · p. 12
  • A-only and A+B loss trajectories reflect different caption corpora and do not resolve whether pseudo-caption supervision improves held-out generation or explains the six-condition results.NF4 runs in the evaluated environment, but BF16/hybrid NaN failure limits caption-quality comparison; metric details and Dataset-B provenance also constrain reproducibility.Source: §7 · p. 12

8 Conclusion

Predicted-class injection produces a 0.9067 ground-truth class-label mention rate across all three captioner conditions, while classifier accuracy differs by source and caption metrics rank models differently. Qualitative examples show both retained identities and propagated classification errors, motivating evaluation of label retention alongside visual-description grounding.

8 Conclusion

  • 0.9067 is the ground-truth class-label mention rate across all three predicted-class-injected captioner conditions.The six-condition evaluation also reports a 22.57-percentage-point source-specific classifier accuracy gap, whose interpretation depends on differing label inventories and training support.Source: §8 · p. 13
  • The reported 22.57-percentage-point source-specific classifier accuracy gap is interpreted in light of different label inventories and training support.Class-level macro-F1 analysis also depends on the averaging convention.Source: §8 · p. 13
  • Caption metrics produce different model rankings, while qualitative examples show both correct identity retention and propagated classification errors.The findings support assessing class-label retention alongside the factual grounding of visual descriptions, within the generalization limits of the component-specific data partitions.Source: §8 · p. 13

A Class Inventory and Per-Class Performance

The appendix identifies Dataset A's regional classes and notes that reproducing its class-index mapping requires the original label encoding. Table 7 covers Dataset-B per-class classification results, with 15 test images per class.

A Class Inventory and Per-Class Performance

B Pseudo-Caption Generation Prompt

The pseudo-caption prompt injects each image's class label and specifies a concise description of visible batik features. It also restricts cultural interpretation, but the exact evaluation prompts for every condition are unspecified.

B Pseudo-Caption Generation Prompt

Related papers

Loading 2610.30661v1…