PaperlayerView exact revision on arXiv

Source-linked AI summary

Compensating for Scarce Historical Images in Cross-Domain Cultural Heritage Retrieval Using Synthetic Aging

Marcin Iwanowski, Adam Mazgaj, Ferdynand Gorski, Sabina Szymoniak

arXiv:2609.08766v1cs.CV

TL;DR

Cultural heritage retrieval must link records of the same object despite visual differences and scarce historical imagery. This study evaluates degradation-based synthetic aging for replacing or completing historical training observations; complete replacement underperformed genuine data, while completion helped most when genuine coverage was scarce.

  • Problem

    Historical records are scarce, and differences in viewpoint, acquisition, reproduction, framing, resolution, and degradation make linking them to contemporary images difficult.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates degradation-based synthetic aging for complete replacement, partial completion, and full-coverage real-plus-synthetic training.

    Source: §2 · p. 2

  • Results

    Complete replacement reduced bidirectional mean R@1 from 92.15% to 87.94%; at 10% coverage, completion gained 12.30 pp over the exposure-matched real-only control, with its advantage effectively absent at 90%.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Genuine and synthetic observations are complementary: genuine images provide authentic historical variability, while synthetic observations extend cross-domain supervision where archival records are missing.

    Source: §6 · p. 11

    The controls do not isolate restored old–new pairing from the effect of retaining a larger, more diverse identity set.

    Source: §5.5 · p. 11

Abstract

from arXiv · show

Cultural heritage collections often contain contemporary and historical visual records of the same physical object. Linking these records is difficult because corresponding images may differ in viewpoint, acquisition conditions, color reproduction, framing, resolution, and degradation, while genuine historical images are frequently scarce. This study investigates whether synthetically aged contemporary images can replace or complement missing historical training data in bidirectional instance-level retrieval. Synthetic old-domain images are generated using degradation-oriented transformations. An EfficientNetV2-M model is evaluated on identity-disjoint training, validation, and test sets across three dataset partitions and three training seeds. Mixed real-synthetic training is compared with real-only baselines using proportionally scaled and fixed 300-batch-per-epoch schedules. Complete replacement of genuine historical images reduced bidirectional mean R@1 from 86.56% to 81.27%, showing that synthetic aging does not reproduce the full genuine old-domain variability. Increasing the number of independently generated synthetic variants provided no consistent improvement. Under controlled scarcity, however, synthetic completion improved mean R@1 by 3.69 percentage points at 25% genuine historical coverage and by 2.92 points at 50%, relative to the proportionally scaled real-only baselines. At 75%, the gain decreased to 2.00 points, while performance remained comparable to the complete-real-data reference. Fixed-schedule real-only controls did not reproduce these improvements. The results indicate that genuine and synthetic observations are complementary. Synthetic completion primarily benefits retrieval by extending cross-domain identity coverage rather than by increasing training exposure, with its contribution gradually decreasing as genuine historical coverage increases.

1. Introduction

Cultural heritage records of the same object are difficult to link across domains because their appearance varies and genuine historical images are scarce. The study examines degradation-based synthetic aging as a substitute for, or complement to, missing historical observations.

1. Introduction

  • Cross-domain instance retrieval must match the same physical object despite substantial visual variation, while scarce historical records leave many identities without genuine observations in both domains.Such identities cannot provide genuine cross-domain positive pairs for metric-learning training.Source: §1 · pp. 1–2
  • Synthetic aging can create identity-preserving old-domain observations, but preserves the contemporary source's viewpoint, composition, and geometry and may not reproduce genuine historical differences.Even imperfect synthetic observations may still provide cross-domain supervision for identities lacking historical counterparts.Source: §1 · p. 2

2. Research aim

  • The study addresses whether synthetic images can replace historical images, whether more stochastic variants help, and how completion behaves as genuine historical coverage changes.It also separates identity-coverage effects from training exposure and general augmentation.Source: §1 · p. 2
  • The research aim is to evaluate degradation-based synthetic aging under complete replacement, partial completion, and full-coverage real-plus-synthetic training.The stated hypothesis is that synthetic aging cannot fully reproduce genuine historical variability, but can complement scarce data, with benefits diminishing as genuine coverage increases.Source: §2 · p. 2

3. Materials and methods

This section describes experiments testing synthetic old-domain images through replacement, augmentation, and controlled-scarcity settings, with identity-disjoint evaluation and paired comparisons. It also outlines statistical analysis using a descriptive two-percentage-point loss margin and limiting uncertainty interpretation to the evaluated test identities and training seeds.

3.1. Dataset and cross-domain retrieval scenario

  • The dataset contains 1,077 identities represented in both domains, split by identity into three partitions with pairwise-disjoint test identities and genuine-only evaluation images.Each partition has 808 training, 108 validation, and 161 test identities; together, the test sets cover 483 distinct identities. Synthetic images are restricted to training.Source: §3.1 · pp. 2–3

3.2. Synthetic aging of contemporary images

  • Synthetic old-domain images are generated offline from contemporary training images using degradation effects; OLDIFY1–3 share a pipeline but use independently sampled parameters.The procedure adds effects such as noise, scratches, blur, tonal changes, and paper texture; generated images are fixed per partition, source image, and variant across training seeds.Source: §3.2 · p. 3

3.3. Retrieval model

  • An ImageNet-pretrained EfficientNetV2-M produces 1,280-dimensional normalized embeddings, trained with batch-hard triplet loss and one old–new positive pair per identity in each batch.The final two backbone blocks and projection layer are trainable; training uses three seeds, validation mean R@1 checkpointing, and early stopping.Source: §3.3 · pp. 3–4

3.4.1. Experiment 1: complete replacement and full-coverage augmentation

  • Complete-coverage experiments compare genuine-only training, full synthetic replacement using one or three variants, and source-balanced genuine-plus-synthetic augmentation.All conditions use 808 identities and 463 batches per epoch; each identity still contributes one old-domain image per batch, so variant count changes synthetic variability, not sampling frequency.Source: §3.4.1 · p. 4, §3.4 · p. 4

3.4.2. Experiment 2: controlled scarcity and synthetic completion

  • Scarcity experiments vary genuine old-domain identity coverage from 0% to 100% in 10-point increments, assigning OLDIFY1 images to identities without genuine observations while retaining all 808 training identities.Genuine subsets are category-stratified, nested, and fixed across seeds within each partition; all coverage conditions use 463 batches per epoch.Source: §3.4.2 · p. 5

3.4.3. Experiment 3: real-only controls for identity coverage and training exposure

  • Real-only controls remove identities lacking genuine old-domain images, with A variants scaling batches to retained identities and B variants keeping the full 463-batch schedule.Mixed-versus-A compares completion at matched per-identity exposure, B-versus-A tests repeated exposure, and mixed-versus-B compares broader identity coverage at matched steps.Source: §3.4.3 · p. 5, Table 2 · p. 5

3.5. Retrieval evaluation

  • Bidirectional retrieval uses genuine old-domain queries against contemporary galleries and vice versa; query-weighted R@k is reported, with bidirectional mean R@1 as the primary metric.Only genuine images appear in queries and galleries, and compared models use identical queries and galleries within each partition.Source: §3.5 · pp. 5–6

3.6. Statistical analysis

  • Performance is summarized across three partitions and three seeds, while paired-difference confidence intervals use 10,000 identity-level bootstrap resamples and resample seeds within partitions.Partitions are not resampled; identities with multiple queries are retained together to avoid pseudo-replication.Source: §3.6 · p. 6
  • The acceptable-loss criterion uses the lower bound of a 95% bootstrap confidence interval to assess a prespecified two-percentage-point margin, and is not a formal non-inferiority test.It identifies the lowest evaluated genuine-coverage level whose uncertainty interval remains within that margin.Source: §3.6 · pp. 6–7
  • Table 3 reports Experiment 1 retrieval performance as mean ± sample standard deviation across three dataset partitions and three training seeds.Source: Table 3 · p. 7
  • The three partitions have disjoint test identities but may share training or validation identities, so intervals do not represent population-level uncertainty across independent datasets.Partition-level results were also examined for consistency across test-set compositions.Source: §3.6 · p. 7

4. Results

Synthetic completion most benefits retrieval when genuine historical identity coverage is scarce, with its advantage diminishing as coverage rises. Complete synthetic replacement underperforms genuine historical observations, while adding synthetic data at full coverage shows no clear benefit.

4.1. Complete replacement and full-coverage augmentation

  • Complete synthetic replacement reduced mean R@1 from 92.15% for Real to 87.94% for Synth-1, with reductions in both retrieval directions.The paired effect was −4.21 pp (95% CI: [−6.19, −2.23] pp).Source: §4.1 · p. 7
  • Increasing the synthetic pool from one to three OLDIFY variants produced essentially no rank-1 change: Synth-3 reached 87.91% mean R@1, with a −0.03 pp paired difference from Synth-1.The 95% CI for that difference was [−2.39, +2.43] pp.Source: §4.1 · p. 7
  • At complete genuine coverage, Real+Synth reached 94.05% mean R@1 versus 92.15% for Real, but its 95% CI crossed zero.The paired difference was +1.90 pp (95% CI: [−0.10, +4.09] pp).Source: §4.1 · p. 7

4.3. Paired effects and acceptable-loss analysis

  • At 10% genuine coverage, the mixed condition exceeded real-only controls by 12.30 pp and 13.99 pp; the completion advantage decreased as coverage rose.At 30%, the gains were 4.67 pp and 5.47 pp, with both confidence intervals above zero.Source: Figure 2 · p. 8, §4.3 · p. 8

4.2. E

  • At 10% genuine coverage, R10_S90 achieved 89.44% mean R@1 versus 77.14% for R10_A and 75.45% for R10_B.The mixed-condition advantage diminished with increasing genuine coverage; conditions from 70% to 90% remained within approximately 0.3 percentage points of the complete-genuine-data mean.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3. Paired effects and acceptable-loss analysis

  • Increasing the budget for reduced real-only identity sets did not reproduce completion gains: B–A estimates ranged from −1.68 to −0.14 pp, and every CI included zero.At 50% and 70% coverage, completion’s advantage over A had CIs including zero, while the 70% advantage over B was +2.29 pp (95% CI: [+0.37, +4.36] pp).Source: Table 4 · p. 8, §4.3 · p. 8
  • At 70% genuine coverage, the paired effect versus R100_S0 was +0.03 pp (95% CI: [−1.78, +1.88] pp), within the prespecified 2-pp acceptable-loss margin.This was the lowest evaluated coverage whose 95% CI remained within that margin; the analysis was descriptive, not a formal non-inferiority test.Source: §4.3 · p. 8

4.4. Variability across dataset partitions

  • The decreasing completion benefit as genuine coverage increased was reproduced across all three test partitions, despite differences in absolute performance.At 10% coverage, mixed-versus-A split-level gains were +10.09, +14.63, and +12.19 pp; by 90%, effect direction was inconsistent across partitions.Source: §4.4 · pp. 8–9

5. Discussion

Synthetic completion was most beneficial when genuine historical coverage was scarce, while complete replacement remained inferior to genuine historical training. Its benefits diminished as genuine coverage increased, and broader cross-domain identity coverage appears to be a primary contributor under severe scarcity.

5.1. Complete replacement and synthetic diversity

  • Complete replacement with OLDIFY1 reduced bidirectional mean R@1 by 4.21 percentage points, while increasing the sampling pool from one to three variants changed mean R@1 by only −0.03 pp.The replacement reduction was from 92.15% to 87.94%; the variant-count difference had a 95% confidence interval of [−2.39, +2.43] pp.Source: §5.1 · p. 9

5.2. Synthetic completion under historical-data scarcity

  • At 10% genuine coverage, mixed training exceeded the exposure-matched A control by 12.30 pp and the full-budget B control by 13.99 pp; gains diminished at higher coverage.At 30%, the corresponding gains were 4.67 and 5.47 pp; 70% was the lowest evaluated coverage within the prespecified acceptable-loss margin.Source: §5.2 · pp. 9–10

5.3. Identity coverage and augmentation effects

  • At severe scarcity, completion gains are consistent with preserving a broad set of cross-domain training identities rather than repeatedly sampling a smaller genuine subset.By 90% genuine coverage, little difference remained between mixed and real-only training.Source: §5.3 · p. 10
  • At complete genuine coverage, adding OLDIFY1 raised mean R@1 from 92.15% to 94.05%, but its +1.90 pp paired effect did not establish a clear augmentation benefit.The 95% confidence interval was [−0.10, +4.09] pp; augmentation may contribute alongside identity completion, with relative importance depending on scarcity.Source: §5.3 · p. 10

5.4. Dataset composition and retrieval implications

  • The completion effect decreased across all three pairwise disjoint test partitions, but absolute performance and effect magnitude varied with dataset composition.At 10% and 30% coverage, mixed training beat its exposure-matched A control in every partition; differences became much smaller at 50% and 70%, and effectively disappeared at 90%.Source: §5.4 · p. 10
  • Genuine records may differ in informational value, and the study leaves selection of an informative genuine subset unresolved.Retrieval difficulty, domain discrepancy, visual diversity, and expected information gain are suggested as future selection criteria; synthetic completion may also suit expert-assisted candidate retrieval.Source: §5.4 · p. 10

5.5. Limitations

  • Generalization is bounded by evaluation on one assembled collection and one backbone, metric-learning objective, and degradation-based aging pipeline.The coverage relationship requires validation on independent collections with different object types, acquisition processes, and domain shifts.Source: §5.5 · pp. 10–11
  • The controls cannot isolate restored old–new pairing from greater identity diversity, and simulated coverage percentages are not universal thresholds.Scarcity was simulated with nested identity subsets; the confidence intervals describe evaluated identities and seeds, not population-level uncertainty across independent collections.Source: §5.5 · p. 11

6. Conclusions

Synthetic completion helps most when genuine historical coverage is scarce, but synthetic images do not replace the variability of genuine records. The conclusions favor using synthetic data to fill identity gaps while treating coverage thresholds as specific to this dataset and protocol.

6. Conclusions

  • 4.21 pp: complete replacement of genuine historical data reduced bidirectional mean R@1 from 92.15% to 87.94%.Increasing the synthetic sampling pool from one to three independently generated variants did not improve retrieval.Source: §6 · p. 11
  • 12.30 pp at 10% coverage and 4.67 pp at 30%: synthetic completion improved mean R@1 over exposure-matched real-only controls, with gains decreasing as genuine coverage increased.At 90% coverage, the advantage was effectively absent.Source: §6 · p. 11
  • Repeated exposure to reduced genuine identity sets did not reproduce the gains, while the controls could not fully separate identity coverage from training-set size and diversity.The results are consistent with broader cross-domain identity coverage contributing to synthetic completion's value, especially under severe scarcity.Source: §6 · p. 11
  • 70% genuine old-domain identity coverage was the lowest evaluated level whose 95% bootstrap confidence interval stayed within the prespecified two-point acceptable-loss margin relative to complete genuine coverage.The threshold is specific to this dataset and protocol, not a universal requirement.Source: §6 · p. 11
  • 1.90 pp: adding OLDIFY1 to complete genuine training data increased mean R@1 from 92.15% to 94.05%, but the paired 95% confidence interval included zero.The conclusions treat genuine and synthetic observations as complementary: genuine images supply authentic historical variability, while synthetic observations extend supervision where records are missing.Source: §6 · p. 11
  • Synthetic data are most valuable for filling specific informational gaps, so training-set construction should consider missing identities and domains, not only image counts.Potential applications include artwork matching, collection integration, catalog enrichment, and provenance research.Source: §6 · pp. 11–12

Data availability

The study uses images from the Louvre Collections and Lost Art Database, subject to their institutions' access and copyright terms. Dataset partitions, experimental configurations, aggregated results, and source code are available from the corresponding author upon reasonable request.

Data availability

  • Study images come from the publicly accessible Louvre Collections and Lost Art Database.Source: Data availability · p. 12
  • Access, reuse, and redistribution remain subject to the respective source institutions' terms and copyright restrictions.Source: Data availability · p. 12
  • Dataset partitions, experimental configurations, aggregated results, and synthetic-image generation and evaluation code are available from the corresponding author upon reasonable request.Source: Data availability · p. 12

Funding

The work was supported by a Polish Agency for Enterprise Development grant co-funded by the European Union.

Funding

  • The work was supported by PARP Grant FENG.01.01-IP.02-3717/23, co-funded through the European Funds for a Modern Economy 2021–2027 (FENG), SMART Path.Source: Funding · p. 12

S1. Synthetic aging procedure

OLDIFY images were generated offline from contemporary training images. Transformation parameters were sampled independently for each image, and OLDIFY1–3 were independent realizations of the same stochastic procedure.

S1. Synthetic aging procedure

  • Images with a larger spatial dimension above 1500 pixels were proportionally downscaled to a maximum dimension of 1500 pixels; smaller images retained their original resolution.Source: S1. Synthetic aging procedure · p. 13
  • Transformation parameters were sampled independently from the Table S1 ranges, using continuous uniform or discrete uniform distributions; all stages were enabled, though some effects could be negligible.Source: S1. Synthetic aging procedure · p. 13, Table S1 · p. 13
  • OLDIFY1, OLDIFY2, and OLDIFY3 used identical transformation definitions, ranges, and ordering, differing only in sampled parameter values.Source: S1. Synthetic aging procedure · p. 13

S1.1. Processing sequence

The aging pipeline applies a fixed sequence of image transformations, including replicated borders, noise, artifacts, blur, perspective changes, texture, light effects, and sepia-to-grayscale conversion.

S1.1. Processing sequence

  • Replicated borders are added by extending the image's outermost pixels by 4–15 pixels.Source: S1.1. Processing sequence · p. 13
  • The fixed sequence combines border replication, noise, vignetting, scratches and dust, blur, perspective changes, paper texture, light leaks, and sepia-to-grayscale conversion.The sequence includes Gaussian and impulse noise before later tonal and geometric transformations.Source: S1.1. Processing sequence · p. 13
  • OLDIFY1 is used for controlled-scarcity and synthetic-completion experiments, while OLDIFY1–3 are combined only for Synth-3 complete replacement.Source: S1.1. Processing sequence · p. 13

S2. Partition-level retrieval results

Across three disjoint test partitions, synthetic completion performs best relative to exposure-matched real-only controls under severe historical-data scarcity, with its advantage generally diminishing as genuine coverage increases.

S2. Partition-level retrieval results

  • At 10% genuine coverage, mixed–A gains were +10.09, +14.63, and +12.19 pp across Splits 1–3, and at 30% were +6.29, +4.36, and +3.35 pp.The benefit at low genuine coverage was reproduced in all three pairwise-disjoint test partitions.Source: S2. Partition-level retrieval results · p. 14
  • At 50% genuine coverage, mixed–A differences remained positive in all partitions (+0.87, +1.30, and +2.79 pp), as they did at 70% (+0.76, +2.06, and +1.63 pp).Source: S2. Partition-level retrieval results · p. 14
  • At 90% genuine coverage, mixed–A differences were small and their direction was no longer consistent across partitions.Within individual partitions, retrieval curves were not strictly monotonic.Source: S2. Partition-level retrieval results · p. 14
  • Synthetic completion's largest and most consistent benefit occurred at low genuine coverage, and diminished as coverage approached completeness.Dataset composition affected absolute retrieval difficulty and effect magnitude, but the scarcity-related pattern was stable.Source: S2. Partition-level retrieval results · p. 14

S3. Detailed and secondary retrieval results

This section provides supplementary numerical results on the complete genuine-coverage series, real-only controls, an acceptable-loss comparison, and retrieval at larger rank depths.

S3. Detailed and secondary retrieval results

S3.1. Complete scarcity-series results

The supplementary results show that synthetic completion improves with increasing genuine coverage but not strictly monotonically; 70% is the lowest evaluated coverage meeting the descriptive acceptable-loss criterion, while rank-10 performance remains high even under complete synthetic replacement.

S3.1. Complete scarcity-series results

  • The mixed completion series showed an overall increase in retrieval performance with greater genuine old-domain coverage, but the relationship was not strictly monotonic.The largest improvements relative to real-only controls occurred at low genuine coverage; differences became negligible at 90%.Source: S3.1. Complete scarcity-series results · p. 14, Table S2 · p. 14

S3.2. Performance relative to complete genuine coverage

S3.3. Rank-dependent retrieval performance

  • Even complete synthetic replacement achieved 98.43% bidirectional mean R@10, versus 99.23% for the complete genuine-data reference.Across the mixed series, R@10 remained close to 99%, while R@1 varied more strongly with genuine coverage.Source: Figure S3 · p. 15, S3.3. Rank-dependent retrieval… · p. 15
  • Synthetic aging often preserved enough identity information to place the correct object among highly ranked candidates, even when it did not rank that identity first.The authors note potential use in expert-assisted workflows with short candidate lists for manual verification.Source: S3.3. Rank-dependent retrieval… · p. 15
Loading 2609.08766v1…