PaperlayerView exact revision on arXiv

Source-linked AI summary

Composed Historical Image Retrieval by Modeling Temporal Representations

Adrià Molina Rodríguez, Oriol Ramos Terrades, Josep Lladós Canet

arXiv:2608.18694v1cs.CVcs.AIcs.IR

TL;DR

Historical-photo retrieval needs representations that preserve both when an image was made and what it depicts. TDIR separates temporal and categorical information into orthogonal subspaces and provides theoretical and empirical support for interpretable composed retrieval.

  • Problem

    Historical-photo archives need methods that jointly capture photograph dates and depicted objects, because date metadata alone cannot fully contextualize archival images.

  • Method

    TDIR decomposes image embeddings into orthogonal temporal and categorical subspaces, enabling temporal information to be extracted and injected without knowing the source year.

  • Results

    TDIR yields interpretable behavior in composed retrieval of in-the-wild historical photographs, while ablations show swapping improves transitivity and both date and object subspaces.

  • Takeaways & Limitations

    The paper formalizes temporal decomposability in visual embeddings and establishes composed historical image retrieval as a new archival evaluation setting.

  • Takeaways & Limitations

    TDIR assumes shared temporal displacement across object categories, and its additive transplantation becomes less faithful at large temporal distances.

Abstract

from arXiv · show

While time evolves linearly, the geometry of neural embedding spaces is inherently multi-dimensional, often chaotic, and difficult to interpret. In principle, one could constrain an embedding space to a single temporal dimension; however, such a reduction would sacrifice performance on downstream tasks, as one-dimensional embeddings cannot retain sufficient expressive capacity. This paper asks whether it is possible to learn representations that preserve temporal structure while remaining effective for image and object retrieval, and answers this question by building the mathematical foundations of such a system. We propose Temporally Decomposable Image Representations (TDIR), a representation learning algorithm that decomposes historical photographs into separate date and content components through orthogonal subspaces. We define and prove the conditions under which such a decomposition is achievable, characterize the error incurred when those conditions are only partially met, and show that orthogonality between temporal and categorical subspaces emerges naturally from the joint optimization, without requiring it to be imposed explicitly. Beyond its geometric properties, TDIR enables a class of transitive operations on embedding spaces: the temporal information of one image can be extracted and injected into the representation of another, with no label supervision required. All theoretical properties are grounded and validated in the real-world problem of Composed Image Retrieval on historical photographs, where a query simultaneously specifies object content and a target time period, either through labels or through example images. This in-the-wild setting serves as a concrete backing for the propositions we derive, offering an intuitive and interpretable way to navigate photographic archives while maintaining competitive performance in both date estimation and object retrieval.

1 Introduction

Historical photographs are important archival resources whose visual information makes date estimation and object-sensitive contextualization difficult for conventional metadata and neural representations. TDIR addresses this by separating temporal and categorical embedding components into orthogonal subspaces and enabling composed retrieval across object content and target time periods.

  • Motivation: Around 5-10% of archival material is estimated to be non-textual, with photographs covering a shorter historical span than printed or handwritten documents.Photographs nevertheless comprise an important portion of archival data despite emerging late during the 19th century.
  • Motivation: Photographs encode most information visually, making discrete metadata restrictive and positioning photograph date as a prominent historical-photography task.Digital Humanities projects therefore pursue semantic search and automatic metadata completion for non-textual collections.
  • Motivation: Complete archival description requires identifying both when a photograph was taken and how its depicted objects relate to their historical period.This object-sensitive interpretation contributes to a document’s probative value by linking images to events, people, technologies, or phenomena.
  • Problem: Standard neural representations entangle colour, texture, objects, and date, while current retrieval systems do not support composing an object query with a target time period.The desired system should reflect how archivists naturally formulate archival queries.
  • Contribution: TDIR separates temporal and categorical components of image embeddings into orthogonal subspaces, formalizes when decomposition holds, and characterizes error under partial satisfaction of those conditions.The framework proves that orthogonality emerges naturally from joint optimization and presents the first formal treatment of temporal decomposability in visual embeddings, according to the passage.
  • Application: TDIR supports composed image retrieval on in-the-wild historical photographs, where queries specify object content and a target time period simultaneously.Its joint proxy optimization interprets year centroids as displacement vectors that transport representations to different dates without altering categorical content.

2 Related Work

Prior photographic date-estimation methods use visual or semantic features but do not formalize ideal temporal representations. Existing datasets also lack the combination of diverse object categories and year-level annotations needed for category-date composed retrieval.

  • Photographic Date Estimation: Earlier date-estimation methods relied on colour histograms and film grain, whereas newer approaches use semantic features.These works do not formalize the desired properties of an ideal temporal representation.
  • Photographic Date Estimation: DEW frames date estimation as regression but does not characterize the geometric structure of temporal representations.DEW provides date annotations without object labels.
  • Existing Datasets: Yearbook datasets are restricted to faces and therefore provide only a single semantic category.The cited datasets are yearbook-based and face-focused.
  • Existing Datasets: EUFCC-CIR supports composed retrieval for cultural heritage collections but lacks temporal metadata, preventing category-date composition.No existing benchmark simultaneously provides object-category and year-level annotations across a diverse photographic archive.

3 Methodology

TDIR learns image representations by separating category and year information into orthogonal components, using category and year proxies with residual-based temporal transformations. Its theoretical results establish decomposability under ideal assumptions, characterize data-dependent residual entanglement, and show orthogonality emerges from joint optimization.

  • Representation decomposition: TDIR decomposes each embedding into category and temporal components occupying orthogonal subspaces, with mutually uninformative centroids whose sum captures label-relevant content.The ideal representation satisfies I(µc; µy) = 0 and decomposes information about category and year into independent contributions.
  • Proxy-based training: Training uses category proxies, year proxies on ℓ2-normalized residuals, and learnable year displacements shared across categories.Category proxies form year-agnostic centroids; year proxies encode temporal information through cosine similarity, while displacement vectors enforce transitivity during training and are discarded at inference.
  • Practical learnability: Proposition 2 shows that the discrepancy δ ≜I(ε; c,y) depends only on training-data statistics, allowing centroids to be learned under ⟨µc, µy⟩= 0.Representations become more temporally decomposable as δ decreases.
  • Residual construction: The method constructs direct residuals by subtracting category centroids, then uses cross-year residuals and auxiliary displacements to enforce temporal transitivity.Re-centering removes category information before temporal modeling, while the swapping trick forms cross-year residuals for transitive training.
  • Emergent orthogonality: Joint minimization yields exact TDIR when I(ε ; c,y) = 0 and asymptotic TDIR when I(ε ; c,y)̸ = 0, with orthogonality emerging from centering and swapping.The theorem formally guarantees that the training algorithm can yield TDIR in both ideal and practical cases without explicitly enforcing orthogonality.
  • Zero-label inference: In zero-label image-plus-image inference, a temporal residual from a reference image is added to a category image, retrieving same-category images from the reference period.Temporal transitivity preserves the category direction while displacing the query toward the reference image’s year, even when neither year is known.

4 Experimental Set-Up

The experiments use an object-centric, date-annotated extension of DEW and evaluate TDIR across multiple visual backbones and inference modes. Evaluation measures categorical and temporal retrieval fidelity, with date correctness defined within a five-year lustrum and a 7.14% random baseline.

  • Proposed Dataset: The proposed dataset contains 7,239,083 medium-to-high-resolution detections spanning 586 objects, with date annotations from DEW.The experiments focus on the top-50 predominant objects, totaling 4,478,887 detections.
  • Proposed Dataset: Objects are detected with OWLv2 using the DETR HuggingFace implementation and a 30% detection threshold, while retaining the original resolution criterion.This broadens object coverage beyond the constrained six-category set used in prior object-centric DEW processing.
  • Backbone Architectures: TDIR is evaluated with ConvNeXt-Base, ViT-B/32, ResNet50, and VGG19-BN backbones initialized with ImageNet pre-trained weights.The architectures include both convolutional and transformer-based visual representations.
  • Evaluation Metrics: Evaluation measures categorical fidelity and temporal fidelity across all three inference modes, using top-K retrieval precision metrics.A retrieved date is counted as correct when it falls within the same five-year lustrum as the target.
  • Evaluation Metrics: 7.14% is the random baseline for correctly classified dates, given DEW’s 1930–1999 range and five-year correctness window.In label-based inference, categorical and temporal targets come from query labels; in image-based modes, the category comes from the content image and the date from the target year or transferred residual.

5 Results

The results evaluate the framework across backbone comparisons, ablations, calibration, and qualitative retrieval behavior. They show that swapping improves learned subspaces and transitivity, while temporal normalization contributes little and remaining errors are usually temporally nearby but can involve category ambiguity.

  • ConvNeXt and ViT emerge as the top-performing and broadly comparable architectures among evaluated computer-vision backbones, while CLIP baselines remain competitive.
  • The swapping trick improves transitivity and produces better date and object embedding subspaces for both image-based and label-based inference.The label-based gains do not depend on injecting a date into a temporal embedding.
  • The swapping-trick component marked with ∗) is especially important for Image+Image inference, whereas Image+Label inference shows only a marginal performance gap.The passage attributes this difference to train–test mismatch at inference time.
  • Temporal-component normalization is least impactful overall, though it slightly improves Image+Image date estimation, and Label+Label object retrieval is maximized only with all three components trained.
  • Predicted-year errors generally remain within the same lustrum, but object retrieval can show category ambiguity despite qualitatively correct temporal transplantation.The reported limitation is that an object-agnostic temporal subspace enforces the same temporal displacement across objects, while boundary-near images can cross categories.

6 Conclusions

TDIR decomposes historical image embeddings into orthogonal temporal and categorical subspaces, with theoretical guarantees and naturally emerging orthogonality. The framework is grounded in a new compositional retrieval task and benchmark, while its limitations motivate nonlinear and category-specific temporal modeling.

  • Framework contributions: TDIR formalises historical image embeddings as orthogonal temporal and categorical subspaces, with proven decomposition conditions and characterised error under partial violation.Orthogonality emerges naturally from joint optimisation rather than being explicitly imposed.
  • Task and benchmark: Composed Historical Image Retrieval queries jointly specify object content and a target time period, addressing a natural archival task without an existing evaluation benchmark.The setting is introduced as a novel problem for historical photographic archives.
  • Task and benchmark: Extending DEW with object-level detections across 50 categories enables the first evaluation of compositional retrieval over a diverse photographic archive.The extension provides the benchmark needed to evaluate the proposed retrieval setting.
  • Limitations and future work: TDIR assumes shared temporal displacement across object categories, and its additive transplantation operator loses fidelity at large temporal distances.These limitations may fail to reflect uneven historical change across objects.
  • Limitations and future work: Future work will investigate nonlinear temporal operators, category-specific temporal subspaces, and zero-shot benchmark evaluation over unseen object categories.These directions directly address the framework’s temporal modeling and evaluation limitations.

Supplementary Material … A.2 Temporally Decomposable Image Representations (TDIR)

The supplementary material provides theoretical derivations, empirical analyses, and additional experimental details, including mutual-information foundations and proofs for temporally decomposable representations. In TDIR, category and year information are designed to be separable, yielding decomposed representations and orthogonal centroid subspaces under the stated assumptions.

  • Supplementary Material: The supplementary material is organized around derivations, photographic calculations, object-centric analysis, CLIP baselines, and frequently asked questions.It supports the main manuscript with theoretical derivations, empirical analyses, and additional experimental details.
  • A.1 Chain rule of mutual information: The chain rule of mutual information decomposes information in a joint representation into information from one component and additional information from another after conditioning.The derivation uses three general random variables, A, B, and C.
  • A.2 Temporally Decomposable Image Representations (TDIR): TDIR treats category and year classes as decomposable by design, with class centroids distributed without influence from date information and vice versa.This is the model assumption used in Definition 1.
  • A.2 Temporally Decomposable Image Representations (TDIR): Proposition 3 establishes the image-representation equality under the model assumptions of Definition 1.Its proof applies the chain rule for mutual information repeatedly to decompose the relevant terms.
  • A.2 Temporally Decomposable Image Representations (TDIR): The category-centroid term simplifies to I(µc; c) when µc ⊥ y | c, while conditional mutual information expands as I(µy; c,y | µc) = I(µy; c | µc)+I(µy; y | µc,c).The derivation relies on the category centroid carrying no date information and on the Markov and independence assumptions.
  • A.2 Temporally Decomposable Image Representations (TDIR): Combining the expansions recovers the target decomposition exactly if and only if the initial assumption in Eq. (28) holds.Conditioning on (µc,c) does not alter the dependence between µy and y under the stated structure.
  • A.2 Temporally Decomposable Image Representations (TDIR): Lemma 2 states that category and temporal centroids occupy orthogonal subspaces under Definition 1.The proof derives independence, Cov(µc,µy) = 0, and an additive decomposition of total representational variance.

A.3 Estimating the error in real scenarios · A.4 TDIR as an Emergent Property of the Joint Loss

In practical settings, TDIR’s error is governed by label-information leakage in a noise component induced by training-data correlations, while orthogonal category and year proxies remain learnable. Joint optimization therefore yields exact TDIR without leakage and asymptotic TDIR with leakage, with swapping enforcing cross-category orthogonality.

  • A.3 Estimating the error in real scenarios: Practical TDIR models embeddings as v = µc + µy + ε, where ε entangles category and year information through training-data correlations.The assumption permits label-dependent noise while allowing an independent proxy vector to be learned.
  • A.3 Estimating the error in real scenarios: δ depends only on the training data’s statistical structure, not on the choices of proxy vectors µc or µy.Thus, δ is a data-dependent constant during optimization.
  • A.3 Estimating the error in real scenarios: The category and year centroids can satisfy ⟨µc, µy⟩ = 0 without affecting the error bound, and δ →0 when ε is disentangled from both labels.The practical mutual-information relation is therefore consistent with TDIR up to an upper bound characterized by training-data entanglement.
  • A.4 TDIR as an Emergent Property of the Joint Loss: The joint optimization of Lc and Ly implicitly enforces ⟨µc, µy⟩ = 0 without imposing orthogonality explicitly.This establishes orthogonality as an emergent property of the joint loss.
  • A.4 TDIR as an Emergent Property of the Joint Loss: If I(ε ; c,y) = 0, joint minimization of L = Lc +Ly implies exact TDIR; if I(ε ; c,y)̸ = 0, it implies asymptotic TDIR.The theorem distinguishes the ideal and noisy cases according to whether the residual noise contains label information.
  • A.4 TDIR as an Emergent Property of the Joint Loss: Near convergence, squared-error surrogates locally approximate the smooth, strictly convex proxy loss, with conclusions holding to the Taylor remainder’s asymptotic order.The local approximation has the form ∥u−µ∗∥2 +O(∥u− µ∗∥3).
  • A.4 TDIR as an Emergent Property of the Joint Loss: Under Theorem 3’s conditions, shared displacement vectors ky enforce ⟨µc,µy⟩→0 uniformly across all categories.The swapping constraint cannot be globally satisfied with nonzero category-dependent temporal projections because ε is not an optimizer control variable.

B Photographic material calculation

A survey of three major public archives estimates that photographic material comprises roughly 5–13% of relevant collections, depending on the archive and corpus definition.

  • Method: The study estimates photographic-material proportions by surveying three major public archives.The estimates draw on an available digitisation sample, catalogue holdings, and reported portal classifications.
  • Archives Nationales (France): Archives Nationales’ sample contained 68 strictly photographic items, approximately 13% of the full sample.The sample included 123 non-textual documents, representing 23% of approximately 535 total documents.
  • Library of Congress (USA): The Library of Congress’s textual-adjacent holdings included approximately 2.5M photographs out of 27.7M items, or approximately 9%.The calculation excludes the bulk audiovisual collection and combines digitised and non-digitised holdings.
  • Portal de Archivos Españoles (Spain): Portal de Archivos Españoles reports approximately 2M photographic objects among 40M digital objects, yielding approximately 5%.Videos, posters, and other non-textual formats are excluded from the photographic count.

C Object-Centric Analysis · D CLIP Baseline

The object-centric analysis identifies category-specific weaknesses in temporal and object retrieval, while the CLIP baseline uses a frozen, normalized OpenCLIP encoder with a shared Annoy retrieval index.

  • C Object-Centric Analysis: Reference objects with low Year MAE, including tires, shorts, glasses, and faces, provide reliable temporal signals with minimal categorical-temporal entanglement.These objects act as temporal vessels for date injection.
  • C Object-Centric Analysis: Object retrieval errors concentrate on visually confusable categories, including glasses, registration plates, socks, and footwear, whose shared features create spurious correlations.These categories are visually sub-specific or overlap with related classes.
  • C Object-Centric Analysis: Target objects with high temporal error are ambiguous, making them difficult to anchor to a specific period regardless of the source image.Figure 6 decomposes average Year Error by reference and target object in Image+Image inference.
  • C Object-Centric Analysis: Precision selectively drops for objects lacking distinctive temporal signatures, such as generic architectural elements and accessories.The degradation is marked and category-specific.
  • C Object-Centric Analysis: Date-estimation precision declines toward earlier decades in Image+Label and Image+Image modes, consistent with sparse pre-1950 representation in the DEW dataset.Image+Image reverses the usual relationship in which date precision exceeds object precision under label-based modes.
  • C Object-Centric Analysis: Object-retrieval degradation from Image+Label to Image+Image is broadly distributed, indicating a largely stable category subspace with exceptions from high inter-class visual overlap.Figures 9 and 10 analyze per-object drops for date estimation and object retrieval.
  • D CLIP Baseline: The CLIP baselines use a frozen OpenCLIP ViT-B/32 encoder pre-trained on LAION-2B, without fine-tuning on DEW or historical imagery.This setup defines the baseline encoder used for comparison.
  • D CLIP Baseline: CLIP image embeddings are ℓ2-normalised to unit norm, and a single Annoy index with Ntrees = 50 trees covers the full training split.The indexed images are identical to those used for TDIR training.

D.1 CLIP Arithmetic Baseline · D.2 CLIP Prompting Baseline · E Frequently Asked Questions (FAQs)

The CLIP arithmetic and prompting baselines use pretrained vision–language embeddings or staged retrieval, but both remain limited by entangled temporal and categorical signals. The FAQ section frames peer-review questions as guidance for interpreting the method, data, and results.

  • D.1 CLIP Arithmetic Baseline: CLIP arithmetic replaces TDIR’s learned category and year proxy vectors with normalized text embeddings for object prompts and 14 five-year year bins spanning 1930–1999.The prompts are “A photo of {c}” and “A photo in {year}”.
  • D.1 CLIP Arithmetic Baseline: Label+Label arithmetic forms a query by summing the category and year CLIP text embeddings, mirroring TDIR’s proxy-sum query.The query is explicitly un-normalised.
  • D.1 CLIP Arithmetic Baseline: Image+Image arithmetic infers the reference category from CLIP text similarities, subtracts its category embedding, and adds the residual to the category-image embedding.This zero-label procedure uses a uniformly sampled training-index reference image.
  • D.1 CLIP Arithmetic Baseline: The arithmetic baseline has poor date precision because CLIP was not trained for orthogonal category–year subspaces, leaving categorical content in the residual and encoding historical period weakly.The passage states that CLIP can carry rankable temporal information while still failing to disentangle the two signals.
  • D.2 CLIP Prompting Baseline: The baseline contrast attributes TDIR’s advantage to learned orthogonal decomposition, since object and temporal retrieval remain entangled in CLIP regardless of composition strategy.The prompting pool is computed once per test image and reused across all 14 year bins in Image+Label mode.
  • D.2 CLIP Prompting Baseline: Across label and image modes, prompting preserves competitive object precision but remains substantially below TDIR in date precision because reranking cannot recover temporally relevant images absent from the category-retrieved pool.Image+Image reranking uses the raw reference-image embedding instead of a residual and avoids category inference.
  • E Frequently Asked Questions (FAQs): The FAQ section answers questions raised during peer review and discussions to help interpret the paper’s data, results, and method.It is presented as a series of interpretive answers.

E.1 Joint metric: Def. 1’s independence assumption holds empirically. … E.4 Ordinal structure & unseen years.

The reported results support independent category and date correctness, with proxy geometry closely matching orthogonality expectations. Year proxies also exhibit smooth ordinal structure and local extrapolation to unseen years, while weakly changing objects encode little temporal information.

  • E.1 Joint metric: Def. 1’s independence assumption holds empirically.: Precision@K requiring simultaneous category-and-date correctness was added across all three retrieval modes to test Def. 1’s independence assumption directly.The Label+Label evaluation used K=10 and a stratified 7 407-crop pool.
  • E.1 Joint metric: Def. 1’s independence assumption holds empirically.: Joint P@10 is 0.349 versus 0.350 predicted by multiplying Obj P@10 = 0.488 and Date P@10 = 0.718 under independence.The relative gap is 0.4%, within noise, supporting empirical independence of object- and date-correctness.
  • E.3 Baselines and Previous Work: At epoch 1, DINOv3 yields cross-block mean |cos| = 0.173 and replicates the same ordinal decay, indicating the decomposition is not an ImageNet-supervised-pretraining artifact.DINOv3 weights were loaded into the same ConvNeXt-Base backbone under the identical 50-category protocol.
  • E.3 Baselines and Previous Work: Baseline [17]’s smooth-nDCG loss can replace the year proxy loss but cannot replace the category loss or compose concepts.The passage states that [17] requires a relevance function derived from a numerical variable.
  • E.4 Ordinal structure & unseen years.: Year-proxy cosine similarity decays smoothly with temporal gap despite training enforcing only pairwise transitivity.This behavior supports an ordinal organization of the temporal representation.
  • E.4 Ordinal structure & unseen years.: At all 12 interior years, each true proxy is closer to its neighbors’ midpoint than to either neighbor alone, including cos = 0.66 versus 0.57 at 1935.The leave-one-out result indicates local extrapolation to unseen years.
  • E.4 Ordinal structure & unseen years.: Window, bus, door, boat, and tower show weak temporal signatures because their slow-changing architectural appearance carries little date information.Their low year-entanglement is therefore consistent with the representation encoding limited temporal content for these categories.
Loading 2608.18694v1…