PaperlayerView exact revision on arXiv

Source-linked AI summary

CIIR - Class-Indexed Interface Representation Learning

Doo Yon Kim, Christian Wallraven

arXiv:2610.23151v1cs.AIcs.LG

TL;DR

Multimodal systems often rely on paired observations, while independently collected datasets lack instance-level correspondence. CIIR connects them through a shared class-indexed address space; post-hoc SVHN attachment achieves 90.24% MNIST→SVHN and 82.29% SVHN→MNIST End-to-end Address Transfer Accuracy.

  • Problem

    Many multimodal methods learn shared representations from paired or co-occurring observations, while independently collected datasets lack instance-level correspondence, motivating an interface for their communication.

    Source: Introduction · p. 1

  • Method

    CIIR connects datasets through input-dependent representations in a shared class-indexed address space, preserving dataset-specific native output modules and enabling post-hoc attachment without retraining earlier components.

    Source: Introduction · p. 1, Method · p. 3

  • Results

    CIIR achieved label-free realization, post-hoc SVHN transfer, and exact collision-free CIFAR-100 code recovery, with 90.24% MNIST→SVHN and 82.29% SVHN→MNIST End-to-end Address Transfer Accuracy.

    Source: Discussion · p. 6

  • Takeaways & Limitations

    The results support CIIR as an interface for modular generation and reuse without fixed address vectors or exhaustive paired multimodal supervision.

    Source: Conclusion · p. 7

    CIIR assumes predefined mappings from local class indices to shared addresses, and larger or differently structured class spaces require manually specified multi-address codes.

    Source: Discussion · p. 6

Abstract

from arXiv · show

Multimodal learning commonly assumes paired or synchronized observations, yet independently collected datasets rarely provide such correspondence. We introduce CIIR, a modular encoder--decoder framework that connects heterogeneous datasets through a class-indexed address space while preserving each dataset's native input format and output module. A known local index selects address $k$, and an input-dependent 128-dimensional address representation Z_k(x) is written to that address. To demonstrate CIIR, we learn this address space from handwritten digits, clothing images, spoken-digit spectrograms, and hand signs without synchronized or instance-matched samples. Newly trained label-free receivers recover the requested addresses with 94.7% macro-average Address Realization Accuracy. We then attach the unseen SVHN dataset after freezing the base address space and train only lightweight adapters, achieving 90.2% MNIST>SVHN and 82.3% SVHN>MNIST end-to-end Address Transfer Accuracy without target-side decoder labels. To demonstrate generalizability of CIIR, we reuse the ten addresses as code elements in collision-free arithmetic and hierarchical address codes for all 100 CIFAR-100 classes and evaluate direct arithmetic-code realization through the existing receivers. These results establish CIIR as a compact interface for modular cross-dataset communication and post-hoc dataset attachment without exhaustive paired multimodal supervision.

Introduction

CIIR addresses the difficulty of integrating independently collected datasets by providing a reusable class-indexed interface that does not require instance-level correspondence. It preserves dataset-specific output modules while enabling post-hoc connections through a shared address space.

Motivation and contrast

  • CIIR replaces instance-level correspondence with a reusable interface through which independently collected datasets can communicate.Prior multimodal methods commonly learn shared representations from paired or co-occurring observations.Source: Introduction · p. 1

CIIR framework

  • A known local class index k selects address k, while input-dependent Z_k(x) carries information that receivers decode into dataset-native outputs.The unified encoder uses ten addresses, each with a 128-dimensional representation; equal addresses need not share semantics across datasets.Source: Introduction · p. 1
  • CIIR preserves dataset-specific output modules while connecting MNIST, Fashion-MNIST, FSDD, and Sign Language MNIST through the shared address space.Inputs with different native resolutions meet only in the fixed-dimensional address space.Source: Introduction · p. 1

Evaluation plan

  • CIIR is evaluated through label-free address recovery, bidirectional SVHN transfer using a frozen address space, and arithmetic and hierarchical codes for CIFAR-100.The post-hoc SVHN connection uses lightweight read/write adapters; the CIFAR-100 codes reuse the same ten addresses.Source: Introduction · p. 1

Contributions

  • The stated contributions are reusable cross-dataset addresses, native-format generation without paired samples, input-dependent representations, and reuse for new datasets and larger class spaces.The input-dependent representations distinguish CIIR from fixed class codes.Source: Introduction · pp. 1–2

Related Work

CIIR differs from prior multimodal approaches by assigning known local class indices to shared addresses rather than learning semantic correspondence across datasets. It separates each address from its input-dependent representation, which independently defined modules can read and write.

Multimodal Representation Learning

  • Prior multimodal representation methods typically learn shared semantic structure from corresponding modalities, whereas CIIR shares addresses whose local meanings may differ across datasets.Related approaches include multimodal VAEs and methods for shared–private factorization, contrastive alignment, posterior modeling, and aggregation.Source: Multimodal Representation Learning · p. 2

Unpaired and Extensible Multimodal Learning

  • Unlike methods that infer semantic alignment or use limited pairing, CIIR assigns each dataset’s known local class index to an address without instance-level pairing.Equal addresses can denote different dataset-local classes; any coarse overlap, such as digit identity, is neither inferred from paired examples nor required.Source: Unpaired and Extensible Multimodal… · p. 2

Modular and Composable Representations

  • CIIR distinguishes an assigned address from its input-dependent representation, testing whether independently defined modules can read and write those representations.Unlike fixed label codes, class embeddings, or prototypes, Z_k(x) varies with the input; the later CIFAR-100 study uses predefined address codes without learned semantic composition.Source: Modular and Composable Representations · p. 2

Method

CIIR preserves each dataset’s native output module and connects modules through a shared class-indexed address space, where a local index selects an address but need not share semantics across datasets. The method learns this interface from four datasets, evaluates label-free receivers, then reuses the frozen space for SVHN attachment and CIFAR-100 address codes.

Method

  • CIIR links native-domain decoders through shared addresses, with input-dependent representations written to the address selected by a known local index.An ordered sequence of addresses can also serve as a code for a larger class space.Source: Method · p. 3

Learning the Address Space

  • Four datasets learn the base space: frozen private decoders are connected by a unified encoder that writes the selected address representation without receiving the local index as input.The base datasets use ten local indices each; matching indices provide a common interface, not shared class semantics. Local indices must be available during training and address selection.Source: Learning the Address Space · pp. 3–4

Base Address-Space Evaluation Protocols

  • Label-free receivers are tested on selected 128-dimensional representations and fixed MNIST-derived address prototypes, with independent judges assessing outputs in each native domain.The protocol reports ARA, SSIM, L1, and diversity retention; it also compares input-dependent representations with learned fixed-vector baselines on MNIST.Source: Base Address-Space Evaluation Protocols · p. 4

Post-hoc Attachment of SVHN

  • For post-hoc SVHN attachment, CIIR freezes the base modules and trains dataset-specific routing and write/read adapters without giving target decoders class labels.Each added dataset requires its own adapter pair, but the base address space and previous adapters need not be retrained.Source: Post-hoc Attachment of SVHN · p. 4

CIFAR-100 Address-Code Evaluation

  • CIFAR-100 fine classes are represented by ordered sequences of existing addresses, evaluated for code recovery and collisions, then passed through existing receivers for ARA and CRA.This tests code realization, not CIFAR-100 image generation, classification, or semantic alignment.Source: CIFAR-100 Address-Code Evaluation · p. 5

Results

CIIR achieved high address realization with input-dependent representations, transferred addresses across an unseen SVHN dataset after freezing the base space, and used the ten addresses in collision-free CIFAR-100 codes.

Base Address-Space Evaluation

  • 94.7% macro-average Address Realization Accuracy was achieved by label-free receivers using input-dependent representations, ranging from 89.1% on FSDD to 98.6% on MNIST.These representations also preserved substantially greater reconstruction fidelity and within-class diversity than fixed MNIST-derived address prototypes.Source: Base Address-Space Evaluation · p. 5, Table 1 · p. 5
  • 94.7% macro-average ARA followed address substitution, while the original address was retained in only 0.2% of cases; zero and random representations scored 10.0% and 10.3%.This verifies control by the continuous representation: replacing it with one from another address redirects the output to that substituted address.Source: Base Address-Space Evaluation · p. 5
  • Input-dependent representations increased macro-average SSIM from 0.4927 to 0.5278, reduced L1 error from 0.1341 to 0.1132, and retained 19.2% of real within-class variation.Fixed modality-specific prototypes retained none; because the known address was supplied in both conditions, SSIM, L1, and diversity retention were the primary evidence of observation-specific information.Source: Base Address-Space Evaluation · p. 5, Table 2 · p. 6
  • CIIR achieved 98.6% mean ARA on MNIST, compared with 77.0% for FV-FR and 84.0% for FV-TR, while fixed-vector baselines retained effectively no within-class variation.CIIR also achieved substantially higher SSIM and lower L1 error; jointly fine-tuning the receiver did not restore sample-specific variation.Source: Base Address-Space Evaluation · p. 5, Table 3 · p. 6

Post-hoc SVHN Attachment

  • 90.24% MNIST→SVHN and 82.29% SVHN→MNIST End-to-end Address Transfer Accuracy were achieved after post-hoc attachment.In both directions, observed end-to-end errors arose during source-side address assignment rather than target realization.Source: Post-hoc SVHN Attachment · p. 6
  • The SVHN dataset was attached after freezing the base address space, training only new routing and read/write components; prior-method comparisons were contextual, not strictly metric-identical.The supplied passage describes CIIR as reaching comparable or higher address-preservation rates, but does not provide the comparison values.Source: Post-hoc SVHN Attachment · p. 6

CIFAR-100 Address-Code Evaluation

  • Both CIFAR-100 address-code schemes achieved 100% Round-trip Recovery Accuracy with zero collisions, and the hierarchical scheme achieved 100% Superclass Prefix Consistency.Arithmetic-code ARA/CRA were 100.0/100.0% for MNIST, 80.0/64.0% for Fashion-MNIST, 40.0/16.0% for FSDD, and 10.0/1.0% for Sign Language MNIST; lower non-MNIST scores reflected receiver compatibility, not code-assignment errors.Source: CIFAR-100 Address-Code Evaluation · p. 6

Discussion

CIIR connects independently collected datasets through a shared, class-indexed address space while preserving modality-specific decoders, and supports post-hoc dataset attachment. The discussion also identifies predefined class-to-address mappings as a limitation and proposes hybrid semantic codes as future work.

  • 90.24% MNIST→SVHN and 82.29% SVHN→MNIST End-to-end Address Transfer Accuracy were achieved after post-hoc SVHN attachment, with target-side ARA of 99.18% and 100%, respectively.The address space was frozen, and only lightweight read/write adapters were used.Source: Discussion · p. 6, Table 4 · p. 7
  • Label-free receivers, representation substitutions, original decoder evaluation, and fixed-vector baselines show that learned address representations carry class-readable and input-specific information beyond fixed address vectors.Source: Discussion · p. 6
  • Both arithmetic and hierarchical CIFAR-100 codes achieved exact round-trip recovery with zero collisions, while receiver realization depended on compatibility with the MNIST-derived prototype bank.Table 5 distinguishes deterministic code structure from address and class-code realization.Source: Discussion · p. 6, Table 5 · p. 7
  • CIIR assumes predefined mappings from dataset-local class indices to the shared address vocabulary, so larger or differently structured class spaces require manually specified multi-address codes.Future work should learn hybrid codes combining class-index addresses with reusable semantic components while preserving the modular interface.Source: Discussion · p. 6

Conclusion

CIIR learns input-dependent representations in a shared class-indexed space for communication among independently collected datasets. Label-free realization, representation substitutions, post-hoc SVHN attachment, and CIFAR-100 address coding demonstrate modular generation and reuse without fixed address vectors or exhaustive paired multimodal supervision.

  • CIIR learns input-dependent representations in a shared class-indexed space for communication among independently collected datasets.Source: Conclusion · p. 7
  • Label-free realization, representation substitutions, post-hoc SVHN attachment, and CIFAR-100 address coding demonstrate modular generation and reuse without fixed address vectors or exhaustive paired multimodal supervision.Source: Conclusion · p. 7
  • For systems with frozen, independently maintained, or non-poolable modules, existing identifiers can serve as shared addresses for lightweight cross-module activation rather than full-system retraining.The passage names industrial, medical, robotic, and multi-sensor systems, and generation, retrieval, diagnosis, or control as application contexts.Source: Conclusion · p. 7

Technical Supplement

The supplement specifies CIIR’s network designs, training and dataset protocols, and label-free receiver evaluation, including the scope of the exploratory CIFAR-100 encoder.

Network Architectures

  • The unified encoder accepts inputs with spatial dimensions that vary across datasets and produces ten 128-dimensional address representations; its label-free MNIST receiver takes only one 128-dimensional representation.The encoder also has a ten-way auxiliary index head.Source: Network Architectures · p. 10
  • The SVHN attachment uses a write adapter that maps address prototypes plus Gaussian noise to SVHN latents, and a read adapter that maps selected latents to address logits.All base CIIR components remain frozen during attachment.Source: Network Architectures · p. 10, Training Hyperparameters · p. 11

Training Hyperparameters

  • Training uses Adam; the base address space has 10 addresses of dimension 128, with 500 samples per local index and batch size 64.Unified-encoder updates process all four source modalities and sample two of the remaining three target decoders per source modality; model selection uses mean cross-module SSIM.Source: Training Hyperparameters · p. 10

Base Datasets and Preprocessing

  • The base address space uses MNIST, Fashion-MNIST, FSDD, and Sign Language MNIST, with 500 samples per local class; equal index values do not imply shared semantics.FSDD waveforms are resampled to 8 kHz and converted to 36 × 36 × 1 mel-spectrograms; the other inputs are 28 × 28 × 1 images.Source: Base Datasets and Preprocessing · p. 11

Private Encoders and Dataset-Specific Decoders

  • Each dataset first trains a private encoder–decoder pair, with the decoder conditioned on the private latent and a learned embedding of the local class index; afterward, private encoders are discarded and decoders frozen.Private latents are 128-dimensional and class-index embeddings are 64-dimensional.Source: Private Encoders and Dataset-Specific… · p. 11

Unified Encoder and Address-Space Objective

  • The unified encoder writes the selected address representation for a local index, while its auxiliary index logits contribute only to training; updates use independently sampled observations without instance correspondence.Two of the other three datasets are sampled as targets in each update.Source: Unified Encoder and Address-Space… · p. 11

Label-Free Receiver and Judge Protocol

  • Label-free receivers are trained on selected address representations without local indices, and independent judges use disjoint splits; test samples are unseen by both receiver and judge training.The test split is not guaranteed to be unseen by the original unified encoder; FSDD clips are split before augmentation, which is applied only to training data.Source: Label-Free Receiver and Judge Protocol · p. 12
  • Address-representation substitution evaluates target-address realization and original-address retention alongside the original-representation ARA and zero/random controls.Source: Label-Free Receiver and Judge Protocol · p. 12

Extended SVHN Attachment Method

The method attaches SVHN through its own address encoder and label-free decoder, then uses write and read adapters to route between SVHN and the frozen MNIST receiver.

SVHN Latent Space

  • SVHN uses a separate ten-address encoder and label-free decoder, with the known local digit index selecting its corresponding latent representation.For each digit, the method estimates a latent mean and standard deviation.Source: SVHN Latent Space · p. 12

Write Adapter

  • The write adapter maps a frozen address prototype and private noise into the SVHN latent space, using a sampled diagonal class-conditional target.Separate SVHN judges support adapter optimization and final evaluation.Source: Write Adapter · p. 12

Read Adapter and Transfer Metrics

  • The read adapter predicts an address whose prototype is passed to the label-free MNIST receiver, supporting routing, target-side realization, and end-to-end transfer metrics.Transfer Retention is end-to-end Address Transfer Accuracy divided by Address Routing Accuracy; interventions use the correct, permuted, zero, or random latent prototype.Source: Read Adapter and Transfer Metrics · p. 12

Extended CIFAR-100 Address-Code Method

The section evaluates arithmetic and hierarchical address-code reuse across CIFAR-100’s 100 fine classes and 20 superclasses, without generating CIFAR-100 images or decomposing their semantics.

Extended CIFAR-100 Address-Code Method

  • The CIFAR-100 experiment reuses arithmetic and hierarchical codes for 100 fine-class identities and 20 superclasses, without image generation or semantic decomposition.The code definitions are given in Eqs. (10) and (11) of the main paper.Source: Extended CIFAR-100 Address-Code Method · p. 12
  • Each requested address selects its frozen prototype, which a label-free receiver decodes before an independent judge predicts the realized address.Source: Extended CIFAR-100 Address-Code Method · p. 12
  • Evaluation reports target-side Address Realization Accuracy, Class-Code Realization Accuracy, Round-trip Recovery Accuracy, Collision Count, and Superclass Prefix Consistency.Source: Extended CIFAR-100 Address-Code Method · p. 13

Extended Results

Extended results show that CIIR’s supplied address representations control address realization, while fixed address codes support collision-free CIFAR-100 class coding. Transfer depends on routing compatibility, and generating requested address representations directly from CIFAR images remains difficult.

Base Address-Space Intervention

  • 0.951 original-representation macro ARA in substitution evaluation (not the base receiver’s 0.9468) accompanied near-disappearance of the original address; zero and random representations remained at chance.This separate intervention run, together with label-free receiver evaluation, supports that the supplied continuous address representation controls the realized address rather than a decoder-side address label.Source: Base Address-Space Intervention · p. 13

SVHN Transfer Decomposition and Intervention

  • 99.18% target-side ARA for SVHN and 100% for MNIST followed when the correct address was available; permuting the prototype redirected outputs with 99.02% ARA.Forward transfer was limited by MNIST-to-address routing and reverse transfer by SVHN-to-address routing. Secondary SVHN generation diagnostics were SSIM = 0.8017 and FID = 62.50, against a real–real FID reference of 19.66.Source: SVHN Transfer Decomposition and… · p. 13

CIFAR-100 Address-Code Analysis

  • The arithmetic and hierarchical assignments both achieved 100% Round-trip Recovery Accuracy with zero collisions; the hierarchical assignment additionally achieved 100% Superclass Prefix Consistency.Source: CIFAR-100 Address-Code Analysis · p. 13
  • Direct arithmetic-code execution was exact through the MNIST receiver; Fashion-MNIST realized most requested addresses, while FSDD and Sign Language MNIST were less compatible with MNIST-derived prototypes.The reported differences reflect receiver compatibility rather than ambiguity in the deterministic address codes.Source: CIFAR-100 Address-Code Analysis · p. 13

Exploratory CIFAR-Conditioned Address-Representation Generation

  • 45.4% ARA and 20.0% CRA were the exploratory expansion encoder’s best results, obtained through the Fashion-MNIST receiver.The encoder was conditioned on a grayscale CIFAR-100 image and its known fine-class identity; the result marks this generation task as harder than direct execution of predefined address codes.Source: Exploratory CIFAR-Conditioned… · p. 13, Table 12 · p. 13

Scope of the Address-Code Claim

  • A CIFAR-100 code such as [5, 2] identifies class 52 without claiming semantic composition; CIIR reuses a fixed address vocabulary when class ontology is known and exhaustive cross-modal pairing is unavailable or expensive.The experiment does not introduce one new address for every class.Source: Scope of the Address-Code Claim · pp. 13–14

Related papers

Loading 2610.23151v1…