PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtoSemImage: Image-Valued Prototypes with Deformable Row Alignment for Interpretable Document Classification

Mohammad Zare, Pirooz Shamsinejadbabaki

arXiv:2610.11460v1cs.CVcs.AI

TL;DR

The paper asks whether interpretable class prototypes can preserve document arrangement, which vector prototypes discard. ProtoSemImage represents classes with document-shaped HSV archetypes and matches them by deformable row alignment; image prototypes beat vector prototypes in every paired seed, while a classifier diagnostic locates a major shortfall in distance-based matching.

  • Problem

    Vector prototypes are not readable as whole objects and discard the arrangement of their members, motivating the question of whether a prototype can instead be a full document-shaped image.

    Source: §I · p. 1

  • Method

    ProtoSemImage learns four-channel HSV document images and class archetypes, then classifies through deformable soft-DTW row alignment.

    Source: §I · p. 1, F. Deformable Row Alignment · p. 3

  • Results

    Image prototypes averaged 0.3538 (standard deviation 0.0587) versus 0.2778 (0.0460) for vector prototypes and led in all three seeds by 4.3–11.8 points; on CatRev, softmax reached 62.1% versus 61.8% for TextCNN while image prototypes reached 41.5%.

    Source: D. Results · pp. 5–6

  • Takeaways & Limitations

    On CatRev-Order, image prototypes read arrangement better than their vector counterpart, but HAN still leads by 7.9 points.

    Source: §VII · p. 7

    CatRev-Order holds the bag of words fixed, making it an artificial diagnostic, and whether the advantage carries to naturally order-dependent problems remains open.

    Source: §VII · p. 7

Abstract

from arXiv · show

Prototypes in classification models are almost always vectors, and a vector has no readable form. This paper asks what happens when a prototype is an image. Documents give the question a natural form, because a document can be rendered as a multi-channel image in which every token becomes a pixel, so a class representative can take the same shape and the same channel semantics as the inputs it stands for. ProtoSemImage represents each class by one or more visual archetypes: prototype images in a four-channel HSV space whose channels carry named linguistic factors. A Skip-Gram objective learns that color space end to end through a four-dimensional bottleneck, discourse boundary rows become differentiable typed difference rows, and classification reduces to 2D visual template matching: a deformable row alignment between a document image and the archetype bank, in the spirit of dynamic time warping. Because the match is a spatial pattern comparison rather than a linear readout, the model reports where an input departs from its archetype and along which channel, and a generative head decodes each archetype back into text. The image representation works: it beats an otherwise identical model with vector prototypes in all three paired seeds, by between 4.3 and 11.8 points on a ten-class task. The distance-based matching does not. A diagnostic that keeps the representation fixed and swaps only the classifier recovers the sequence baselines, which locates a 20.6-point shortfall in the matching rather than in the color compression, and a benchmark built so that a pair of documents shares a bag of words and differs only in arrangement confirms the layout-preservation it was designed for. We report both directions, because for a representation whose whole purpose is inspect ability, the failure modes are as informative as the gains.

I. INTRODUCTION

The paper asks whether a prototype can be a full document-shaped image, preserving arrangement and channel meaning rather than reducing a class to an unreadable vector. ProtoSemImage learns image-valued archetypes and tests both their representational contribution and the remaining classifier shortfall.

I. INTRODUCTION

  • A vector prototype cannot be inspected as a whole and discards arrangement, while a full image prototype can share a document’s shape and channel semantics.The paper frames this as the question of whether a class prototype can be a full image.Source: §I · p. 1

A. Text as an Image

  • SemImage already rendered sentences as colored pixel rows, but used an opaque softmax, auxiliary channel-separation losses, and frozen-encoder boundary rows.ProtoSemImage replaces the softmax and fixed boundary computation with archetype matching and learned boundaries.Source: §I · p. 1, A. Text as an Image · p. 2

I. INTRODUCTION

  • ProtoSemImage uses displayed four-channel HSV archetypes, a Skip-Gram-trained bottleneck, differentiable typed boundaries, soft-DTW row alignment, and a generative head that decodes archetypes to token distributions.The channels carry named linguistic factors, and the alignment path supports a localized account of disagreement.Source: §I · pp. 1–2
  • The paper isolates the image representation’s contribution with a vector-prototype counterpart and diagnoses the remaining accuracy shortfall by swapping only the classifier.The comparison is designed to separate prototype shape from other shared model components.Source: §I · p. 2

C. Discourse Structure and Alignment

ProtoSemImage turns documents into images, learns a four-channel color space, and classifies by matching each image against archetypes. Its bottleneck compresses token context into pixels trained to predict neighboring words, while row alignment provides explicit correspondences.

C. Discourse Structure and Alignment

  • ProtoSemImage replaces fixed, separately computed discourse boundaries with learned boundaries and uses soft-DTW along image rows to produce an explicit correspondence path.The path becomes the basis of the explanation, unlike self-attention’s lack of an explicit alignment correspondence.Source: C. Discourse Structure and Alignment · p. 2

III. METHODOLOGY

  • The pipeline renders documents as images, learns a meaningful color space, and classifies by comparison with a bank of archetypes.The overview identifies image construction and archetype comparison as the core stages.Source: §III · p. 2

A. End-to-End Continuous HSV Bottleneck

  • A four-dimensional HSV bottleneck is trained with Skip-Gram so each token’s pixel vector predicts context words, grounding the color space in corpus statistics.The context window stays within a sentence, leaving row structure for the alignment stage.Source: A. End-to-End Continuous HSV Bottleneck · p. 3
  • Only the bottleneck carries each token’s distributional information, while auxiliary losses on pooled hue and saturation keep channel assignments honest.The four pixel values are the representation a person sees.Source: A. End-to-End Continuous HSV Bottleneck · p. 3

B. Dynamic Discourse Boundary Learning

Sentence means from learned pixels provide the basis for differentiable boundary rows whose channels encode distinct types of discourse change.

B. Dynamic Discourse Boundary Learning

C. Document Image Construction

Documents alternate sentence rows with full-width boundary bands, and class archetypes share the document image’s shape and channel semantics. A shared encoder preserves channel-wise interpretability at its first convolution.

C. Document Image Construction

  • Sentence and boundary rows alternate in a document image of height 2M-1, with each boundary vector repeated across the row’s columns.The repeated vector makes a discourse break a full-width horizontal band.Source: C. Document Image Construction · p. 3

D. Visual Archetype Bank

  • Each class has K archetypes because one representative may poorly represent multiple modes; archetypes are seeded from class clusters and anchored against drift.The anchoring penalty is intended to prevent prototypes from becoming constant arrays or noise.Source: D. Visual Archetype Bank · p. 3

E. Shared Encoder with Channel-Grouped Convolution

  • The shared encoder uses one first-convolution group per input channel, preventing early cross-channel mixing that would make channel-wise difference maps unreliable.Features are L2-normalized so squared alignment distances do not swamp the classification objective.Source: E. Shared Encoder with Channel-Grouped… · p. 3

F. Deformable Row Alignment

ProtoSemImage matches documents to archetypes by aligning their ordered rows, then uses channel-specific difference maps to show where and how they diverge.

F. Deformable Row Alignment

  • Soft-DTW aligns document and archetype rows despite differing heights and shifted positions, replacing the non-differentiable hard minimum used for ordered structures.The alignment path is retained to identify corresponding rows.Source: F. Deformable Row Alignment · pp. 3–4
  • Dividing by document sentence count M yields a mean per-aligned-row cost and keeps distances comparable across length buckets without changing class rankings.Within a bucket, M is constant across documents and archetypes have a fixed height; each class distance uses its nearest archetype before a softmax over negated distances.Source: F. Deformable Row Alignment · p. 3

G. Typed Difference Maps

H. Generative Head

A generative head turns pixel vectors into token distributions, while the training objective keeps archetypes class-relevant, separated, decodable, and tied to channel semantics. Training proceeds through distributional pretraining, archetype seeding, and joint optimization.

H. Generative Head

  • The generative head reuses the Skip-Gram decoder direction to produce token distributions, allowing archetypes to be read as words and divergent cells to show expected tokens.The decoder maps pixel vectors toward vocabulary; the head applies this direction to archetypes and document cells where they diverge.Source: H. Generative Head · p. 4

I. Training Objective

  • The objective combines distance-based classification, class-archetype attraction, and separation of distinct archetypes, while also requiring decodability, channel supervision, and proximity to seeds.Channel supervision applies to documents and archetypes alike so archetypes inherit the channel semantics.Source: §I · p. 4

J. Training Procedure

  • Training first updates the image-mapping and decoder components with an unlabeled Skip-Gram objective, then seeds archetypes by clustering each class's images after PCA.The procedure next jointly optimizes all parameters, including archetypes and the learned scale, using the full objective.Source: J. Training Procedure · p. 4
  • During joint optimization, each document is aligned to each archetype with soft-DTW before the objective is computed and parameters are updated.The procedure batches documents by length.Source: J. Training Procedure · p. 4

IV. EXPERIMENTAL SETUP AND RESULTS

The experiments use three collected benchmarks and a constructed order-sensitive test to probe the model's linguistic factors and its sensitivity to arrangement. Dataset properties and document lengths inform the evaluation design.

A. Datasets

  • CatRev-Order pairs documents with identical bags of words but opposite segment order, labeling each by its first topic to test whether arrangement affects classification.CatRev itself cannot test this claim because its labels do not depend on arrangement.Source: A. Datasets · p. 4
  • CatRev combines five product-category topic labels with star-rating polarity, while 20 Newsgroups isolates topic and IMDB isolates sentiment.The latter benchmarks set the sentiment or topic weight to zero, respectively; Table 1 lists statistics for all four benchmarks.Source: A. Datasets · pp. 4–5, Table 1 · p. 5
  • The review corpus uses five Amazon categories and star ratings, matching the two-label structure of MLR but not its collection, so their accuracy figures are not directly comparable.Table 1 is identified as reporting dataset statistics.Source: A. Datasets · p. 5, Table 1 · p. 5
  • CatRev documents have a median of five sentences and range from two to sixteen, motivating one encoder feature per sentence and alignment instead of padding.The image is correspondingly shallow because reviews are short.Source: A. Datasets · p. 5

B. Baselines and Protocol

The comparison uses matched splits and training budgets, including a vector-prototype control that differs from ProtoSemImage only in prototype representation. Implementation choices and the soft-DTW runtime cap constrain the reported accuracy.

B. Baselines and Protocol

  • The vector-prototype control shares ProtoSemImage's embedding, ColorMapper, encoder, boundary construction, and schedule, differing only in representing each class as a point rather than an image.All main-comparison models use identical splits and budgets; TextCNN and a hierarchical attention network are also included.Source: B. Baselines and Protocol · p. 5

C. Implementation

  • The model uses a four-block convolutional encoder, 32-token sentence width, 15-row archetypes, and a linear vocabulary projection for the generative head.The ColorMapper is a two-layer perceptron with hidden width 256.Source: C. Implementation · p. 5
  • Runs use a 10,000-document sample per dataset for six joint epochs after one Skip-Gram pretraining epoch, with documents bucketed by sentence count.Joint optimization uses a learning rate of 1e-3, compared with 2e-4 during Skip-Gram pretraining.Source: C. Implementation · p. 5
  • The training sample is capped below corpus size because soft-DTW dominates runtime, limiting absolute accuracy while preserving internal comparability across Table 2 rows.The same cap applies to every row in the table.Source: C. Implementation · p. 5, Table 2 · p. 5

D. Results

Image prototypes outperform vector prototypes in the paired comparison, but the distance-based matcher remains a source of substantial accuracy loss. A same-representation classifier diagnostic separates that matching shortfall from the color bottleneck, while order-sensitive results show that arrangement can matter.

D. Results

  • 0.3538 versus 0.2778: image prototypes outperform vector prototypes in all three seeds, by 4.3–11.8 points.The reported means and standard deviations are 0.3538 (0.0587) and 0.2778 (0.0460), respectively; the paired comparison carries the claim because single runs vary by several points.Source: D. Results · p. 5
  • The image model trails sequence baselines by 23.2 points, a gap that remains despite its advantage over vector prototypes.The comparison is reported plainly as a limitation of the model's accuracy.Source: D. Results · p. 5
  • On 20 Newsgroups, image and vector prototypes reach 17.8% and 17.0%, respectively, versus 57.2% for the better sequence baseline.The paper identifies twenty-way topic classification as the setting where the four-channel bottleneck does the most damage.Source: D. Results · p. 5
  • On IMDB, image prototypes reach 69.8% versus 68.9% for vector prototypes, while sequence baselines reach 84.9%.The paper describes binary polarity as the easiest task for the bottleneck because its supervision survives compression better than twenty topics do.Source: D. Results · p. 5
  • 62.1% versus 61.8%: on CatRev, the same-bottleneck softmax variant reaches TextCNN-level accuracy, while image prototypes score 41.5%.This diagnostic attributes the CatRev shortfall to distance-based matching rather than color compression.Source: D. Results · pp. 5–6, Table 3 · p. 6
  • On the order-sensitive benchmark, image prototypes beat vector prototypes by 9.3 points; sentence-order permutation drops ProtoSemImage from 83.7% to 40.7%.On CatRev, the same permutation changes accuracy by only 0.3 points, reported as inconclusive.Source: D. Results · p. 6, Figure 3 · p. 6

V. ABLATION STUDY

The ablations identify grouped first convolution as the most consequential component and show that adding archetypes can cause class-representation collapse. The evidence also supports layout-sensitive matching and readable archetypes, while locating the remaining accuracy shortfall in distance-based matching.

V. ABLATION STUDY

  • 13.8 points is the largest CatRev ablation loss, from removing the grouped first convolution.Source: Table 5 · p. 6, §V · p. 6
  • K = 4 rises from 22.0% to 37.9%, and K = 8 from 15.2% to 23.6% with intra-class diversity; two archetypes remains the best setting.Source: §V · p. 7

A. Reading an Archetype

  • An archetype's row-wise hue, saturation, and value patterns let readers inspect typical discourse progression, unlike a vector prototype.Source: A. Reading an Archetype · p. 7

B. Locating Disagreement

  • Aligned difference maps locate channel-specific departures, such as matching topic hue alongside a contiguous saturation divergence.Source: B. Locating Disagreement · p. 7

C. What the Evidence Supports, and What It Does Not

VII. LIMITATIONS AND OPEN QUESTIONS

The limitations section identifies four constraints, including provisional performance magnitudes, an artificial order-only benchmark, fragile archetype-count behavior, and a review corpus that differs from earlier work. The conclusion reports that image prototypes beat vector prototypes in every tested seed and proposes several next steps.

VII. LIMITATIONS AND OPEN QUESTIONS

  • The 10,000-document, six-epoch training cap makes absolute results provisional, although the authors expect model rankings to survive it.The cap reflects soft-DTW recursion cost; a longer schedule may narrow the gap to sequence baselines.Source: §VII · p. 7
  • CatRev-Order isolates arrangement by fixing the bag of words, but real documents rarely differ only in the order of their halves.HAN still leads ProtoSemImage by 7.9 points on this benchmark, leaving performance on naturally order-dependent tasks open.Source: §VII · p. 7
  • With four or more archetypes, alignment can be satisfied by a close representative without discrimination, and performance collapses toward chance.The authors say one archetype is too few, two works, and the mechanism remains poorly understood.Source: §VII · pp. 7–8
  • The review corpus shares the earlier benchmark’s two-label structure but comes from a different source, so published accuracies are not comparable baselines.Source: §VII · p. 8

VIII. CONCLUSION AND FUTURE WORK

  • Image prototypes beat vector prototypes in every tested seed on the benchmark where two factors must be held apart.The authors identify distance-based matching as what keeps the model behind sequence baselines.Source: §VIII · p. 8
  • Proposed next steps include cross-attention for alignment and a naturally order-sensitive benchmark to test layout modeling.The authors also propose studying archetype communication, class growth, vision-transformer encoders, and archetype-generated training examples.Source: §VIII · p. 8
Loading 2610.11460v1…