Source-linked AI summary
Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning
Kishor Kumar Bhaumik, Nicolas Roque dos Santos, Neil Shah, Jia Chen, Evangelos E. Papalexakis
TL;DR
Self-supervised fine-tuning commonly relies on computationally costly multiview negatives or auxiliary graphs and networks. SILK instead trains on a simplified silhouette loss using corpus-level k-means centroids, and reports lower training cost with competitive linear-probe results.
Problem
Contrastive self-supervised fine-tuning uses multiple views and in-batch negatives, while negative-free approaches require auxiliary mechanisms; whether a CVI can guide text fine-tuning without either remains an open question.
Source: §1 · p. 1Method
SILK clusters unlabeled corpus embeddings and regresses a simplified-silhouette CVI toward a target, comparing each document with k centroids using one view and no negative pairs.
Source: §1 · pp. 1–2, §2.2 · p. 2Results
SILK stays competitive with the best baselines on three linear-probe tasks, trains 1.46× faster per epoch than SCD, and uses 45.4% less memory than SimCSE.
Source: §5 · p. 4, §3.3 · p. 4, §3.2 · p. 4Takeaways & Limitations
These results suggest cluster validation indices can be used to learn representations as well as to judge them.
Source: §5 · p. 4Experiments cluster a corpus of about 18.8k documents; larger-corpus mini-batch or streaming k-means would make centroids approximate, and whether gains survive is untested.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.
1. INTRODUCTION AND RELATED WORK
Existing self-supervised approaches use multiview negatives or auxiliary collapse-prevention mechanisms, while prior clustering-guided methods face graph or full-corpus computation constraints. SILK uses a cluster validation index to guide text representations, with one view per document and no negative pairs.
1. INTRODUCTION AND RELATED WORK
- Prior clustering-guided methods rely on fixed graphs or full-corpus computation, mismatching independent text sequences and Transformer memory limits.The paper therefore investigates whether CVI-based learning transfers to text.Source: §1 · p. 1
- SILK clusters unlabeled corpus embeddings and optimizes a simplified silhouette using distances to k centroids, reducing distance work from O(n2) to O(nk).The corpus is clustered once per epoch, with gradients accumulated across batches.Source: §1 · pp. 1–2
- The silhouette objective captures compactness and separation preferences without pairs, using one view per document and no augmented views or negative pairs.Contrastive methods construct two views and compare them against in-batch negatives.Source: §1 · pp. 1–2
- Clustering once per epoch and accumulating batch gradients keeps peak activation memory batch-scaled while providing a consistent clustering for updates.The method compares documents with corpus-level centroids rather than other documents.Source: §1 · p. 2
- Per-epoch time and peak GPU memory fall below every evaluated baseline, while linear-probe accuracy remains competitive with the strongest.The supplied contribution statement gives no numerical values for these comparisons.Source: §1 · p. 2
2. PROPOSED METHOD
SILK encodes and projects documents, clusters the resulting embeddings, and backpropagates a simplified-silhouette CVI loss to fine-tune BERT. Its objective uses centroid distances, and its training scheme separates corpus-level clustering from batchwise gradient computation.
2. PROPOSED METHOD
- SILK encodes documents with BERT, projects them, clusters the embeddings with k-means, and backpropagates the CVI loss to fine-tune the encoder.The method is shown in Figure 1.Source: §2 · p. 2
2.1. Setup
- Documents are BERT-base [CLS] outputs projected by a two-layer MLP into normalized embeddings; the projector is discarded at evaluation.Evaluation uses the 768-dimensional encoder features.Source: §2.1 · p. 2
2.2. Silhouette-Guided Objective
- The simplified silhouette replaces document-to-document distances with own-centroid and nearest-rival-centroid distances, yielding O(nkd) time and O(nk) memory.Regressing the score toward a bounded target avoids maximizing silhouette to 1, which can collapse clusters and erase useful within-cluster variation.Source: §2.2 · pp. 2–3
- The score’s gradient pulls a document toward its assigned centroid and pushes it from the nearest competing centroid, with stronger repulsion for documents caught between clusters.The objective contrasts documents against corpus-level summaries rather than individual documents, so it requires no negative sampling.Source: §2.2 · p. 3
2.3. Clustering under a Bounded Computation Graph
- An inference pass refreshes corpus-level centroids once per epoch; a second pass computes gradients batchwise against frozen centroids, keeping peak activation memory proportional to batch size.The scheme uses one optimizer step per epoch and pays for an extra forward pass.Source: §2.3 · p. 3
3. EXPERIMENTS
In frozen-feature linear probing, SILK is competitive across three downstream tasks and reduces training time and memory versus the evaluated baselines. Its best cluster count is roughly an order of magnitude above the downstream class count on all three tasks.
3.2. Downstream Accuracy
- 1.83 points on TREC-6 and 1.34 on Snippets over the strongest baseline, but 0.92 behind SimCSE on StackOverflow.These are linear-probe test-accuracy comparisons.Source: Table 2 · p. 3, §3.2 · p. 4
3.1. Setup
- All methods were fine-tuned on 20 Newsgroups, then evaluated by fitting a linear classifier to frozen encoder features on TREC-6, Snippets, and StackOverflow.The downstream datasets differ from the fine-tuning domain.Source: §3.1 · p. 3
3.3. Training Cost
- 1.46× faster per epoch than SCD and 45.4% less memory than SimCSE, despite an extra clustering forward pass each epoch.The comparison reports per-epoch time and peak memory under matched conditions.Source: Table 3 · p. 4, §3.3 · p. 4
3.4. Number of Clusters
- The best k is roughly an order of magnitude above the downstream class count on all three tasks, with gains stopping and curves turning down beyond the peak.Table 2 uses the k at each validation-split accuracy peak.Source: Figure 2 · p. 4, §3.4 · p. 4
4. LIMITATIONS AND FUTURE WORK
The experiments cluster a small corpus, and the authors identify scaling to approximate clustering and choosing k per corpus as open practical boundaries.
4. LIMITATIONS AND FUTURE WORK
- Experiments used a small clustering corpus of approximately 18.8k documents; whether gains survive approximate centroids from mini-batch or streaming k-means remains to be tested.The authors identify this as the first test for scaling to much larger corpora.Source: §4 · p. 4
- The method depends on k, which must be chosen separately for each corpus.Source: §4 · p. 4
- Applying the recipe to images and other modalities is proposed as future work, not evaluated here.Source: §4 · p. 4
5. CONCLUSION
SILK fine-tunes a language encoder with a Cluster Validation Index loss, using epoch-level clustering and accumulated batch gradients. It trains faster and uses less memory than the evaluated baselines while remaining competitive on linear-probe accuracy.
5. CONCLUSION
- SILK fine-tunes a language encoder using a Cluster Validation Index as its training loss.Source: §5 · p. 4
- Clustering once per epoch and accumulating gradients across batches keeps memory bounded while updates use one consistent clustering.Source: §5 · p. 4
- SILK trains faster and uses less memory than every evaluated baseline while staying competitive on linear-probe accuracy.The authors suggest Cluster Validation Indices can be used to learn representations, as well as judge them.Source: §5 · p. 4