PaperlayerView exact revision on arXiv

Source-linked AI summary

Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach

Che Shen, Junwei Su, Lingpeng Kong, Chuan Wu

arXiv:2608.15224v1cs.LG

TL;DR

Post-hoc evaluation needs representations that distinguish task-relevant labels beyond broad semantic similarity in frozen text embeddings. PGCL learns prototype-guided task-adapted representations without updating the encoder, improving over raw frozen embeddings across three proxy tasks and showing its largest direct-baseline margin on AmazonReviews.

  • Problem

    Frozen embeddings can place text examples with different task labels near each other because broad semantic similarity does not capture task-specific boundaries.

  • Method

    PGCL maps frozen embeddings into a compact task-specific space using learnable class prototypes, supervised contrastive learning, prototype margins, and stream regularization.

  • Results

    PGCL improves over raw frozen embeddings on GoEmotions, AmazonReviews, and ToxicComment, with its largest positive direct-baseline margin on AmazonReviews.

  • Takeaways & Limitations

    The results support PGCL as a lightweight frozen-embedding method for post-hoc principle-evaluation proxy tasks, with prototype-margin theory limited to sufficient conditions in the mapping space.

  • Takeaways & Limitations

    PGCL is not a replacement for supervised encoder adaptation, since LoRA and full fine-tuning obtain higher scores on GoEmotions and ToxicComment.

Abstract

from arXiv · show

Reliable post-hoc evaluation asks whether already generated text satisfies a target criterion after generation. In this paper we study a focused frozen-embedding setting using principle-evaluation proxy tasks: toxicity detection, fine-grained emotion categorization, and ordinal review rating. General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples in overlapping regions of the representation space. We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings. The module combines a semantic stream, a prototype-anchor attention stream, supervised contrastive learning, offset-based prototype-margin regularization, and stream regularization to produce a compact task-adapted representation without updating the base encoder. Controlled experiments show that PGCL improves over raw frozen embeddings on all three datasets and gives the clearest direct-baseline margin on AmazonReviews, while remaining competitive with strong direct frozen metric-learning baselines on GoEmotions and ToxicComment. We also add supervised residual-adapter, encoder-LoRA, full fine-tuning, objective ablation, sensitivity, and fully logged few-shot LLM protocol diagnostics to define the boundary of the claim. The theoretical analysis is revised as a sufficient-condition account for prototype-margin behavior under explicit assumptions in the prototype-mapping space, rather than as an unconditional training or final-embedding separation guarantee.

1 Introduction

The introduction frames post-hoc evaluation as a task-relevant representation problem for lightweight toxicity, emotion, and review-rating proxies. It proposes PGCL, which adapts frozen embeddings into a structured task-specific subspace using learnable prototypes and a combined training objective without updating the base encoder.

  • Motivation: Post-hoc evaluation determines whether generated text satisfies target criteria in applications including moderation, safety auditing, rating prediction, and output monitoring.
  • Scope: The study narrows this problem to lightweight classification or rating proxies covering toxicity detection, selected emotion categories, and ordinal review ratings.
  • Limitations of Existing Methods: General-purpose embeddings can smooth over subtle task-critical distinctions, causing semantically similar texts with different evaluation targets to remain entangled.
  • Proposed Method: PGCL maps frozen base embeddings into a low-dimensional, task-specific subspace through an attention-based transformation with learnable prototype anchors as class-level references.
  • Contributions: The modular objective combines supervised contrastive learning, offset-based prototype margins, and stream regularization while leaving the underlying encoder unchanged.

2 Related Work

Prior work motivates PGCL through task-specific subspace learning, supervised metric learning, and prototype-based classification, while distinguishing it from encoder-updating PEFT and generation-time principle alignment. The section frames PGCL as a post-hoc method for adapting already generated embeddings without changing the base encoder.

  • General-Purpose Text Embeddings and Subspace Learning: General-purpose embeddings capture broad semantic context but can conflate subtle task-specific distinctions, motivating task-relevant subspace learning and projections.
  • Parameter-Efficient Fine-Tuning (PEFT): PEFT methods such as LoRA can adapt pretrained models when raw text, labels, and compute are available, but they update the encoder or add trainable modules inside its computation.PGCL instead addresses a post-hoc setting with already generated embeddings and a fixed base encoder.
  • Principle Alignment and LLM-Based Evaluation: Principle-alignment work often constrains generation through RLHF or Constitutional AI, whereas evaluating already generated text is a distinct post-hoc problem.

3 Methodology

PGCL adapts frozen embeddings through a dual-stream, prototype-guided projection onto a low-dimensional hypersphere, combining semantic preservation with label-anchor alignment. Its composite objective regularizes clustering, prototype margins, stream complementarity, and ordinal intensity, while theory gives sufficient—not unconditional—geometric guarantees.

  • Motivation: Frozen semantic embeddings can place lexically similar texts with different labels nearby because broad semantic distance does not encode task-specific boundaries.
  • Prototype-guided projection: PGCL learns a lightweight mapping and class prototypes in a lower-dimensional hypersphere without updating the base encoder.The mapping is fθ: R^D → S^(d−1), with d ≪ D.
  • Ordinal extension: For ordinal tasks, ordered prototype initialization and magnitude regularization encourage intensity structure before final hyperspherical normalization.
  • Dual-stream architecture: The dual-stream extractor preserves contextual semantics through an MLP while dynamically aligning inputs with optimizable class-level prototype anchors.Attention computes soft alignment coefficients, and the fused vector is normalized onto the target hypersphere.
  • Geometric objective: The composite objective combines supervised contrastive clustering, offset-based prototype-margin regularization, orthogonality between streams, and optional magnitude regularization for ordinal progressions.The Offset Loss constrains prototype mappings relative to their target anchors, including an intra-class safe-radius penalty.
  • Theoretical guarantees: The theoretical results provide sufficient-condition bounds: inter-prototype margins, bounded intra-class deviation, and small optimization error imply structured prototype-space separation and potentially positive Silhouette Score.The account does not claim unconditional training or final-embedding separation guarantees.

4 Experiment

The experiments evaluate PGCL’s downstream performance, geometry, efficiency, component sensitivity, baseline comparisons, and few-shot LLM behavior under a frozen-embedding protocol. Across three principle-evaluation proxy tasks, PGCL improves task-adapted representations while showing task-dependent limitations and competitive baselines.

  • Experimental design: The evaluation uses confusable emotion, ordinal rating, and imbalanced toxicity proxy tasks to test downstream probes and empirical geometry.The study also examines direct metric-learning baselines, objective sensitivity, and a fully specified few-shot LLM protocol.
  • Downstream results: 0.776 ± 0.032 LR F1 follows 0.726 ± 0.031 on the emotion diagnostic when optimized embeddings replace raw embeddings.Standard classifiers on raw embeddings plateau around Overall F1 0.72-0.73.
  • Downstream results: PGCL lowers MAE, increases QWK, and reduces severe errors on AmazonReviews versus raw embeddings and the strongest direct SupCon baseline.The optimized embeddings also improve AmazonReviews regression metrics such as MSE and RMSE.
  • Downstream results: PGCL improves Average F1 and Minority F1 across most ToxicComment classifiers, although center loss remains highly competitive.This result concerns recovery of minority-class signal under strong class imbalance.
  • Baseline comparisons: PGCL has its largest direct-baseline margin on AmazonReviews, while paired bootstrap intervals support positive differences on AmazonReviews and GoEmotions but overlap zero for ToxicComment center loss.A supervised residual adapter favors GoEmotions, whereas PGCL improves over it on AmazonReviews and ToxicComment.
  • Geometry and diagnostics: PGCL yields a 0.358 Within/Between Ratio and 0.0975 GQI on AmazonReviews, compared with SimCSE’s ratio above 1.0 and GQI of -0.0005.t-SNE plots qualitatively show task-specific organization and ordered Amazon clusters, while retraining and architecture diagnostics indicate task-dependent component usefulness.
  • Deployment efficiency: Mapping 1024-dimensional raw vectors to a 64-dimensional task-specific subspace can reduce downstream training and inference cost for lightweight classifiers.The representation is presented as a reusable intermediate layer for the frozen-embedding post-hoc pipeline.

5 Conclusion and Future Work · A Implementation Details

The conclusion frames PGCL as a lightweight frozen-embedding method with bounded empirical support across three proxy tasks, while distinguishing it from supervised encoder adaptation. It identifies dataset scope, objective selection, broader-impact risks, and implementation details as priorities for future work.

  • 5 Conclusion and Future Work: PGCL learns a lightweight prototype-guided mapping that regularizes frozen embeddings into a compact task-specific representation without updating the base encoder.Its theory is presented as a sufficient-condition analysis in prototype-mapping space, assuming bounded within-class spread, sufficient inter-prototype separation, and small optimization residuals.
  • 5 Conclusion and Future Work: PGCL improves over raw frozen embeddings on GoEmotions, AmazonReviews, and ToxicComment.Against direct frozen metric-learning baselines, it has the largest positive margin on AmazonReviews and a smaller positive margin on GoEmotions.
  • 5 Conclusion and Future Work: PGCL remains competitive with center loss on ToxicComment, where the bootstrap confidence interval for the comparison overlaps zero.This result bounds the strength of the empirical claim against direct frozen metric-learning baselines.
  • 5 Conclusion and Future Work: LoRA and full fine-tuning can obtain higher scores than PGCL on GoEmotions and ToxicComment, so PGCL is not a replacement for supervised encoder adaptation.The method should instead be understood as a frozen-embedding post-hoc approach.
  • 5 Conclusion and Future Work: The evaluated datasets are proxies for emotion distinctions, ordinal review intensity, and toxicity detection rather than general alignment, fairness, helpfulness, or human-value benchmarks.Future work should study validation-driven objective selection and broader multi-seed checks because ablated component effects are task-dependent.
  • 5 Conclusion and Future Work: Content-moderation deployment risks false negatives, false positives, and prototype geometry that inherits or concentrates training-label bias.These risks are especially relevant for minority dialects, non-standard expressions, protected-attribute language, and high-throughput evaluation using flawed principle definitions.
  • A Implementation Details: The implementation appendix documents the neural principle extractor, prototype-anchor initialization, training hyperparameters, loss configurations, computational complexity, and task-specific subspace dimension.These details support the methods referenced in the main paper.

A.1 Neural Network Architecture Details · A.2 Prototype-Anchor Initialization Details

The neural principle extractor maps 1024-dimensional input embeddings to 64-dimensional principle-aware representations using a shared MLP and prototype-anchor attention. Prototype anchors use task-specific initialization: randomized separated points for classification and geodesic angular progression for ordinal ratings.

  • A.1 Neural Network Architecture Details: The neural principle extractor maps input text embeddings Xi ∈R1024 to d = 64 dimensional principle-aware representations ei.Its architecture combines a shared Multi-Layer Perceptron (MLP) with an attention mechanism.
  • A.1 Neural Network Architecture Details: The shared MLP computes the semantic basis si using two fully connected layers with LeakyReLU activations, Batch Normalization, and Dropout.Dropout follows each hidden layer for regularization.
  • A.1 Neural Network Architecture Details: The first hidden layer projects R512 →R256, followed by LeakyReLU, Batch Norm, and Dropout.This layer is the first stage of the shared MLP.
  • A.1 Neural Network Architecture Details: The second hidden layer projects R256 →Rd with d = 64, using LeakyReLU, Batch Norm, and Dropout, and outputs si.The Dropout rate throughout the MLP is 0.2.
  • A.1 Neural Network Architecture Details: The attention mechanism linearly transforms input embeddings and prototype anchors into queries, keys, and values for scaled dot-product attention.A scalar α initialized to 0.05 weights the semantic basis and prototype-anchor mapping in final fusion.
  • A.2 Prototype-Anchor Initialization Details: The K learnable prototype anchors ck ∈Rd are initialized according to task type to encourage structured learning.The initialization strategy differs between classification and ordinal regression tasks.
  • A.2 Prototype-Anchor Initialization Details: For classification tasks, anchors are randomized on the unit hypersphere with a targeted minimum pairwise Euclidean distance of 2, without guaranteeing final orthogonality or separation.This applies to the GoEmotions five-emotion subset, Amazon Reviews classification, and ToxicComment.
  • A.2 Prototype-Anchor Initialization Details: For ordinal ratings, anchors follow sequential angular progression along a geodesic path, initializing distant ratings farther apart than adjacent ratings.This supplies an ordinal inductive bias rather than proving strict global ordering in learned representations.

A.3 Training and Loss Function Details

The appendix specifies the composite training objective and configurations for the neural principle extractor. Ltotal balances supervised contrastive, offset, orthogonality, and ordinal magnitude components under end-to-end optimization.

  • Training configuration: Training uses AdamW with learning rate 1e-4, weight decay 1e-5, batch size 128, and a 100-epoch cap with early stopping.Cosine Annealing or ReduceLROnPlateau adjusts the learning rate based on validation performance.
  • Contrastive loss: SupCon pulls together same-label samples and repels negatives using cosine similarity with temperature τ = 0.1.The positive set contains batch samples sharing the anchor label, while negatives are the remaining samples.
  • Offset loss: Offset loss combines intra-class and inter-class penalties, with margins δintra and δinter typically ranging from [0.1, 0.5].λinclass and λcrossclass control the relative penalty strengths.
  • Auxiliary regularization: The ordinal magnitude loss enforces natural ordering through a learnable scaling factor λscale and numerical label intensity mapping I(yi).The orthogonality loss promotes soft orthogonality between semantic basis si and prototype mapping mi, with δorthogonal annealed from 0.5 to 0.05.
  • Composite objective: Ltotal combines supervised contrastive, offset, orthogonality, and magnitude losses, with the final objective optimized by balancing these components.The magnitude term applies exclusively to ordinal regression tasks.

A.4 Computational Complexity Analysis · A.5 Justification for Principle Subspace Dimension (d = 64)

The framework has batch training complexity dominated by extractor passes and loss calculations, while inference scales linearly with input dimension and principle count. The d = 64 subspace reduces downstream dimensionality from 1024 dimensions while retaining task-relevant information according to preliminary validation experiments.

  • A.4 Computational Complexity Analysis: The shared MLP costs O(D · h1 + h1 · h2 + h2 · d) per sample with D = 1024, d = 64, h1 = 512, and h2 = 256.The attention mechanism additionally includes projection, attention-score, and weighted-summation operations involving K principles.
  • A.4 Computational Complexity Analysis: Contrastive loss costs O(B2 · d) in standard form, often optimized to O(B2) or O(B · P · d) with P positives per sample.Offset loss costs O(B · K · d), while orthogonality, classification, and magnitude losses cost O(B · d) or O(B).
  • A.4 Computational Complexity Analysis: Training complexity is roughly O(B · (D · hmax + K · d) + B2 · d) in the worst case, or O(B · (D · hmax + K · d)) with typical optimizations.The analysis describes this as comparable to other deep metric-learning or contrastive-learning frameworks.
  • A.4 Computational Complexity Analysis: Inference costs O(D · h1 + h1 · h2 + h2 · d + K · d) per sample and is linear with respect to D and K.This single forward pass produces the optimized embedding efficiently.
  • A.4 Computational Complexity Analysis: Using d = 64 instead of D = 1024 reduces the computational cost of downstream classifiers and contributes to faster downstream training and inference.The passage identifies this as a practical benefit for many standard classifiers in the reported pipeline.
  • A.5 Justification for Principle Subspace Dimension (d = 64): Preliminary validation experiments compared output dimensions including 32, 64, 128, and 256, with d = 64 providing a practical balance.The stated balance combines material dimensionality reduction from 1024 dimensions with retention of enough task-relevant information for downstream probes and reduced computational cost.

A.6 Compute Resources

Experiments used a four-GPU RTX 4090 system with 128GB RAM and an Intel Xeon Platinum 8336C CPU on Ubuntu 24.04 LTS. Neural Principle Extractor training was lightweight, typically taking 3–15 minutes on one RTX 4090.

  • Experiments ran on four NVIDIA RTX 4090 GPUs with 24GB VRAM each, 128GB system RAM, an Intel Xeon Platinum 8336C CPU, and Ubuntu 24.04 LTS.
  • 3 to 15 minutes was the typical full training time for the Neural Principle Extractor on a single RTX 4090, depending on dataset size and complexity.Multiple GPUs can further reduce training time, while inference requires one forward pass per sample.

B Sufficient-Condition Proofs … C.2 Amazon Reviews Per-Rating Performance

The appendix reframes the geometric theorems as sufficient-condition results in prototype-mapping space, with final fused-embedding behavior left to empirical evaluation. It also supplies detailed per-principle GoEmotions and per-rating Amazon Reviews diagnostics alongside the broader experimental results.

  • B Sufficient-Condition Proofs: Theorem 1 assumes bounded within-class distance and sufficient prototype-anchor separation; these assumptions interpret the objective’s geometric bias but are not automatic consequences of gradient-based training.The theorem concerns prototype-mapping space m_i rather than an unconditional guarantee from training.
  • B Sufficient-Condition Proofs: If δinter > 2δintra + 2ε, the derived lower bound on inter-sample distance is positive.The condition follows from combining within-class radius and inter-anchor separation bounds with residual optimization error ε.
  • B Sufficient-Condition Proofs: The theorem does not directly guarantee separation in every final fused normalized embedding because semantic, prototype-anchor, fusion-gate, and normalization operations can reshape the prototype-mapping geometry.Final-embedding behavior is therefore assessed through downstream and geometry diagnostics.
  • B.1 Proof of Theorem 2: If δinter > 4δintra + 2ε, the minimum inter-cluster distance exceeds the maximum intra-cluster diameter and the Silhouette Score is lower-bounded.The Within/Between Ratio is upper-bounded by the maximum intra-cluster distance divided by the minimum inter-cluster distance.
  • B.1 Proof of Theorem 2: Theorem 2’s margin condition is a sufficient design constraint, while achieved geometry depends on architecture capacity, optimization, class imbalance, and hyperparameter choices.The appendix motivates reporting both geometry diagnostics and experimental results because the condition is not itself an empirical guarantee.
  • C Detailed Experimental Results: The appendix provides supplementary detailed results for the experiments presented in Section 4.These detailed results include per-principle and per-rating analyses in the following subsections.
  • C.1 GoEmotions Per-Principle Performance: GoEmotions per-principle results show the largest improvements for challenging, semantically similar principles such as Disappointment and Sadness, while Gratitude shows more modest gains.The analysis covers five selected emotion principles and reports Mean ± Standard Deviation F1 scores.
  • C.2 Amazon Reviews Per-Rating Performance: Amazon Reviews diagnostics report per-rating F1 for star ratings 1–5 and per-rating Mean Squared Error for ordinal regression using raw and optimized embeddings.The per-rating analyses supplement the summarized classification and ordinal regression results in Section 4.2.

C.3 Amazon Reviews Classification Results … D.6 Broader Impact Statement

The appendices expand Amazon Reviews classification diagnostics, document reproducible evaluation protocols and statistical checks, and delimit the interpretation of LLM comparisons. They also identify operational risks from bias, thresholding, and automated principle evaluation.

  • C.3 Amazon Reviews Classification Results: Amazon Reviews treats 1–5 star ratings as discrete categories, with optimized embeddings improving F1 across most ratings, especially the challenging 2-, 3-, and 4-star classes.The appendix supplements the main text’s ordinal-regression analysis with overall and per-rating classification diagnostics.
  • D Additional Experimental Details: The additional experiments define asset licensing, few-shot LLM procedures, split auditing, statistical comparisons, prompt sensitivity, and threshold diagnostics around the core frozen-embedding evaluation.These appendices position the results as controlled, reproducible diagnostics rather than unrestricted claims.
  • D.1 Details on Used Assets and Licenses: The study states that its external datasets, embedding models, and language models are used according to their respective licenses and terms, with researchers directed to original documentation where needed.The listed assets include GoEmotions, Amazon Reviews, Toxic Comment, Jina Embeddings v3, and a locally hosted Qwen-family model.
  • D.2 Few-shot LLM Protocol Details: The few-shot LLM diagnostic uses deterministic local Qwen inference, fixed demonstrations and label sets, leakage exclusions for AmazonReviews, and telemetry reporting with $0.00 API cost.The protocol fixes temperature=0.0, top_p=1.0, max_tokens=8, and random seed 42; invalid rate is 0.000 in the audited run.
  • D.3 Split Protocol and Leakage Audit: The leakage audit verifies saved embedding splits before evaluation, restricts PGCL training to training labels, and reserves test labels for final metrics after all selections are fixed.Validation labels support early stopping, checkpoint and representation selection, and threshold selection where applicable.
  • D.4 Additional Statistical and Geometry Diagnostics: Paired bootstrap diagnostics find positive PGCL differences over raw embeddings on all datasets, while ToxicComment’s 0.0009 difference versus center loss has a 95% CI of [-0.0012, 0.0029].Against the strongest direct metric-learning baseline, AmazonReviews differs by 0.0904 and GoEmotions by 0.0256; ToxicComment is therefore described as competitive rather than statistically significant.
  • D.6 Broader Impact Statement: The broader-impact discussion warns that false positives, false negatives, annotator or platform bias, and prototype-guided anchors may operationally persist harms affecting minority dialects and protected groups.It recommends human oversight, validation-set threshold calibration, periodic bias audits, and channels for contesting or reviewing automated decisions.

D.7 Limitations and Scope · D.8 Data Distribution

The paper bounds its claims to frozen-embedding post-hoc evaluation on three principle-evaluation proxy tasks, with theory, ablations, and few-shot LLM comparisons interpreted under explicit limitations. Leakage-controlled evaluation uses fixed embedding splits with reserved test labels and task-specific train/validation/test distributions.

  • D.7 Limitations and Scope: The evaluated tasks test representation quality but do not directly benchmark general alignment, safety, fairness, helpfulness, or human-value adherence.They comprise toxicity detection, selected emotion categorization, and ordinal review rating.
  • D.7 Limitations and Scope: PGCL’s primary claim concerns improving a fixed embedding store with a lightweight task-specific module in a frozen-embedding post-hoc setting.LoRA and full supervised fine-tuning use a different encoder-updating deployment setting.
  • D.7 Limitations and Scope: Encoder-updating methods can obtain higher scores than PGCL on GoEmotions and ToxicComment, so they are treated as a separate setting rather than direct frozen-embedding baselines.The distinction follows the different deployment settings of encoder-updating methods and PGCL.
  • D.7 Limitations and Scope: The theoretical analysis is limited to sufficient conditions for prototype-margin separation in the prototype-mapping space.This limitation narrows the interpretation of the theoretical account.
  • D.7 Limitations and Scope: The full composite objective is a configurable design, while objective-ablation and sensitivity results do not establish that every loss term is uniformly necessary across retraining configurations.The manuscript motivates validation-driven objective selection and broader multi-seed checks as future work.
  • D.7 Limitations and Scope: The few-shot LLM comparison supports only a bounded efficiency and performance comparison under the reported fixed local protocol.Different model families, prompting strategies, calibration procedures, and API settings may produce different outcomes; direct alignment benchmarks and multimodal spaces remain future work.
  • D.8 Data Distribution: GoEmotions uses 5126/906/881, AmazonReviews 2271/487/487, and ToxicComment 44256/15073/22609 examples for train/validation/test in fixed embedding arrays.The arrays were recovered from project artifacts and checked against expected shapes and label distributions.
  • D.8 Data Distribution: Test labels are reserved for final metric computation in the leakage-controlled evaluation protocol.The audit uses fixed train/validation/test embedding arrays and verifies their expected shapes and label distributions.
Loading 2608.15224v1…