PaperlayerView exact revision on arXiv

Source-linked AI summary

A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation

Lorenzo Zangari, Davide Picca

arXiv:2609.26527v1cs.CLcs.AI

TL;DR

Whole-text similarity may miss how texts frame the same expression through contextual meaning and salient discourse references. The paper encodes those profiles with Semiotic Embeddings and compares them using Semiotic Fidelity and Semiotic Coverage. Experiments report lower coverage than fidelity and strongest alignment with human-curated data at low sampling temperatures, with higher temperatures reducing that alignment.

  • Problem

    Whole-text lexical or similarity comparisons may fail to detect differences in how two texts frame the same expression.

    Source: §1 · p. 1

  • Method

    The framework encodes contextual meaning and salient discourse references in Semiotic Embeddings, then compares profiles with Semiotic Fidelity and Semiotic Coverage.

    Source: §1 · p. 2

  • Results

    Coverage is typically lower than fidelity, and lower sampling temperatures yield the strongest alignment with human-curated data while higher temperatures reduce it.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The scores provide a computational basis for assessing how closely two agents’ textual outputs are semiotically aligned.

    Source: §6 · p. 9

    The controlled diagnostic and LLM-judge analysis do not replace direct validation against human judgments.

    Source: §7 · p. 9

Abstract

from arXiv · show

When two texts describe the same expression, standard metrics based on lexical overlap or whole-text similarity may fail to detect meaningful differences in how that expression is framed. We propose a framework to evaluate semiotic alignment between texts, where a semiotic profile encompasses both the contextual meaning and the discourse references made salient by a text. Our approach yields two scores, Semiotic Fidelity and Semiotic Coverage, estimating how much of one text's profile is supported by the other and how much of the other's profile it recovers. Experiments show that coverage is typically lower than fidelity, and that alignment between LLMs and human-curated data is highest at low sampling temperatures, while higher temperatures reduce this alignment.

1 Introduction

Texts about the same expression can assign it different contextual meanings and make different discourse entities salient, differences that whole-text similarity metrics may conflate with fluency, style, length, and topic. The paper proposes semiotic profiles and embeddings to compare those framings, reporting that coverage is the main bottleneck relative to fidelity and that lower sampling temperatures yield stronger alignment with human-curated data.

1 Introduction

  • Texts about the same expression can diverge in contextual meaning and in the entities they make salient, even when both are fluent and coherent.“Venus” may be framed as a Roman goddess, a planet, or an astrological symbol.Source: §1 · p. 1
  • Whole-text overlap and similarity metrics conflate framing with fluency, style, length, and topical content.The motivating question is how to estimate overlap between two agents’ output-level framings given the same input.Source: §1 · p. 1
  • A semiotic profile combines an expression’s contextual meaning with the discourse entities made salient through that meaning.The framework compares an evaluated agent’s profile with an anchor agent’s profile using Semiotic Embeddings.Source: §1 · p. 2
  • The framework operationalizes Peircean semiotics and estimates unsupported evaluated content as well as anchor-supported content the evaluated agent fails to recover.It introduces model-agnostic Semiotic Embeddings as proxy representations of framing and salient discourse entities.Source: §1 · p. 2
  • Coverage is the main bottleneck relative to fidelity, and lower temperatures yield the strongest alignment with human-curated data while higher temperatures reduce it.Higher temperatures also increase model-to-model agreement.Source: §1 · p. 2
  • Three frontier LLMs used as judges showed positive correlations between their ratings and the proposed scores.The supplied passage reports positive correlations but gives no values here.Source: §1 · p. 2

2 Background

The background adopts Peirce’s triadic account of signs and maps it to a text-based profile of contextual meaning and salient references. It then motivates masked language-model representations as a way to encode these components for comparing agents’ outputs.

Peircean Semiotics

  • Peirce’s model relates a representamen, the material form of a sign, to an object and an interpretant, the meaning arising for an agent.The framework adopts the triad to capture how different agents frame the same input.Source: §2 · p. 2

Semiotic modeling

  • The paper maps the triad to Expression, Meaning, and Reference, with Expresses, Denotes, and Interprets linking those elements.Meaning is the contextual sense assigned to an expression; Reference is the entity or situation made accessible through interpretation.Source: §2 · p. 2

Semiotic profile

  • A semiotic profile encodes an agent’s situated interpretation as a context-dependent Meaning and the relevant Reference identified through it.The profile is text-based.Source: §2 · pp. 2–3

Masked Word Embedding

  • Masked word embeddings serve as proxies for Meaning and Reference by representing what context expects at a masked target position.The problem setup compares evaluated and anchor agents’ profiles through Semiotic Fidelity and Semiotic Coverage.Source: §2 · p. 3

3 Related work

Prior NLG metrics assess overlap, contextual similarity, diversity, or distributions of generated texts, but often treat each whole text as the comparison unit. This paper instead compares the Meaning–Reference configuration of a target expression, distinguishing its contribution from global text quality and diversity measures.

Prior NLG metrics

  • Lexical-overlap and embedding-based metrics compare generated text with references, while diversity metrics use surface n-gram statistics that provide only a coarse account of semantic diversity.LLM-as-a-judge protocols aggregate quality scores across tasks.Source: §3 · p. 3

Distributional approaches

  • Distributional approaches compare sets of texts, with precision and recall separating generated samples within real-text support from real samples covered by generated-text support.MAUVE summarizes a divergence frontier in one score, while Le Bronnec et al. (2024) estimate supports to recover distinct precision and recall scores.Source: §3 · p. 3

Comparison with prior work

  • Unlike whole-text distributional approaches such as Le Bronnec et al. (2024) and Pillutla et al. (2021), this framework evaluates a target expression’s Meaning–Reference configuration.The authors state that this can identify fluent outputs that reduce a polysemous topic to fewer senses and paraphrases that preserve the same configuration under different surface forms.Source: §3 · p. 3

4 Methodology

The framework encodes an expression’s contextual meaning and salient references together, then compares evaluated and anchor agents’ semiotic profiles. It uses the resulting support overlap to estimate Semiotic Fidelity and Semiotic Coverage.

4 Methodology

  • Semiotic Embeddings combine an expression’s Meaning Embedding with Reference Embeddings represented relative to that meaning, preserving distinctions between identical displacements attached to different meanings.When there is no reference displacement, the embedding stores only the Meaning Embedding.Source: §4 · pp. 4–5
  • Meaning Embeddings use a masked-language-model Transformer to represent an expression’s contextual frame, averaging the hidden representations of its masked occurrences as a proxy for meaning.The resulting representation abstracts away from the expression’s lexical form.Source: §4 · p. 4
  • PCA retains the smallest number of Semiotic Embedding components explaining at least 90% of variance before local support estimation.Each point’s support hypersphere uses its distance to its k-th nearest neighbor as the radius.Source: §4 · p. 5
  • Semiotic Fidelity is the fraction of evaluated points within the anchor support, while Semiotic Coverage is the fraction of anchor points covered by the evaluated support.Coverage uses radii calibrated to the anchor’s local density because sparse evaluated references could otherwise expand the evaluated support.Source: §4 · pp. 5–6
  • Anchor-calibrated Coverage is monotonic: removing an evaluated point removes its hypersphere without changing the others, so coverage cannot increase.The final scores average per-expression Fidelity and Coverage across expressions.Source: §4 · p. 6

5 Experiments

The experiments assess whether Semiotic Fidelity (SF) and Semiotic Coverage (SC) behave as expected and compare semiotic alignment across LLM and human-authored texts. Results vary by task and temperature, with coverage often weaker than fidelity and simplification producing asymmetric scores.

5 Experiments

  • The experiments use 10 random seeds, vary LLM-generation temperatures from 0.2 to 1.6 for R1 and R2, and evaluate human-anchor, LLM-anchor, and human-to-human settings.The study includes WIKIPOLY, COMMENS, SWIPE, OSE, and PEIRCE, with k = 3 for support estimation.Source: §5 · p. 6

5.2 R1: LLMs vs. Human-curated data

  • At T = 0.8, Phi-4 achieves the highest R1 harmonic mean overall (0.26), while higher temperatures destabilize several models and Tiny-Aya’s high average SF (0.30) coexists with the lowest SC.Qwen has the highest average SC (0.23), while Gemma’s SC stays close to 0.20 over most of the temperature range.Source: Table 2 · p. 7, §5.2 · p. 7

5.1 Controlled Diagnostic

  • In the controlled diagnostic, SF and SC follow their expected patterns across four transformations; the ablation without logarithmic-map Reference displacement scores Q4 lower despite preserved senses and referents.The proposed method yields 0.74/0.77 on Q1, while the ablation yields 0.61/0.58; prior methods stay high on Q2, where the proposed scores show reduced SF and high SC.Source: §5.1 · p. 7

5.3 R2: LLMs vs. LLMs

  • On WIKIPOLY, LLM alignment is low on average (SF/SC 0.28/0.23; mean pairwise harmonic score 0.23), while Phi-4 is the strongest evaluated agent (harmonic mean 0.31).Tiny-Aya combines high SF (0.38) with low SC (0.17); the harmonic mean falls to 0.13 at T = 1.2 before recovering to 0.37 at T = 1.6.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
  • On COMMENS, alignment is higher than on WIKIPOLY (average SF/SC 0.32/0.29; harmonic mean 0.29), and Qwen is the strongest, most balanced evaluated agent (0.38/0.38; harmonic mean 0.37).The passage attributes the higher alignment tentatively to shared vocabulary and referential structure around technically constrained Peircean concepts.Source: §5.3 · p. 8

5.4 R3: Comparing Human-Curated Data

  • On SWIPE and OSE, simplified texts are typically faithful to their complex anchors while covering less, and OSE agreement decreases from Intermediate-to-Advanced to Advanced-to-Elementary simplification.OSE-AI has the highest agreement; OSE-AE has the lowest, consistent with the stronger semantic reduction in the direct Advanced-to-Elementary step.Source: §5.4 · p. 8
  • On PEIRCE, scores are low across texts from different periods (SF/SC 0.27/0.22); “Critical-common-sensism” scores 0.65/0.69, whereas “categories” scores 0.05/0.06.The two periods emphasize different aspects of Critical-common-sensism, while older and later texts frame categories differently.Source: §5.4 · p. 8, Figure 5 · p. 9

5.5 Correlation with LLM Judgments

  • At T = 0.2, SF correlates with median LLM-judge ratings at ρ = 0.67 and SC at ρ = 0.53 for sampled R1 outputs.Three LLMs rated outputs for 30 sampled expressions across seven models.Source: §5.5 · pp. 8–9

6 Conclusion

The conclusion presents a framework for comparing how two agents characterize the same expression. Its Semiotic Embeddings capture contextual meaning and salient references, producing scores intended to assess semiotic alignment.

6 Conclusion

  • The authors present the framework as a computational basis for assessing how closely two agents’ textual outputs are semiotically aligned.Evaluating how LLMs frame visual signs in multimodal settings is identified as future work.Source: §6 · p. 9
  • The framework encodes texts as Semiotic Embeddings that capture an expression’s contextual meaning and the references made salient through that meaning.It yields Semiotic Fidelity and Semiotic Coverage scores for comparing agents’ characterizations.Source: §6 · p. 9

7 Limitations

The framework has constraints around support estimation, reference extraction, and validation. Its computationally heavier preprocessing can be cached, leaving agent comparisons unaffected.

7 Limitations

  • Stable manifold support estimation benefits from dense samples and can be affected by outliers or sparse regions.The framework pools multiple semiotic points per document and calibrates SC on the anchor distribution; k sets the estimated support’s neighborhood scale.Source: §7 · p. 9
  • Changing the reference extractor or progressively omitting References varied absolute scores, but preserved dataset ordering for both measures.Documents with few or no extracted References still contribute their Meaning component.Source: §7 · p. 9
  • The controlled diagnostic and LLM-judgment analysis do not replace direct validation against human judgments.The authors leave a dedicated human annotation study for future work, although the supplied passage does not complete that statement.Source: §7 · p. 9
  • Reference extraction and masked-language-model forward passes make the method more computationally demanding than single whole-document embeddings.This cost falls on preprocessing that can be run offline and cached; document embeddings are independent and can be computed in parallel.Source: §7 · pp. 9–10

8 Ethical Considerations

The framework measures alignment relative to a documented anchor, not absolute truth or quality. The implementation uses specified hardware, generation settings, and an encoder that retains a masked-language-model interface.

8 Ethical Considerations

  • SF and SC are directional, anchor-dependent evidence of semiotic alignment, not universal measures of truth, objectivity, or quality.Because anchors may be incomplete, biased, or domain-specific, their selection should be documented with care, especially in culturally sensitive or contested domains.Source: §8 · p. 10

B.1 Environment

  • Cached-output evaluation took less than 12 GPU hours on a single GPU per experiment across datasets, excluding LLM text generation.Experiments ran on a server with one NVIDIA A100-PCIE-40GB GPU and 503 GiB of system RAM.Source: §B.1 · p. 12

B.2 Comparison with prior work

  • The prior approaches compared in Section 5.1 used Le Bronnec et al.’s (2024) publicly available GitHub implementation under GPL-3.0.Source: §B.2 · p. 12

B.3 Generative Models

  • Generation varied temperature from 0.2 to 1.6 in 0.2 increments, with 10 responses per target, model, task, and temperature.The setup left top_p and top_k at defaults and used max_new_tokens = 512.Source: §B.3 · p. 12

B.4 Encoder-only Transformer

  • ModernBERT served as the encoder-only Transformer, with maximum input length set to 512 tokens.The framework can use other encoders if their MLM-pretrained backbone retains the MLM interface needed for Meaning and Reference Embeddings.Source: §B.4 · p. 13

B.5 Correlation with LLM Judgments of Semiotic Fidelity and Coverage

The appendix reports positive rank associations between framework scores and LLM judge ratings, and details the reference extraction, support-estimation, and generation procedures used in the experiments.

B.5 Correlation with LLM Judgments of Semiotic Fidelity and Coverage

  • ρ = 0.67 for SF and ρ = 0.53 for SC in Spearman correlations with median LLM judge ratings on 30 WIKIPOLY expressions at T = 0.2.Three judges rated SF and SC separately on a 1–5 scale after seeing complete reference and candidate collections, without framework scores or model identities.Source: §B.5 · p. 13

B.6 Reference Extraction Algorithm

  • SRL extracts predicate–argument frames, from which the procedure selects relevant argument spans and simplifies them into noun-centered chunks.It identifies subject mentions for exclusion, filters uninformative chunks and subject mentions, then normalizes and merges repeated References.Source: §B.6 · pp. 13, 15

B.7 Other implementation details

  • The experiments use k = 3 nearest neighbors to define each hypersphere’s radius and estimate local support.At coincident Meaning and Reference embeddings or antipodal points, the displacement is set to zero rather than choosing an arbitrary tangent direction.Source: §B.7 · p. 15

B.8 LLM generation prompts

  • For Wikipedia lead generation, prior outputs were included in the prompt to discourage restating a lead and encourage another sense or complementary aspect.The Peircean semiotics task used no generation history because some concepts lack a single unambiguous definition and Peirce’s terminology shifted across writings.Source: §B.8 · pp. 15–16

C Datasets

The evaluations use English datasets for aggregate evaluation, including COMMENS materials and WIKIPOLY, a collection designed to test semantic coverage across polysemous concepts. The appendix also identifies figures reporting SF and SC for R2.

C Datasets

  • The datasets are English and used only for aggregate evaluation; R1 and R2 instances were generated 10 times per temperature from 0.2 to 1.6 in 0.2 increments.Source: C Datasets · p. 16

C.1 Commens

  • The COMMENS materials are used for non-commercial research and evaluation under a Creative Commons Attribution–NonCommercial–ShareAlike license.The R2 concepts are selected from concepts with available definitions, and R3 collects publicly available definitions.Source: §C.1 · pp. 16–17

C.2 WIKIPOLY

  • WIKIPOLY pairs each polysemous concept with three to six sense-specific Wikipedia articles, favoring concepts with more distinct senses.It is designed to evaluate coverage of a concept’s semantic range rather than defaulting to its most common interpretation.Source: §C.2 · p. 17
  • The controlled diagnostic includes variants that add senses, remove part of the anchor’s semantic coverage, or change surface form while preserving represented senses and referents.The WIKIPOLY construction also describes a variant combining a same-sense textual substitution with a new sense.Source: §C.2 · p. 17

D.1 More results on R2

  • Figures 9 and 10 show Semiotic Fidelity and Semiotic Coverage in the R2 setting.Source: §D.1 · p. 17

D.2 Results with other MLMs

Across four additional MLM-pretrained encoders, the R3 dataset ordering was unchanged, and harmonic-mean patterns varied only slightly across encoders.

D.2 Results with other MLMs

  • Encoder sensitivity was tested on R3, where both text collections are human-authored, by comparing ModernBERT with four additional MLM-pretrained encoders.Source: §D.2 · pp. 17–18
  • SF, SC, and their harmonic mean preserve the dataset ordering OSE-AI > OSE-IE > SWIPE > OSE-AE > PEIRCE across encoders.The result is reported for human-authored anchor and evaluated texts in R3.Source: §D.2 · p. 18
  • The harmonic mean varies only slightly across encoders within each dataset, suggesting a consistent overall pattern across representation spaces.Source: §D.2 · p. 18

D.3 Analysis of Reference Extraction

In R3, SRL-based and NP+NER Reference extraction preserve the same dataset ordering, while Reference omission changes scores progressively without changing that ordering.

D.3 Analysis of Reference Extraction

  • The alternative NP+NER procedure collects noun chunks and named entities instead of SRL argument spans; only Reference selection changes, with subsequent scoring steps held constant.Source: §D.3 · pp. 18–19
  • Both Reference extractors yield the ordering OSE-AI > OSE-IE > SWIPE > OSE-AE > PEIRCE for fidelity and coverage, with Kendall’s τb = 1.0.NP+NER generally gives lower absolute scores, with the largest difference on PEIRCE and smaller differences across OSE comparisons.Source: §D.3 · p. 19, Table 6 · p. 19
  • The omission analysis independently removes 10%, 20%, or 30% of Reference points from anchor and evaluated collections over 20 random seeds, retaining all Meaning points.Source: §D.3 · p. 19
  • At 10%, 20%, and 30% Reference omission, the largest mean absolute SF/SC changes are .02/.02, .04/.04, and .06/.07, respectively, while dataset ordering remains unchanged.Kendall’s τb = 1.0 indicates complete agreement with the baseline ordering across the tested omission rates.Source: §D.3 · p. 19, Table 7 · p. 19

D.4 Analysis of k

Both scores rise with k, although most model rankings remain stable. The accompanying appendix proves SC’s monotonicity and describes the semiotic and geometric representations underlying the framework.

D.4 Analysis of k

  • Both SF and SC rise with k for every model; 357 of 378 pairwise model orderings (94.4%) are preserved across the tested k values and measures.No model shifts by more than two positions; rankings are compared with k = 3.Source: §D.4 · p. 19, Figure 11 · p. 20

E Monotonicity of Semiotic Coverage

  • Removing points from the evaluated collection cannot increase SC: anchor-calibrated radii remain fixed, while the evaluated support can only shrink under sparsification.Source: E Monotonicity of Semiotic Coverage · p. 20

F.1 More on Semiotics

  • Peirce’s triad treats meaning as a relation among representamen, object, and interpretant, addressing the agent and object roles left outside the Saussurean account described here.Source: §F.1 · pp. 20–21

F.2 Logarithmic Map

  • The logarithmic map represents each Reference Embedding as a tangent-space displacement from the Meaning Embedding, encoding both direction and magnitude.Source: §F.2 · pp. 21–22
Loading 2609.26527v1…