PaperlayerView exact revision on arXiv

Source-linked AI summary

T2LSC-Bench: Benchmarking Localized Semantic Control in Text-to-Image Generation

Yan Wang, Xinyi Hou, Weiguo Lin, Junjun Si, Siwei Ma

arXiv:2609.02255v1cs.CV

TL;DR

Text-to-image benchmarks have largely measured whether text is readable and correctly placed, not whether its semantics remain confined to the designated anchor. T2LSC-Bench addresses this gap with a controlled, factorized benchmark and separate text and semantic evaluations, finding substantial leakage despite nearly stable text-rendering accuracy.

  • Problem

    Existing visual-text evaluations primarily assess readability, spelling, layout, and rendering robustness, leaving semantic containment largely unexamined despite localized-text requirements.

  • Method

    T2LSC-Bench uses factorized cases across seed subjects, semantic relations, scene openness, prompt modes, and languages, evaluated with OCR–VLM text verification and structured VLM semantic judgments.

  • Results

    Stress-test cases raise SLR from 1.2% to 18.1% while TAA changes only from 91.4% to 90.9% across six models.

  • Takeaways & Limitations

    Accurate target-text rendering does not guarantee local containment of its semantics, making semantic containment a complementary evaluation dimension.

Abstract

from arXiv · show

Recent text-to-image models have become increasingly capable of rendering explicit text, but reliable localized text control requires more than generating the correct string. In applications such as product labeling, signage, and interface design, target text should be rendered within a designated text-bearing region without altering the predefined subject identity or surrounding scene semantics. We refer to violations of this requirement as target-text-associated semantic leakage, in which target-text semantics are expressed through non-textual visual content beyond the designated anchor. Existing visual-text benchmarks primarily evaluate readability, spelling accuracy, and layout, leaving this form of semantic leakage largely unexamined. We introduce T2LSC-Bench, a controlled diagnostic benchmark comprising 50 seed subjects and 1,200 prompt cases per model, yielding 7,160 evaluated images across six models. Its factorized design varies semantic relation, scene openness, prompt mode, and language. A dual-branch protocol combines OCR-VLM text verification with structured VLM semantic judgments to measure Text-at-Anchor Accuracy (TAA), Semantic Subject Preservation (SSP), Semantic Leakage Rate (SLR), and Conditional Semantic Leakage Rate (cSLR). Under stress-test conditions, SLR increases from 1.2% to 18.1% and cSLR from 1.3% to 18.2%, whereas TAA decreases only from 91.4% to 90.9%. Anti-leakage prompting reduces SLR from 16.6% to 8.4% without degrading rendering accuracy. Human validation on 420 images shows strong agreement between automatic and adjudicated annotations. These results show that accurate text rendering does not guarantee local containment of target-text semantics.

I. INTRODUCTION

T2LSC-Bench addresses a gap in visual-text evaluation: models must render target text locally while preserving the subject and surrounding semantics. Its controlled benchmark and dual-branch protocol expose semantic leakage that text-rendering accuracy alone can miss.

  • The explicit localized-text contract requires accurate text at the designated anchor without altering subject identity or surrounding scene conditions.
  • Existing visual-text evaluations emphasize readability, spelling, placement, and rendering robustness rather than semantic containment.
  • T2LSC-BENCH varies semantic relation, scene openness, prompt mode, and language across 50 seed subjects and 1,200 prompt cases per model.
  • Its dual-branch protocol separates anchor-text correctness from semantic containment using OCR–VLM verification, blinded VLM assessment, and human validation.
  • 18.1% SLR and 18.2% cSLR under stress tests contrasted with nearly unchanged TAA values of 91.4% and 90.9%.

II. BACKGROUND AND RELATED WORK

Prior visual-text work improves local textual fidelity and evaluates readability, spelling, layout, and robustness, while related control evaluations focus on identity or local controllability. T2LSC-Bench complements these settings by separately evaluating anchor-text accuracy, subject preservation, and non-textual semantic leakage.

  • Visual text generation methods improve textual fidelity through character-aware representations, glyph guidance, spatial control, and related techniques.
  • Existing benchmarks primarily assess readability, spelling, layout, prompt complexity, and rendering robustness.
  • The proposed protocol evaluates anchor-text accuracy, subject preservation, and non-textual semantic leakage separately.

III. BENCHMARK CONSTRUCTION

T2LSC-Bench is a controlled diagnostic testbed for confining target-text semantics to a designated anchor while preserving subject identity and scene context. Its factorized construction separates these properties under comparable generation conditions.

  • Its factorized design supports separate analysis of local rendering, subject preservation, and target-text-associated changes to non-textual content.
  • Each case specifies a text-bearing subject, predefined identity and context, designated anchor, and target text.
  • Target-text-associated semantic leakage is defined as visible non-textual cues outside the anchor that are semantically linked to the target text but not naturally attributable to the subject context.
  • The benchmark distinguishes subject preservation from semantic leakage, since leakage can occur on a recognizable subject or in its surrounding scene.

B. Seed and Text-Anchor Design

Seed and anchor design prioritizes concrete, recognizable subjects with stable identities, native text regions, and clear default meanings. Controlled factors and conflict probes then test leakage under varied semantic and scene conditions.

  • Each seed requires a stable visual identity, native text-bearing region, and clear default semantic or functional interpretation.
  • Seeds use concrete subjects and native anchors, such as storefronts, service vehicles, product containers, and warning signs.
  • The benchmark varies semantic relation, prompt mode, scene openness, and language to test complementary leakage hypotheses.
  • Each seed pairs one aligned target text with two conflicting texts that are semantically incompatible with the default subject interpretation.

2) Prompt Mode:

T2LSC-BENCH instantiates controlled cases by varying prompt mode, scene openness, language, and semantic relation while preserving the underlying subject-anchor design. Quality checks ensure semantic compatibility, visual specificity, and cross-language consistency.

  • 2) Prompt Mode:: Prompt mode compares a natural base prompt with an anti-leakage condition that adds constraints against semantic propagation and scene reinterpretation.All remaining prompt content is unchanged, isolating the effect of semantic-boundary constraints.
  • 3) Scene Openness:: Scene openness varies contextual richness from a subject-centered mid-shot with 1–3 surrounding objects to a mid-to-far shot with 4–6 objects.The main subject remains identifiable in both settings.
  • 3) Scene Openness:: Richer surroundings provide more opportunities for target-text-associated objects or contextual cues to appear outside the designated text anchor.
  • 4) Language:: Each case is instantiated in English and Chinese using language-specific subject descriptions, anchors, target texts, and prompt templates.Subject identity, semantic relation, scene openness, and prompt mode remain paired across languages.
  • 4) Language:: The factor definitions jointly form the controlled condition space for factorized case instantiation and analysis.
  • D. Case Instantiation and Quality Control: Each of 50 seeds expands into three target-text instances across two scene-openness settings, two prompt modes, and two languages, producing 1,200 prompt instances per model.
  • D. Case Instantiation and Quality Control: Quality control checks subject concreteness, anchor naturalness and specificity, target-text compatibility, and preservation of the underlying cross-language design.

A. Experimental Setup

The experiments evaluate six models under a shared automatic protocol, combining text-rendering verification with independent semantic assessment and targeted human validation. OCR and VLM evidence are fused with fixed rules, while unresolved disagreements receive manual review.

  • A. Experimental Setup: Six text-to-image models receive the same 1,200 benchmark cases, yielding 7,160 available images from 7,200 planned generations.Unavailable outputs involved one Disney storefront seed and provider-side content-safety filters.
  • A. Experimental Setup: All images use a shared protocol and judge configuration combining OCR evidence, anchor-text VLM verification, and VLM semantic assessment.Results are analyzed across the four controlled factors.
  • A. Experimental Setup: The protocol applies independent text-rendering and semantic branches to each image, assigning uncertainty when visible evidence is insufficient.The semantic branch evaluates subject preservation and leakage independently of text-rendering outcome.
  • A. Experimental Setup: The VLM version, prompts, confidence thresholds, OCR–VLM fusion rules, and parsing procedures remain fixed across images and generators.
  • A. Experimental Setup: PaddleOCR filters detections below 0.5 before comparing retained text with the target string and designated anchor region.The configuration supports Chinese and Latin characters with angle classification enabled.
  • A. Experimental Setup: An anchor-aware VLM reports string, visibility, readability, placement, character deviations, ambiguity, and confidence for the designated region.
  • A. Experimental Setup: A VLM match with confidence ci ≥ τ recovers likely OCR false negatives when the target text is visibly correct at the anchor.The threshold is τ = 0.7.
  • A. Experimental Setup: Unresolved OCR–VLM disagreements undergo manual review, after which visible deviations are classified as minor error, failure, or ambiguous.

2) Semantic Evaluation:

Semantic evaluation separates subject preservation from target-text-associated non-textual evidence outside the designated anchor. The resulting metrics distinguish text accuracy, subject identity, overall leakage, and leakage conditional on exact rendering.

  • 2) Semantic Evaluation:: The blinded semantic branch judges non-textual target-text evidence outside the anchor while withholding semantic relation, prompt mode, prompt, and generator identity.
  • 2) Semantic Evaluation:: The judge first assesses subject identity, then describes external non-textual cues and tests whether they support target-text semantics beyond the default subject context.
  • 2) Semantic Evaluation:: Readable text outside the anchor, generic blur, deformation, or identity loss alone do not count as semantic leakage without supporting non-textual target-text evidence.
  • 3) Evaluation Metrics:: TAA measures exact target-string rendering at the designated anchor, while SSP measures preservation of the predefined subject identity.
  • 3) Evaluation Metrics:: SLR measures valid leakage judgments in which external non-textual content expresses target-text semantics beyond the default subject context.
  • 3) Evaluation Metrics:: cSLR measures the same leakage outcome specifically among samples with exact anchor-text rendering.It therefore assesses containment after the target string has been rendered exactly.
  • 3) Evaluation Metrics:: SLR includes both exact and non-exact renderings, whereas cSLR restricts leakage measurement to exact renderings with valid leakage judgments.
  • 3) Evaluation Metrics:: Higher TAA and SSP indicate better performance, while lower SLR and cSLR indicate stronger semantic containment.Uncertain judgments use metric-specific denominator rules, and cSLR requires exact text rendering.

A. Overall Results

Across six models, text-rendering accuracy and semantic control diverge: conflicting target text substantially increases leakage while TAA changes little. Model profiles and controlled-factor analyses show that semantic relation is the strongest stressor, with prompting offering partial mitigation.

  • Overall performance: 18.1% aggregate SLR and 18.2% conflicting cSLR show substantial leakage under conflicting conditions, despite TAA changing only from 91.4% to 90.9%.The seed-clustered SLR difference is 16.9 percentage points, with a 95% confidence interval of [13.4, 20.7].
  • Model profiles: Seedream 5.0 combines 97.1% TAA with 22.0% conflicting cSLR, whereas Gemini 3.1 achieves the strongest overall performance but retains 13.1% conflicting cSLR.Wan2.6 records the lowest SSP and highest conflicting cSLR among the reported model profiles.
  • Qualitative variation: Matched examples show object insertion, subject reinterpretation, or scene reconstruction across models even when anchor-text accuracy is held constant.Examples include food-related cues for “SNACK BOX,” an automotive-fluid interpretation of a sports bottle, and a storefront reconstructed as a laundry.
  • Evaluation coverage: 7,160 of 7,200 planned generations were evaluated, and excluding the unavailable Disney storefront seed changes every reported metric by less than 0.5 percentage points.The 40 unavailable outputs were associated with provider-side content-safety filtering.
  • Controlled factors: Semantic relation produces the largest SLR difference, followed by prompt mode, while scene openness and language produce smaller differences.All confidence intervals exclude zero; open scenes have 13.6% SLR versus 11.4% for closed scenes, and English has 14.1% versus 10.9% for Chinese.
  • Controlled factors: 8.4% SLR under explicit anti-leakage prompting is lower than 16.6% under the natural prompt, while TAA remains nearly unchanged.The seed-clustered difference is −8.1 percentage points, with a 95% confidence interval of [−10.2, −6.2], and the reduction varies across models.

C. Human Validation Results

Automatic evaluation agrees strongly with adjudicated human annotations for anchor-text correctness and subject preservation, while leakage assessment remains more difficult. The SLR judge nevertheless achieves substantial agreement and detects most human-confirmed leakage cases.

  • Agreement: TAA and SSP achieve 95.5% and 95.0% agreement with human annotations, respectively, with Cohen’s κ values of 0.842 and 0.823.Precision, recall, and F1 scores exceed 96% for both dimensions.
  • Assessment boundary: SLR is more challenging because the evaluator must distinguish target-text-associated non-textual cues from context naturally compatible with the predefined subject.False positives mainly reflect overattribution of ambiguous or natural contextual cues to target-text semantics.
  • Leakage assessment: The automatic SLR judge achieves 90.0% agreement, Cohen’s κ of 0.800, and an F1 score of 89.3%.Its recall is 96.2%, while precision is 83.3%.

D. Qualitative Observations

Qualitative comparisons make the leakage decision boundary concrete: leakage requires visible non-textual target-text cues outside the anchor that are not naturally explained by the default subject. The benchmark is positioned for model comparison and mitigation evaluation, but its current scope is bounded.

  • Leakage examples: Leakage cases introduce target-text-associated non-textual cues outside the anchor, whereas contained outputs render the same text without those cues.Examples include a water tap, pharmacy shelving, an ATM and coins, an excavator, gasoline pumps, and an ice-cream cone.
  • Decision boundary: SLR identifies leakage only when target-text semantics appear in visible non-textual content outside the anchor and cannot be naturally explained by the default subject context.This operational boundary distinguishes semantic leakage from ordinary subject-compatible visual content.
  • Uses and scope: T2LSC-BENCH supports cross-model comparison, regression testing, and evaluation of prompting or training strategies for reducing leakage.The stated application scope includes advertising and design variants, controlled synthetic data, and personalized visual content.
  • Scope boundary: The current benchmark focuses on stable subjects with predefined anchors and relatively short English and Chinese target strings.Future extensions named by the paper include longer strings, additional languages, less structured subjects, and image or video editing.
Loading 2609.02255v1…