Source-linked AI summary
Telligram: Text-Driven Calligram Generation via Diffusion-Guided Skeleton Optimization
Tianci Shi, Pengfei Xu
TL;DR
Text-only calligram generation must form a semantic figure without a supplied contour while preserving readable letters. Telligram separates occupancy formation from glyph realization in a training-free, two-stage framework. Its reported comparison favors hierarchical gradient projection over direct or smoothed joint updates for semantic alignment, and examples show diverse recognizable figurative layouts without an input contour.
Problem
Text-only calligram generation lacks a supplied contour and must balance semantic shape formation with readable letters, which can conflict under single-stage optimization.
Source: §1 · pp. 1–2Method
Telligram uses a training-free two-stage pipeline that forms a diffusion-guided semantic occupancy prior, then converts it into per-letter regions and readable glyphs through geometric processing.
Source: §1 · p. 2, §4 · p. 4, §4.2 · p. 5Results
Hierarchical gradient projection provides better semantic alignment than direct joint or smoothed joint updates while avoiding the heavy tuning burden of manually staged schedules.
Source: §7 · p. 10Takeaways & Limitations
Qualitative examples show visual diversity and recognizable figurative layouts generated without an input contour image.
Source: §6.2 · p. 10Silhouette-and-skeleton representations can lose fine appearance cues and may not restore letters after severe deformation; low-resolution DeepFloyd guidance can yield similar global shapes across prompts.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Compact calligram generation aims to form a semantic shape while keeping letters recognizable. Most existing methods are shape-conditioned and mainly solve downstream letter layout inside a given contour. We study text-only calligram generation without an input contour. This setting is difficult because semantic shape formation and letter readability strongly interfere with each other when optimized in a single stage. Pushing the word toward a clear figure can easily damage glyph structure, while preserving readable letters can weaken the target shape. To address this difficulty, we present Telligram, a training-free, low-tuning, two-stage framework composed of Semantic Occupancy Prior Formation and Readability-Constrained Glyph Realization. The first stage uses Variational Score Distillation (VSD) with structured skeleton optimization and hierarchical gradient projection to produce a semantic occupancy prior. The second stage converts this occupancy prior into per-letter regions and reconstructs readable glyph layouts through lightweight geometric processing. The framework generates coherent and creative word-level semantic calligrams directly from text prompts.
1. Introduction
Text-only calligram generation must infer a semantic shape without a supplied contour while balancing readable letters against a recognizable figure. Telligram addresses this setting with a training-free, two-stage framework that separates semantic occupancy formation from glyph realization.
1. Introduction
- Shape-only optimization can compromise letter integrity, while readability-only optimization can weaken the intended figure, making semantic fidelity, topological coherence, and spatial coverage difficult to balance.Unlike long-text micrography, calligrams require geometric control over individual glyph placement and deformation.Source: §1 · p. 1
- Existing methods often assume a target contour or manually selected image, leaving automatic semantic shape acquisition and letter-level structural writability unresolved.Image-first pipelines can leave thin branches and boundary details uncovered by letters.Source: §1 · pp. 1–2
- Text-only calligram generation takes a target word and semantic prompt without a manually provided contour, shifting the task from fixed-container layout to semantic-to-structure generation.The difficult front-end step is identifying forms that are both semantically appropriate and geometrically writable.Source: §1 · pp. 1–2
- Telligram uses a training-free two-stage framework: diffusion-guided occupancy formation first, followed by readability-constrained geometric glyph realization.Its stated contributions include hierarchical gradient projection to stabilize occupancy formation and lightweight processing to convert the prior into fillable letter regions.Source: §1 · p. 2
2. Related Work
Related work spans shape-conditioned layout, semantic typography, glyph modeling, and large-scale text layout. These approaches provide useful geometric and typographic tools but generally do not solve image-free, whole-word calligram generation.
2.1. Shape-Conditioned Calligram and Image-Guided Text Composition
- Shape-conditioned calligram methods can fit letters to known contours, but rely on external target geometry and do not generate the shape from semantic input.Calligraphic Packing partitions a container; Legible Compact Calligrams [ZCR∗16] combine contour skeletons, protrusion handling, and letter-anchor matching.Source: §2.1 · p. 2
2.2. Semantic Typography and Word-as-Image
- Semantic typography can link glyph forms to concepts, but its focus on individual letters or short groups falls short of coherent, readable whole-word figure construction.Word-as-Image [IVH∗23] reduces dependence on external contours; Khattat [HEH∗25] targets semantic stylization and local readability rather than full figurative structure.Source: §2.2 · pp. 2–3
2.3. Font Generation, Glyph Modeling, and Layout Synthesis
- Font and glyph generation improves local glyph quality and editability, while text-layout synthesis organizes scale, transforms, and composition; neither alone induces word-level figurative structure.The cited font methods remain primarily in font space, and text-logo layout [WPL∗22] addresses character-level composition.Source: §2.3 · p. 3
2.4. Large-Scale Text Layout and Word Clouds
- Word-cloud methods offer reusable layout tools for bounded regions, but treat words as independent units rather than preserving letter-level continuity and reading order in whole-word calligrams.Shape-bounded settings such as ShapeWordle [WCZ∗20] also assume external shape constraints.Source: §2.4 · p. 3
3. Prior Knowledge
The prior-knowledge section describes diffusion-based semantic guidance and Voronoi partitioning as tools for structured calligram optimization and geometric realization.
3.1. VSD Semantic Prior
- VSD extends SDS from point-estimate optimization to distribution-level variational matching, with the stated aim of improving gradient stability and diversity.In Telligram, VSD gradients guide rendered glyph occupancy toward the target semantic concept while geometric parameterization keeps optimization structured.Source: §3.1 · p. 3
3.2. Voronoi Partition
- Voronoi partition assigns each point to its nearest site, creating non-overlapping regions that Telligram uses as boundary-aware separators during geometric realization.When sites are sampled from skeletons or centerlines, the regions help assign spatial support to neighboring structures and reduce unwanted merging.Source: §3.2 · p. 3
4. Method
Telligram separates semantic occupancy formation from readable glyph realization in a training-free, two-stage text-only pipeline. Hierarchical gradient projection stabilizes the first stage, while lightweight geometric processing converts its occupancy prior into readable glyphs.
4. Method
- Telligram uses a training-free two-stage pipeline: diffusion-guided occupancy formation precedes geometric realization of readable glyphs.The stages separate semantic structure formation from topology-sensitive glyph realization.Source: §4 · p. 4
4.1. Semantic Occupancy Prior Formation
- Parameterized letters are rendered as soft occupancy masks, then soft closing and connectivity and overlap regularization make regions semantically usable without requiring true letter overlap.The regularizers discourage excessive inter-letter interference and prevent one enlarged letter from dominating while others shrink into isolated blobs.Source: §4.1 · pp. 4–5
- Hierarchical projection sends coarse semantic motion first into affine variables, then only smooth residuals to skeleton joints, avoiding unstable high-frequency deformation while retaining semantic freedom.The high-frequency residual is discarded in the final update; stroke width remains fixed during this stage.Source: §4.1 · p. 5, Figure 3 · p. 6
4.2. Readability-Constrained Glyph Realization
- Convex hulls bound fitted skeletons, and occupancy pixels are assigned to the nearest fitted skeleton to create per-letter regions for filling.The fitted skeletons follow allocated semantic mass while avoiding excessive distortion.Source: §4.2 · pp. 5–6
- Candidate skeleton thickening balances recognizability against region coverage, while a narrow protection band preserves local details and inter-letter separation.A small radius can leave regions underfilled, while a large radius can erase thin structures and reduce readability.Source: §4.2 · p. 6
5. Experiments
The experiments evaluate optimization behavior, semantic alignment, OCR readability, and qualitative effects of Stage 1 modules. Across these tests, hierarchical updates lead the compared methods in semantic alignment, while OCR results vary with recognizer and boundary style.
5. Experiments
- Evaluation combines CLIP ViT-B/32 scores averaged over 10 prompts and 15 seeds with character-crop OCR from CRNN and PARSeq.OCR uses separate recognizers from Stage 2’s TrOCR candidate-selection step and serves as an auxiliary readability measure.Source: §5 · p. 6, §5.3 · pp. 7–8
5.1. Coarse-Deformation Ratio Analysis
- Earlier participation of skeleton deformation in the staged affine schedule consistently improves semantic alignment; high early affine ratios favor dominant-letter shortcuts.Max region ratio and region variance describe area concentration and imbalance, not direct quality.Source: Table 3 · p. 7, §5.1 · p. 7
5.2. Hierarchical Projection Analysis
- The hierarchical update achieves the highest CLIPScore, with structured, flexible letter-size adaptation rather than unstable joint-wise noise.Its max ratio and region variance can exceed those of direct joint updates; reducing size imbalance is not its objective.Source: Table 4 · p. 7, §5.2 · p. 7
5.4. Qualitative Ablation Study
- Qualitative ablations associate prolonged early affine updates with dominant-letter shortcuts, skeleton-only deformation with weaker semantic alignment, and removing Lsingle or Loverlap with connectivity or interference problems.Together, the ablations indicate that the full Stage 1 design balances coarse motion, local adaptation, whole-word connectivity, and inter-letter separation.Source: Figure 4 · p. 8, §5.4 · p. 8
5.3. OCR-Based Readability Evaluation
- Ours outperforms Berio et al. under the same no-mask setting and is comparable to Fleming on PARSeq and CRNN mean ground-truth posterior.OCR is an auxiliary measure: CRNN top-1 accuracy is lower for smooth-boundary results than for sharper-contour baselines.Source: Table 5 · p. 8, §5.3 · p. 8
6. Comparison and Results
Telligram forms semantic structure and letter placement jointly from text, unlike contour-conditioned methods that require an input shape. Qualitative comparisons and additional examples show skeleton-guided shape formation, multi-line layouts, and varied outputs across prompts and arrangements.
6.1. Comparison with Prior Methods
- Telligram jointly forms semantic outer structure and letter placement from text alone, while contour-conditioned methods require a given image or contour.The comparison is qualitative because the prior methods lack official code.Source: §6.1 · p. 8, Figure 5 · p. 9
- Skeleton-guided semantic prior formation reduces branches too thin for readable glyphs while preserving semantic contour quality.Direct silhouette prompting can produce recognizable contours with branches that are difficult to fill.Source: §6.1 · p. 8, Figure 6 · p. 10
- With the same “boat” concept and word “successful,” Telligram forms the figure through semantic occupancy optimization, whereas Zou et al. require guiding lines for multiple text lines.The comparison uses matched semantic information and the same word.Source: §6.1 · pp. 8, 10, Figure 7 · p. 10
6.2. Additional Qualitative Results
- Additional examples show varied, recognizable figurative layouts across prompts without an input contour image.The examples span diverse semantic targets.Source: §6.2 · p. 10
- Outputs vary across random seeds, line arrangements, and word orders under the same prompt, without collapsing to one repeated template across prompts.The examples include different line arrangements that preserve the same letter set.Source: §6.2 · p. 10
7. Conclusion
The conclusion reports that hierarchical gradient projection improves semantic alignment over direct or smoothed joint updates while avoiding the heavy tuning burden of manually staged schedules. It also identifies limits in category recognition, glyph restoration, and prompt-specificity, and points to learned readability guidance as a possible direction.
Conclusion
- Outputs vary across random seeds, line arrangements, and prompts, with additional qualitative examples spanning diverse semantic targets.The examples also vary the order of the same letters under one prompt.Source: Figure 8 · p. 10, Figure 9 · p. 11
- Hierarchical gradient projection provides better semantic alignment than direct or smoothed joint updates while avoiding the heavy tuning burden of manually staged schedules.The conclusion presents this as the experiments’ main finding.Source: §7 · p. 10
- Silhouette-and-skeleton representation can lose fine appearance cues, while 96×96 semantic resolution may make different prompts produce similar global shapes.Severely deformed letters may also remain difficult to restore.Source: §7 · p. 10
- 72.03% top-5 semantic recognition rate is only a coarse category-level indicator; severe deformation and residual branch-like support remain failure modes.The reported test uses 36 target categories and 15 random seeds per category.Source: §7 · pp. 10–11, Figure 10 · p. 11
- A possible extension is learned readability guidance that supplies identity-preserving gradients in Stage 1, alongside finer boundary-refinement variables.The authors tie current limitations in part to keeping the pipeline training-free.Source: §7 · p. 11