PaperlayerView exact revision on arXiv

Source-linked AI summary

GALA: Generation-Aware Cross-Modal Alignment for Text-to-Time-Series Synthesis

Haochen Zhang, Gengwei Zhang, Laura Yao, Nicholas Knoz, Tianlong Chen

arXiv:2608.13741v1cs.CLcs.LG

TL;DR

Existing text-to-time-series generators do not deliberately align conditioning representations with the signal modality, limiting their suitability for generation. GALA separates alignment from synthesis, using generation-aware cross-modal training before flow-matching generation, and achieves state-of-the-art results across fidelity and caption adherence.

  • Problem

    Existing text-conditioned generators use frozen or indirectly adapted text embeddings without deliberately matching conditioning representations to the time-series modality.

  • Method

    GALA contrastively aligns adapted text and time-series encoders with an auxiliary generative loss, then freezes the text embedding for a flow-matching generator.

  • Results

    GALA ranks first in 30 of 36 metric columns, with average ranks of 1.08/1.08/1.42 at lengths 24/48/96 versus 1.92/2.00/1.75 for VerbalTS.

  • Takeaways & Limitations

    Generation-aware alignment improves fidelity and caption adherence together, supporting dedicated alignment as a necessary component of text-to-time-series generation.

  • Takeaways & Limitations

    The learned embedding retains text-relevant, level-, and range-related cues, but the authors do not claim it is an optimal captioning feature.

Abstract

from arXiv · show

Synthesizing time series from natural language is emerging as the most expressive form of controllable time series generation. However, existing text-conditioned generators either take caption embeddings frozen from off-the-shelf text encoders, or adapt the encoder end-to-end, letting the denoising loss shape the embeddings only as a by-product. In either case, the conditioning representation is never deliberately matched to the signal modality, leaving it ill-suited to guide generation. We address this by introducing GALA: Generation-Aware cross-modaL Alignment for text conditional time series generation. GALA is a two-stage approach that first contrastively couples a pretrained text encoder with a time-series foundation model into a shared embedding space with both encoders adapted to generation by an auxiliary generative loss, and then freezes the resulting caption embedding to drive a flow-matching generator. On TSFragment-600K, spanning four domains and three fragment lengths, GALA sets a new state of the art, ranking first in 30 of 36 metric columns and reaching an average rank of 1.08/1.08/1.42 at lengths 24/48/96 against 1.92/2.00/1.75 for the strongest baseline. We further find that generator-internal text encoders force a trade-off between fidelity and caption adherence, whereas conditioning on the aligned embedding breaks it: FID, CTTP, and JFTSD all improve at once. Ablating the auxiliary loss degrades FID, CTTP and JFTSD together, it indicates the generative term is a necessary component of the alignment rather than an add-on.

1 Introduction

Text-to-time-series generators commonly use frozen or generation-only-adapted text representations, leaving them explicitly unaligned with the time-series modality and limiting controllability. GALA addresses this gap with a two-stage, generation-aware cross-modal alignment that conditions generation on a shared embedding.

  • Motivation: Unconditional generators are not controllable, while controllable synthetic time series support augmentation, privacy-preserving data sharing, and simulation.These applications span domains including energy, traffic, and healthcare.
  • Problem: Existing text-conditioned methods either use frozen text-only caption features or adapt text representations only through denoising, without explicit time-series alignment.This treatment is shared by latent-space and observation-space generation approaches.
  • GALA: GALA uses two stages: generation-aware contrastive alignment with an auxiliary generation loss, followed by generation conditioned on the resulting text embedding.The auxiliary objective keeps the text embedding sufficient to synthesize time series while decoupling alignment from generation.
  • Problem: Fine-tuning generator-internal text encoders improves caption adherence at the cost of fidelity, showing that generation loss alone insufficiently shapes the shared latent space.The introduction identifies this trade-off as a challenge specific to text-conditional time-series generation.
  • GALA: GALA contrastively aligns a pretrained text encoder and time-series foundation model, adding magnitude-aware encoding to preserve absolute level and scale referenced by captions.The alignment is designed as a shared embedding space rather than a retrieval-only representation.

2 Method

GALA separates generation-aware cross-modal alignment from synthesis: Stage 1 aligns adapted text and time-series encoders while preserving generation-relevant information, and Stage 2 freezes the text encoder to condition a flow-matching generator. Its auxiliary generative loss complements contrastive alignment by encouraging embeddings that retain fine-grained information needed for reconstruction, improving fidelity and caption adherence together.

  • Motivation: The alignment error is R(E) = I(X; C | U) = I(X; C) − I(X; U), so Stage 1 seeks caption embeddings that are informative about their associated time series.The error reaches zero when the embedding is a sufficient statistic of the caption for the time series.
  • Stage 1: Alignment: Stage 1 adapts pretrained text and time-series foundation models into a shared space using contrastive alignment plus an auxiliary generation loss.The text encoder is Embedding-Gemma-300M and the time-series encoder is Chronos-2; both are frozen backbones adapted with LoRA.
  • Generation-aware alignment: Contrastive loss alone can produce discriminative but generation-unready embeddings because its mutual-information bound saturates at log B once negatives are easily ranked.The auxiliary generation loss continues rewarding finer information needed for reconstruction, rather than only information useful for ranking.
  • Two-stage architecture: GALA uses two decoupled stages: Stage 1 learns a generation-aware shared embedding space, while Stage 2 freezes the aligned text encoder and trains a conditional flow-matching generator.The generator receives the frozen caption embedding as its conditioning signal.
  • Stage 2: Generation: Stage 2 represents each time series as an invertible delay-embedded square image and trains a DiT-style generator under the JiT flow-matching recipe.Sampling uses a 50-step deterministic Heun solver before applying the inverse delay embedding.

3 Experiments

GALA is evaluated on TSFragment-600K across four domains and three fragment lengths against five adapted text-to-time-series baselines. It achieves state-of-the-art aggregate performance, while ablations show that dedicated, auxiliary-loss-trained alignment—especially tuning the text encoder—improves generation and caption adherence.

  • Experimental setup: TSFragment-600K covers four sources and lengths 24, 48, and 96, with 90/10 training and held-out test splits.The sources are ETTh1s, ETTm1, electricity, and traffic.
  • Main results: GALA ranks first in 30 of 36 metric columns, with average ranks 1.08/1.08/1.42 at lengths 24/48/96 versus VerbalTS at 1.92/2.00/1.75.All other baselines have average ranks ≥3.0.
  • Main results: GALA obtains the highest CTTP in all 12 dataset–length settings, the lowest JFTSD in 11 of 12, and the best FID in 7 of 12.FID, CTTP, and JFTSD evaluate fidelity and caption adherence in an evaluation aligner’s embedding space.
  • Ablations: At its best auxiliary-loss weight, every backbone-adaptation architecture improves FID, CTTP, and JFTSD over λdw=0; LoRA at λdw=1 improves them by 3.1/2.9/5.4%.The three percentages correspond to FID/CTTP/JFTSD.
  • Ablations: Leaving the text encoder frozen collapses caption adherence: Tune-TS reaches CTTP 0.405 versus 0.632 for Tune-Text and 0.641 for Tune-Both.Tune-Both is the default, and the accompanying comparison also reports JFTSD 0.215 for Tune-TS.

4 Related Work

Time-series generation has progressed from unconditional synthesis to structured and free-form text conditioning. Existing text-conditioned methods leave representations insufficiently aligned with the series modality, whereas GALA separates modality alignment from generation and targets generation-specific conditioning.

  • Time-Series Generation: Time-series generation progressed from unconditional synthesis to structured controls such as discrete classes or low-dimensional attribute metadata, and then to free-form text.Structured examples include TTS-CGAN, TimeVQVAE, and Time Weaver.
  • Text-Conditioned Generation: Recent text-conditioned generators include DiffuSETS, T2S, VerbalTS, and BRIDGE, covering clinical-report ECG synthesis, latent diffusion, multi-view denoising, and multi-agent text control.These methods represent recent approaches to flexible free-form conditioning.
  • Text-Conditioned Generation: Existing methods use either frozen text-only encoders or representations trained solely by reconstruction, so conditioning is never deliberately aligned with the time-series modality.GALA instead decouples alignment from generation through a dedicated contrastive stage.
  • Time-Series Multimodal Alignment: Prior time-series multimodal alignment targets forecasting, question answering, or explanation, whereas GALA uses alignment to supply the conditioning signal for generation.Time-LLM and TimesCLIP align series with LLM spaces for forecasting; TEST and ChatTS use text prototypes or synthetic captions for question answering and explanation.

5 Conclusion · Appendix

The conclusion identifies an implicit language–signal correspondence as a blind spot in existing text-to-time-series generation. GALA addresses it through a dedicated alignment stage separated from synthesis, with an information-theoretic rationale for this decomposition and for augmenting contrastive learning.

  • 5 Conclusion: Existing methods leave language–signal correspondence implicit.The blind spot arises from relying on text-only conditioning or shaping representations through generative loss alone.
  • 5 Conclusion: Text-only encoders provide the conditioning signal without explicitly matching language to the signal modality.This is identified as one source of the implicit correspondence problem.
  • 5 Conclusion: Generative loss alone shapes conditioning representations without making language–signal correspondence explicit.The conclusion distinguishes this limitation from using a text-only encoder.
  • 5 Conclusion: GALA remedies the blind spot with a dedicated alignment stage held apart from synthesis.The method separates representation alignment from the generation process.
  • 5 Conclusion: The objective factors into a representation term and a generation term.This factorization supplies the information-theoretic rationale for separating alignment from synthesis.
  • 5 Conclusion: A contrastive criterion alone falls short once negatives are easy to rank.The conclusion presents this as an information-theoretic reason contrastive learning needs the broader alignment formulation.

A Datasets and Preprocessing · B Stage-1: Cross-Modal Alignment · C Stage-2: Conditional Generation

The method uses globally normalized TSFragment-600K data to align text and time-series encoders in Stage 1, then conditions Stage 2 generation on the resulting frozen 512-dimensional caption embedding. Stage 1 jointly trains contrastive alignment with an auxiliary text-conditioned denoiser that is discarded afterward.

  • A Datasets and Preprocessing: TSFragment-600K contains four sources and three fragment lengths, with each source-length configuration split 90/10 into training and held-out test sets.The sources are ETTh1s, ETTm1, electricity, and traffic; lengths are L ∈{24, 48, 96}.
  • A Datasets and Preprocessing: Stage 1 replaces per-instance normalization with dataset-level global z-score normalization estimated only from training data, preserving absolute level and scale in embeddings.
  • B Stage-1: Cross-Modal Alignment: The Stage-1 auxiliary denoiser is conditioned on the text embedding, trained jointly with the contrastive objective, and discarded after alignment.
  • C Stage-2: Conditional Generation: Stage 2 conditions the generator only on the frozen 512-d caption embedding and the timestep.
  • B Stage-1: Cross-Modal Alignment: Both the Embedding-Gemma-300M text encoder and Chronos-2 series encoder use LoRA adaptation with shared embedding dimension d=512.The listed LoRA configuration uses rank r=64, α=16, dropout 0.05, and target modules {q, k, v, o} in both towers.
  • C Stage-2: Conditional Generation: The alignment objective combines symmetric InfoNCE with λdw Lgen, using fixed temperature τ=0.07 and default λdw=1.0.
  • C Stage-2: Conditional Generation: The discarded auxiliary denoiser uses a 2-D JiT backbone with hidden size 64, depth 2, heads 4, patch size 2, and image side 10.Its timestep schedule is logit-normal with Pmean=0.5 and Pstd=1.2.
  • C Stage-2: Conditional Generation: In the fine-tuning-method ablation, FT-Full uses learning rate 1 × 10−5 while all other settings remain unchanged.

D Length- and Dataset-Specific Settings

This section specifies length- and dataset-specific implementation settings for delay embeddings and logit-normal sampling. Fragment lengths determine the embedding configuration, while dataset–length pairs determine the logit-normal mean.

  • Delay embedding maps each length-L fragment to a square m × m image with column stride s.
  • Table 7 lists delay-embedding values separately for each fragment length.
  • Table 8 specifies the logit-normal mean Pmean for each dataset–length pair, with Pstd = 1.2 throughout.

E Baseline Implementations and Hyperparameters … F Evaluation Protocol

The paper implements all baselines in a shared harness while preserving each method’s official architecture and released configuration. It details method-specific conditioning, generation, optimization, and pretraining choices, then evaluates every method with identical data handling and embedding-space scoring.

  • E Baseline Implementations and Hyperparameters: All five baselines and GALA share data loading, train/validation splits, checkpointing, generation evaluation, and metric embedding space.Each method’s generator and conditioning path follows its official architecture and released configuration.
  • E.1 VerbalTS: VerbalTS uses frozen LongCLIP per-token features, a trainable text projector, multi-scale patch denoising, and deterministic Heun sampling for 50 steps.Its flow-matching setup uses X-prediction plus velocity matching, logit-normal timesteps with Pstd=1.2, and noise scale σ 1.0.
  • F Evaluation Protocol: Checkpoint selection and metric embedding follow the Evaluation Protocol, ensuring no method is scored in its own embedding space.The common protocol is shared by all five baselines and GALA.
  • E.2 BRIDGE: BRIDGE is an observation-space DDPM whose 1-D U-Net cross-attends to domain prototypes and fuses pooled text through a gated conditioning MLP.Harness-level condition dropout is disabled because BRIDGE already uses classifier-free guidance with a learnable null embedding.
  • E.3 T2S: T2S pretrains a length-adaptive VAE and trains a rectified-flow Transformer in the fixed (64, 30) latent geometry.Conditioning uses the dataset’s 128-d TextEmbedding column rather than a text encoder trained in this study.
  • E.4 DiffuSETS: DiffuSETS trains one VAE per source and length, then uses a latent DDPM U-Net whose cross-attention reads the dataset’s 128-d TextEmbedding column.The released setting keeps guidance scale at 1.0, meaning no classifier-free guidance, with condition dropout 0.
  • E.5 Text2Motion: Text2Motion pretrains a movement autoencoder per source and length, then generates in its latent space with a text-conditioned autoregressive CVAE.It uses the dataset’s 128-d TextEmbedding column and released Comp trainer settings: plain Adam, gradient clipping 0.5, and no weight decay.
  • E.6 Stage-1 Pretraining for the Two-Stage Baselines: T2S, DiffuSETS, and Text2Motion each require stage-1 autoencoder pretraining before generator training, using the same validation ratio and split seed as stage 2.Thus, each stage-1 model is trained without seeing a held-out fragment; Table 15 contains the settings.

F.1 The Evaluation Aligner · F.2 Checkpoint Selection and Aggregation · G Variability and Significance Testing

The paper evaluates fidelity and caption adherence in a frozen, held-out text–time-series embedding space, then selects checkpoints by FID and aggregates repeated samples to quantify sampling variability. It additionally reports dispersion and statistical tests to assess whether close metric-based rankings are supported.

  • F.1 The Evaluation Aligner: FID measures fidelity, while CTTP (↑) and JFTSD (↓) measure caption adherence using a single evaluation aligner.The aligner is a text–time-series dual encoder used only for measurement, with metrics computed under the ConTSG-Bench reimplementation.
  • F.1 The Evaluation Aligner: The evaluation aligner is trained without the held-out fragments used for scoring, then frozen and reused unchanged across methods, ablations, and figures.This keeps reported numbers commensurable and prevents scoring a model in an embedding space it helped produce.
  • F.2 Checkpoint Selection and Aggregation: Generation metrics are evaluated every 100 epochs for the online, ema1, and ema2 variants.Checkpoint evaluation covers all three model variants throughout training.
  • F.2 Checkpoint Selection and Aggregation: For each run, the selected checkpoint is the epoch × variant with the lowest FID on the full held-out split.Selection is based on the minimum FID across evaluated epochs and variants.
  • F.2 Checkpoint Selection and Aggregation: 3 independent sampling seeds (100/200/300) produce the reported mean ± sample standard deviation for each selected checkpoint.Both stages use fixed seed 0, so the reported variation represents sampling variance rather than training variation.
  • G Variability and Significance Testing: The appendix reports dispersion and tests whether the metric ordering in Table 1 is statistically supported.This addresses columns whose rankings depend on differences in the third decimal, whereas Table 1 reports means only.

G.1 Per-Cell Sampling Variability · G.2 Wilcoxon Signed-Rank Tests Against the Baselines

GALA’s sampling variability is small relative to method gaps, with most per-cell comparisons remaining statistically significant against VerbalTS. Holm-corrected Wilcoxon tests confirm broad gains across metrics, while fidelity against VerbalTS remains indistinguishable.

  • G.1 Per-Cell Sampling Variability: Table 17 measures sampling dispersion across 216 cells spanning 6 methods, 4 datasets, 3 lengths, and 3 metrics.The reported statistic is the sample standard deviation over 3 sampling seeds.
  • G.1 Per-Cell Sampling Variability: 0.0033 is GALA’s largest standard deviation, versus 0.0165 overall; 98.1% of cells are below 0.01.Median standard deviations are 0.0011 for FID, 0.0026 for CTTP, and 0.0024 for JFTSD.
  • G.1 Per-Cell Sampling Variability: 29 of GALA’s 30 wins against VerbalTS remain significant at α = 0.05 after Holm correction across 36 per-cell Welch tests.Across winning cells, the median gap is 34× the pooled standard deviation.
  • G.1 Per-Cell Sampling Variability: The unresolved cells are traffic-48 JFTSD, where GALA wins, plus the FID cells of ETTh1s-96 and ETTm1-96, where GALA loses.In all three cases, the methods are within one standard deviation of each other.
  • G.2 Wilcoxon Signed-Rank Tests Against the Baselines: Wilcoxon signed-rank tests compare GALA with each baseline separately for FID, CTTP, and JFTSD over 12 settings, with Holm correction across five comparisons per metric.Keeping metrics separate avoids weighting them according to arbitrary numeric ranges.
  • G.2 Wilcoxon Signed-Rank Tests Against the Baselines: FID against VerbalTS is the only nonsignificant comparison, so the two methods are indistinguishable on fidelity.Per-length tests pool the three metrics, and the L = 96 result reflects this FID parity rather than weaker caption adherence.
  • G.2 Wilcoxon Signed-Rank Tests Against the Baselines: At L = 96, the per-length result is 8/12 wins, 12.8%, with p = 3.7 × 10−6 across all 36 pooled columns, corresponding to 30/36 wins and 28.6%.Per-cell Welch tests are used for dataset-length pairs because a signed-rank test on only three columns cannot attain p < 0.125.

H Morphological Diversity of the Benchmark Sources · I Sweeps · J Software and Hardware

The appendix characterizes benchmark shape diversity, showing Traffic is morphologically degenerate and where GALA’s gains are thinnest, while documenting sweeps and implementation details. It also cautions that diversity–performance associations are not causally established.

  • H Morphological Diversity of the Benchmark Sources: The protocol z-normalizes 2000 fragments per source-length configuration, then evaluates PCA2, PC90, effective rank, and k=6 KMeans clustering.Z-normalization removes level and amplitude, isolating morphology.
  • H Morphological Diversity of the Benchmark Sources: Traffic needs 2–4 principal components for 90% shape variance, versus 6–11 for other sources, with effective rank 2.3–2.6 and silhouette 0.35–0.43.Random fragment-pair mean |r| is 0.55–0.59 for Traffic versus 0.31–0.45 elsewhere.
  • H Morphological Diversity of the Benchmark Sources: Averaged across lengths, GALA’s relative gain over VerbalTS on Traffic is +21.7% CTTP, +0.1% JFTSD, and −11.1% FID, versus +30.8%, +54.5%, and +20.7% elsewhere.Traffic is the only dataset-length pair where a caption-adherence metric favors VerbalTS, and three of GALA’s six losses occur there.
  • H Morphological Diversity of the Benchmark Sources: Across 12 configurations, effective shape rank was not significantly correlated with relative gain: Spearman ρ = +0.01 CTTP, +0.29 JFTSD, and +0.34 FID, all p > 0.25.The appendix presents diversity as benchmark characterization and a plausible interpretation, not a demonstrated mechanism.
  • I Sweeps: The reported sweeps vary auxiliary-loss weight λdw ∈{0, 1, 2, 4, 8}, fine-tuning method, adapted tower, and series normalization.The fine-tuning methods are LoRA, FT-Full, and FT-Head; adapted towers are text-only, series-only, and both.
  • I Sweeps: Each sweep’s default-configuration cell reproduces the corresponding Table 1 result.Normalization choices are global z-score and per-sample.
  • J Software and Hardware: All experiments ran on one NVIDIA RTX PRO 6000 Blackwell GPU using Python 3.11, PyTorch 2.11.0, and the listed current-generation libraries.The implementation also used Transformers 4.57.6, PEFT 0.18.1, chronos-forecasting 2.2.2, Accelerate 1.12.0, and NumPy 2.1.3.
Loading 2608.13741v1…