Source-linked AI summary
Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language
Yunkai Zhang, Yawen Zhang, Ming Zheng, Kezhen Chen, Chongyang Gao, Ruian Ge, Siyuan Teng, Amine Jelloul, Jinmeng Rao, Xiaoyuan Guo, Chiang-Wei Fang, Zeyu Zheng, Jie Yang
TL;DR
Large-scale datasets pairing general time-series data with language descriptions are lacking, while time-series windows have no original captions and are difficult to describe directly in natural language. The model starts from pretrained LLaVA weights, finetunes them for time series, converts windows into line plots, and maps vision embeddings into language space for response generation. The authors report that their 100,000-sample dataset can enable existing LMMs to align time-series data with textual descriptions and perform detailed analysis.
Problem
Method
The model starts from pretrained LLaVA weights, finetunes them for time series, converts windows into line plots, and maps vision embeddings into language space for response generation.
Source: §3 · p. 4Results
The authors report that their 100,000-sample dataset can enable existing LMMs to align time-series data with textual descriptions and perform detailed analysis.
Source: §5 · p. 5Takeaways & Limitations
The dataset provides a large-scale resource for time-series language alignment and a starting point for further LMM research on time-series analysis.
Source: §1 · p. 2, §5 · p. 5The current workflow focuses on trend descriptions; describing other properties and multifeature time series remains an open challenge.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Time-series data is critical across many scientific and industrial domains, including environmental analysis, agriculture, transportation, and finance. However, mining insights from this data typically requires deep domain expertise, a process that is both time-consuming and labor-intensive. In this paper, we propose \textbf{Insight Miner}, a large-scale multimodal model (LMM) designed to generate high-quality, comprehensive time-series descriptions enriched with domain-specific knowledge. To facilitate this, we introduce \textbf{TS-Insights}\footnote{Available at \href{https://huggingface.co/datasets/zhykoties/time-series-language-alignment}{https://huggingface.co/datasets/zhykoties/time-series-language-alignment}.}, the first general-domain dataset for time series and language alignment. TS-Insights contains 100k time-series windows sampled from 20 forecasting datasets. We construct this dataset using a novel \textbf{agentic workflow}, where we use statistical tools to extract features from raw time series before synthesizing them into coherent trend descriptions with GPT-4. Following instruction tuning on TS-Insights, Insight Miner outperforms state-of-the-art multimodal models, such as LLaVA \citep{liu2023llava} and GPT-4, in generating time-series descriptions and insights. Our findings suggest a promising direction for leveraging LMMs in time series analysis, and serve as a foundational step toward enabling LLMs to interpret time series as a native input modality.
1 Introduction
Time-series analysis has relied on statistical methods, while many LLM approaches focus on numerical outputs rather than natural-language explanations. The paper addresses the lack of a general time-series alignment dataset by proposing a dataset for LMMs to generate descriptive insights.
1 Introduction
- Traditional statistical tools such as ARIMA, STL, and state space models support forecasting, seasonality detection, and trend interpretation across domains.The passage names economics, meteorology, and transportation as fields where these techniques are prevalent.Source: §1 · p. 2
- Existing LLM time-series work targets forecasting, classification, anomaly detection, or numerical outputs, which does not by itself enable natural-language explanations of data.The paper contrasts those approaches with LMM alignment work in finance and its aim to generalize beyond a specific domain.Source: §1 · p. 2
- The paper presents a time-series analysis dataset for descriptive insights and a large-scale repository aligning time-series data with language.Its stated contributions are to enable faithful, descriptive insights and support future LMM research on time-series analysis.Source: §1 · p. 2
2 TS-Insights Dataset
TS-Insights addresses the lack of large-scale time-series and language-description pairs with a dataset for general time-series analysis. Its workflow extracts trend information before GPT-4 generates descriptions, and the resulting corpus is expanded through augmentation and rephrasing.
2 TS-Insights Dataset
- TS-Insights is presented as the first dataset specifically curated for general time-series analysis alignment, addressing the absence of large-scale time-series and language-description pairs.Time-series windows lack original captions, are difficult to convert into an input suitable for language-only models, and have semantics that are harder to describe in natural language.Source: §2 · p. 2
- The dataset pairs sampled time-series windows with question-answer instruction examples, with window length and feature count sampled from the source datasets.The sampling description defines each window as a sequence of time steps and features, then formats the window and question as an instruction-following example.Source: §2 · p. 2
2.1 Trend Generation Workflow
- Because GPT-4 did not reliably extract trend, seasonality, and residuals from raw values, the workflow decomposes series with STL and currently generates trend descriptions.It generates descriptions from one component at a time; trend description is the paper's proof of concept.Source: §2 · p. 3
- For windows without seasonality, the workflow fits a Gaussian Process and uses its posterior mean as the extracted trend before smoothing, downsampling, and prompting GPT-4.The trend is smoothed with a Gaussian kernel, downsampled to 25 values, rounded to one decimal place, and paired with the original window.Source: §2.1 · pp. 3–4
2.2 Trend Description Dataset
- The authors generate 10,000 initial samples from 29 Monash datasets, reserve 11 datasets for evaluation, and expand the corpus to 100,000 samples through augmentations and rephrasing.Initial windows are drawn from the first 70% of each dataset's time steps; each original sample receives nine augmentations, and GPT-3.5-turbo rephrases its description.Source: §2.2 · p. 4
3 Insight Miner
Insight Miner adapts LLaVA to time-series inputs by representing each series as a line plot. It aligns those inputs with language by fine-tuning the projection layer while freezing the vision encoder and language model.
3 Insight Miner
- Insight Miner starts from pretrained LLaVA weights and converts each time-series window into a line plot for the vision encoder.A linear projection maps vision output into the language embedding space, where it is combined with language instructions before response generation.Source: §3 · p. 4
- Training fine-tunes only the linear projection layer, keeping the vision encoder and language model frozen, to predict GPT-generated trend descriptions from line plots.The language instruction asks the model to describe the trend; the resulting model is named Insight Miner.Source: §3 · p. 4
- The authors report training Insight Miner on eight A100 40GiB GPUs, with each epoch taking around an hour and low inference cost after training.The passage characterizes training cost as relatively affordable and deployment inference cost as low.Source: §3 · p. 4
4 Experiments
The experiments assess the faithfulness of generated trend descriptions using human ratings on test and holdout windows. The supplied passages describe the evaluation setup but do not report comparative scores.
4 Experiments
- The evaluation uses 119 windows: 69 from test splits of training datasets and 50 from held-out datasets.The test examples come from the final 30% of the same datasets used for training; holdout datasets are not used for training.Source: §4 · p. 4
- The comparison includes pretrained LLaVA, one- and three-epoch vision fine-tunes, and GPT-4 supplied with extracted features.The paper identifies the two vision fine-tunes as versions of Insight Miner.Source: §4 · pp. 4–5
- Three domain experts score each model's description as correct, partially correct, or incorrect, and the summed scores are normalized for test and holdout samples.Descriptions are shuffled in random order for each sample before expert scoring.Source: §4 · p. 5
5 Discussions
The paper presents TS-Insights as a large-scale resource for aligning time series with language and enabling detailed analysis by existing multimodal models. It also identifies unpretrained time-series encoders and multi-feature descriptions as areas for future work.
5 Discussions
- TS-Insights provides 100k training samples pairing time series with natural-language descriptions for general time-series analysis.The dataset is presented as enabling existing large multimodal models to align time series with text and perform detailed analysis.Source: §5 · p. 5
- Replacing LLaVA’s vision encoder with OneFitsAll led to incoherent descriptions for most samples, likely because the time-series encoder was not pretrained.The authors leave pretraining the time-series encoder as future work.Source: §5 · p. 5
- The workflow suggests extensions to volatility-change and residual-based outlier descriptions, while multi-feature descriptions using cross-correlations are more challenging.Source: §5 · p. 5
A Trend Dataset Details
This section describes the dataset’s augmentations, holdout sets, and evaluation sample coverage. It also notes that augmented windows were not saved, though the authors say they can be regenerated from the script.
A Trend Dataset Details
- Each listed augmentation may be applied with 50% probability, and multiple augmentations may be combined on a window.Source: A Trend Dataset Details · p. 7
- Jittering adds Gaussian noise based on a size-4 local rolling-window standard deviation; scaling multiplies the series, and shifting adds a constant.Source: A Trend Dataset Details · p. 7
- Smoothing convolves the window with a randomly sized average kernel, while downsampling keeps every other k steps.Source: A Trend Dataset Details · p. 7
- The holdout sets include Electricity Demand, M3, M4, Traffic, and Weather datasets at the listed frequencies.The section identifies 20 datasets involved in generating TS-Insights.Source: A Trend Dataset Details · p. 7
- Ten windows per holdout and test dataset are released, but evaluation used only the first three samples from each dataset.Only original windows were saved; the authors say augmented windows can be regenerated using the provided script.Source: A Trend Dataset Details · pp. 7–8