MINT: A Universal Zero-Shot Predictor for Transaction Data

Parameswaran Kamalaruban; Viktor Drobnyi; Maeve Madigan; Julia Rozanova; David Sutton; Stuart Burrell

Source-linked AI summary

MINT: A Universal Zero-Shot Predictor for Transaction Data

Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan, Julia Rozanova, David Sutton, Stuart Burrell

arXiv:2608.14198v1Aug 14, 2026cs.LGcs.CL

TL;DR

Transaction reasoning must handle increasingly inefficient text serialization of long histories while using rich transaction representations. MINT connects a pretrained transaction encoder to a decoder-only LLM through embedding injection and achieves state-of-the-art zero-shot predictive QA with favorable efficiency.

Problem
Text serialization of transaction histories becomes inefficient as histories grow, motivating more scalable transaction reasoning.
Method
MINT connects a pretrained transaction sequence encoder to a decoder-only LLM through an embedding connector and parameter-efficient adaptation.
Results
MINT achieves state-of-the-art zero-shot predictive QA across in-distribution and out-of-distribution settings while reducing tokens, latency, and GPU memory usage.
Takeaways & Limitations
Compact transaction representations support effective reasoning across extractive QA, predictive QA, and captioning while maintaining favorable inference efficiency.
Takeaways & Limitations
The study uses proprietary transaction data that cannot be publicly released because of privacy and regulatory constraints.

Abstract

from arXiv · show

Banks analyse sequential financial transaction data to perform many tasks, including fraud prevention, credit risk assessment and offer personalization. To improve the predictive accuracy of these tasks, Payments Foundation Models encode transaction sequence data as rich contextual embeddings, which can then be provided to task-specific models as features. However, these Foundation Models are not designed for flexible zero-shot reasoning across novel downstream prediction tasks, limiting their adaptability and utility. Existing LLM-based approaches to zero-shot prediction often fail to fully exploit the predictive signal within transaction data, while relying on costly text serialization or task-specific architectures that scale poorly. To address these limitations, we present the Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning. We find that MINT achieves state-of-the-art predictive question-answering performance in both in-distribution and out-of-distribution questions, while substantially reducing input tokens, latency, and memory consumption compared to text-serialization baselines. Through comprehensive analyses of representations, alignment strategies, training data, and history length, we establish that compact transaction embeddings are a superior approach to transaction representation than text serialization for multimodal reasoning and zero-shot prediction tasks.

1 Introduction

Transaction data combines heterogeneous fields with long-range temporal dependencies, making it difficult for conventional language-based approaches to exploit its structured signals efficiently. MINT addresses this by linking a pretrained transaction encoder to a decoder-only LLM through compact embeddings, modality alignment, and parameter-efficient adaptation.

Motivation: Transaction histories require modeling heterogeneous within-transaction relationships and temporal dependencies across potentially long behavioral horizons.
Motivation: LLMs enable natural-language interaction, instruction following, and explanations, but serializing transaction histories becomes inefficient as histories grow longer.
Motivation: Text-based approaches often struggle to exploit structured numerical and temporal signals, motivating compact transaction representations for multimodal reasoning.
Foundation models: Self-supervised transaction foundation models learn transferable behavioral representations and perform strongly across diverse downstream financial tasks.
Contributions: MINT integrates a pretrained transaction sequence encoder with a decoder-only LLM through embedding injection, modality alignment, and parameter-efficient adaptation.
Contributions: MINT uses transaction embeddings instead of textual serialization for predictive reasoning and reports state-of-the-art predictive QA performance in both ID and OOD settings.

2 Related Work

Prior work models transactions as structured event sequences and connects specialized sequence encoders with LLMs for multimodal reasoning. MINT instead decouples transaction representation learning from reasoning-model training, using a lightweight projector and transaction-language alignment for generative tasks.

Event sequence foundation models: Hierarchical architectures separately encode field-level information within transaction events and temporal dependencies across events.Large-scale self-supervised pretraining on transaction corpora has also established pretrained transaction sequence encoders.
Multimodal event sequence reasoning: Multimodal event-sequence systems connect specialized sequence encoders to LLMs through projection layers, cross-attention modules, or other fusion mechanisms.This work addresses challenges including long-context processing, numerical reasoning, and faithful generation.
Training strategy: MINT first pretrains a transaction sequence encoder on large-scale transaction data and then freezes it during reasoning-model training.Prior approaches either jointly optimize the encoder, connector, and LLM or use more complex fusion architectures.
Modality alignment and instruction tuning: This separation simplifies training and enables a lightweight MLP projector instead of more complex architectures such as Q-Former.It avoids repeatedly processing massive datasets through the LLM while learning transaction representations.
Modality alignment and instruction tuning: Unlike task-conditioned approaches using task identifiers, task embeddings, or task-specific control tokens, MINT treats transaction understanding as part of multimodal reasoning training.Transaction-language alignment serves as an intermediate stage targeting generative tasks such as transaction captioning and question answering.

3 Model Architecture

MINT combines a frozen Transformer-based transaction encoder, a trainable connector, and a decoder-only LLM adapted with LoRA. Projected transaction embeddings replace placeholder tokens, enabling compact conditioning for autoregressive generation.

Model components: MINT comprises a frozen transaction sequence encoder, a trainable connector into the LLM hidden space, and a decoder-only LLM adapted through LoRA.Base LLM weights remain frozen, with gradients flowing only through LoRA and connector parameters.
Transaction encoder: A field-level Transformer models within-transaction interactions, while a sequence-level Transformer captures temporal dependencies across transactions.Numerical attributes are log-transformed and categorical attributes use field-specific lookup tables.
Transaction encoder: The encoder is pretrained with autoregressive next-event prediction and remains frozen during multimodal training as a transaction feature extractor.This pretraining learns customer behavioral patterns before multimodal adaptation.
Modality projector: The connector uses a normalized multilayer perceptron to map each transaction embedding e_i ∈ R^d_e into the LLM hidden space R^d.Its capacity is varied in Section 6.4 ablations.
Embedding injection: Each <emb> placeholder is replaced with a projected transaction embedding, allowing long histories to enter the LLM as compact embeddings rather than serialized text.The resulting embedding sequence is processed for autoregressive generation.

4 Datasets

MINT training data unifies transaction captioning and question answering under an instruction-following format grounded in transaction histories. The datasets combine proprietary anonymized card transactions with programmatically generated, LLM-synthesized captions, QA examples, and filtered reasoning rationales.

Task Families: MINT trains on transaction captioning and multiple-choice transaction question answering, covering customer behavior summaries and questions grounded in transaction history.Captioning includes spending patterns, merchant preferences, and temporal trends; QA includes extractive questions.
Transaction Data: The transaction corpus contains anonymized card transactions from a real-world payment network, with timestamps, numerical attributes, and high-cardinality categorical features.Examples include transaction amount, merchant identity, merchant category, and location; specific attribute names are withheld for sensitivity.
Caption Data: 134k/14k/19k train/validation/test examples comprise the captioning dataset generated from attribute-based templates and teacher-LLM semantic synthesis.Structured reports use transaction attributes and temporal aggregates, while semantic captions use textualized transaction histories and Mistral-7B-Instruct-v0.3.
Reasoning Supervision: 94k/16k/26k extractive QA examples span 8 question types in the final chain-of-thought datasets after answer-consistency, length, and manual-quality filtering.A teacher LLM generates step-by-step rationales conditioned on the transaction history, question, and ground-truth answer.

5 Training

MINT trains in two stages: modality alignment maps transaction representations into the LLM token space, followed by supervised fine-tuning of the connector and LoRA adapters on captioning and QA data. The transaction encoder remains frozen, enabling efficient adaptation from reusable representations with a few hundred thousand supervised examples.

Modality Alignment: MINT first trains a connector to map frozen transaction representations into embeddings compatible with LLM token representations.During alignment, only the connector is updated; the base LLM is frozen and LoRA is disabled.
Supervised Fine-Tuning (SFT): Supervised fine-tuning jointly optimizes the connector and LoRA adapters on a mixture of captioning and QA datasets.Captioning and QA examples are interleaved to preserve generation quality while improving grounded reasoning.
Supervised Fine-Tuning (SFT): For QA examples with rationales, targets concatenate the reasoning trace and final answer, whereas answer-only examples contain only the short answer.This target construction supports both rationale-based and concise answer supervision.
Decoupled Representation and Reasoning Learning: MINT decouples transaction representation learning from language-model reasoning by keeping the pretrained transaction sequence encoder frozen throughout multimodal training.The encoder is pretrained on unlabeled transaction sequences with a next-transaction prediction objective, while multimodal training updates the connector and LoRA adaptation.
Decoupled Representation and Reasoning Learning: A few hundred thousand supervised transaction captioning and QA examples suffice to align reusable pretrained transaction representations with the LLM for domain-specific reasoning tasks.Direct embedding injection avoids repeatedly processing large transaction corpora during multimodal training.

6 Experiments

MINT outperforms text-serialized LLM SFT on predictive QA and OOD extractive QA, while requiring substantially less inference cost. Ablations show that compact history, rationale supervision, and suitable connector capacity support performance, but domain adaptation reduces general-purpose reasoning.

Predictive QA: MINT (ℎ= 1) outperforms LLM SFT (ℎ= 1, ℎ= 5) on both ID and OOD predictive QA.A single injected transaction embedding is sufficient to outperform text-serialized histories for forecasting future behavior.
Extractive QA: Under OOD evaluation, MINT (ℎ= 1) surpasses LLM SFT (ℎ= 1, ℎ= 5), while LLM SFT is relatively stronger on ID extractive QA.Both methods improve monotonically with increasing history length for extractive QA.
History length: For predictive QA and OOD tasks, additional history embeddings provide no consistent benefit and can hurt performance, whereas extractive QA (ID) improves monotonically with history length.The results indicate that MINT extracts relevant signal from very limited context and that history length should be adapted by task.
Efficiency: MINT (ℎ= 5) achieves 18% lower TTFT, 2.4× higher decode throughput, 11% lower peak VRAM, and 26% fewer input tokens than LLM SFT (ℎ= 1).Embedding injection scales more efficiently than tokenized serialization as history length increases.
Ablations: Rationale-augmented QA data considerably improves performance across tasks, while connector hidden size ℓ= 512 improves both extractive and predictive QA (OOD).Varying LoRA rank has little overall effect, except that r= 32 performs considerably better on predictive QA (OOD).
Limitations: GSM8K accuracy drops from 0.412 to 0.026 (MINT) and 0.033 (LLM-SFT) after adaptation, reflecting a trade-off between transaction-domain specialization and general-purpose reasoning.ARC-Challenge and BoolQ also decline relative to the base Qwen3-1.7B model.

7 Conclusion

MINT integrates pretrained transaction embeddings with a decoder-only LLM to enable efficient multimodal reasoning across transaction tasks. Its findings establish complementary roles for embeddings and text serialization, while identifying task-, context-, and distribution-dependent design choices and important deployment limitations.

Contributions: MINT integrates a pretrained transaction sequence encoder with a decoder-only LLM through embedding injection, supporting extractive QA, predictive QA, and transaction captioning.The framework maintains favorable inference efficiency while enabling effective reasoning with compact transaction representations.
Contributions: MINT achieves state-of-the-art zero-shot predictive QA and systematically studies history representation, modality projection, training data, prompt design, and inference trade-offs.Its training incorporates both captioning and chain-of-thought supervision.
Key findings: Transaction embeddings outperform text serialization for predictive reasoning, whereas serialized histories remain strong for extractive retrieval.The predictive advantage is hypothesized to reflect autoregressive self-supervised pretraining aligned with forecasting future behavior.
Key findings: Embedding injection achieves comparable or stronger performance with substantially fewer input tokens, lower latency, and less GPU memory than text serialization.This supports more efficient scaling of transaction reasoning systems.
Key findings: Increasing transaction history consistently improves extractive QA but offers limited gains for predictive QA, making optimal context size task dependent.Earlier encoder layers improve in-distribution performance, while final-layer embeddings generally strengthen out-of-distribution generalization.
Key findings and limitations: Adding chain-of-thought data broadly improves performance across tasks, particularly for extractive QA.The conclusion also notes privacy, profiling, fairness, and misuse risks, alongside proprietary data constraints that limit direct reproducibility.