PaperlayerView exact revision on arXiv

Source-linked AI summary

TEFM: Token-Efficient Faithful Modeling for Structured Data

Zhichao Hou, Lingdao Sha, Xueyu Mao, Yang Liu, Peijie Qiu, Rui Song

arXiv:2609.09552v1cs.CLcs.LG

TL;DR

LLMs applied to structured data face token-efficiency and faithfulness obstacles in critical domains. TEFM compresses observations into Behavioral Codes and jointly optimizes compressed prediction with grounded rationales. Across clinical and security evaluations, it achieves competitive accuracy with dramatic token reduction and faithful rationale extraction.

  • Problem

    Structured-data applications face excessive token consumption and unreliable, insufficiently grounded predictions in critical domains.

  • Method

    TEFM combines hierarchical Behavioral Code compression, BC–text alignment pretraining, supervised fine-tuning, and dual-fidelity rationalization.

  • Results

    TEFM achieves competitive accuracy across clinical and security tasks while producing faithful rationales and dramatically reducing token consumption.

  • Takeaways & Limitations

    TEFM jointly addresses context efficiency and rationalization for LLM-based classification over high-dimensional structured data.

Abstract

from arXiv · show

In this paper, we solve two fundamental obstacles in applying LLMs to critical domains: token efficiency and faithfulness. To address both constraints jointly, we present TEFM (Token-Efficient Faithful Modeling), a framework designed for structured data analysis in critical domains. TEFM achieves token efficiency by compressing lengthy structured observations into compact Behavioral Code tokens, dramatically reducing token consumption with minimal information loss. Moreover, TEFM enables faithful rationalization through a dual-fidelity objective that jointly optimizes code-level reconstruction and prediction-level fidelity, identifying minimal sufficient feature subsets grounded in input data. Comprehensive experiments across various domain datasets and model backbones (Qwen3, Gemma-2, Phi-4) show that TEFM achieves competitive classification accuracy with dramatic token reduction (approximately 1\% token retention in clinical and 2\% in security domains) while producing faithful rationales.

1 Introduction

Applying LLMs to structured data in critical domains is limited by token inefficiency and faithfulness risks. TEFM addresses both by compressing observations into Behavioral Codes and jointly learning accurate predictions with grounded rationales.

  • Structured medical records and network logs consume excessive tokens when serialized, exhausting context windows and increasing API and memory costs.
  • Unfaithful predictions in healthcare and security can create serious risks, so users need evidence that outputs are grounded in input data.
  • Prior context-extension and rationalization methods address these obstacles separately, while compression approaches can still scale with sequence length or lose information.
  • TEFM proposes a framework that reduces token consumption at the source with minimal information loss while providing faithful rationales grounded in input data.
  • Hierarchical tokenization compresses lengthy observations into compact Behavioral Codes, achieving approximately 99% token reduction with minimal information loss.
  • Dual-fidelity rationalization jointly optimizes code-level reconstruction and prediction-level fidelity to identify minimal sufficient feature subsets without post-hoc perturbation.

2 TEFM: Token-Efficient Faithful Modeling

TEFM combines hierarchical Behavioral Code tokenization, LLM alignment, task-specific fine-tuning, and dual-fidelity rationalization to analyze structured data efficiently and faithfully.

  • Framework overview: TEFM comprises four stages: hierarchical Behavioral Code tokenization, BC–text alignment pretraining, supervised classification fine-tuning, and dual-fidelity variational rationalization.The pipeline is presented as a unified framework for structured-data analysis.
  • Hierarchical High-Fidelity Tokenization: Hierarchical tokenization exploits shared structure in observations to compress lengthy records into compact codes while preserving task-relevant information.Records are vectorized and encoded with an RQ-VAE encoder-decoder, whose residual levels capture dominant patterns and progressively refine them.
  • Hierarchical High-Fidelity Tokenization: Eight length-6 binary observations illustrate the scheme: three hierarchical codes use six vocabulary tokens to encode eight patterns, with three tokens per observation.The example uses three codebook layers of size two.
  • Token-Efficient Training: BC–text alignment pretraining adds Behavioral Code tokens and trains only the embedding layer before supervised fine-tuning on compressed representations.The alignment stage uses paired code–description data, while downstream classification replaces raw records with BC-encoded inputs.
  • Dual-Fidelity Rationalization: Rationalization seeks minimal sufficient feature subsets by preserving predictive power while enforcing compactness through a sparse rationale mask.The learned extractor identifies features responsible for both behavioral-code assignment and prediction.
  • Dual-Fidelity Rationalization: The tractable dual-fidelity design separates prediction fidelity into behavioral-code preservation and prediction consistency on the compressed representation.A learned extractor produces a soft mask, and the two fidelity terms support faithful rationales without directly optimizing repeated discrete re-encoding and LLM inference.

3 Experiments

TEFM is evaluated for tokenization fidelity, alignment, prediction, rationalization, and ablations across clinical and security tasks and multiple LLM backbones. Results show strong compression, competitive performance, faithful rationales, and benefits from alignment and configuration choices.

  • Experimental Setup: TEFM is evaluated on clinical mortality prediction and network intrusion detection using Qwen3, Gemma2, and Phi-4 backbones.The evaluation covers tokenization, BC-text alignment, hierarchical structure, prediction, rationalization, and ablations.
  • Tokenization: 90.6% clinical alignment and over 96.9% security alignment show that BC tokens can be recovered reliably from natural-language descriptions.Tolerant alignment accuracy exceeds 91.7% across domains.
  • Prediction: 99% token reduction is achieved across model families, while five of six models lose less than 2% accuracy and Phi-4-mini loses 0.08%.Qwen3-4B improves from 0.8048 to 0.8117, indicating compression need not reduce performance in every setting.
  • Rationalization: At 5% feature retention, Dual Fidelity reaches 0.7472 accuracy versus SHAP’s 0.7051, a 6.0% advantage under aggressive compression.At 10% retention, Dual Fidelity achieves 0.7704 versus SHAP’s 0.7429, while at 20% the methods are nearly tied.
  • Rationalization: The clinical rationale example combines respiratory, cardiovascular, neurological, and infectious dysfunction with mechanical ventilation and severe sepsis.The case is presented as a 5% retention rationale for a rapidly deteriorating patient.
  • Ablation Study: BC alignment pretraining improves clinical accuracy by 6.37% and security accuracy by 0.09–0.24%, with larger gains on the clinical task.The ablation attributes robust representation learning to aligning BC tokens with the LLM embedding space.
  • Ablation Study: K3-C128 provides the best overall balance, achieving 82.1% F1 for prediction, competitive 90.6% alignment, and 94.4% slot accuracy with superior reconstruction.K4-C256 reaches 94.8% slot accuracy, but K3-C128 avoids higher computational overhead and performs better downstream.

4 Related Work

Related work addresses efficient context handling, discrete semantic tokenization, and rationalization, but these lines of work target different aspects of structured-data modeling. TEFM builds on them by combining hierarchical codes, LLM alignment, and faithful feature selection.

  • TEFM Positioning: TEFM connects these directions by using hierarchical behavioral codes, BC-text alignment, and dual-fidelity feature extraction for structured-data classification.The cited related-work passages motivate the framework’s combination of compression and rationalization.
  • Context-Efficient Models and Tokenization: Context-efficient architectures use memory tokens, sparse attention, or recurrence to extend context handling but still scale with sequence length.These approaches can also lose information under aggressive compression.
  • Semantic Identifiers and Discrete Tokenization: Semantic identifiers compress high-dimensional observations into discrete tokens, with VQ-VAE and residual quantization preserving hierarchical structure.Prior work also aligns discrete item tokens with LLM embeddings for efficient sequential recommendation.
  • Rationalization: Rationalization methods identify input information driving predictions, often selecting minimal sufficient subsets for interpretability.Extractive rationalization commonly relies on REINFORCE-based training, which can suffer from high variance.

5 Conclusion

TEFM jointly addresses token efficiency and rationalization for LLM classification over high-dimensional structured data. Its framework combines hierarchical compression, BC-text alignment, and dual-fidelity feature extraction.

  • 5 Conclusion: TEFM jointly optimizes context efficiency and rationalization for LLM-based classification over high-dimensional structured data.The framework targets both token consumption and faithful explanation in one modeling pipeline.
  • 5 Conclusion: The framework combines Residual Quantized VAE compression, BC-text alignment pretraining, and dual-fidelity feature extraction.These are identified as TEFM’s three key components.

A Datasets & Task

The paper evaluates TEFM on clinical mortality prediction using MIMIC-III and on network intrusion detection using CIC-IDS2017. Both tasks represent observations as structured sequences for classification.

  • Datasets and Tasks: MIMIC-III is used for mortality prediction, with each patient represented as a sequence of hospital admissions and clinical records.The task predicts whether the patient dies during the final hospitalization.
  • Datasets and Tasks: CIC-IDS2017 is used for intrusion detection on DDoS and PortScan traffic represented as sequences of network flows.The task focuses on classifying whether the final flow belongs to an attack.

B Cross-Domain Evaluation Results

TEFM maintains high classification accuracy while retaining only a small fraction of the original input tokens across DDoS and PortScan detection. The reported results support token-efficient prediction across security tasks.

  • Security-domain token efficiency: 1.72% token retention is reported for DDoS detection with 99.6–99.9% accuracy across models.Table 10 identifies the retained-token fraction relative to the original Qwen3 input.
  • Security-domain token efficiency: 1.71% token retention is reported for PortScan detection with 99.4–99.8% accuracy across models.Table 11 identifies the retained-token fraction relative to the original Qwen3 input.

C Additional Rationalization Cases

The additional cases illustrate rationale extraction at 5% retention for patients with differing critical-care trajectories and clinical profiles. The cases describe both prolonged deterioration and rapid multi-organ collapse.

  • Case 2: At 5% rationale retention, Patient ID 68529 had prolonged ICU illness with acute respiratory failure, cardiac arrhythmia, and UTI.The patient survived initial resuscitation but progressively deteriorated across an extended critical illness course.
  • Case 3: At 5% rationale retention, Patient ID 1855 had a short hospital stay, multi-day ICU care, CHF, ESRD, atrial fibrillation, diabetes, and anemia in CKD.The case included intubation, mechanical ventilation, hemodialysis, and cardioversion during one ICU admission.

D.1 RQ-VAE Configuration Ablation on Security Domains

Security-domain ablations show that compact Behavioral Code configurations can achieve strong performance and high codebook utilization. K3-C128 is highlighted as an efficient configuration for both DDoS and PortScan.

  • Security-task performance: 97–99% slot AUC and >99.9% tolerant accuracy characterize the security tasks.The passage attributes these results to lower dimensionality and binary features.
  • Security-task performance: K3-C128 achieves 97.3% slot AUC on DDoS with 87.5% utilization and 98.6% on PortScan with 72.9% utilization.These results are presented as efficient configurations that do not require larger codebooks.

D.2 BC Alignment Configuration Ablation on Security Domains

Alignment remains high across security configurations, with limited gains beyond K3-C128. PortScan shows stronger alignment than DDoS, while tolerant accuracy stays high across configurations.

  • Alignment quality: K3-C128 achieves 96.9% alignment on DDoS and 98.5% on PortScan.The PortScan result is reported as outperforming DDoS due to binary feature clarity.
  • Alignment quality: 96.9–98.9% alignment is achieved across security tasks, with marginal gains beyond K3-C128.The alignment range covers the reported security configurations.
  • Alignment quality: Tolerant accuracy exceeds 99.6% across security configurations, with alignment errors typically off-by-one.The passage reports this pattern alongside uniformly high alignment.

D.3 Context-efficient Configuration Ablation on Security Domains

Security tasks remain robust across K3 configurations, with F1 varying only minimally. The selected K3-C64 configuration balances model capacity and performance while maintaining high accuracy.

  • 99.1–99.6% F1 across K3 configurations shows minimal performance variation on security tasks.The results indicate that network intrusion patterns are more forgiving of codebook size changes.
  • Network intrusion patterns are more forgiving of codebook size changes than the security-task results might otherwise suggest.
  • K3-C64 is selected for security tasks to balance model capacity with performance while maintaining high accuracy.
Loading 2609.09552v1…