Source-linked AI summary
From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs
Usha Shrestha, Dmitry Ignatov, Radu Timofte
TL;DR
Data-aware augmentation remains difficult for LLM code synthesis because existing approaches rely on heuristics or exhaustive search. The paper uses a closed-loop, performance-aware process that fine-tunes an LLM on empirically evaluated transformations and better-than-worse pairings. It reports up to 600× fewer evaluated candidates than brute-force discovery while maintaining competitive peak accuracy and learning semantic performance cues.
Problem
Data-aware augmentation remains a limiting challenge for LLM code synthesis, which has commonly relied on heuristic design or brute-force approaches.
Method
The method iteratively generates and evaluates PyTorch transformations, then fine-tunes an LLM with LoRA using pairs ordered by downstream accuracy.
Results
Up to 600× fewer candidates were evaluated than with brute-force discovery while mean accuracy increased from 0.4317 at epoch A0 to 0.56 at epoch A27 and peak performance remained competitive.
Takeaways & Limitations
Empirical, non-textual feedback can shift LLM generation toward task-aligned transformations and semantic performance cues without reinforcement learning, reward models, or symbolic objectives.
Takeaways & Limitations
The evaluation is limited to a ResNet on a single dataset, with augmentations implicitly specialized to that configuration; one training epoch also limits assessment of long-term convergence stability.
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches. We introduce a performance-aware, closed-loop solution in the NNGPT ecosystem of projects that enables LLMs to autonomously engineer optimal transformations by internalizing empirical performance cues. We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions, each annotated solely by downstream model accuracy. Training uses pairwise performance ordering (better-worse transformations), enabling alignment through empirical feedback without reinforcement learning, reward models, or symbolic objectives. This reduces the need for exhaustive search, achieving up to 600x times fewer evaluated candidates than brute-force discovery while maintaining competitive peak accuracy and shifting generation from random synthesis to task-aligned design. Ablation studies show that structured Chain-of-Thought prompting introduces syntactic noise and degrades performance, whereas direct prompting ensures stable optimization in performance-critical code tasks. Qualitative and quantitative analyses demonstrate that the model internalizes semantic performance cues rather than memorizing syntax. These results show that LLMs can exhibit task-level reasoning through non-textual feedback loops, bypassing explicit symbolic rewards.
1 Introduction
The paper frames data preprocessing and augmentation as important determinants of neural-network generalization and convergence. It extends LLM-based neural architecture synthesis toward empirically guided data-transformation generation.
- Data preprocessing and augmentation affect neural-network generalization and convergence.
- Prior methods use reinforcement learning or predictive meta-models to identify transformations for particular tasks.
- The study curates diverse PyTorch transformations and links each code snippet to its measured training impact.
- The resulting performance metadata is used to fine-tune an LLM to better understand data variability and performance effects.
2 Related Works
Prior augmentation systems range from fixed discrete-policy search to instance-specific generation, learned transformations, and LLM adaptation. This progression increasingly emphasizes transformation-space diversity, task adaptation, and training-data quality.
- AutoAugment searches approximately 10^32 policies over 14–16 standard image-processing operations.Each policy specifies sequential operations, application probabilities, and magnitudes.
- RandAugment reduces generation to two interpretable hyperparameters, N and M, while matching AutoAugment’s performance.
- Instance-specific frameworks generate a distinct transformation policy for each image rather than one policy for the dataset.A Deep Q-Network iteratively selects transformations or a stop action for each sample.
- Later approaches expand the transformation space through learned functions and generative models such as Spatial Transformer Networks.
- LLM adaptation research emphasizes instruction tuning, supervised fine-tuning, balanced data, and prompt structure as factors affecting generalization and output quality.
3 Methodology
The methodology constructs and evaluates thousands of PyTorch transformation combinations, then iteratively fine-tunes an LLM using performance-ordered examples. Evaluation metadata is generated under a fixed image-classification configuration and reused in the closed loop.
- 3.3 Fine Tuning LLM: Generation used zero-shot, role, constraint, and Chain-of-Thought prompting, alongside configured LoRA and sampling hyperparameters.
- 3.2 Brute Force Approach: The system generated 6,000 transform files, with 2,000 containing one, two, or three variable transforms.Fixed resize, tensor conversion, and normalization operations were included in every file.
- 3.2 Brute Force Approach: The brute-force pipeline automatically generates transformations, evaluates them under fixed training settings, and stores each function with its accuracy.
- 3.3 Fine Tuning LLM: The iterative loop alternates between generating transformations, evaluating them, and refining the LLM with the resulting metadata.LoRA provides parameter-efficient adaptation instead of full-parameter fine-tuning.
- 3.3 Fine Tuning LLM: Training pairs are formed by matching a transform A with an add-on transform B whose accuracy is higher, creating “B better than A” instruction examples.
4 Experiments and Results
Experiments evaluated brute-force and fine-tuned LLM generation on CIFAR-10 with ResNet, showing that iterative fine-tuning improved average transformation quality while retaining competitive peak accuracy. The fine-tuned model also achieved substantially greater sample efficiency than brute-force search.
- Evaluation Setup: All transformations were evaluated on CIFAR-10 image classification with a ResNet trained for one epoch under fixed hyperparameters.The setup used batch size 64, learning rate 0.01, momentum 0.9, and dropout 0.2.
- Brute-Force Baseline: The brute-force pipeline evaluated 6,000 transforms and reached a maximum accuracy of 0.6124 with RandomPosterize, Resize, ToTensor, and Normalize.Single-transform configurations generally outperformed compositions of two or three transforms, with higher accuracy and lower variance.
- Fine-Tuning Process: The fine-tuning dataset began with 2,361 transform-evaluation pairs, removed duplicates, added 1,180 Resize(256) samples, and expanded with generated transforms exceeding 0.55 accuracy.The model generated 10 transforms per epoch across 28 fine-tuning epochs.
- Fine-Tuning Process: Mean accuracy increased from 0.4317 at epoch A0 to 0.56 at epoch A27, while maximum accuracy remained relatively stable across fine-tuning epochs.The narrowing gap between mean and maximum accuracy indicated fewer low-quality outliers and more consistently effective transformations.
- Qualitative Analysis: The model’s generated code reflected semantic links between parameters such as Resize(256) and high accuracy rather than syntax memorization.The generated transformations used varied resolution parameters, including 224, 256, and 32.
- Efficiency: Fine-tuning produced competitive transformations from 280 candidates, compared with 6,000 brute-force candidates, while consistently exceeding the 0.55 accuracy threshold.The fine-tuned model converged on high-performing strategies without exploring thousands of random possibilities.
5 Ablation Study
The ablation study shows that curated data improves fine-tuning stability and accuracy, while direct prompting outperforms verbose Structured Chain-of-Thought prompting. Redundant samples and reasoning constraints hinder semantic optimization and code-generation stability.
- Dataset composition: Curated fine-tuning consistently outperformed the unfiltered dataset, achieving mean accuracies exceeding 0.56.
- Dataset composition: Duplicate transformation files hindered optimization, encouraging memorization of frequent patterns rather than semantic features linked to higher accuracy.
- Prompt engineering: Direct prompting produced better stability and convergence than Structured Chain-of-Thought prompting during fine-tuning.
- Prompt engineering: Structured prompting fell below 0.48 by epoch A10 after its mean accuracy peaked at epoch A6, suggesting that extra analysis tokens introduced noise.
- Prompt engineering: Direct prompting peaked above 0.55 at epoch A10, while removing negative constraints and reasoning requirements concentrated the signal on transform code.
6 Conclusion
The paper presents a closed-loop framework that grounds LLM-generated data transformations in empirical downstream performance. Fine-tuning over evaluated augmentations reduces search while supporting semantic, task-aligned generation without symbolic rewards or reinforcement learning.
- The framework fine-tunes LLMs with Low-Rank Adaptation on over 6,000 empirically evaluated PyTorch augmentation functions.
- Iterative fine-tuning shifts generation from random synthesis toward informed, task-aligned design.
- Up to 600× fewer candidates were evaluated than with brute-force discovery, while mean accuracy improved from 0.43 to 0.56 and peak performance remained competitive.
- Analyses indicate that the model internalized semantic performance cues, such as benefits from resolution scaling, rather than memorizing transformation syntax.
- Grounding LLMs in non-textual empirical feedback loops provides an alternative to symbolic or reward-based alignment for complex downstream objectives.
7 Limitations
The experiments were limited to one ResNet architecture and one dataset, with augmentations specialized to that configuration. A single-epoch training constraint also limits assessment of long-term convergence stability.
- The generative fine-tuning loop and evaluation used only the ResNet architecture and a single dataset.
- The generated augmentations are implicitly specialized for this architecture–dataset configuration rather than validated across alternatives.
- Because alternative architectures and datasets were not tested, the model’s ability to adaptively discover best-fit solutions cannot be quantified.
- Single-epoch training limits assessment of long-term neural-network convergence stability.