Source-linked AI summary
LEMUR 2: Unlocking Neural Network Diversity for AI
Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Muhammad A. Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram, Harsh R. Moradiya, Alice Arslanian, Dmitry Ignatov, Radu Timofte
TL;DR
Existing NAS repositories cover narrow architectural regions and rarely connect architecture specifications with task and device evaluation. LEMUR 2 addresses this gap by combining diverse architecture generation with cross-task evaluation and deployment-aware benchmarking. The resulting corpus contains over 14,000 models and more than 750,000 structured training records, with deployment and task metadata.
Problem
Existing NAS repositories cover narrow, methodologically homogeneous architectural regions and rarely provide an end-to-end path from architecture specification to task evaluation and device deployment.
Method
LEMUR 2 combines programmatic editing, reinforcement learning, evolution, fractal construction, retrieval-augmented module extraction, LLM synthesis, multimodal task pipelines, and deployment benchmarking.
Results
LEMUR 2 contains over 14,000 architectures and more than 750,000 structured training records spanning model, task, and deployment metadata.
Takeaways & Limitations
LEMUR 2 provides a public corpus linking diverse architectural origins with task outcomes and platform measurements for automated AI design.
Takeaways & Limitations
Evaluations use bounded training schedules, so reported scores reflect compute-budgeted performance rather than uniformly converged endpoints.
Abstract
from arXiv · showhide
Existing NAS benchmarks (e.g., NAS-Bench, NATS-Bench) cover only narrow, task-specific regions of the architectural design space and lack cross-domain or deployment-aware evaluation. LEMUR 2 introduces a large-scale, extensible framework unifying generative, evaluative, and deployment pipelines to unlock neural-network diversity. It comprises over 14,000 distinct architectures and more than 750,000 structured training records documenting model performance, hyperparameters, and task outcomes. These models were produced through AST-based code mutation, genetic and reinforcement-learning evolution, generation of fractal architectures, and synthesis guided by a Large Language Model (LLM). This includes deep models generated with the retrieval-augmented system NN-RAG, which derived and used architectural motifs from over 900 PyTorch modules extracted from public repositories. LEMUR 2 further employs NN-VR and NN-Lite pipelines for automated deployment and latency benchmarking on heterogeneous mobile and Unity-based VR platforms, providing real-device performance metadata. It spans multimodal tasks, image captioning, text-to-image synthesis, and language modeling, supporting cross-domain analysis of architectural transferability. By linking diverse architectures, tasks, and deployment data, LEMUR 2 provides the data foundation for LLM fine-tuning and coupling diverse architectural origins with large-scale, cross-platform empirical validation. This dataset defines a new basis for reproducible and data-driven AI design, advancing the emerging paradigm of LLM-driven AutoML and architectural generalization across modalities and hardware.
1. Introduction
LEMUR 2 addresses the narrow, methodologically homogeneous scope of existing neural-architecture repositories by combining diverse architecture generation with task and deployment evaluation. It expands the corpus across models, modalities, transformations, and hardware platforms as a public resource for automated AI design.
- Existing repositories cover narrow, homogeneous architectural regions and rarely connect architecture specifications to task evaluation and device deployment.
- LEMUR 2 combines programmatic editing, reinforcement learning, evolution, fractal construction, retrieval-augmented extraction, and LLM synthesis to generate diverse models.
- Over 14,000 models are generated through several automated methodologies, including genetic algorithms, reinforcement learning, AST editing, and fractal-inspired generation.
- NN-RAG extracts validated, self-contained PyTorch modules from external codebases, creating a library of over 900 reusable components.
- The framework spans image captioning, text-to-text and text-to-image synthesis, and multimodal architectural research, supporting a public resource for automated AI design.
- NN-Lite benchmarks more than 7,500 models on Android, while NN-VR extends automated conversion and performance evaluation to Unity-based virtual-reality contexts.
2. Methodology
LEMUR 2 extends the original LEMUR corpus through a modular generation and integration framework. A fixed-schema data interface supports uniform processing across tasks, architectures, metrics, runtime, and transformations, while heterogeneous generators produce structurally distinct variants.
- LEMUR 2 builds on the original experiment manager with a modular framework for generating candidates and persisting task results.
- Its data interface exposes records through a fixed-schema pandas DataFrame for uniform filtering and aggregation across tasks, datasets, metrics, architectures, epochs, accuracy, runtime, parameters, and transformations.
- Heterogeneous generators and sources prevent the corpus from centering on a single backbone family.
- Seed-architecture mutation uses lightweight reference templates to support high-throughput evaluation and controlled attribution of operator effects while producing structurally distinct variants.
2.1. Task Extensions
LEMUR 2 extends its evaluation scope beyond image classification to image captioning, text-to-image synthesis, language generation, and mixture-of-experts models. These task pipelines use standardized interfaces and task-specific metrics to evaluate generated architectures.
- Image Captioning: NN-Caption generates encoder–decoder architectures coupling convolutional image encoders with LSTM, GRU, or Transformer decoders.
- Image Captioning: NN-Caption contributes 357 captioning architectures trained on MS COCO and evaluated with BLEU-4.
- Text-to-Image Synthesis: The text-to-image pipeline evaluates diffusion, GAN, and CVAE-GAN families conditioned on natural-language text within a unified framework.
- Natural Language Generation: The text-to-text pipeline evaluates recurrent language models on WikiText using perplexity and BLEU, with LEMUR metrics normalized to [0, 1].
- Mixture-of-Experts: LEMUR integrates eight mixture-of-experts variants spanning homogeneous, AlexNet-based, and heterogeneous backbone configurations.
- Mixture-of-Experts: Heterogeneous mixture-of-experts models combine AlexNet, AirNet, DenseNet, and BagNet with soft routing that weights every expert.
2.2. Achieving Diversity
LEMUR 2 achieves architectural diversity through code mutation, reinforcement learning, genetic evolution, fractal construction, retrieval-augmented module mining, and few-shot LLM prompting. These pipelines combine structural variation with validation, deduplication, and bounded evaluation.
- AST-based mutation edits convolutional and linear-layer channel dimensions while preserving tensor-shape compatibility through symbolic tracing and source maps.
- Reinforcement Learning: The reinforcement-learning pipeline masks layer-instantiation code while preserving class structure and forward logic, producing skeletons paired with complete source-code ground truth.
- Reinforcement Learning: Generated completions are validated for compilation and tensor-shape consistency, then trained for one mini-epoch on CIFAR-10 to measure accuracy.
- Reinforcement Learning: The reinforcement-learning reward assigns −1.0 to validation failure, +0.2 per passed stage, and +1.0 × ∆accuracy for improvement over baseline.
- Genetic Algorithm: Genetic evolution searches AlexNet-style architectures by crossover and mutation over parameter vectors, using validation accuracy and architecture checksums for selection and deduplication.
- Fractal Architectures: FractalNet-style generation samples recursive multi-column blueprints and scales to over 1,200 architectures using mixed precision and gradient checkpointing.
- NN-RAG: NN-RAG mines self-contained PyTorch modules from repositories by extracting torch.nn.Module subclasses and resolving dependencies without code execution.
- Few-Shot Prompting: Few-shot prompting supplies high-performing LEMUR exemplars, while canonicalization and MD5 hashing remove exact duplicate architectures before training.
2.3. Cross Domain Deployment and Testing
LEMUR 2 integrates automated model validation, deployment, and benchmarking across Android and Unity-based virtual-reality environments. These pipelines connect model conversion and compatibility checks with latency, memory, and device analytics.
- NN-VR: NN-VR parses ONNX exports, verifies operator and shader compatibility, and prepares models for Unity-based virtual-reality deployment.It also profiles GPU memory and records diagnostic logs during automated porting and inference-scene configuration.
- Data augmentation: Augmentation pipelines are generated by permuting fixed transforms and appending a standard preprocessing suffix.Figure 5 summarizes the brute-force construction procedure.
- NN-Lite: NN-Lite provides an end-to-end automated workflow for converting PyTorch models, deploying them on Android, and reporting benchmark results.The pipeline manages virtual-device execution and combines latency metrics with device analytics in structured reports.
- NN-Lite: Over 7,500 LEMUR models were processed during a documented 48-hour NN-Lite session.Statistical consolidation aggregates task-level accuracy with on-device latency to support unified software and deployment analysis.
3. Evaluation
The evaluation repeatedly tests supported tasks and datasets across architectures and hyperparameter configurations. Because training schedules are bounded, results measure compute-budgeted performance rather than uniformly converged endpoints.
- Evaluation environment: Computer-vision experiments run in AI Linux Docker environments on NVIDIA GeForce RTX 3090/4090 GPUs with 24 GB memory.Experiments use Kubernetes and dedicated workstations.
- Evaluation protocol: Every evaluation run varies hyperparameters such as batch size, epochs, learning rate, momentum, dropout, and transformation pipeline.Accuracy and runtime are logged for systematic comparison across configurations.
- Evaluation protocol: Bounded training schedules make reported scores indicators of compute-budgeted performance rather than uniformly converged endpoints.Comparisons with fully trained baselines should use matched training horizons.
4. Results
LEMUR 2’s generators produce competitive models across image classification and multimodal tasks, but performance and reliability vary substantially by generation strategy and evaluation setting.
- Cross-Method Comparison: Optimization-guided generators produce the strongest and most consistent image-classification results, whereas prompt-based and programmatic mutation methods show higher variance.Best-per-run compares peak accuracy across training epochs rather than final-epoch values.
- Image Classification: Evolutionary models attain the highest median image-classification accuracy, while LLM-generated models occasionally match top groups but show greater variance.AST mutations yield the lowest median accuracy, consistent with sensitivity to the seed architecture.
- Image Captioning: 50% of generations remained runnable with 5–10 prompt snippets, and the best ResNet-50 plus Transformer variant reached BLEU-4 = 0.317 at 50 epochs.With longer training, the score approached 0.3246, illustrating a trade-off between accuracy and architectural diversity.
- Text-to-Image: 0.2751 was the peak CLIP score for the CVAE-GAN text-to-image model using a CLIP encoder, PatchGAN discriminator, and adversarial losses.The configuration prioritized semantic alignment over photorealistic textures.
- Mixture-of-Experts: 93.9% accuracy was achieved by a tuned homogeneous MoE, while a heterogeneous MoE reached 93.13% and outperformed its constituent backbones individually.The heterogeneous system combined AlexNet, AirNet, DenseNet, and BagNet experts.
- Genetic Algorithm: 80.04% test accuracy was achieved by evolving block-type structures, compared with 62.76% for a hyperparameter-restricted genetic search.The champion used batch normalization, 3×3 convolutions, and mixed pooling.
- NN-RAG: 941 of 1,289 candidate blocks passed execution checks in NN-RAG, yielding a library of executable attention, convolutional, and normalization modules.The resulting validation rate was 73.0%.
- Few-Shot Prompting: 53.1% balanced mean accuracy was achieved with three exemplars, whereas six exemplars caused a 99.8% failure rate from context overflow.Three exemplars also improved CIFAR-100 performance by 11.6pp (p = 0.001).
5. Conclusion
LEMUR 2 extends LEMUR into a modular framework combining diverse architecture generation, broad task coverage, and deployment-aware benchmarking. It provides a large public corpus while deferring implementation details, ablations, and extended results to supplementary material.
- Corpus: LEMUR 2 contains over 14,000 models and more than 750,000 structured training records spanning architectures, configurations, and task-specific metrics.The records include metrics such as accuracy.
- Scope Boundary: Further implementation details, ablation studies, and extended results are deferred to supplementary material because of space constraints.