NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation

1. Introduction

• Bespoke scripts and glue code make synthetic-data pipelines difficult to interpret, share, and reproduce.Synthetic data is used across pretraining, post-training, agentic, and multimodal development.Source: §1 · pp. 1–2
• NDD is an open-source, general-purpose multimodal synthetic-data framework with declarative dataset specifications and built-in text, code, structured-output, image, and embedding columns.Statistical samplers steer diversity; validators and judges verify generated values, and plugins add column types.Source: §1 · p. 2
• Users preview a small sample, inspect it, revise the configuration, and generate the full dataset from that same specification.NDD resolves column dependencies, schedules endpoint calls, and retries failed requests.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Statistical sampler columns draw from user-specified distributions and provide direct control over dataset composition and diversity.Source: §1 · p. 3
• A flexible plugin system adds column types and functionality, while built-in and custom columns express many generation strategies.Source: §1 · pp. 2–3
• NDD-built datasets support structured, agentic, multimodal, and domain-specialized tasks, including Nemotron development and enterprise deployments.Source: §1 · p. 3

2. Architecture

• NDD’s architecture separates dataset design, execution, and external services, with plugins extending the configuration and model endpoints, tools, and validators outside its process boundary.Source: §2 · p. 3, Figure 2 · p. 4

2.1. Overview

• The DAG compiler validates configurations and resolves dependencies from explicit settings and template references, rejecting cycles before generation.The asynchronous runtime schedules ready work and routes model tasks through its runtime layer.Source: §2.1 · p. 3

2.2. Design Interface

• Dataset designs can be declared in YAML or Python, with column dependencies encoded in data rather than imperative orchestration.Execution settings can be overridden separately, so the same design supports preview and full-scale runs.Source: §2.2 · p. 4

2.3. Columns as Pipeline Building Blocks

• Seed and sampler columns establish inputs and variation; dependent columns build candidates, and validators and judges gate records for transformation or rejection.Rejected records retain failed checks and scores, while model-interaction traces remain inspectable.Source: §2.3 · pp. 4–5, Figure 3 · p. 5

2.4. Models and Providers

• Model aliases let columns name logical model roles independently of endpoints, while provider and model configurations separate serving details from model choice and inference policy.Source: §2.4 · p. 5

2.5. Async Runtime

• The asynchronous runtime runs dependent work as soon as inputs are ready, allowing independent branches to overlap without changing the declarative configuration.Row-group checkpoints support resuming interrupted runs; retry policies handle retryable failures, while permanent cell failures remove affected rows and downstream work.Source: §2.5 · p. 6

2.6. From Generation to Usable Data

• Validators and judges assess generated values as graph stages, while processors reshape outputs and materialization collects datasets, traces, checkpoints, and diagnostics.Figure 4 contrasts one NDD process producing an artifact set with independently run shards assembled downstream.Source: §2.6 · pp. 6–7, Figure 4 · p. 7

2.7. Trust and Deployment Boundaries

• External services and installed plugins form trust boundaries; provisioning and distributed orchestration remain external, and larger runs can use scheduler-launched shards.Sensitive workflows should account for data sent to services and service-dependent variation that can prevent byte-for-byte regeneration.Source: §2.7 · pp. 7–8

3. Pipeline Design

• Pipelines define the record contract, make variation explicit, preview before scaling, validate aggressively, and preserve provenance for accepted and rejected data.Important variation axes are represented as seed or sampler fields before generation, rather than delegated to a prompt asking for diversity.Source: §3 · p. 9

3.1. Design Loop

• The design loop encodes downstream record requirements, seeds, coverage axes, staged generation, and quality gates, with preview() checking samples and outputs before full-scale runs.After a full run, evaluation results and rejected-record diagnostics feed the next iteration.Source: §3.1 · p. 9

3.2. Sampling and Staging

• Seed curation and sampler columns make coverage axes explicit, while staged workflows reserve model calls for semantic work and use deterministic transformations for parsing and bookkeeping.Sampler constraints can enforce joint conditions by construction instead of generate-and-discard filtering that may skew the accepted distribution.Source: §3.2 · pp. 9–10

3.3. Validation and Provenance

• Validation uses artifact-specific gates and rejection sampling, with coverage rebalanced through seed and sampler curation when filtering skews the intended mix.Preserved intermediate fields, seed identifiers, inputs, scores, and dropped examples support inspection of selected data and agent traces.Source: §3.3 · p. 10

3.4. Export and Reproducibility

• Export and processor stages produce downstream views without changing source generation columns, while an auditable design recipe supports inspection and reruns despite changing endpoints, settings, and tools.The Hugging Face Hub upload includes builder_config.json, a serialized representation of the declarative design state.Source: §3.4 · pp. 10–11

4.1. Structured Outputs and Model Usability

• Training with NDD-generated structured-output data improved Nemotron Nano v3 from 80.2% to 86.9% on JSONSchemaBench and from 64.5% to 72.1% on StructEval-Text; Nemotron 3 Ultra's StructEval-T accuracy rose from 78.6% to 82.1%.The JSON-schema tasks used exact schema conformance as the reward; although the released dataset is JSON-only, StructEval-Text gains extended across formats, with TOML and XML remaining weakest.Source: §4.1 · pp. 11–12

4.2. Text-to-SQL

• The SQL workflow retained 96.5K of 300K candidates; CrowdStrike-trained Llama-3.3-Nemotron-Super-49B-v1.5 achieved 96% valid-query accuracy and a 2.50/5 semantic score, versus Claude Sonnet 4.5's 94% and 2.35/5.NDD's staged generation uses dialect-specific validation and five LLM critics; the CrowdStrike case converted privacy-scrubbed enterprise queries into supervised data for a specialized production task.Source: §4.2 · pp. 13–14

4.3. Search-Agent Trajectories

• From 50K Wikidata seeds, the search workflow produced approximately 7K valid trajectories, averaging 12 tool calls, with complete search traces incorporated into Nemotron 3 Super supervised fine-tuning.Intermediate entities are obfuscated in generated questions, and successful rollouts preserve the Thought–Action–Observation process.Source: §4.3 · pp. 14–15, Figure 6 · p. 15

4.4. Long-Document Vision-Language Model Understanding

• Across successive data-generation, supervised fine-tuning, and reinforcement-learning interventions, MMLongBench-Doc improved from 26.32% to 59.00%; released Nemotron 3 Nano Omni reports 57.5% in reasoning-on mode.The workflow used complementary text, visual, single-page, multi-page, and whole-document QA streams, with evaluation failures guiding new capability targets and an independent VLM judge filtering examples.Source: §4.4 · pp. 16–17

4.5. Prompt Robustness Through Diverse Instructions

• A training mix with 25% canonical and 75% varied prompts was associated with roughly 2× lower prompt-sensitivity variation across GPQA, MMLU-Pro, competition mathematics, and LiveCodeBench.The variation pool used sampled axes including tone, verbosity, domain, and prompt ordering, with judges filtering for format compliance and linguistic quality.Source: §4.5 · p. 17

4.6. Nemotron-Personas

• Nemotron-Personas-USA contains one million records mirroring complex real-world distributions, covering approximately 29K ZIP Code Tabulation Areas and more than 560 occupations; an LLM-only baseline failed on both simple and complex joint distributions.The broader collection has ten regional datasets and 15 language or script editions; it provides reusable grounding for training-data synthesis and evaluation, and Nemotron 3 Nano extends this pattern to precise instruction following.Source: §4.6 · p. 18

4.7. Broader applicability and SDG Design Principles

• Across research and production workflows, NDD has supported approximately 10 trillion tokens of synthetic data, while its design principles emphasize explicit coverage, representative seeds, controlled variation, and iterative quality selection.The persona framework's underlying SDG-PGMs approach is entity-agnostic and can model structured relationships for products, transactions, devices, and proprietary business objects.Source: §4.7 · pp. 19–20, §4.6 · p. 19

5. Conclusion

• NDD lets humans and agents define columns, samplers, dependencies, and quality gates, inspect records, and revise a preview before scaling without rewriting the workflow.The same design produces reusable artifacts.Source: §5 · p. 20
• The case studies support treating synthetic-data generation as an explicit, inspectable, reusable data-design process rather than isolated prompts.The cases contribute to Nemotron model development and enterprise applications, with outcomes measured differently.Source: §5 · p. 20
• Pipeline choices influence what a model ultimately learns, making data-pipeline construction part of model building.Source: §5 · p. 20

6. Limitations

• Synthetic data is weaker for poorly understood or rapidly changing targets, and validators can miss semantic errors, encode bias, or fail to verify multimodal grounding.Real data and human review remain essential for unknown failure modes, judge calibration in high-stakes settings, and deployment measurement.Source: §6 · p. 20
• Rejection sampling can remove rare or difficult cases and skew a dataset; samplers alone do not ensure population or task-distribution fidelity.Distribution-sensitive workflows should use representative seed data or external statistics and audit accepted and rejected records.Source: §6 · p. 20
• Multi-stage pipelines that generate excess candidates add compute cost, wall-clock time, and operational complexity; adaptive concurrency does not eliminate model-call or validation costs.Endpoint and tool behavior can also change generated data, so reproducibility requires an auditable recipe.Source: §6 · pp. 20–21
• Synthetic data is not automatically private, and generated records can leak sensitive details, overlap benchmark-like sources, or satisfy a schema without improving a model.Privacy-sensitive workflows need explicit scrubbing, access controls, review, overlap checks, and task-specific downstream evaluation.Source: §6 · p. 21

7. Broader Impact

• With seed data and validators, NDD can reduce manual annotation effort and the marginal cost of producing new dataset variants, though total savings depend on inference, validation, and review costs.It can help organizations with constrained annotation budgets build datasets where expert labels are scarce and make underrepresented cases easier to specify and test.Source: §7 · p. 21
• Synthetic-data pipelines can produce material for harmful automation or create false confidence when generated records are treated as real observations.Responsible use calls for provenance tracking, validation, bias and privacy review, access controls, and clear documentation of intended representation.Source: §7 · p. 21

8. Availability

• NDD is open-source software under Apache 2.0, described as version 0.9.0, supports Python 3.10 and later, and can use user-provided hosted or self-hosted model endpoints.Documentation includes reference material, installation instructions, tutorials, and recipes.Source: §8 · p. 21

A. Installation Examples

• The installation examples show package installation, provider-configuration checks, and a minimal builder workflow that previews a few records before revision and full generation.The setup is an interface example, not a complete experimental protocol.Source: A. Installation Examples · p. 27

B. Configuration Examples

• Configuration examples cover sampler-controlled generation, structured outputs, multimodal context, image generation, validation, export views, and MCP trace capture.The examples are not a complete recipe for any one public case study.Source: B. Configuration Examples · p. 28

B.1. Declarative Configuration and Builder Equivalence

• A dataset design can be represented either as a declarative schema or a Python builder, while preview() and create() calls remain outside the specification in both forms.Source: §B.1 · p. 28

B.2. Model Providers and Aliases

• ModelProvider objects define endpoint connectivity, while ModelConfig aliases assign inference policies that columns reference without depending on the underlying model configuration.Listing 6 uses two aliases with different temperatures for the same model.Source: §B.2 · pp. 29–30

B.3. Structured Output

• LLMStructuredColumnConfig supports either Pydantic models or JSON schemas as output contracts; Listing 7 shows a compact Pydantic-based excerpt.The excerpt defines price as a float constrained to be at least 10 and at most 1000.Source: §B.3 · p. 30

B.4. Multimodal Context and Image Generation

• Image columns can generate media from sampler-controlled prompts, and downstream vision-language columns can receive that image through context dependencies.The context dependency makes caption generation wait for the image; the selected aliases must support image generation and image input, respectively.Source: §B.4 · p. 31

B.5. Validation Columns

• Validation columns attach executable checks to generated fields, with interfaces for Python code, SQL dialects, local callables, and remote HTTP validators.The excerpt retains the validation result because drop=False.Source: §B.5 · p. 32

B.6. Export Views

• Processors can remove intermediate fields from the primary output or write an additional schema-transformed dataset view.The examples show both cleanup and chat-format export patterns.Source: §B.6 · p. 32

B.7. MCP Tool Traces

• MCP tools can be exposed to an LLM column, with the conversation captured as a side-effect trace column.The provider must be registered by the application, and the example writes the trace to fact_response__trace.Source: §B.7 · p. 33

B.8. Minimal Column-Generator Plugin

• A reusable column-type plugin comprises a configuration class, implementation, plugin descriptor, and package entry point.The index-multiplier example shows these components and their use in a dataset configuration; the files remain separate in the installed package.Source: §B.8 · pp. 33–34

• The appendix details how a ready model-backed cell reaches an external endpoint without changing the declarative column graph or its data-design semantics.Source: C. Model Runtime and Request Admission · p. 35

C.1. Model Facade and Client Stack

• The ModelRegistry resolves aliases to lazily constructed ModelFacades, which provide a common interface for chat, embeddings, and image generation and centralize request construction and output handling.The facade also supports structured-response validation, multimodal context, MCP tool-use loops, traces, and usage accounting.Source: §C.1 · pp. 35–36
• Provider adapters translate canonical NDD requests and responses into provider wire formats and normalize provider-specific responses and errors.The same registry and facade path supports OpenAI-compatible APIs, the Anthropic Messages API, hosted services, gateways, and self-hosted compatible servers.Source: §C.1 · p. 36

C.2. Adaptive Request Admission and AIMD

• A ModelRequestExecutor acquires a lease from shared admission control before each concrete provider attempt, using a request resource keyed by provider, underlying model, and operation domain.Leases are released with classified outcomes, while requests without capacity wait in a queue.Source: Figure 10 · p. 35, §C.2 · p. 36
• AIMD adapts concurrency below the configured provider-model ceiling, reducing active capacity after HTTP 429 and restoring it additively after successful requests.An optional startup ramp begins at one concurrent call; cooldown uses Retry-After or a configured fallback.Source: Figure 10 · p. 35, §C.2 · p. 36
• HTTP 429 responses remain visible to drive AIMD, while eligible transient failures and selected server errors use separate retries with exponential backoff.Every retry attempt passes through request admission to avoid adding load while the endpoint signals over-capacity.Source: §C.2 · p. 36
• Aliases for the same provider and underlying model share a global request cap, while adaptive state is maintained separately for chat, embedding, image, and health-check domains.The dataset scheduler chooses ready cells; request admission determines when a cell can issue a provider request.Source: §C.2 · p. 36

D. Operational Deployment and Capacity Planning

• Deployment plans own job partitioning, resource allocation, submission, and output consolidation, while each NDD process manages dependencies, scheduling, retries, validation, and artifacts.Separate runtime state and artifact namespaces, non-overlapping shards, and bounded aggregate request ceilings are needed when jobs share a gateway or serving pool.Source: D. Operational Deployment and Capacity… · p. 37

D.1. Pilot-Based Capacity Planning

• Representative single-job pilots measure accepted-record throughput, model calls, latency, retries, throttling, resource use, artifact growth, and rejection rates before scheduler submission.For self-hosted endpoints, pilots also vary serving settings; tensor parallelism may help fit a model or reduce latency, while additional replicas may yield higher throughput when the model fits on fewer GPUs.Source: §D.1 · p. 37
• Pilot measurements guide job count and shard size, but throughput may not scale linearly; plans must account for shared endpoints, quotas, resources, storage, network, and cost.Recording configuration version, shard size, per-job resources, endpoint topology, request ceilings, and artifact layout supports reproducibility or revision.Source: §D.1 · p. 37

D.2. Scheduler Integration

• A user-supplied launcher can map Slurm array tasks to shards, invoke NDD with shard-specific names and paths, and enable checkpoint reuse through resume options.Distinct task-ID dataset names prevent overwrites under a shared artifact root; reliable consolidation retains shard metadata, rejection diagnostics, and trace provenance.Source: §D.2 · p. 38