Source-linked AI summary
Compile by Training: Turning Natural-Language Specifications into Local Neural Functions
Yuntian Deng, Pengyu Nie, Stuart Shieber
TL;DR
Recurring text functions can be difficult to encode with rules yet costly to run through a remote large model. The paper compiles natural-language specifications into reusable local neural functions using teacher-generated examples and a compact interpreter adapter. On FuzzyBench-Hard, compile by training reaches 83.6% semantic accuracy, with compilation taking roughly a minute rather than seconds.
Problem
Recurring text functions may be too fuzzy for conventional code but too narrow and frequent to justify a large-model call at every invocation.
Method
Teacher models synthesize task-specific examples that train a lightweight adapter and scaffold for a shared frozen interpreter, producing a packaged reusable program.
Results
83.6% semantic accuracy is achieved on FuzzyBench-Hard, where the PAW fast compiler produced no exact matches.
Takeaways & Limitations
Compiled functions can run without teacher models and serve as components in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English–Claudish translator.
Takeaways & Limitations
Synthetic supervision may inherit teacher errors, so applications requiring guaranteed correctness should validate outputs or retain deterministic control paths.
Abstract
from arXiv · showhide
Many recurring text functions are easy to describe but difficult to implement with rules, while calling a large remote model for every input introduces repeated cost, latency, and dependency on a provider. We present compile by training, which turns a natural-language specification into a reusable neural function. At compile time, teacher models generate task-specific examples that are used to train a small adapter for a compact interpreter. The resulting function runs without the teachers and can be stored, versioned, and composed like ordinary software. On FuzzyBench-Hard, a subset on which the Program-as-Weights fast compiler produced no exact matches, compile by training reaches 83.6% semantic accuracy. This higher accuracy comes with a higher compile-time cost: roughly a minute rather than seconds for the fast compiler. We deploy the compiler in a public interactive service and demonstrate compiled functions in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.
1 Introduction
Compile by training targets recurring text functions that are too fuzzy for conventional rules yet too frequent to justify a remote large-model call. It uses teacher-generated examples to specialize a compact interpreter, trading longer compilation for reusable local execution.
- Many recurring text functions are too fuzzy for conventional code but too narrow and frequent to justify a large-model call at every invocation.
- Compile by training uses large models once to build a smaller, reusable neural function from a natural-language specification.
- Teacher models generate desired-behavior examples, and gradient descent specializes a compact interpreter for the specified function.
- The resulting function processes new inputs without calling the teachers again, making adaptation a software build step rather than a run-time dependency.
- The system evaluates correctness on FuzzyBench-Hard, measures interactive latency, and demonstrates composition in three applications.
2 System Overview
The system separates compilation from execution: a natural-language specification becomes a packaged program that can be invoked repeatedly through a local SDK. Compilation is hosted, while future local execution does not send inputs to the PAW service or teachers.
- A developer describes desired behavior in natural language and compiles it into a reusable program for repeated invocation on new inputs.
- The interface maps a specification s to a compiled program p_s, then maps each new input x to the program output ŷ.
- Each compiled program supplies an adapter and prompt that specialize a shared frozen language-model interpreter for one function.
- The workflow is specify, submit and monitor a compile job, then open or download the completed program and invoke it through the SDK.
- Compilation sends the specification to the PAW service and teacher APIs, whereas local SDK execution does not send future inputs to them.
3 Compile by Training
Compile by training converts a natural-language specification into a reusable neural program through teacher-generated supervision, adapter specialization, and artifact packaging. Shared interpreter components keep per-function training and storage lightweight.
- From Specification to Supervision: Teacher models synthesize task-specific input-output examples because a specification alone lacks enough labeled data for direct training.
- From Specification to Supervision: Each synthesized pair illustrates how the function should map an input x_i to an output y_i.
- From Specification to Supervision: Structured JSON requests and validation allow accepted teacher examples to enter training automatically, with mixed teachers providing complementary supervision.
- Specialization and Packaging: All programs share a frozen Qwen3-0.6B interpreter, while each function uses a lightweight LoRA adapter and compiler-generated run-time scaffold.
- Specialization and Packaging: The amortized PAW prediction initializes the adapter and scaffold, which are refined using the synthesized dataset before packaging into program p_s.
4 Interactive Minute-Scale Compilation
The minute-scale compiler is implemented as a persistent background service so users can continue working while synthesis and training proceed. Streaming, shared workers, caching, and durable artifacts keep the interactive interface responsive.
- Interactive Compilation: Interactive compilation overlaps teacher synthesis with model loading and training instead of waiting for all examples sequentially.
- Interactive Compilation: Training starts when the first batch is available and the scheduler prioritizes incoming examples needed by earlier batches.
- Service Architecture: A persistent API record, shared queue, GPU workers, teacher-output cache, and artifact store coordinate long-running compilation jobs.
- Service Architecture: Separating coordination from workers keeps the API responsive while GPUs execute builds.
- User Experience: Users can continue browsing while queue position and training progress are reported, with job state persisting across navigation and reloads.
5 Evaluation and Validation
The evaluation uses semantic correctness judgments on FuzzyBench-Hard and controlled supervision sweeps. Compile by training substantially improves mean LEM over PAW’s fast compiler, but requires longer compilation.
- 5.1 Metric and benchmark: LEM measures the fraction of predictions judged semantically correct by an LLM according to the specification, input, reference output, and prediction.The selected GPT-5.5 judge reached 0.977 accuracy with Cohen’s κ = 0.946 against 128 author labels.
- 5.2 Does training improve correctness?: 0.836 mean LEM was achieved by compile by training on FuzzyBench-Hard, improving 0.612 over PAW’s 0.224.FuzzyBench-Hard contains specifications where PAW’s fast compiler produced no exact matches, although some predictions remained semantically correct.
- 5.4 Is compilation fast enough for interaction?: 50.9 seconds was required for compile by training, compared with 3.5 seconds for PAW’s fast amortized compiler.The correctness improvement therefore comes with longer compilation.
- 5.3 How do supervision choices affect correctness?: A 2:1 mixture of GPT-5.4-mini and GPT-5.5 supervision raised development mean LEM from 0.746 to 0.851 relative to GPT-5.4-mini alone.In data scaling, mean LEM reached 0.866 with 7200 unique pairs, compared with 0.821 using 1440.
- 5.4 Is compilation fast enough for interaction?: Four concurrent compile jobs completed with a mean queue wait of 1.01 seconds and even utilization across workers.Cold compile latency was 50.9 seconds on a B300, 68.2 seconds on an H200, and 99.2 seconds on an RTX GPU.
6 Deployment and Applications
The system is deployed as a queued compilation service and demonstrated through composed applications. These applications combine compiled neural functions with deterministic code for websites, avatar control, and bidirectional translation.
- 6.1 Composing many compiled functions: The deployed service uses a shared queue, GPU workers, cached teacher outputs, and centralized artifact storage to manage compilation jobs.Users receive queue and training progress through persistent job records.
- 6.1 Composing many compiled functions: The deployed website helper contains 30 compiled programs, with 28 participating in live routing across four websites.A router uses the current website and page as context to send each question through only a small part of the program tree.
- 6.1 Composing many compiled functions: In paw-helper, compiled functions make fuzzy decisions while ordinary code handles retrieval, caching, and branch control.For a course question, compiled functions draft and evaluate answers while BM25 retrieval searches Piazza.
- 6.2 Generating executable behavior: Avatar Director translates natural-language instructions into an action DSL that the browser validates and executes for 3D animation.The DSL supports sequences, durations, repetition, and compatible parallel motions.
- 6.3 Translating between English and Claudish: The English-Claudish translator uses separately finetuned programs for each translation direction.Both programs are powered by specifications and adapters for the shared 0.6B interpreter.
- 6.3 Translating between English and Claudish: The public translation demo completed 100,747 successful requests between August 22 and September 2, 2026.The programs can also be downloaded and run locally, with supporting code available in a public repository.
7 Related Work
Compile by training combines synthetic supervision with parameter-efficient adaptation to compile user-defined functions into reusable programs.
- Synthetic supervision and parameter-efficient adaptation: Teachers generate input-output pairs from a natural-language function specification, and those pairs train a LoRA adapter for a compact interpreter.This applies instruction-example synthesis and knowledge distillation to one user-defined function.
8 Conclusion
Compile by training converts user-provided function descriptions into reusable neural programs by synthesizing examples and finetuning a LoRA adapter. It reaches 83.6% semantic accuracy on FuzzyBench-Hard while taking roughly a minute rather than seconds, and supports several demonstrated applications.
- Compile by training converts a user-provided function description into a neural program.
- Teacher models synthesize examples, which are used to finetune a LoRA adapter for a smaller interpreter model.
- 83.6% semantic accuracy is reached on FuzzyBench-Hard, where the PAW fast compiler produced no exact matches.
- Roughly one minute of compilation adds a higher-accuracy option to PAW’s seconds-scale fast compiler.
- Compiled programs support a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English–Claudish translator.
9 Limitations
The paper identifies two limitations: synthetic supervision may inherit teacher errors, and its application evidence does not include systematic user studies.
- Synthetic supervision may inherit teacher errors, so guaranteed-correctness applications should validate outputs or retain deterministic control paths.
- Application evidence focuses on composition and structured execution, while systematic user studies remain future work.
A Additional Experimental Details
The appendix provides interpretive context, including the LEM definition and references to configuration and grader-validation tables.
- The appendix collects details intended to help interpret the main paper.
- Table 2 is labeled the Public Finetuned Standard configuration.
- LLM Exact Match is the mean of binary, specification-grounded semantic-correctness judgments over specifications, inputs, reference outputs, and model outputs.
- Table 3 is labeled the Compact LEM grader validation.
C Supervision Sweep Protocol
The supervision sweep protocol fixes training settings for teacher-mixture comparisons and evaluates outputs with specification-grounded semantic judgments. The evaluator treats the specification as authoritative while allowing permitted formatting variation and rejecting substantive errors.
- Teacher-mixture sweeps compare GPT-5.4-mini and GPT-5.5 example counts in 6400-example training sets built from 3600 unique pairs.
- Teacher-mixture sweeps hold batch 64, learning rate 2 × 10^-4, and 100 steps fixed, while data-scaling rows use batch 48, the same learning rate, and 100 steps.
- The benchmark evaluates a natural-language specification, input, reference output, and model output.
- The evaluator decides whether the model output is a correct application of the specification to the input.
- The specification is authoritative, and any output valid under it is accepted even when it differs cosmetically from the reference.
- Whitespace, JSON spacing, indentation, and unspecified object key order are ignored as cosmetic differences.
- Substantive differences such as changed values, missing or extra items, structural errors, logical errors, or refusals are not ignored.
- Explicitly required formats are enforced, while unspecified output containers may vary when they preserve the same values and order.