Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Source-linked AI summary

Program-as-Weights: A Programming Paradigm for Fuzzy Functions

arXiv:2607.02512v1

TL;DR

Developers often outsource fuzzy functions to remote LLM APIs, which are costly, fragile, difficult to reproduce, and not self-contained. Program-as-Weights compiles natural-language specifications into small neural programs executed by a fixed lightweight interpreter, achieving 73.78% exact match versus 68.70% for Qwen3-32B prompting at roughly 50× less inference memory.

Problem
Fuzzy functions are increasingly implemented through remote LLM APIs, creating cost, fragility, reproducibility, and self-containment concerns.
Method
Program-as-Weights compiles a natural-language specification into a pseudo-program and task-specific PEFT module executed by a frozen interpreter, trained using FuzzyBench.
Results
73.78% exact match versus 68.70% for Qwen3-32B prompting, with approximately 50× less inference memory.
Takeaways & Limitations
The paradigm supports locally executable, versionable fuzzy-function artifacts and is illustrated across five case studies, with an extension to image-conditioned tasks by swapping the compiler.
Takeaways & Limitations
All evaluations cover only single-step functions, leaving multi-step and long-horizon reasoning unvalidated.

Abstract

from arXiv · show

1 Introduction

Program-as-Weights replaces fragile, costly remote-LLM calls for fuzzy functions with locally executable neural programs compiled from natural-language specifications. Its compiler emits PEFT-based programs for a frozen interpreter, achieving stronger exact-match performance than direct prompting with substantially lower inference memory.

Remote LLM APIs conveniently implement fuzzy functions but are costly, fragile, non-reproducible under silent provider updates, and not self-contained.
PAW compiles a natural-language function description into a small neural binary that a frozen, lightweight interpreter runs locally like a user-defined function.The compiled program can be cached, version-controlled, and called locally.
PAW programs combine a natural-language pseudo-program with a task-specific PEFT module that retunes the frozen interpreter; the current system uses compiler-generated LoRA.The pseudo-program shields the interpreter from typos and ambiguity in the original specification.
The compile pipeline uses an untrained 4B Qwen3 pseudo compiler for task rewriting and a trained 4B Qwen3 LoRA compiler trained on the released 10M-example FUZZYBENCH dataset.
73.78% vs. 68.70% exact match: a Qwen3-0.6B interpreter running PAW programs outperforms direct prompting of Qwen3-32B at roughly one fiftieth the inference memory.Quantized, the system runs at 30 tokens per second on a MacBook M3 using a ∼430 MB shared GGUF base and a 23 MB per-program LoRA adapter.
PAW is illustrated through case studies in output triage, custom classification, fuzzy search, agent preprocessing, and creative generation, including 93% on TOOLCALL-15.

2 Programs as Weights

Program-as-Weights compiles fuzzy-function specifications into neural programs executed by a fixed neural interpreter, avoiding repeated LLM invocation. These programs combine discrete pseudo-program tokens with continuous parameter-efficient weights and can be packaged and used like software modules.

Compilation and execution: Introducing a new fuzzy function requires compiling a new learned parameter blob rather than repeatedly invoking or retraining the interpreter.This division mirrors classical compilation, but the executable is learned parameters and the runtime is a neural network.
Compilation and execution: A neural Compiler maps a natural-language specification, optionally with examples, to a program p that a fixed neural Interpreter executes on inputs x.The interpreter produces outputs ŷ without retraining for each new fuzzy function.
Hybrid programs: Concrete PAW programs are hybrid objects combining a variable-length discrete token sequence with a continuous component implemented using PEFT methods such as LoRA.The discrete component acts as a self-contained pseudo-program supplied to the interpreter as part of its input.
Program artifacts: ∼23 MB at Q4_0 is the size of a compiled PAW program for a 0.6B interpreter, plus a one-time shared base.Programs can be saved, version-controlled, distributed via package managers, and called from Python or JavaScript with a two-line API.

3 The Compiler–Interpreter System

PAW separates program construction into a prompted pseudo compiler, a trained PEFT compiler, and a frozen interpreter that executes hot-swappable program modules. It instantiates PEFT modules with prefix-tuning or LoRA, with LoRA achieving the strongest controlled-comparison result and becoming the scaled configuration.

3 The Compiler–Interpreter System: PAW’s pipeline uses a pseudo compiler, a PEFT compiler, and a frozen interpreter to transform a specification s into output ŷ.The pseudo compiler produces pdiscrete, the PEFT compiler produces pcontinuous, and the interpreter attaches it to target modules before running input x.
3 The Compiler–Interpreter System: The off-the-shelf Qwen3-4B-Instruct-2507 pseudo compiler rewrites each specification into a task restatement with representative input-output examples.It is never trained and is shared by both PEFT instantiations.
3.2 LoRA compiler: The trained LoRA compiler processes [s | pdiscrete | EOS | τ1, . . . , τT ] with T = 64 learned prefix tokens and emits depth-aligned hidden states for the mapper.The mapper mean-pools these states, applies an MLP, and generates mixing coefficients over shared LoRA bases; r = 64 and N = 64 inject approximately 38.5M parameters per fuzzy function.
3.2 LoRA compiler: The frozen interpreter attaches the generated LoRA to target modules, prepends pdiscrete to x, and autoregressively generates outputs, allowing one device-resident runtime to serve unboundedly many programs.The LoRA is hot-swappable while the interpreter remains frozen.
3.3 Prefix compiler: 65.7% at r=64 was LoRA’s FuzzyBench exact-match result, versus 50.4% for prefix-tuning, 56.5% for LoRA at r=18, and 9.8% for no-compiler prompting.The controlled comparison used the same amount of training compute; LoRA was selected for scaling to the full training data.

4 Training

Training updates only the PEFT compiler while keeping the pseudo compiler and interpreter frozen. The compiler learns LoRA adapters through a supervised target-output likelihood objective, with gradients propagated through the frozen interpreter.

Training setup: Only the PEFT compiler is trained; the pseudo compiler and interpreter remain frozen.The compiler produces an adapter injected into the interpreter alongside a fixed pseudo-program.
Training objective: For each training triple (s, x, y), the objective minimizes the negative mean-token log-likelihood of target y under the frozen interpreter.A pre-generated pseudo-program is processed by CL, whose prefix-position hidden states are mapped to a LoRA adapter before interpreter injection.
Optimization: The gradient flows through the frozen interpreter into the LoRA mapper and then into CL’s hidden states.The trainable parameters are CL and the LoRA mapper, while interpreter parameters remain fixed.

5 FuzzyBench: A 10M-Example Dataset of Fuzzy Functions

FuzzyBench is a 10M-example dataset for compiling fuzzy functions from specifications, generated through a two-stage gpt-5.2 pipeline and organized across seven task families. It also includes multi-axis noisy test variants and establishes empirical ceilings of 96.09% for gpt-5.2 and 91.87% for gpt-5-mini.

Dataset construction: FuzzyBench contains 10M triples of specifications, inputs, and target outputs for training PAW-style fuzzy-function compilation.The dataset addresses the lack of a public benchmark for compiling a fuzzy function from a specification.
Dataset construction: A two-stage gpt-5.2 pipeline generates natural-language specifications, then eight input/output pairs for each specification, with 80/10/10 spec-level train/validation/test splits.Each prompting call produces eight specifications, and repeated calls use different category constraints.
Thematic coverage: Across 29 incremental versions, FuzzyBench covers seven high-level fuzzy-task families spanning text processing, parsing, agentic tool use, web intelligence, code generation, and verification.The first version establishes 277 base categories, while the final dataset covers more than 800 sub-categories.
Robustness variants: Eight noise axes, each at light, medium, and heavy intensity, provide robustness test variants covering typos, grammar errors, ambiguity, formatting drift, combined noise, terse or casual phrasing, and paraphrase.The released variants are evaluated for robustness in Section 8.
Empirical ceiling: 96.09% is achieved by gpt-5.2 and 91.87% by gpt-5-mini, bounding the performance of compiled functions trained on FuzzyBench.These results are presented as the dataset’s empirical ceiling.

6 Main Results

PAW achieves 73.78% exact match on FuzzyBench with a 0.6B interpreter, surpassing Qwen3-32B prompting while using approximately 50× less inference memory. The compiler-interpreter abstraction also supports multimodal fuzzy functions without changing the interpreter and outperforms VLM baselines on three diagram tasks.

Main result: 73.78% exact match: a 0.6B-parameter interpreter executing PAW programs surpasses Qwen3-32B prompting at 68.70% while using approximately 50× less inference memory.Inference memory is approximately ∼1.2 GB at bf16 versus ∼60 GB.
Cross-interpreter scaling: 54%: GPT-2 124M achieves this despite having one-fifth Qwen3 0.6B’s parameters and no instruction tuning, while Qwen3 0.6B remains the strongest of three interpreters.The result suggests compiler-generated LoRAs can encode usable task adaptations in very small, weakly capable bases.
Multimodal generalization: The same Qwen3 0.6B interpreter and LoRA mapper support image-conditioned fuzzy functions when the compiler is swapped from Qwen3-4B-Instruct to Qwen3-VL-4B.Image conditioning is fully encoded in the PEFT module emitted by the VL compiler, so the small text interpreter never sees the pixels.
Image-conditioned results: PAW (LoRA) outperforms the best VLM baselines on CoSyn diagram tasks: Circuit 0.274 vs. 0.196, Chemical 0.414 vs. 0.258, and Music 0.552 vs. 0.470.On Im2LaTeX, PAW (LoRA) is weaker than prefix-tuning, 0.181 vs. 0.391, because long pseudo-program examples crowd the small interpreter’s context.

7 Ablations

Ablations show that the simplest LoRA mapper is strongest, while PAW’s advantage comes specifically from compiler-generated LoRA rather than the shared base model or training setup. The pseudo-program also protects interpretation under noisy specifications, with only slight degradation under combined heavy noise.

Architectural variants: The simplest mean-pooling, single-residual-MLP, shared-basis mapper outperformed more expressive architectural variants.These variants were expected to help but all underperformed the default design.
Compiler vs. no compiler: PAW exceeded full fine-tuning by 15.4 percentage points on FuzzyBench.The comparison used the same data, base model, and training budget, differing only by compiler removal.
Compiler vs. no compiler: PAW exceeded the strongest fixed LoRA by 21.7 points on FuzzyBench.The no-compiler baselines included per-task fixed LoRAs at three ranks.
Noise robustness: PAW degraded only slightly under combined heavy noise in 8-axis specification variants.The 8-axis variants modified the specification while leaving the input unchanged.
Noise robustness: On heavy-typo specifications, feeding the raw specification was 4.5 points worse than feeding the pseudo-program to the interpreter.The pseudo-program therefore protects the interpreter from noisy specifications.

8 Robustness to Noisy Specifications

The paper evaluates PAW on specifications corrupted across seven noise axes and finds that performance degrades only slightly under heavy noise. Evidence from a bypass variant supports the hypothesis that the discrete pseudo-program mediates this robustness by converting noisy specifications into clean restatements before interpretation.

Noise evaluation: PAW is evaluated on noise-perturbed test_clean specifications across seven axes: typos, grammar, ambiguity, formatting, all-noise combined, terse, and paraphrase.The evaluation targets realistic developer-written specifications containing typos, ambiguity, and grammar errors.
Robustness results: PAW degrades only slightly under heavy noise, according to the robustness results in Table 6.The reported evaluation covers the seven perturbation axes applied to test_clean specifications.
Mechanism: A bypass variant that feeds the raw specification directly to the interpreter confirms that robustness is mediated by the discrete pseudo-program.The proposed mechanism is that the 4B compiler converts the noisy specification into a clean restatement before the small interpreter receives it.

9 Local Execution

PAW provides a developer interface for compiling, downloading, caching, and executing programs locally after the first download, without external API calls. On-device execution is supported by quantization with minimal measured accuracy loss and reported MacBook M3 latency, alongside five local-function use cases.

Developer interface: A PAW program is a single downloadable file accessible through small Python or JavaScript APIs, with execution occurring locally after its first download.paw.compile sends a specification to a compiler service and returns a serializable program object, while paw.function loads it as a Python callable.
Quantization without measurable accuracy loss: A Q6_K base plus Q4_0 LoRA adapter is statistically indistinguishable from bf16 on the 0.6B Qwen3 interpreter.The quantization findings were validated on a 4096-example test subset.
Quantization without measurable accuracy loss: 1.3 points: Q4_K_M base plus Q4_0 LoRA loses only 1.3 points relative to bf16, with approximately 484 MB base and 23 MB per-program adapter.Table 8 reports approximately 507 MB total disk for this configuration.
Latency on a MacBook M3: 31.6 tokens/s: Q5_K_M base plus Q4_0 adapter runs at 31.6 tokens/s on a MacBook M3 with Metal acceleration, with a 0.48 s cold load.Full per-quantization tables for GPT-2 124M and Qwen3.5 0.8B appear in Appendix K.
Case studies: PAW was applied to five use cases, including event-driven log monitoring, intent-based site navigation, and semantic-search reranking without an LLM API call per request.The log monitor fires only on relevant lines, and reranking adds intent-aware fuzzy search to an existing keyword index without putting an LLM in the request path.

10 Related Work

PAW relates to hypernetworks, parameter-efficient fine-tuning, synthetic instruction generation, model distillation, and neural programs. Its distinguishing setup is compiling programs directly into shareable neural weights executed by a fixed interpreter.

Hypernetworks: Hypernetworks generate target-network weights from embeddings, while later systems emit prefixes, adapters, LoRAs, or activation-steering modules from instructions or examples.These systems include HyperTuning, Text-to-LoRA, Generative Adapter, HyperSteer, Gist, and MEND.
Parameter-efficient fine-tuning: PAW’s compiler emits established parameter-efficient components, including adapters, prefix-tuning, prompt tuning, LoRA, and AdaLoRA.These methods respectively use bottlenecks, learned attention key–value pairs, soft embeddings, or low-rank projection updates.
Synthetic instruction-data generation: FuzzyBench-10M follows prior work that uses LLMs to generate synthetic instruction data for training smaller models.Self-Instruct and Unnatural Instructions automatically generate instructions, while Textbooks Are All You Need advocates synthetic textbook-style data.
Model distillation: PAW shares model-distillation efforts’ goal of amortizing LLM usage but compiles directly into neural weights, enabling fuzzy functions that resist symbolic encoding.ALCHEmist distills labelling logic into Python programs, whereas Binder translates inputs into SQL/Python programs with embedded language-model API calls.
Neural programs: Unlike prior neural-program approaches, PAW produces programs on demand through a single compiler, executes them on a fixed interpreter, and allows free sharing.This setup is presented as one realization of replacing API LLMs with small, locally executed models.

11 Conclusion … D Image Processing

Program-as-Weights compiles fuzzy functions into compact neural binaries executed locally by a fixed interpreter, achieving strong FuzzyBench efficiency while extending to image-conditioned tasks. The appendices describe hosted compilation, interactive testing, offline execution, benchmark-prompt construction, compiler prompting, and multimodal component reuse.

11 Conclusion: PAW compiles a fuzzy function once into a small neural binary executed locally on a fixed interpreter.The paradigm is illustrated through five case studies.
11 Conclusion: 50× less inference memory and 30 tok/s on a MacBook M3 characterize the 0.6B-parameter PAW interpreter versus Qwen3-32B prompting on FuzzyBench.The PAW model uses quantized GGUF.
A Web Interface for PAW Compilation: The hosted interface accepts fuzzy specifications, compiles them on a GPU-backed server, supports interactive testing, and exports serialized weights or program identifiers.Users do not need to provision GPUs locally.
A Web Interface for PAW Compilation: After download, compiled programs execute entirely offline through a simple Python API without requiring internet access.The workflow comprises compilation, interactive validation, and local execution.
B FuzzyBench Construction Prompts: FuzzyBench generation mixes specifications with and without exemplar examples, producing more diverse specifications than either style alone.Input–output examples are generated conditioned on each specification.
C Compiler and Interpreter Prompts: The examples compiler prompt supports reference rollouts during training, whereas the trained PAW compiler uses the minimal prompt at inference.The interpreter also uses a single minimal prompt.
D Image Processing: Image-task PAW replaces only the compiler base, changing Qwen3-4B-Instruct to Qwen3-VL-4B while retaining the Qwen3 0.6B device-resident interpreter.These materials support the multimodal generalization experiments in Table 3.

D.1 Component decomposition of image-task PAW (prefix-tuning era) … O Broader Impacts

The appendix combines component ablations, architecture and training details, exploratory evaluations, case studies, limitations, and broader impacts of PAW. Results show task-dependent benefits of discrete programs versus continuous PEFT, while deployment emphasizes local execution through a shared interpreter.

D.1 Component decomposition of image-task PAW (prefix-tuning era): Discrete pseudo-programs add 5–40 EM points on short-output image tasks, but removing them recovers 6–8 EM points on long structured sequences.The pseudo-program acts as an inductive bias for short phrases, while its examples can crowd the small interpreter’s context budget for Im2SMILES and Im2LaTeX.
F FuzzyBench-10M Dataset Versions: FuzzyBench grows through 29 incremental versions, each adding 100K–500K examples from a new fuzzy-task family.The released benchmark contains over 800 sub-categories, with 2,000 new validation and 2,000 new test specifications per version.
E Prefix-tuning Precursor Architecture: The prefix-tuning precursor compiles specifications and representative I/O examples into a per-example KV prefix that a frozen interpreter executes locally.The KV prefix is described as the compiled program or “neural binary.”
G Training Configuration: The main Qwen3 PAW training uses three epochs over 10M examples with effective batch 48 and negative mean-token log-likelihood under a frozen interpreter.The configuration uses no policy-gradient term or group baseline.
I Compiler Scaling and Freezing (Inconclusive): Compiler-scaling results are inconclusive because performance is non-monotonic: unfreezing the 4B compiler beats a frozen 32B compiler, while frozen gpt-oss-20B underperforms frozen Qwen3-4B-Instruct-2507.The study did not include a controlled large-data-scale comparison.
L Qualitative Analysis: Qualitative inspection found 12/20 perfect GPT-2 rollouts, 8/20 for Qwen3 0.6B, and 13/20 for Qwen3.5 0.8B, with failures including numeric computation and position tracking.The 0.8B model was strongest on structured outputs, multi-label classification, pattern transformation, and explicit-case logical reasoning.
M Full Case-Study Walkthroughs: Case studies compose PAW functions for log monitoring, site navigation, semantic reranking, tool calling, and word guessing, with compilation occurring once and inference running locally.The site-navigation pipeline uses five sequential functions, while the word-guessing project vetted a 361-word bank.
N Limitations / O Broader Impacts: PAW remains coupled to its compiler–interpreter pair, exposes only its discrete component for inspection, and has not validated learned multi-step reasoning; its best PEFT is task-dependent.Broader impacts include reduced API dependency and cost, reproducibility, and offline availability through a roughly 500 MB device-resident interpreter.