Neural Compilation from Natural-Language Specifications
Program-as-Weights (PAW) compiles a natural-language function specification into a reusable neural program. PAW introduces fuzzy-function programming: define a function in natural language, compile it once, and call the saved program on new inputs with a small local interpreter. Its fast compiler generates program weights in one forward pass.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. NeurIPS 2026.
First released · PAW BibTeX
Neural compilers differ in what they produce and how they build it. A task description can become generated weights, compressed input context, or saved activations. That representation determines what is reused and what must run for every new input.
Generated weights and adapters
HyperNetworks uses one network to generate weights for another. Hypter generates task-specific adapters from descriptions. HyperTuning generates adaptation parameters from few-shot examples. HINT encodes instructions through a hypernetwork.
Text-to-LoRA maps task descriptions to LoRA adaptations. PAW places generated adapters inside a programming interface: the compiler creates a neural program, software saves and composes it, and a lightweight interpreter executes it on new inputs. PAW’s programs combine a text pseudo-program with learned parameters. Read the compiler–interpreter system in the original PAW paper.
From generated adapters to reusable functions
Hypter, HINT, HyperTuning, and Text-to-LoRA generate adaptations from descriptions or examples. PAW uses generated parameters as part of a program that can be saved, called from ordinary code, and reused by a fixed interpreter.
Compilation through training
Compile by Training, by Yuntian Deng, Pengyu Nie, and Stuart Shieber, extends PAW with training at compile time. It was first released on September 3, 2026 and published at EMNLP 2026 System Demonstrations. Teacher models synthesize examples from a natural-language function specification. The compiler uses those examples to refine an adapter initialized by PAW’s fast compiler, then packages it with the runtime scaffold for repeated local execution. The shared Qwen3-0.6B interpreter stays frozen during this training procedure.
PAW’s fast compiler predicts program parameters. Compile by Training adds optimization for the specification while preserving PAW’s program format, interface, and local runtime. Read Weight-Space Programming and Program-as-Weights for the connection between task descriptions, generated weights, and reusable functions. The evaluation guide compares compilation cost and accuracy.
Fast prediction and training
PAW’s fast compiler predicts a program. Compile by Training starts from that prediction, generates examples, and trains an adapter. Both use the same runtime interface; the extra training improves semantic accuracy on FuzzyBench-Hard at a higher compilation cost.
PAW compiler implementations · Compile by Training method · Compile by Training evaluation
Context compression
Processing the same instruction for every input repeats work. Gisting compresses prompts into a small number of gist-token activations. Generative Adapter produces contextual parameter updates.
To compare their cost, count the work needed to create the representation and the computation that remains for each input. The conditioning information also matters: an instruction, a demonstration, and a document provide different information about the task.
What gets reused?
Gisting caches a few instruction activations. Generative Adapter stores context in parameter updates. PAW saves a function as a program containing text and a neural module.
Gist paper · Generative Adapter paper · PAW program definition
Prefixes and LoRA adapters
Prefix-Tuning learns continuous prefix activations for a frozen language model. LoRA represents adaptation through low-rank updates to model weights. Both original methods learn task adaptations from examples.
PAW generates prefixes or LoRA adapters from a specification. Compile by Training refines an initialized LoRA using synthesized examples. Prefixes and LoRA define the program’s representation; the compiler determines how it is produced; the programming interface determines how software saves and calls it.
Program form and compiler
LoRA represents adaptation as low-rank weight updates; prefix tuning uses continuous activations. Their original methods learn these parameters from examples. PAW generates both forms from specifications.
Comparing compilers
For the PAW compiler, read how task descriptions become LoRA program weights and how teacher-generated examples train the compiler. The same build-then-run interface supports two-stage decision models with fast local inference.
For each method, compare its inputs, its output, whether a new task requires optimization, what stays frozen, and what is reused across calls. The paper pages describe these components. The evaluation guide covers accuracy, robustness, and execution cost.