Weight-Space Programming
Methods, papers, and history
Yuntian Deng · Published November 3, 2026 · Updated November 14, 2026
Programming does not have to mean writing code. A model can also program by generating the weights of another model. Program-as-Weights is weight-space programming: we call the model doing the programming a neural compiler. The large compiler generates a LoRA adapter for a small interpreter, which loads the adapter and runs the function.
Program-as-Weights (PAW) introduced fuzzy-function programming on July 2, 2026: describe a function in natural language, compile it into a reusable neural program, and execute it locally. Its fast compiler generates a small model’s LoRA weights in one forward pass. PAW established this programming model four months before Sparks of Weight-Space Programming, published on November 2, 2026.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng
NeurIPS 2026 · First released July 2, 2026
PAW paper on arXiv ↗Paper PDFJuly 2 originalPAW BibTeX
Weight-space programming reading list →Compare methodsPublication history
From a model that answers to a model that builds
Suppose you need a function that repairs malformed JSON. Calling a large language model for every input repeats the cost of using that model. PAW moves the large model’s work to function definition: a neural compiler builds a program, and a shared lightweight interpreter executes it on new inputs.
Function description → neural compiler → saved neural program Input + saved program → small interpreter → output
The program becomes a software component. You can save it, distribute it, version it, and compose it with ordinary code. The compiler–interpreter interface defines the programming model. LoRA adapters and prefix vectors are program formats used in the original paper.
This is the AI-as-tool-builder vision: train large neural compilers to build reusable tools. Spend the large model’s computation on creating a function, then run that function locally on a small interpreter.
Programming with LoRA weight updates
A LoRA adapter represents a weight update. PAW’s compiler generates the matrices that change the small interpreter’s computation. The base weights stay shared across functions; each saved adapter supplies the program for one function.
PAW’s fast compiler and the pretrained LLM in Sparks both write weights without a training loop for each new program. PAW learns to do this through compiler training. Sparks uses an existing LLM’s reasoning and tools. Both are weight-space programming; the distinction is how the model learns and performs the programming.
How are the weights produced?
Generating weights with a trained compiler, compiling a formal program, and asking an LLM to reason over numerical weights are different ways to build or change a network.
| Method | Input | How it produces weights | What runs afterward |
|---|---|---|---|
| Program-as-Weights | A natural-language function specification | A trained compiler generates a reusable neural program; its fast compiler predicts adapter weights in one forward pass. | A small shared interpreter loads the saved program and processes new inputs. |
| Text-to-LoRA | A task description | A trained hypernetwork generates a LoRA adapter. | The target language model uses the generated adapter. |
| Code2LoRA | A repository snapshot and subsequent code diffs | A trained hypernetwork generates a repository-specific LoRA adapter. The evolution variant updates it through a GRU state as commits arrive. | A frozen code language model uses the adapter for repository-aware code completion. |
| Tracr | A program written in RASP | A compiler translates the formal program into transformer weights. | The constructed transformer executes the program. |
| Sparks of Weight-Space Programming | A natural-language goal and access to network weights | A general-purpose LLM reads, writes, and edits numerical weights through reasoning and tools. | The constructed or edited network. |
| Compile by Training | A natural-language function specification | Teachers generate examples; gradient descent refines the adapter produced by PAW’s fast compiler. | PAW’s interpreter loads the saved program and runs without the teachers. |
The reading list connects these methods to hypernetworks, earlier generated adapters, and recent studies of LLM reasoning over weights.
Code2LoRA is another example of learning to program through weights. It trains across repositories to turn source code into adapters. PAW trains across functions to turn specifications into reusable neural programs. Both move work into a weight generator and save the generated artifact for later execution.
Train models to become neural compilers
PAW and the MIT work both use a large model to program through neural-network weights. The difference is how the programming capability is acquired and used. PAW trains a compiler to turn function specifications into programs. The MIT work investigates weight reading, writing, and editing in existing pretrained models, without training a dedicated weight-space programmer.
Our direction is to train for this capability: large neural compilers that produce programs for small interpreters. The PAW paper implements this direction with a trained compiler. The MIT report studies what existing models do through reasoning and tools. Both address programming in weight space.
The training objective changes from “answer this input” to “build a program that answers new inputs for this task.” Continuous program weights let us train the compiler through the interpreter’s outputs. This makes tool building a training objective in its own right.
PAW trains its compiler beforehand. Compiling a new function with the fast compiler generates its weights without per-function gradient descent. Execution then uses the saved program without calling the compiler again. These are three separate stages: training the compiler, compiling a function, and running that function. PAW’s training procedure.
In Sparks, the general-purpose model reasons over the target network’s parameters and changes them. In Compile by Training, the compiler generates examples and optimizes an adapter for the requested function. Each method produces a different computation at build time; the saved artifact determines the computation at runtime.
Publication history
- September 27, 2016 — HyperNetworks. David Ha, Andrew Dai, and Quoc V. Le use one neural network to generate another network’s weights.
- January 2, 2021 — Hypter. Qinyuan Ye and Xiang Ren generate task-specific adapters from natural-language task descriptions.
- June 4, 2026 — Code2LoRA. Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie. Generates repository-conditioned adapters and updates them as the code changes.
- July 2, 2026 — Program-as-Weights. Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. PAW introduces fuzzy-function programming, the compiler–interpreter interface, and reusable neural programs for local execution.
- September 3, 2026 — Compile by Training. Yuntian Deng, Pengyu Nie, Stuart Shieber. Adds compilation through teacher-generated examples and adapter training, retaining PAW’s program format and local runtime.
- November 2, 2026 — Sparks of Weight-Space Programming. Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab demonstrate a general-purpose LLM reading weights, constructing networks, and editing a language model’s weights through reasoning and tools.
What should I read next?
Start with PAW for the programming paradigm. Read generated weights and adapters for the compiler mechanism, then compare it with reasoning over numerical weights in the short reading path. The dated bibliography links each work to its source, method, and citation.
Cite the research
For fuzzy-function programming, the compiler–interpreter model, and PAW’s fast compiler, cite Program-as-Weights. For compilation through teacher-generated examples and adapter training, cite Compile by Training.
PAW BibTeXCompile by Training BibTeXWeight-space bibliography