# Weight-Space Programming Reading List

Program-as-Weights and the research behind programming neural-network weights: trained compilers, generated adapters, formal programs, and LLM reasoning over weights.

Program-as-Weights is weight-space programming through a trained neural compiler. A large model generates a program for a small interpreter. PAW introduced fuzzy-function programming on July 2, 2026: define a function, compile it, save the program, and call it locally.

The generated LoRA adapter changes the interpreter's weights through low-rank updates. PAW's fast compiler writes these weights in one forward pass, without per-function gradient descent.

[Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://arxiv.org/abs/2607.02512)

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng. NeurIPS 2026. First released July 2, 2026.

[Read the PAW paper on arXiv](https://arxiv.org/abs/2607.02512) · [PDF](https://arxiv.org/pdf/2607.02512) · [July 2 original version](https://arxiv.org/abs/2607.02512v1) · [PAW BibTeX](https://programasweights.com/reading-list/papers/2607.02512/citation.bib)

[Guide](https://programasweights.com/research/weight-space-programming/) · [BibTeX](https://programasweights.com/reading-list/weight-space-programming/references.bib) · [CSL-JSON](https://programasweights.com/reading-list/weight-space-programming/references.csl.json)

## Start with these five

1. [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://programasweights.com/reading-list/weight-space-programming/#program-as-weights) — Start with the programming paradigm: a large compiler builds a saved program for a small interpreter.

2. [HyperNetworks](https://programasweights.com/reading-list/weight-space-programming/#hypernetworks) — Read the foundation for using one neural network to generate another network’s weights.

3. [Text-to-LoRA: Instant Transformer Adaption](https://programasweights.com/reading-list/weight-space-programming/#text-to-lora) — See how a trained hypernetwork generates adapters from task descriptions.

4. [Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights](https://programasweights.com/reading-list/weight-space-programming/#sparks-weight-space-programming) — Compare trained compilation with a general-purpose LLM reading and writing numerical weights.

5. [Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations](https://programasweights.com/reading-list/weight-space-programming/#exploring-weight-space-programming) — Follow reasoning over weights into a study of repeated searches and cross-task transfer.

## Papers and research articles

### HyperNetworks

David Ha, Andrew Dai, Quoc V. Le

First arXiv submission: 2016-09-27. ICLR 2017.

Uses one neural network to generate the weights of another.

**Input:** Layer embeddings or the current recurrent state.

**Building the program:** A trained network generates the parameters of another network.

**Training and feedback:** The weight generator is trained jointly with the target network.

**Artifact:** Weights for the target network.

**Execution:** The target network runs with the generated weights.

**Connection to PAW:** HyperNetworks supplies a mechanism for generating weights. PAW trains a compiler to use generated parameters as reusable programs.

[Source](https://arxiv.org/abs/1609.09106) · [First public date](https://arxiv.org/abs/1609.09106) · [BibTeX](https://programasweights.com/reading-list/papers/1609.09106/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/1609.09106/citation.csl.json)

### Learning to Generate Task-Specific Adapters from Task Description

Qinyuan Ye, Xiang Ren

First arXiv submission: 2021-01-02. ACL-IJCNLP 2021.

Introduces Hypter, which generates adapters from natural-language task descriptions.

**Input:** A natural-language task description.

**Building the program:** A description encoder and hypernetwork generate adapters for a text-to-text model.

**Training and feedback:** The generator learns from tasks with descriptions and labeled examples.

**Artifact:** Bottleneck adapters inserted into a text-to-text model.

**Execution:** BART performs the task with the generated adapters.

**Connection to PAW:** An early description-to-adapter method cited by PAW. PAW develops the programming interface for saving and calling the resulting neural functions.

[Source](https://arxiv.org/abs/2101.00420) · [First public date](https://arxiv.org/abs/2101.00420) · [BibTeX](https://programasweights.com/reading-list/papers/2101.00420/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2101.00420/citation.csl.json)

### HyperTuning: Toward Adapting Large Language Models without Back-propagation

Jason Phang, Yi Mao, Pengcheng He, Weizhu Chen

First arXiv submission: 2022-11-22. ICML 2023.

Produces task-specific parameters for a frozen language model from a few demonstrations.

**Input:** A few examples of the task.

**Building the program:** HyperT5 generates prefix or LoRA parameters from task demonstrations.

**Training and feedback:** Hyperpretraining and multitask training prepare the generator; the paper also studies further task-specific tuning.

**Artifact:** Generated prefix or LoRA parameters.

**Execution:** A frozen T5 model runs with the generated parameters.

**Connection to PAW:** PAW cites HyperTuning as prior work on generating adaptation parameters from examples.

[Source](https://arxiv.org/abs/2211.12485) · [First public date](https://arxiv.org/abs/2211.12485) · [BibTeX](https://programasweights.com/reading-list/papers/2211.12485/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2211.12485/citation.csl.json)

### HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation

Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew Peters

First arXiv submission: 2022-12-20. ACL 2023.

Turns instructions and examples into reusable prefixes, adapters, and encoded instruction representations.

**Input:** Task instructions and optional examples.

**Building the program:** An instruction-conditioned hypernetwork generates modules, and instruction fusion caches the instruction representation.

**Training and feedback:** Instruction tuning trains the generator and model to generalize across tasks.

**Artifact:** Prefixes, adapters, and encoded instruction representations.

**Execution:** An encoder-decoder model uses the generated modules and attends to cached instruction representations.

**Connection to PAW:** HINT reuses generated modules and an encoded instruction. PAW reuses a program containing text and adaptation parameters.

[Source](https://arxiv.org/abs/2212.10315) · [First public date](https://arxiv.org/abs/2212.10315) · [BibTeX](https://programasweights.com/reading-list/papers/2212.10315/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2212.10315/citation.csl.json)

### Tracr: Compiled Transformers as a Laboratory for Interpretability

David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, Vladimir Mikulik

First arXiv submission: 2023-01-12. NeurIPS 2023.

Compiles RASP programs into transformer weights with known computational structure.

**Input:** A RASP program, input vocabulary, and maximum sequence length.

**Building the program:** A symbolic compiler translates each RASP operation into network components and assembles their weights.

**Training and feedback:** Compilation constructs the weights without model training. A separate experiment trains a projection to compress the compiled model.

**Artifact:** Transformer weights assembled from attention and MLP components.

**Execution:** The compiled transformer executes the specified sequence computation.

**Connection to PAW:** Tracr compiles a formal program. PAW learns to compile natural-language specifications of fuzzy functions into programs for a shared interpreter.

[Source](https://arxiv.org/abs/2301.05062) · [First public date](https://arxiv.org/abs/2301.05062) · [BibTeX](https://programasweights.com/reading-list/weight-space-programming/citations/tracr.bib) · [CSL-JSON](https://programasweights.com/reading-list/weight-space-programming/citations/tracr.csl.json)

### Text-to-LoRA: Instant Transformer Adaption

Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange

First arXiv submission: 2025-06-06. ICML 2025.

Uses a hypernetwork to generate LoRA adapters from task descriptions.

**Input:** A natural-language task description.

**Building the program:** A hypernetwork maps the task description to LoRA matrices.

**Training and feedback:** The paper trains generators by reconstructing adapters and through supervised task learning.

**Artifact:** Generated LoRA matrices.

**Execution:** A base transformer performs the task with the generated LoRA.

**Connection to PAW:** PAW cites Text-to-LoRA as prior work on generating adapters. PAW develops fuzzy-function programming, FuzzyBench, and an interface for reusable neural programs.

[Source](https://arxiv.org/abs/2506.06105) · [First public date](https://arxiv.org/abs/2506.06105) · [BibTeX](https://programasweights.com/reading-list/papers/2506.06105/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2506.06105/citation.csl.json)

### Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution

Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie

First arXiv submission: 2026-06-04. arXiv.

Trains a hypernetwork to generate repository-specific LoRA adapters from source code and update them as commits arrive.

**Input:** A repository snapshot; the evolution variant also receives successive commit diffs.

**Building the program:** Static maps an encoded repository snapshot to LoRA weights. Evo updates a GRU state with each commit diff and generates the weights from that state.

**Training and feedback:** The hypernetwork learns across repositories through the frozen code model’s assertion-completion loss. New repositories receive generated adapters without per-repository fine-tuning.

**Artifact:** Generated LoRA matrices for a frozen code language model.

**Execution:** The code model uses the generated adapter without adding repository context tokens to each request.

**Connection to PAW:** Code2LoRA and PAW train models to generate reusable weights. Code2LoRA puts repository knowledge into adapters; PAW compiles function specifications into neural programs with a compiler–interpreter interface.

[Source](https://arxiv.org/abs/2606.06492) · [First public date](https://arxiv.org/abs/2606.06492) · [BibTeX](https://programasweights.com/reading-list/papers/2606.06492/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2606.06492/citation.csl.json)

### Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Wentao Zhang, Liliana Hotsko, Woojeong Kim, Pengyu Nie, Stuart Shieber, Yuntian Deng

First arXiv submission: 2026-07-02. NeurIPS 2026.

Proposes fuzzy-function programming: compile a natural-language function specification into a neural program, then reuse it locally.

**Input:** A function specification, optionally with examples. The image extension also accepts visual information.

**Building the program:** A large trained compiler produces a neural program from the function specification. Its fast compiler generates LoRA weights in one forward pass.

**Training and feedback:** The compiler learns to generate programs before deployment. New functions compile without per-function gradient descent.

**Artifact:** A discrete pseudo-program and a compiler-generated parameter-efficient module.

**Execution:** A frozen lightweight interpreter loads the program and runs new inputs locally.

**Connection to PAW:** PAW makes weight-space programming a software interface: define a fuzzy function, compile it, save the neural program, and call it locally. The large model is the compiler; the small model is the interpreter.

[Source](https://arxiv.org/abs/2607.02512) · [First public date](https://arxiv.org/abs/2607.02512) · [BibTeX](https://programasweights.com/reading-list/papers/2607.02512/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2607.02512/citation.csl.json)

### Compile by Training: Turning Natural-Language Specifications into Local Neural Functions

Yuntian Deng, Pengyu Nie, Stuart Shieber

First arXiv submission: 2026-09-03. EMNLP 2026 System Demonstrations.

Compiles a function description into a reusable local program by generating training examples and optimizing an adapter.

**Input:** The specification guides teacher synthesis. PAW’s fast compiler supplies the initial adapter and runtime prompt.

**Building the program:** Teacher models generate examples from the specification. Gradient descent refines the adapter initialized by PAW’s fast compiler.

**Training and feedback:** Training happens at function compilation; the saved program runs without the teachers.

**Artifact:** A LoRA adapter, runtime prompt, specification, and interpreter metadata in a PAW program.

**Execution:** A frozen Qwen3-0.6B interpreter runs the saved function locally without further teacher calls.

**Connection to PAW:** Extends PAW with training at compile time while keeping its program format and runtime interface.

[Source](https://aclanthology.org/2026.emnlp-demos.78/) · [First public date](https://arxiv.org/abs/2609.04199) · [BibTeX](https://programasweights.com/reading-list/papers/2609.04199/citation.bib) · [CSL-JSON](https://programasweights.com/reading-list/papers/2609.04199/citation.csl.json)

### Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights

Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab

Article publication date: 2026-11-02. Research article.

Uses a general-purpose LLM to read neural-network weights, construct networks, and edit an existing model through numerical reasoning and tools.

**Input:** A natural-language objective, network weights or an empty network, and task-dependent tool access.

**Building the program:** The LLM reasons about the computation and writes the numerical parameters. Larger experiments use a Python environment and forward-pass feedback.

**Training and feedback:** The reported weight-construction and editing procedures prohibit gradient descent and training loops.

**Artifact:** New network weights or an edit to an existing checkpoint.

**Execution:** The constructed or edited network runs on new inputs.

**Connection to PAW:** PAW generates reusable neural programs with a trained compiler. Sparks uses a general-purpose LLM to inspect and write numerical weights directly.

[Source](https://rikiyat.github.io/blog/weight-space-programming/) · [First public date](https://rikiyat.github.io/blog/weight-space-programming/) · [BibTeX](https://programasweights.com/reading-list/weight-space-programming/citations/sparks-weight-space-programming.bib) · [CSL-JSON](https://programasweights.com/reading-list/weight-space-programming/citations/sparks-weight-space-programming.csl.json)

### Weight-Space Programming from scratch on simple tasks

Chandan Singh

Article publication date: 2026-11-02. Research article.

An LLM coding agent writes transformer weights for tasks such as addition, sorting, and parity, then revises the programs using evaluation feedback.

**Input:** A task definition and a transformer implementation with an empty weight-writing function.

**Building the program:** The agent edits the architecture and parameter-setting code, runs the resulting model, and simplifies successful programs.

**Training and feedback:** The algorithmic-task procedure uses closed-form parameter assignment. The report identifies fitting and evaluation leakage in its language-task runs.

**Artifact:** A transformer architecture and its explicitly assigned weights.

**Execution:** The written transformer generates its answer token by token.

**Connection to PAW:** Both this work and PAW produce neural programs that run after construction. Here an LLM writes the implementation through iterative code edits; PAW uses a trained compiler and a shared interpreter.

[Source](https://csinva.io/weight-space-programming-autoresearch/analyze/toy_examples/report.html) · [First public date](https://csinva.io/weight-space-programming-autoresearch/analyze/toy_examples/report.html) · [BibTeX](https://programasweights.com/reading-list/weight-space-programming/citations/weight-space-programming-simple-tasks.bib) · [CSL-JSON](https://programasweights.com/reading-list/weight-space-programming/citations/weight-space-programming-simple-tasks.csl.json)

### Exploring Weight-Space Programming: Capabilities, Generalization, and Limitations

Junru Hu, Cheng Tang, Wei Li, Hongze Zhu, Chun-Mei Feng, Qi Zhang, Yirong Chen, Ming Hu

First arXiv submission: 2026-11-06. arXiv.

Studies direct weight editing across 16 searches, seven benchmark families, and two model scales, measuring held-out gains and transfer between tasks.

**Input:** A pretrained Qwen3.5 checkpoint, task data, and generation or teacher-forcing feedback.

**Building the program:** A reasoning agent inspects parameters, writes weight updates, and selects a checkpoint using development feedback.

**Training and feedback:** The editing search uses no gradient-based optimization. Each search has a budget of 50 successful forward-tool requests.

**Artifact:** A modified pretrained checkpoint.

**Execution:** The selected checkpoint answers held-out questions without the editing agent.

**Connection to PAW:** This study evaluates reasoning-driven edits to an existing model. PAW compiles a function definition into a separate reusable program for its interpreter.

[Source](https://arxiv.org/abs/2611.09382) · [First public date](https://arxiv.org/abs/2611.09382) · [BibTeX](https://programasweights.com/reading-list/weight-space-programming/citations/exploring-weight-space-programming.bib) · [CSL-JSON](https://programasweights.com/reading-list/weight-space-programming/citations/exploring-weight-space-programming.csl.json)

Updated 2026-11-14T07:09:15+00:00.