Language Models Can Rewrite Their Own Weights

Impact 8 arXiv cs.AI

Summary

  • Researchers introduce self-rewriting, a technique where a 27B language model diagnoses its failures and edits its own weights without gradients or training.
  • The model's self-directed edits increase accuracy by 6.6 points on unseen problems across eight continuous improvement sessions, persisting without added inference cost.
  • The 27B model also successfully repairs a corrupted copy of itself using only text math problems, recovering performance on unseen image benchmarks from 7% to 70% accuracy.
  • Smaller models in the same family (2B, 4B, 9B) failed to perform effective self-rewriting, demonstrating that this capability emerges with model scale.

Why it matters

Self-rewriting bypasses traditional pre- and post-training infrastructure, allowing models to autonomously improve and repair themselves using only inference capabilities on a single GPU.

Who should care