Interpreting Inkling 2 Mini

Impact 7 Goodfire Research

Summary

  • Thinking Machines released Inkling 2 Mini, a small open-weights model featuring 24 layers divided into four blocks with depth-wise attention and looped middle layers.
  • Goodfire Research investigated the model's internal mechanisms, analyzing how information flows through depth-wise and cross-loop attention channels across blocks and loops.
  • The analysis reveals that late layers within each block heavily write for reader weights to tag activations, while looped layers can read from future or past iterations.

Why it matters

Understanding the internal mechanisms of non-vanilla transformer architectures like looped models and depth-wise attention helps researchers map information flow and adapt interpretability tools for evolving model designs.

Who should care