Reading and Writing with Projections
Transformers store, retrieve, and modify data along different feature directions in their model space, via projections.
The Inner Workings of Multihead Latent Attention (MLA)
Multihead Latent Attention (MLA), introduced by DeepSeek in their V2 model, is an alternative to standard attention (and other variants such as MQA and GQA) which dramatically reduces memory bandwidth requirements for the attention calculations.
Patterns and Messages - Part 6 - Vocabulary-Based Analysis
What had me most excited about the merged matrix perspective (and perhaps overly so) was that the patterns and messages are in model space, the same dimension as the vocabulary.
Patterns and Messages - Part 5 - The Residual Stream
Something I find really helpful about this merged-matrix perspective is that it puts everything in “model space”. The patterns and messages and their projection matrices all have the same length as the word embeddings.
Patterns and Messages - Part 4 - Attention as a Dynamic Neural Network
When you reduce Attention down to two matrices instead of four, the pattern and message vectors represent a more familiar architecture–they form a neural network, whose neurons are created dynamically at inference time from the tokens.
Older
Newer