Definition
Self-Attention
Self-attention lets every token in a sequence look at every other token at once and decide which ones matter. Each token computes a Query, Key, and Value vector; comparing Queries to Keys produces relevance scores that weight the Values. The result is a representation of each word informed by its whole context, computed in parallel rather than one word at a time.
Explained in
Chapter 3: Attention Is All You Need FreeThe paper behind every major AI model today. Google invented it. Then watched someone else ship it.
Related terms