Definition

Matrix Multiplication

A neuron computes a weighted sum of its inputs, which is a dot product. A layer with thousands of neurons sharing the same inputs is thousands of dot products, which is by definition a matrix multiplication. Attention, feed-forward layers, embeddings, and gradients are all implemented this way. That is why GPUs, built to multiply large matrices in parallel, became the hardware of AI.