AI LAB / EXPERIMENT 01
Attention Explorer
Follow a token from query–key scores to a weighted mixture of values. Change the controls and inspect the numbers behind every connection.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
Exact calculations, small untrained matrices. This experiment illustrates the mechanism; its weights do not represent learned language relationships.
Interactive controls require JavaScript. Attention compares each query with the keys, normalizes the scores across keys, then combines their value vectors. Read the method and limitations below.
Choose a query token
Each row asks: which value vectors should this token mix?
Attention weights
Rows are queries. Columns are keys. Select any cell to inspect its calculation.
Values in the mixture
Inspect one connection
Inspect the matrices and both heads
All entries below are from the current input. Shapes: X ∈ ℝⁿˣ⁴; WQ, WK, WV ∈ ℝ⁴ˣ²; each head ∈ ℝⁿˣ²; WO ∈ ℝ⁴ˣ⁴. Displayed numbers are rounded; calculations use full precision.
Three experiments to try
- Select cat, then enable Causal mask. Positions to its right become exactly zero; the remaining weights are normalized again.
- Move τ toward 0.25, then 2.00. Compare the distribution while Q, K and V remain fixed.
- Switch between Head 1 and Head 2. Different projections produce different scores and value mixtures from the same input.
What the numbers mean
A query–key dot product is a compatibility score. Scaling controls its magnitude; softmax turns each row into nonnegative weights summing to one. Multiplying by V produces a new vector for each query.
The two head outputs are concatenated and multiplied by WO. This is the attention sublayer’s output, before any residual connection, normalization or feed-forward network.
Model assumptions and limitations
This is an educational toy model with d_model = 4, two heads, and dₖ = dᵥ = 2. It runs locally in your browser without calling a model API.
Tokens are split at spaces. Case-insensitive token hashes generate deterministic four-dimensional inputs between −1 and 1. Repeated words therefore share an input vector. No positional encoding is added; token order matters here only when the causal mask is enabled. The projection matrices are fixed, hand-chosen and untrained.
τ = 1 gives the standard scaled dot-product formula. The additional temperature control divides logits by τ before softmax. Causal masking excludes future positions with −∞. There is no dropout, training, residual connection, layer normalization, feed-forward network or vocabulary prediction in this experiment. Attention weights alone are not a complete explanation of a model’s reasoning.