The backwardpass.
Your loss goes down. Your gradients are wrong. Both can be true at once. To see why, follow one small network through its forward computation, reverse derivatives, and parameter update. The important part is a value used twice—and the contribution you might forget to add.
One prediction.
Three separate jobs.#
A training step contains a model evaluation, a derivative computation, and a parameter update. Keeping those jobs separate makes the mathematics easier to understand—and the implementation easier to debug.
Backpropagation is reverse-mode automatic differentiation applied to a model’s computation graph. It calculates derivatives. Gradient descent uses those derivatives to change parameters. You can run the backward pass without changing a single weight.[1][2]
Use this deliberately small scalar network. The input is x = 2, the target is t = 1, and the trainable parameters are θ = (w, b, v) = (0.5, 0, 1). Every example below uses this same network and this same training pair.
The model maps an input to a prediction: x ↦ ŷ. The training objective maps a parameter setting to a loss: θ ↦ L, with x and t held fixed. Those are different functions. Here, the prediction is 1.761594 and the loss is 0.290013, rounded to six decimals.
The factor ½ cancels the 2 introduced when you differentiate a square. It changes the loss scale, not the parameter settings where this squared error is minimized. This is a single-example objective; a dataset objective would aggregate losses across examples.
Calculate intermediate values, prediction, and loss at the current parameters.
Calculate how the scalar loss changes locally with each parameter.
Choose how far and in which direction to change those parameters.
Notice what you need to retain: the backward rule for tanh uses its forward output a, while multiplication needs its operands. Autodiff systems save selected intermediate tensors for that reason. The graph records dependencies; it is not a diagram of biological neurons.[2]
Multiply locally.
Accumulate globally.#
At each operation, incoming sensitivity is multiplied by a local derivative. When several downstream uses depend on the same value, their contributions add. That addition is the central event in our example.
Write q̄ = ∂L/∂q for each node. This is an adjoint, not a new forward activation. Initialize adjoints to zero and seed the loss with L̄ = 1, because ∂L/∂L = 1. Then visit operations in reverse topological order, after their downstream contributions are available.[3]
Start at the loss
The prediction is too high. A sufficiently small increase in ŷ would increase this loss. Now distribute that sensitivity through ŷ = va + u. Multiplication sends each operand the other operand times the incoming adjoint; addition sends the incoming adjoint to each input.
Return through the nonlinear branch
The derivative of tanh(z) is 1 − tanh²(z). You already computed a = tanh(z) in the forward pass, so you can use 1 − a² = 0.419974. Continue through z = u + b:
The shared value needs both contributions
Changing u changes the direct term in ŷ and the input to tanh. Neither path cancels the other. The total derivative adds their effects before propagating through u = wx:
This is why a tiny autodiff engine uses += when writing adjoints. An assignment that overwrites a previous contribution can silently differentiate the wrong computation. Karpathy’s micrograd makes both reverse traversal and accumulation visible in a small implementation.[3]
A falling loss can
hide a broken gradient.#
Verify two separate claims: whether the backward pass computes the derivative of your forward program, and whether a particular optimizer step improves the objective. Neither test substitutes for the other.
For plain gradient descent, calculate θ′ = θ − ηg, where g = ∇L(θ) and η is the learning rate. Update all three parameters from the original gradient. At η = 0.10, the new parameters are approximately (0.283711, −0.031985, 0.941997), and a fresh forward pass gives L(θ′) = 0.000408.
At η = 0.20, the exact same correct gradient produces a loss of 0.322089—higher than the starting loss of 0.290013. The gradient is a local derivative, not a promise about every finite step. For this smooth objective and nonzero g:
Now deliberately omit the direct contribution to ū. The resulting incorrect gradient is (0.639700, 0.319850, 0.580026). At η = 0.10 it still reduces the loss to 0.134228. Its direction happens to remain downhill; its derivative of w is nevertheless wrong.
Change the step. Break the derivative.
Compare a complete backward pass with one that omits the skip-path contribution. Both use the original forward model to measure the resulting loss.
Complete gradient, η = 0.10: the derivative check passes and loss decreases from 0.290013 to 0.000408.
Ask the forward program to check your work
Central finite differences estimate one parameter derivative by evaluating the loss on both sides of the current value. For parameter i, with ei the corresponding unit vector:
Use ε = 10−6 here. The analytical gradient agrees with this independent calculation to about 10−10 in double precision. The broken backward pass misses ∂L/∂w by about 1.523188. A gradient check tests the derivative; a falling loss does not.
Run it · Python standard library only
import math
theta = (0.5, 0.0, 1.0) # w, b, v; x=2, target=1
def loss(p):
w, b, v = p
u = 2.0 * w
prediction = v * math.tanh(u + b) + u
return 0.5 * (prediction - 1.0) ** 2
w, b, v = theta
a = math.tanh(2.0 * w + b)
error = v * a + 2.0 * w - 1.0
bar_z = error * v * (1.0 - a * a)
bar_u = error + bar_z # Both paths contribute.
gradient = (2.0 * bar_u, bar_z, error * a)
eps = 1e-6
for i, name in enumerate(("w", "b", "v")):
plus, minus = list(theta), list(theta)
plus[i] += eps
minus[i] -= eps
estimate = (loss(plus) - loss(minus)) / (2 * eps)
print(name, gradient[i], estimate)
assert abs(gradient[i] - estimate) < 1e-7
updated = tuple(p - 0.1 * g for p, g in zip(theta, gradient))
print("new loss:", loss(updated)) # about 0.000407694Finite differences are a check, not the training algorithm here. Very small ε can amplify floating-point cancellation; large ε no longer probes the local derivative accurately. Checks near a nondifferentiable point or with changing random inputs require extra care. Keep the forward inputs fixed when comparing results.
The same rules.
A different scale.#
Our three-parameter example exposes the essential mechanism. Tensor libraries apply the same chain rule in blocks, reusing efficient operations instead of enumerating every scalar path.
You usually need a product, not a Jacobian
For y = f(x), with x ∈ ℝⁿ and y ∈ ℝᵐ, the Jacobian J has shape m × n. A reverse rule takes an upstream sensitivity ȳ and returns x̄ = Jᵀȳ. This is a vector–Jacobian product expressed using column vectors. You need its result, not necessarily the full matrix J.[1]
Our tanh rule is the scalar instance. A dense layer makes the tensor version concrete. For z = Wh + b, W has shape m × n, h has n entries, and z̄ has m entries. The reverse rules are:
The outer product gives a gradient for every weight. If h also feeds another branch, add its contribution to h̄—exactly as we added the two contributions to ū. For a batch, reductions depend on whether the loss is a sum or a mean.
The time advantage comes with a memory bill
Coordinate-wise central differences need two model evaluations per parameter. Reverse mode instead traverses the executed graph backward once. When local derivative rules have costs comparable to their forward operations, its arithmetic cost is a constant-factor multiple of forward evaluation, rather than a separate forward run for every parameter.[1]
In return, reverse mode needs intermediate values. Our network retains a for 1 − a²; a deep network can retain many large activations. Activation checkpointing saves selected values and recomputes others during the backward pass. It trades extra computation for less activation storage; it does not remove parameter or optimizer-state memory.[4]
A correct derivative has boundaries
A gradient describes local sensitivity. A zero gradient may mark a minimum, maximum, or saddle. Step-size conditions require assumptions about the objective.[5]
ReLU has no classical derivative at zero. Frameworks use a defined backward convention; PyTorch’s choice there is zero.[2]
Backpropagation serves differentiable computations. It does not describe all machine learning, prove generalization, or establish how biological brains learn.[8]
The history is also broader than one invention story. Modern reverse accumulation has roots in Linnainmaa’s 1970 thesis and 1976 paper. Rumelhart, Hinton, and Williams’ 1986 work became a landmark in learning hidden representations with backpropagated errors.[6][7]
Inspect the machinery.#
The numerical example is fully specified and independently checked with central differences. Rounded labels aid reading; calculations retain floating-point precision. The three-parameter demonstration is not an empirical benchmark or a newly discovered learning algorithm.
- Forward- and reverse-mode autodiff. JAX documentation.Technical guideJVPs, VJPs, scalar gradients, and the computational trade-offs of reverse mode.
- Autograd mechanics. PyTorch documentation, version 2.14.Implementation guideExecuted graphs, saved tensors, and conventions for nondifferentiable operations.
- micrograd: the autograd engine. Andrej Karpathy.Original source codeReverse topological traversal, a scalar seed of one, and gradient accumulation with +=.
- Training Deep Nets with Sublinear Memory Cost. Tianqi Chen and colleagues, 2016.Research paperCheckpointing through recomputation; activation-storage savings depend on the computation and schedule.
- Gradient descent and smoothness. Cornell CS4787 course notes, 2025.Mathematical conditionsThe descent lemma and learning-rate conditions under smoothness assumptions.
- Taylor expansion of the accumulated rounding error. Seppo Linnainmaa · BIT Numerical Mathematics, 1976.Original paperReverse accumulation history; the paper references the author’s 1970 master’s thesis.
- Learning representations by back-propagating errors. David Rumelhart, Geoffrey Hinton, and Ronald Williams · Nature, 1986.Original paperA landmark for learning useful hidden representations, not the sole origin of reverse-mode differentiation.
- Random synaptic feedback weights support error backpropagation for deep learning. Timothy Lillicrap and colleagues · Nature Communications, 2016.Research paperFeedback alignment relaxes precise weight symmetry in artificial networks; it does not establish that brains implement backpropagation.
- The Most Important Algorithm in Machine Learning. Artem Kirsanov.Source videoThe conceptual starting point. This article develops a different worked example, a deliberate gradient bug, and a reproducible numerical experiment.
