Backpropagation.md (1368B)
1 # Backpropogation 2 3 ML D6 4 5 **Definition:** Backpropagation is the combination of reverse-mode autodiff and gradient descent to iteratively improve models based on expected outputs by given inputs by following the gradient for each [Weight](Weight.md) and [Bias](Bias.md). 6 7 When using backpropogation we use many mini-batches. Generally we go through the entire dataset to train multiple times and these passes are called epochs. When using mini-batches we first find the values from the input layer for each input, then we go to the second layer, and so on until reaching the output layer. This is the forward pass stage. An important note is that all intermediate values must be preserved to ensure we can do the backward pass. 8 9 Once the forward pass is completed we then compute a loss function to find the output error. 10 11 Next, we compute how much each bias and connection contributed to this error moving are way backwards from the output layer to the input layer. This is done using the chain rule. 12 13 Lastly, using these error gradients, we do a gradient descent step to tweak the connection weights and biases. 14 15 When doing backpropogration we should replace the MLP's step function with a function that does not have a derivative of 0 in all places (ReLU, Sigmoid, etc) to ensure gradient descent steps can be made. This is referred to as the activation function.