GradientClipping.md (789B)
1 # Gradient Clipping 2 3 ML P569 4 5 **Definition:** Gradient clipping is the process of clipping gradients during backpropogration so they never exceed some threshold. 6 7 This is another technique used to resolve issues relating to [ExplodingGradients](ExplodingGradients.md) particularly for RNNs where batch normalization does not work. 8 9 There are two ways to do gradient clipping either with a threshold cut off or with vector scaling. With vector scaling we retain the direction of the vector and set the minimize the largest value to 1 (if greater than 1) while scaling all other features proprotionally. More commonly, we simply truncate values so if we have [100, .1] with a threshold of (-1,1) we would then scale the vector to [1, .1]. 10 11 Scaling the entire vector is called normalization.