ExplodingGradients.md (502B)
1 # Exploding Gradients 2 3 ML 550 4 5 **Definition:** Exploding gradients is a problem with training neural networks where lower levels have very high gradients and thus the gradient steps diverge from a proper solution. 6 7 This is the opposite of [VanishingGradients](VanishingGradients.md) 8 9 This often occurs for recurrent neural networks. 10 11 ### Solutions 12 13 Use ReLU and better weight initialization (not gaussian distribution with std deviation of 1). 14 15 See [UnstableGradients](UnstableGradients.md) for more.