LeakyReLU.md (958B)
1 # Leaky ReLU 2 3 ML P554 4 5 **Definition:** Leaky ReLU is a variant of ReLU designed to solve the problem of neurons dying due to the use of ReLU. 6 7 Leaky ReLU adds a small (or larger) slope to the function representing values less than 0 for the activation function. This ensures neurons don't die, but they can enter long coma phases. 8 9 ReLU sometimes kills neurons because all inputs for all training samples result in a negative input to the activation function thus causing it to always output 0. 10 11 This can be specified in keras as follows: 12 13 ```python3 14 15 leaky_relu = tf.keras.layers.LeakyReLU(alpha=0.2) # defaults to alpha=0.3 16 dense = tf.keras.layers.Dense(50, activation=leaky_relu, kernel_initializer="he_normal") 17 18 ``` 19 20 Basically, initialize leaky relu with the hyperparameter of slope and then set it as the layer's activation function. Interestingly, when not specified 'Dense' uses a linear activation function which outputs the inputs * weights + bias.