notes

commit 0220388672490820ac52c1a5565dfec5cc0d50d7
parent 2d750a4eb7534b652291f5cc68ec47287381161a
Author: Andrew <andrewlaack1@gmail.com>
Date:   Tue, 25 Jun 2024 14:35:05 -0500

Took some notes

Diffstat:
A
Dropout.md
 | 
14
++++++++++++++
M
MachineLearning.md
 | 
6
+++++-
A
MaxNormRegularization.md
 | 
8
++++++++
A
Tensor.md
 | 
10
++++++++++
A
Transpose.md
 | 
14
++++++++++++++

5 files changed, 51 insertions(+), 1 deletion(-)
diff --git a/Dropout.md b/Dropout.md
@@ -0,0 +1,14 @@
+:ml:
+# Dropout
+
+ML P604
+
+## Notes
+
+**Definition:** Dropout is a regularization technique for deep neural networks where upon each pass every neuron has a constant probability of being 'dropped out' meaning the output is 0.
+
+This works very well with a rate somewhere between 10%-50%. With RNNs we often do 20%-30% and with CNNs we use 40%-50%.
+
+This form of regularization ensures the model has multiple neurons that perform similar functions instead of being dependent upon one or only a few neurons to do important things.
+
+When using dropout we never drop output neurons. Additionally, it is common to only dropout neurons from the first few layers.
diff --git a/MachineLearning.md b/MachineLearning.md
@@ -12,6 +12,7 @@ Links to ML Notes
 	- [[ReinforcementLearning.md]]
 3. Create a walking model
 	- [[ReinforcementLearning.md]]
+4. Dropout where you simply skip the gradient update?
 
 ## Good Info
 
@@ -115,7 +116,6 @@ Concepts:
 [[RandomPatches.md]]
 [[RandomSubspaces.md]]
 [[ExtraTrees.md]]
-[[Boosting.md]]
 [[AdaBoost.md]]
 [[GradientBoosting.md]]
 [[HistogramBasedGradientBoosting.md]]
@@ -150,3 +150,7 @@ Concepts:
 [[NAG.md]]
 [[AdaGrad.md]]
 [[Adam.md]]
+[[Dropout.md]]
+[[MaxNormRegularization.md]]
+[[Tensor.md]]
+[[Transpose.md]]
diff --git a/MaxNormRegularization.md b/MaxNormRegularization.md
@@ -0,0 +1,8 @@
+:ml:
+# Max-Norm Regularization
+
+ML P612
+
+## Notes
+
+**Definition:** Max-norm regularization is a regularization technique for neural networks that limits the combination (euclidean norm) of all incoming weights to a predefined range. If a step goes beyond this the weights are scaled accordingly to ensure compliance. 
diff --git a/Tensor.md b/Tensor.md
@@ -0,0 +1,10 @@
+:ml:
+# Tensor
+
+ML P626
+
+## Notes
+
+**Definition:** A tensor is a multidimensional array of any dimensionallity. 
+
+Tensors can be 0-dim (scalars), 1-dim (vectors), 2-dim (matrix), and higher dimensions as well.
diff --git a/Transpose.md b/Transpose.md
@@ -0,0 +1,14 @@
+:ml:
+# Transpose
+
+ML P627
+
+## Notes
+
+**Definition:** The transpose of a matrix is the matrix flipped over the diagnol by switching the rows and columns. 
+
+2 4 1    2 3 4
+3 7 2 -> 4 7 6
+4 6 3    1 2 3
+
+As you can see, the first value remains and across the top we have the first column.