commit 582fc1030ad7b8d472428d3bbd00e6fca62ca5b5
parent ab87b381b8a20905e186b6336c9380f20be8afe6
Author: Andrew <andrewlaack1@gmail.com>
Date: Thu, 18 Apr 2024 17:41:35 -0500
Simple gradient descent implementation
Diffstat:
6 files changed, 105 insertions(+), 2 deletions(-)
diff --git a/GradientDescent.md b/GradientDescent.md
@@ -1 +1,16 @@
-:todo:
+:ml:
+# Gradient Descent
+
+ML L2
+
+## Notes
+
+**Definition:** Gradient Descent is an algorithm used to find a 'near' optimal approach to the given problem. This is used with [[LinearRegression.md]] to optimize the function by selecting a set of parameters $\theta$ and then repeatedly finding the direction that results in the fastest movement towards a cost function's value nearest to 0. This will find a local optimum. With linear regression however there will not be local optimum but only global.
+
+General idea is to start with some $\theta$ (parameters) and keep changing it to reduce J($\theta$). (Find J in [[LinearRegression.md]])
+
+More specifically, you pick a starting point, see what direction you should go to get closer to 0 the fastest. You then repeat this algorithm. It's not perfect, but it's fast.
+
+This is the algorithm used for [[LinearRegression.md]] to minimize the cost function (also defined in linear regression).
+
+For simple (imperfect) implementation of gradient descent with third degree polynomials see [[GradientDescentCode.md]].
diff --git a/GradientDescentCode.md b/GradientDescentCode.md
@@ -0,0 +1,56 @@
+:code:
+# Gradient Descent Implementation
+
+This approach works for 3rd degree polynomials. It is imperfect, but works and illustrates the ideas behind gradient descent.
+
+## Code
+```python
+import random
+print("ax^3 + bx^2 + cx + d")
+
+a = float(input("Enter a: "))
+b = float(input("Enter b: "))
+c = float(input("Enter c: "))
+d = float(input("Enter d: "))
+
+
+def calculateValue(x):
+ return x**3 * a + x**2 * b + x * c + d
+
+
+def printResult(x, y):
+ print("x: " + str(x) + " y: " + str(y))
+
+
+# Find the x at the bottom
+def descend(x, depth):
+
+ if depth > 100:
+ return x
+
+ midX = x
+ rightX = x + .00000001
+ leftX = x - .00000001
+
+ midY = calculateValue(midX)
+ rightY = calculateValue(rightX)
+ leftY = calculateValue(leftX)
+
+ printResult(leftX, leftY)
+ printResult(midX, midY)
+ printResult(rightX, rightY)
+
+ depth += 1
+ if rightY > leftY:
+ return descend(x - (1 / depth), depth)
+ else:
+ return descend(x + (1/depth), depth)
+
+
+currSearch = random.random() * 10
+xVal = descend(currSearch, 0)
+
+printResult(xVal, calculateValue(xVal))
+
+
+```
diff --git a/LinearRegression.md b/LinearRegression.md
@@ -0,0 +1,11 @@
+:ml:
+# Linear Regression
+
+ML L2 - Also referred to as ordinary least squares
+
+## Notes
+
+**Definition:** Fitting a straight line to data which allows for arbitrary inputs in the valid domain but not necessarily in the training set, to get accurate outputs.
+
+The goal is to find a $\theta$ (parameters) that minimizes $J(\theta)=\frac{1}{2}\sum_{i=1}^{m}(h(x_i) - y_i)^2$. This is called the cost function.
+
diff --git a/MachineLearning.md b/MachineLearning.md
@@ -10,6 +10,22 @@ Links to ML Notes
1. How do I create new ML models
2. Create chess ML program
+## Good Info
+
+x = vector of inputs. Known as features
+
+y = output also known as target variable
+
+(x,y) = Training example
+
+m = Number of training examples
+
+n = # of features (size of inputs/features/x)
+
+h(x) = this is the function with an input of x this should be about the correct y.
+
+
+
## Main Links
ML Categories:
@@ -21,6 +37,8 @@ ML Categories:
Concepts:
[[RegressionProblem.md]]
+[[LinearRegression.md]]
+[[GradientDescent.md]]
[[ClassificationProblem.md]]
[[SupportVectorMachine.md]]
[[ClusteringAlgorithms.md]]
@@ -32,4 +50,3 @@ To do:
[[Kernels.md]]
[[Backpropagation.md]]
[[EigenVector.md]]
-[[GradientDescent.md]]
diff --git a/RegressionProblem.md b/RegressionProblem.md
@@ -8,3 +8,5 @@ ML L1
**Definition:** A regression problem is a problem where the value trying to be predicted is continuous (think graphing not yes/no).
Yes/no problem is a [[ClassificationProblem.md]].
+
+Also see for a more specific example [[LinearRegression.md]]
diff --git a/SupervisedLearning.md b/SupervisedLearning.md
@@ -8,3 +8,5 @@ ML L1
**Definition:** Training a model by giving it inputs and valid associated outputs.
Most widely used form of model training.
+
+
