notes

commit faf24c4de83aa05b44fd4cafe2a1c048fd424a68
parent 582fc1030ad7b8d472428d3bbd00e6fca62ca5b5
Author: Andrew <andrewlaack1@gmail.com>
Date:   Thu, 18 Apr 2024 18:27:42 -0500

Updated gradient descent code and added information about learning rate

Diffstat:
M
GradientDescent.md
 | 
4
+++-
M
GradientDescentCode.md
 | 
41
++++++++++++++++++++++-------------------
A
LearningRate.md
 | 
11
+++++++++++

3 files changed, 36 insertions(+), 20 deletions(-)
diff --git a/GradientDescent.md b/GradientDescent.md
@@ -13,4 +13,6 @@ More specifically, you pick a starting point, see what direction you should go t
 
 This is the algorithm used for [[LinearRegression.md]] to minimize the cost function (also defined in linear regression). 
 
-For simple (imperfect) implementation of gradient descent with third degree polynomials see [[GradientDescentCode.md]].
+For a simple implementation of gradient descent using a [[LearningRate.md]] for third degree polynomials see [[GradientDescentCode.md]].
+
+
diff --git a/GradientDescentCode.md b/GradientDescentCode.md
@@ -1,17 +1,23 @@
 :code:
 # Gradient Descent Implementation 
 
-This approach works for 3rd degree polynomials. It is imperfect, but works and illustrates the ideas behind gradient descent.
+This approach implements a [[LearningRate.md]] parameter to narrow in upon a local minimum of the given third degree polynomial.
 
 ## Code
 ```python
+import sys
 import random
-print("ax^3 + bx^2 + cx + d")
 
+RECURSION_LIMIT = 1500
+
+
+sys.setrecursionlimit(RECURSION_LIMIT)
+print("ax^3 + bx^2 + cx + d")
 a = float(input("Enter a: "))
 b = float(input("Enter b: "))
 c = float(input("Enter c: "))
 d = float(input("Enter d: "))
+learningRate = float(input("Learning Rate: "))
 
 
 def calculateValue(x):
@@ -19,38 +25,35 @@ def calculateValue(x):
 
 
 def printResult(x, y):
-    print("x: " + str(x) + " y: " + str(y))
-
-
-# Find the x at the bottom
-def descend(x, depth):
+    print("x: " + str(x) + "\ty: " + str(y))
 
-    if depth > 100:
-        return x
 
-    midX = x
+def limit(x):
     rightX = x + .00000001
     leftX = x - .00000001
 
-    midY = calculateValue(midX)
     rightY = calculateValue(rightX)
     leftY = calculateValue(leftX)
 
-    printResult(leftX, leftY)
-    printResult(midX, midY)
-    printResult(rightX, rightY)
+    return ((rightY - leftY) / .00000002)
+
 
+def descend(x, depth):
+    # Need - 15 because recursion includes other function calls...
+    if depth >= RECURSION_LIMIT - 15:
+        return x
+    lim = limit(x)
+    printResult(x, calculateValue(x))
     depth += 1
-    if rightY > leftY:
-        return descend(x - (1 / depth), depth)
+
+    if lim > 0:
+        return descend(x - learningRate * lim, depth)
     else:
-        return descend(x + (1/depth), depth)
+        return descend(x + learningRate * lim, depth)
 
 
 currSearch = random.random() * 10
 xVal = descend(currSearch, 0)
 
 printResult(xVal, calculateValue(xVal))
-
-
 ```
diff --git a/LearningRate.md b/LearningRate.md
@@ -0,0 +1,11 @@
+:ml:
+# Learning Rate
+
+ML L2
+
+## Notes
+
+**Definition:** The learning rate is a constant used to narrow in upon some value based on it's distance from an expected value. The further away from the value, the larger the change for a parameter(s) will be.
+
+
+See [[GradientDescentCode.md]] and [[GradientDescent.md]] for an example of when a learning rate would be used and an implementation of it.