Backpropagation with Andrej Karpathy: Part 1 | by Kavishka Abeywardana | Feb, 2024

Disclaimer: This generally is a summary of Andrej Karpathy’s The spelled-out Intro to Neural Networks and Backpropagation: Establishing Micrograd video. Go to: https://www.youtube.com/watch?v=VMj-3S1tku0&t=612s

Backpropagation is far further regular than the neural networks. It’s a mathematical machine.

First, we’ve got to rearrange our workspace.

import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Let’s define a simple function and try to plot it.

def f(x):
return 3*x**2-4*x+5
xs = np.arange(-5,5,0.25)
ys = f(xs)
plt.plot(xs,ys)

We want to calculate the spinoff of the function at each degree. On this occasion, we’re in a position to explicitly calculate the spinoff of the function. Nonetheless, this methodology simply isn’t environment friendly with regards to neural networks. What’s a by-product?

We give a slight bump to the variable x and look at how the function responds to it. By-product reveals the sensitivity of a function to a change in its variables.

We’re going to do a small numerical approximation of the spinoff.

h=0.01
x=3.0
(f(x+h)-f(x))/h

The reply is 14.029999999999632. If we take smaller and smaller h values, we’re in a position to converge to the true gradient (14). Nonetheless, while you use a too-small amount you’ll get an error as a result of floating degree limitations.

At an optimum degree, the sensitivity is zero! The function won’t switch in any path on account of altering the enter.

Suppose we’ve acquired a variety of enter variables.

#further difficult occasion
a = 2.0
b = -3.0
c = 10.0
d = a*b + c
print(d)

We want to calculate the spinoff of the function wrt a.

h = 0.001
d1 = a*b + c
a +=h
d2 = a*b + c
print('d1 = ',d1)
print('d2 = ',d2)
print('spinoff="(d2-d1)/h)

Since b is a minus amount, from d1 to d2, the price of f is decreased. Thus, the gradient/spinoff is unfavorable. We are going to try to do this to b and c.

To deploy a neural neighborhood, we’d like separate data constructions. Let’s develop a simple data development to deal with the micrograd.

class Value:
def __init__(self,data):
self.data = data
def __repr__(self):
return f"Value(data={self.data})"
def __add__(self,totally different):
sum = Value(self.data+totally different.data)
return sum
def __mul__(self,totally different):
out = Value(self.data*totally different.data)
return out

This class merely wraps the price and creates an object. Python makes use of __repr__ function to internally signify the merchandise. With out __repr__, the print function will print out the state of affairs of the merchandise.

We define addition using __add__. Every time Python sees a+b, it internally executes a.__add__(b). Multiplication moreover could also be outlined within the similar methodology.

Now we’re in a position to execute the following code.

a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
d = a*b+c

Now we’ve got to create the connective tissue of the system. We now have to retailer the operation graph development. We are going to in all probability be enhancing the above code.

class Value:
def __init__(self,data,_children=(),_op="',label=""):
self.data = data
self._prev = set(_children)
self._op =_op
self.label = label
def __repr__(self):
return f"Value(data={self.data})"
def __add__(self,totally different):
sum = Value(self.data+totally different.data,(self,totally different),'+')
return sum
def __mul__(self,totally different):
out = Value(self.data*totally different.data,(self,totally different),'*')
return out

When initializing objects, the _children tuple is empty. Nonetheless, when together with or multiplying, the sooner parts are added to prev set. Moreover, we’ve got to know how a worth was created. Thus, we introduce the op attribute.

Now we’ve got to visualise the development. Thus, we introduce the following code.

from graphviz import Digraph
def trace(root):
nodes,edges = set(),set()
def assemble(v):
if v not in nodes:
nodes.add(v)
for teen in v._prev:
edges.add((teen,v))
assemble(teen)
assemble(root)
return nodes,edges
def dot_draw(root):
dot = Digraph(format="svg", graph_attr={'rankdir':'LR'})
nodes,edges = trace(root)
for n in nodes:
uid = str(id(n))
dot.node(title=uid, label=" data %.4f"%(n.label,n.data),type = 'report')
if n._op:
dot.node(title = uid+n._op,label=n._op)
dot.edge(uid+n._op,uid)
for n1,n2 in edges:
dot.edge(str(id(n1)),str(id(n2))+n2._op)
return dot
a = Value(2.,label="a")
b = Value(-3.0,label="b")
c = Value(10.0,label="c")
e = a*b; e.label="e"
d = e+c; d.label="d"
draw_dot(d)

We are going to make this one layer deeper.

f = Value(-2.0,label="f")
L = d*f; L.label="L"
dot_draw(L)

Now we’re in a position to do a forward transfer and do a mathematical operation. Now we need to start from the highest and calculate the gradients of the output using backpropagation. In a neural neighborhood, L is a loss function and some of the leaf nodes are the weights of the online.

We’re going to extra develop the Value class to calculate the gradients. Initially, we assume that the gradient is zero.

We are going to manually calculate some derivatives.

L.grad = 1.0
f.grad = d.data
d.grad = f.data
dot_draw(L)

We now have to calculate dL/dc. The change of c impacts L by the use of d.

We are going to use the chain rule. The chain rule could possibly be very intuitive. Underneath is a cool clarification given by George F. Simmons.

“If a automotive travels twice as fast as a bicycle and the bicycle is 4 situations as fast as a strolling man, then the automotive travels 2 × 4 = 8 situations as fast because the individual.”

If z will rely upon y and y will rely upon x, z will rely upon x. Thus, we’re in a position to write the fees of changes.

Now we’re ready to make use of the chain rule.

As a result of the native spinoff of an addition is 1, we’re in a position to say that addition merely routes the sooner spinoff to the youngsters nodes.

We always use native information (native derivatives) and some earlier data to calculate the worldwide spinoff.

Thus, we’re in a position to write,

L.grad = 1.0
f.grad = d.data
d.grad = f.data
c.grad = e.grad = d.grad
a.grad = e.grad*b.data
b.gard = e.grad*a.data

In backpropagation, we recursively apply the chain rule backward by the use of the computational graph.

If we nudge the variables inside the path of their gradients, it’s going to enhance L. That’s the concept of optimization. Let’s try.

Now L=8.0000. Let’s change the enter variables, inside the path of their gradient.

a.data += 0.01*a.grad
b.data += 0.01*b.grad
c.data += 0.01*c.grad
f.data += 0.01*f.grad

After working the forward transfer, L=-7.6000. Proper right here, 0.01 is the step dimension.

A neuron is a extremely tough object. Nonetheless, we use a fairly easy model in neural networks.

The synapses have weights (synaptic strengths) on them (w). Inputs are multiplied by the weights and fed to the neuron. Internally they’re added. There’s a further bias (b). Then the consequence’s handed by the use of the squeezing function f. It could be ReLu or tanh.

#enter x1,x2
x1 = Value(2.0,label="x1")
x2 = Value(0.0,label="x2")
#weights
w1 = Value(-3.0,label="w1")
w2 = Value(1.0,label="w2")
#bias the neuron
b = Value(6.7,label="b")
x1w1 = x1*w1; x1w1.label="x1w1"
x2w2 = x2*w2; x2w2.label="x2w2"
x1w1x2w2 = x1w1+x2w2; x1w1x2w2.label="x1w1x2w2"
n = x1w1x2w2+b; n.label="n"

We now have to implement the tanh function. We are going to implement arbitrary difficult options if everyone knows the native spinoff. No need to interrupt it down into atomic gadgets. We’re going to implement the tanh function beneath the Value class.

 def tanh(self):
x = self.data
t = (math.exp(2*x)-1)/(math.exp(2*x)+1)
return Value(t,(self,),'tanh ')

The above code must be added to the Value class. Now if we’re in a position to run the following code.

o = n.tanh()
dot_draw(o);o.label="o"

Now we’ve acquired carried out a whole neuron.

Now when backpropagating, we’ve got to find the native spinoff of the tanh function.

Thus, we’re in a position to write,

o.grad = 1.0
n.grad = 1 - o.data**2
#addition merely distributes the spinoff
b.grad = n.grad
x1w1x2w2.grad = n.grad
x1w1.grad = x1w1x2w2.grad
x2w2.grad = x1w1x2w2.grad
#we solely need the derivatives of the weights
w1.grad = x1w1.grad * x1.data
w2.grad = x2w2.grad * x2.data