Backpropagation Details Pt. 1: Optimizing 3 parameters simultaneously.
StatQuest with Josh Starmer
https://www.youtube.com/watch?v=iyn2zdALii8

Transkript (automatisch erstellt)

0:00 The sun is out and it's nice outside. It's the perfect weather for StatQuest. Yeah. Hello, I'm Josh Starmer and welcome to
0:11 StatQuest. Today, we're going to talk about backpropagation details, part one. Note, this StatQuest assumes that you have already watched Neural Networks
0:21 Part 2, Backpropagation Main Ideas. If not, check out the quest. The link is in the description below. In Backpropagation Main Ideas, we had
0:33 this super simple data set that showed whether or not different drug dosages were effective against a virus.
0:42 Then we had this simple neural network that already had optimal values for all of the parameters except for the last bias term, B sub 3.
0:54 Then, using everything in the neural network except for the last bias, B sub 3, we drew this green squiggle.
1:04 Then we demonstrated the main ideas behind backpropagation by optimizing B sub 3. We first used the chain rule to
1:13 calculate the derivative of the sum of the squared residuals with respect to the unknown parameter, which in this case was B sub 3.
1:23 Then we initialized the unknown parameter with a number, and in this case, we set B sub 3 equal to 0. And used gradient descent to optimize
1:33 the unknown parameter. Hooray! We can optimize the last bias term, B sub 3. Now let's pretend we don't know B sub
1:44 3's optimal value and start working our way backwards so that, along with B sub 3, we optimize the last two weights, W W 3 and W sub 4.
1:58 Note, the goal of this quest is to learn how the chain rule and gradient ascent applies to multiple parameters and to introduce some
2:07 fancy notation. In the next part, we'll go completely bonkers with the chain rule and learn how to optimize all seven parameters in
2:17 this neural network simultaneously. Bam! So, let's go back to not knowing the optimal values for W sub 3, W sub 4, and
2:28 B sub 3. And, just like before, we'll assume that the other weights and biases are already optimized.
2:37 The first thing we do is initialize the weights W sub 3 and W sub 4 with random starting values. And, in this example, that means we
2:47 randomly select two values from a standard normal distribution. Then, we initialize the last bias B sub 3 to zero because bias terms frequently
2:59 start at zero. Now, if we run dosages from zero to one through the connection to the top node in the hidden layer,
3:09 then, just like before, we get the corresponding Y axis coordinates and this blue curve. Now, we multiply the Y axis coordinates
3:21 on the blue curve by W sub 3, which starts out with the random value 0.36. And, we get this new blue curve. Now, if we run dosages from zero to one
3:36 through the connection to the bottom node in the hidden layer, then, just like before, we get the corresponding Y axis coordinates
3:46 for this orange curve. Now, we multiply the Y axis coordinates on the orange curve by W sub 4, which starts with the random value 0.63.
3:59 And we get this new orange curve. Now, we add the blue and orange curves together and get this green squiggle.
4:09 Lastly, since the initial value for B sub 3 is 0, adding it to the Y-axis values on the green squiggle does not change anything.
4:20 In other words, given the current parameters for this neural network, some of which are optimal, and some of which are not optimal,
4:30 we end up with this green squiggle. Now, just like before, we can quantify how well the green squiggle fits the data by calculating the sum of the
4:40 squared residuals. And we get the sum of the squared residuals equals 1.4. Now, even though we have not yet
4:49 optimized W sub 3 and W sub 4, we can still plot the sum of the squared residuals with respect to B sub 3. And just like before, if we change B sub
5:02 3, then we will change the sum of the squared residuals. And that means, just like before, we can optimize B sub 3 by finding the
5:12 derivative of the sum of the squared residuals with respect to B sub 3 and plugging the derivative into the gradient descent algorithm to find the
5:21 optimal value for B sub 3. And just like before, because the predicted values in the sum of the squared residuals
5:30 come from the green squiggle, and the green squiggle is the sum of the blue and orange curves plus B sub 3, then the sum of the squared residuals
5:42 are linked to B sub 3 by the predicted values. So, by the chain rule, the derivative of the sum of the squared residuals with
5:52 respect to B sub 3 is the derivative of the sum of the squared residuals with respect to the predicted values
6:00 times the derivative of the predicted values with respect to B sub 3. Note, this is the exact same derivative that we calculated in backpropagation
6:11 main ideas. The point of this is that even though we are now optimizing more than one parameter,
6:19 the derivatives that we have already calculated with respect to the sum of the squared residuals do not change. Bam!
6:29 Now, let's talk about how to calculate the derivatives of the sum of the squared residuals with respect to the weights W sub 3 and W sub 4.
6:39 Unfortunately, before we can do that, we have to introduce some fancy notation. First, let's remember that the I in this summation notation
6:51 is an index for the data in the data set. For example, when I equals 1, we are talking about observed sub 1, which is
7:00 0. And we are talking about predicted sub 1, which is 0.72. However, we can also talk about dosage
7:10 sub I. And when I equals 1, we are talking about dosage sub 1, which is 0. When I equals 2, we're talking about
7:20 dosage sub 2, which is 0.5. And when I equals 3, we're talking about dosage sub 3, which is 1. And because dosage sub I is the input
7:33 value, we call it input sub I. And that means this connection multiplies input sub I by weight W sub 1, which is 3.34.
7:46 And it adds bias sub 1, which is -1.43 to get an x-axis coordinate for the activation function in the top node in the hidden layer.
7:59 Meanwhile, the other connection multiplies input sub I by weight W sub 2, which is -3.53 and adds bias B sub 2, which is 0.57
8:14 to get an x-axis coordinate for the activation function in the bottom node in the hidden layer. So, we have two different x-axis
8:23 coordinates for input sub I. In order to keep track of things, let's call this x-axis coordinate X sub 1, I where the one in 1, I refers to the
8:37 activation function in the top node. And the I in 1, I tells us that it corresponds to input sub I. Likewise, let's call this x-axis
8:50 coordinate X sub 2, I where the two in 2, I refers to the activation function in the bottom node and the I in 2, I tells us that it
9:02 corresponds to input sub I. For example, if I equals 3, then we're talking about the third dosage, dosage sub 3.
9:14 And that means we're talking about input sub 3, which is one, the maximum dosage. And that means the x-axis coordinate for the activation function in the top node,
9:25 X sub 1, 3 is equal to 1.91. And the x-axis coordinate for the activation function in the bottom node,
9:37 X sub 2,3 is equal to -2.96. Bam. If we plugged in all values for I into
9:48 dosage sub I, we get X sub 1,I values in this red box. And X sub 2,I values in this red box. Now, in order to get the Y axis
10:01 coordinates for the activation function in the top node, we plug X sub 1,I into the activation function,
10:10 which, in this example, is the soft plus function, and that gives us Y sub 1,I. Just like before, the one in 1,I tells
10:22 us that we are talking about the activation function in the top node. And the I tells us which dosage we are talking about.
10:31 Likewise, in order to get the Y axis coordinates for the activation function in the bottom node, we plug X sub 2,I into the activation
10:41 function. And that gives us Y sub 2,I. Bam. Now that we understand the
10:52 fancy notation, we can talk about how to calculate the derivatives of the sum of the squared residuals with respect to the weights W
11:01 sub 3 and W sub 4. First, remember that Y sub 1,I represents the Y axis coordinates for the top activation function.
11:12 And they form this initial blue curve. However, we get the final blue curve by multiplying the Y axis coordinates Y sub 1,I by W sub 3.
11:26 And that means we can plug Y sub 1,I * W sub 3 into the equation for the predicted values. Likewise, W sub 4 multiplies the Y-axis
11:39 coordinates Y sub 2, I from the bottom activation function to create the final orange curve. And that means we can plug Y sub 2, I *
11:52 W sub 4 into the equation for the predicted values. Now, since this sum creates the green squiggle
12:01 and the green squiggle gives us predictions that we evaluate with the sum of the squared residuals
12:09 then the sum of the squared residuals are linked to W sub 3 and W sub 4 by the predicted values. That means we can use the chain rule
12:20 to determine the derivative of the sum of the squared residuals with respect to W sub 3 and with respect to W sub 4.
12:30 The chain rule says that the derivative of the sum of the squared residuals with respect to W sub 3 is the derivative of the sum of the
12:39 squared residuals with respect to the predicted values * the derivative of the predicted values with respect to W sub 3.
12:49 Likewise, the derivative with respect to W sub 4 is the derivative of the sum of the squared residuals with respect to the predicted values
12:58 * of the predicted values with respect to W sub 4. Double bam? Not yet. Note, in both cases, the derivative of the sum of the
13:10 squared residuals with respect to the predicted values is the exact same as the derivative used for B sub 3.
13:19 Just to remind you, we start by substituting the sum of the squared residuals with its equation. Then we use the chain rule
13:28 to move the square to the front and then we multiply that by the derivative of the stuff inside the parentheses with respect to the
13:36 predicted values, -1. Lastly, we simplify by multiplying two by -1. And this is the derivative of the sum of
13:47 the squared residuals with respect to the predicted values. So, we just plug it in. Now, to solve for the derivative of the
13:57 predicted values with respect to W sub three, we plug in the equation for the predicted values and the derivative of the first term
14:07 with respect to W sub three is Y sub one {comma} I. And the derivatives of the other terms are both zero since they do not contain
14:16 W sub three. And we end up with just Y sub one {comma} I. So, we multiply the derivative of the
14:25 sum of the squared residuals with respect to the predicted values by Y sub one {comma} I. Likewise, the derivative of the
14:34 predicted values with respect to W sub four is zero for the first term plus Y sub two {comma} I for the second
14:43 term plus zero for the third term which is just Y sub two {comma} I. So, we multiply the derivative of the
14:53 sum of the squared residuals with respect to the predicted values by Y sub two {comma} I. Double bam!
15:03 Now that we have the derivatives of the sum of the squared residuals with respect to W sub three, W sub four,
15:11 and B sub three, we can plug them into gradient descent to optimize W sub three, W sub four, and B sub three.
15:22 First, we initialize W sub 3 and W sub 4 with random values and set B sub 3 equal to 0. Now, starting with the derivative of the
15:34 sum of the squared residuals with respect to W sub 3, first, we expand the summation. Then we plug in the observed values
15:46 and plug in the predicted values from the green squiggle. Remember, we get the predicted values on the green squiggle by running the
15:54 dosages through the neural network. Now we plug in the Y axis coordinates for the activation function in the top node, Y sub 1, I.
16:06 Lastly, we do the math and get 2.58. Likewise, we calculate the derivative of the sum of the squared residuals with respect to W sub 4
16:18 and with respect to B sub 3. Now we use the derivatives to calculate the new values for W sub 3,
16:29 W sub 4, and B sub 3.
16:43 Now we repeat that process until the predictions no longer improve very much or we reach a maximum number of steps or we meet some other criteria.
16:55 Now let's check out a fancy animation that shows the gradient descent in action. These gray dots represent the data that
17:04 we are using to train the neural network. And the orange and blue curves represent the orange and blue curves.
17:13 And the green squiggle represents the sum of the orange and blue curves plus B sub 3. Now, watch how the green squiggle fits
17:22 the data after 175 steps in gradient descent. Bam! So, after a bunch of steps, we see how gradient descent optimizes the
17:34 parameters. Triple bam! In the next StatQuest, we'll go totally bonkers with the chain rule and show how
17:43 to optimize all of the parameters in a neural network simultaneously. Now, it's time for some shameless self-promotion.
17:54 If you want to review statistics and machine learning offline, check out the StatQuest Study Guides at statquest.org. There's something for everyone.
18:04 Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to
18:14 support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie,
18:24 or just donate. The links are in the description below. All right. Until next time, quest on.