Das Video kommt von YouTube: erst beim Abspielen verbindet sich die Seite mit YouTube (Google).
Backpropagation Details Pt. 1: Optimizing 3 parameters simultaneously.
Das Wichtigste aus dem Video
Tipp auf eine Zeit – das Video springt genau dorthin.
Transkriptautomatisch erstellt · 105 Zeilen
- The sun is out and it's nice outside. It's the perfect weather for StatQuest. Yeah. Hello, I'm Josh Starmer and welcome to
- StatQuest. Today, we're going to talk about backpropagation details, part one. Note, this StatQuest assumes that you have already watched Neural Networks
- Part 2, Backpropagation Main Ideas. If not, check out the quest. The link is in the description below. In Backpropagation Main Ideas, we had
- this super simple data set that showed whether or not different drug dosages were effective against a virus.
- Then we had this simple neural network that already had optimal values for all of the parameters except for the last bias term, B sub 3.
- Then, using everything in the neural network except for the last bias, B sub 3, we drew this green squiggle.
- Then we demonstrated the main ideas behind backpropagation by optimizing B sub 3. We first used the chain rule to
- calculate the derivative of the sum of the squared residuals with respect to the unknown parameter, which in this case was B sub 3.
- Then we initialized the unknown parameter with a number, and in this case, we set B sub 3 equal to 0. And used gradient descent to optimize
- the unknown parameter. Hooray! We can optimize the last bias term, B sub 3. Now let's pretend we don't know B sub
- 3's optimal value and start working our way backwards so that, along with B sub 3, we optimize the last two weights, W W 3 and W sub 4.
- Note, the goal of this quest is to learn how the chain rule and gradient ascent applies to multiple parameters and to introduce some
- fancy notation. In the next part, we'll go completely bonkers with the chain rule and learn how to optimize all seven parameters in
- this neural network simultaneously. Bam! So, let's go back to not knowing the optimal values for W sub 3, W sub 4, and
- B sub 3. And, just like before, we'll assume that the other weights and biases are already optimized.
- The first thing we do is initialize the weights W sub 3 and W sub 4 with random starting values. And, in this example, that means we
- randomly select two values from a standard normal distribution. Then, we initialize the last bias B sub 3 to zero because bias terms frequently
- start at zero. Now, if we run dosages from zero to one through the connection to the top node in the hidden layer,
- then, just like before, we get the corresponding Y axis coordinates and this blue curve. Now, we multiply the Y axis coordinates
- on the blue curve by W sub 3, which starts out with the random value 0.36. And, we get this new blue curve. Now, if we run dosages from zero to one
- through the connection to the bottom node in the hidden layer, then, just like before, we get the corresponding Y axis coordinates
- for this orange curve. Now, we multiply the Y axis coordinates on the orange curve by W sub 4, which starts with the random value 0.63.
- And we get this new orange curve. Now, we add the blue and orange curves together and get this green squiggle.
- Lastly, since the initial value for B sub 3 is 0, adding it to the Y-axis values on the green squiggle does not change anything.
- In other words, given the current parameters for this neural network, some of which are optimal, and some of which are not optimal,
- we end up with this green squiggle. Now, just like before, we can quantify how well the green squiggle fits the data by calculating the sum of the
- squared residuals. And we get the sum of the squared residuals equals 1.4. Now, even though we have not yet
- optimized W sub 3 and W sub 4, we can still plot the sum of the squared residuals with respect to B sub 3. And just like before, if we change B sub
- 3, then we will change the sum of the squared residuals. And that means, just like before, we can optimize B sub 3 by finding the
- derivative of the sum of the squared residuals with respect to B sub 3 and plugging the derivative into the gradient descent algorithm to find the
- optimal value for B sub 3. And just like before, because the predicted values in the sum of the squared residuals
- come from the green squiggle, and the green squiggle is the sum of the blue and orange curves plus B sub 3, then the sum of the squared residuals
- are linked to B sub 3 by the predicted values. So, by the chain rule, the derivative of the sum of the squared residuals with
- respect to B sub 3 is the derivative of the sum of the squared residuals with respect to the predicted values
- times the derivative of the predicted values with respect to B sub 3. Note, this is the exact same derivative that we calculated in backpropagation
- main ideas. The point of this is that even though we are now optimizing more than one parameter,
- the derivatives that we have already calculated with respect to the sum of the squared residuals do not change. Bam!
- Now, let's talk about how to calculate the derivatives of the sum of the squared residuals with respect to the weights W sub 3 and W sub 4.
- Unfortunately, before we can do that, we have to introduce some fancy notation. First, let's remember that the I in this summation notation
- is an index for the data in the data set. For example, when I equals 1, we are talking about observed sub 1, which is
- 0. And we are talking about predicted sub 1, which is 0.72. However, we can also talk about dosage
- sub I. And when I equals 1, we are talking about dosage sub 1, which is 0. When I equals 2, we're talking about
- dosage sub 2, which is 0.5. And when I equals 3, we're talking about dosage sub 3, which is 1. And because dosage sub I is the input
- value, we call it input sub I. And that means this connection multiplies input sub I by weight W sub 1, which is 3.34.
- And it adds bias sub 1, which is -1.43 to get an x-axis coordinate for the activation function in the top node in the hidden layer.
- Meanwhile, the other connection multiplies input sub I by weight W sub 2, which is -3.53 and adds bias B sub 2, which is 0.57
- to get an x-axis coordinate for the activation function in the bottom node in the hidden layer. So, we have two different x-axis
- coordinates for input sub I. In order to keep track of things, let's call this x-axis coordinate X sub 1, I where the one in 1, I refers to the
- activation function in the top node. And the I in 1, I tells us that it corresponds to input sub I. Likewise, let's call this x-axis
- coordinate X sub 2, I where the two in 2, I refers to the activation function in the bottom node and the I in 2, I tells us that it
- corresponds to input sub I. For example, if I equals 3, then we're talking about the third dosage, dosage sub 3.
- And that means we're talking about input sub 3, which is one, the maximum dosage. And that means the x-axis coordinate for the activation function in the top node,
- X sub 1, 3 is equal to 1.91. And the x-axis coordinate for the activation function in the bottom node,
- X sub 2,3 is equal to -2.96. Bam. If we plugged in all values for I into
- dosage sub I, we get X sub 1,I values in this red box. And X sub 2,I values in this red box. Now, in order to get the Y axis
- coordinates for the activation function in the top node, we plug X sub 1,I into the activation function,
- which, in this example, is the soft plus function, and that gives us Y sub 1,I. Just like before, the one in 1,I tells
- us that we are talking about the activation function in the top node. And the I tells us which dosage we are talking about.
- Likewise, in order to get the Y axis coordinates for the activation function in the bottom node, we plug X sub 2,I into the activation
- function. And that gives us Y sub 2,I. Bam. Now that we understand the
- fancy notation, we can talk about how to calculate the derivatives of the sum of the squared residuals with respect to the weights W
- sub 3 and W sub 4. First, remember that Y sub 1,I represents the Y axis coordinates for the top activation function.
- And they form this initial blue curve. However, we get the final blue curve by multiplying the Y axis coordinates Y sub 1,I by W sub 3.
- And that means we can plug Y sub 1,I * W sub 3 into the equation for the predicted values. Likewise, W sub 4 multiplies the Y-axis
- coordinates Y sub 2, I from the bottom activation function to create the final orange curve. And that means we can plug Y sub 2, I *
- W sub 4 into the equation for the predicted values. Now, since this sum creates the green squiggle
- and the green squiggle gives us predictions that we evaluate with the sum of the squared residuals
- then the sum of the squared residuals are linked to W sub 3 and W sub 4 by the predicted values. That means we can use the chain rule
- to determine the derivative of the sum of the squared residuals with respect to W sub 3 and with respect to W sub 4.
- The chain rule says that the derivative of the sum of the squared residuals with respect to W sub 3 is the derivative of the sum of the
- squared residuals with respect to the predicted values * the derivative of the predicted values with respect to W sub 3.
- Likewise, the derivative with respect to W sub 4 is the derivative of the sum of the squared residuals with respect to the predicted values
- * of the predicted values with respect to W sub 4. Double bam? Not yet. Note, in both cases, the derivative of the sum of the
- squared residuals with respect to the predicted values is the exact same as the derivative used for B sub 3.
- Just to remind you, we start by substituting the sum of the squared residuals with its equation. Then we use the chain rule
- to move the square to the front and then we multiply that by the derivative of the stuff inside the parentheses with respect to the
- predicted values, -1. Lastly, we simplify by multiplying two by -1. And this is the derivative of the sum of
- the squared residuals with respect to the predicted values. So, we just plug it in. Now, to solve for the derivative of the
- predicted values with respect to W sub three, we plug in the equation for the predicted values and the derivative of the first term
- with respect to W sub three is Y sub one {comma} I. And the derivatives of the other terms are both zero since they do not contain
- W sub three. And we end up with just Y sub one {comma} I. So, we multiply the derivative of the
- sum of the squared residuals with respect to the predicted values by Y sub one {comma} I. Likewise, the derivative of the
- predicted values with respect to W sub four is zero for the first term plus Y sub two {comma} I for the second
- term plus zero for the third term which is just Y sub two {comma} I. So, we multiply the derivative of the
- sum of the squared residuals with respect to the predicted values by Y sub two {comma} I. Double bam!
- Now that we have the derivatives of the sum of the squared residuals with respect to W sub three, W sub four,
- and B sub three, we can plug them into gradient descent to optimize W sub three, W sub four, and B sub three.
- First, we initialize W sub 3 and W sub 4 with random values and set B sub 3 equal to 0. Now, starting with the derivative of the
- sum of the squared residuals with respect to W sub 3, first, we expand the summation. Then we plug in the observed values
- and plug in the predicted values from the green squiggle. Remember, we get the predicted values on the green squiggle by running the
- dosages through the neural network. Now we plug in the Y axis coordinates for the activation function in the top node, Y sub 1, I.
- Lastly, we do the math and get 2.58. Likewise, we calculate the derivative of the sum of the squared residuals with respect to W sub 4
- and with respect to B sub 3. Now we use the derivatives to calculate the new values for W sub 3,
- W sub 4, and B sub 3.
- Now we repeat that process until the predictions no longer improve very much or we reach a maximum number of steps or we meet some other criteria.
- Now let's check out a fancy animation that shows the gradient descent in action. These gray dots represent the data that
- we are using to train the neural network. And the orange and blue curves represent the orange and blue curves.
- And the green squiggle represents the sum of the orange and blue curves plus B sub 3. Now, watch how the green squiggle fits
- the data after 175 steps in gradient descent. Bam! So, after a bunch of steps, we see how gradient descent optimizes the
- parameters. Triple bam! In the next StatQuest, we'll go totally bonkers with the chain rule and show how
- to optimize all of the parameters in a neural network simultaneously. Now, it's time for some shameless self-promotion.
- If you want to review statistics and machine learning offline, check out the StatQuest Study Guides at statquest.org. There's something for everyone.
- Hooray! We've made it to the end of another exciting StatQuest. If you like this StatQuest and want to see more, please subscribe. And if you want to
- support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying one or two of my original songs or a t-shirt or a hoodie,
- or just donate. The links are in the description below. All right. Until next time, quest on.
Zum Nachlesen
Data ScienceDer Begriff Data Science (dt. etwa Datenwissenschaft) bezeichnet die Wissenschaft von der Extraktion von Wissen aus Daten, um daraus zu lernen.
BackpropagationFehlerrückführung bzw. Rückpropagierung ist ein verbreitetes Verfahren zum Einlernen künstlicher neuronaler Netze. Es gehört in der einfachen Form zur Gruppe …