What Is Reinforcement Learning?
MATLAB
https://www.youtube.com/watch?v=pc-H4vyg2L4

Transkript (automatisch erstellt)

0:00 artificial intelligence machine learning and deep neural networks these are terms that can spark your imagination of a future where robots are thinking and
0:09 evolving creatures in this video we're gonna look at reinforcement learning or RL as I'll sometimes abbreviate it it's a type of machine learning that has the
0:18 potential to solve some really hard control problems you may have heard that the AI company deepmind created a program called alphago it's an AI that
0:26 uses reinforcement learning to beat the world's best go players and then they recently created alpha star which is poised to dominate the Starcraft 2 scene
0:36 so naturally you might be thinking well if it can do that why can't I use reinforcement learning to control my robot or cool my data center or
0:45 stabilize a drone in a highly dynamic and turbulent flow well let's talk about that now a quick warning before we begin this is not intended to be an exhaustive
0:55 look at reinforcement learning I wouldn't be qualified to explain it all to you anyway instead I want to introduce this topic from the point of
1:03 view of a traditionally trained controls engineer and I hope to show you that there's actually a lot of overlap with control theory by the end of this series
1:11 I think you'll be better prepared to answer questions like what is reinforcement learning and why should I consider it when solving my control
1:18 problem how do I set up and solve the reinforcement learning problem and what are some of the benefits and drawbacks of reinforcement learning compared to a
1:26 traditional controls approach alright so with the scope of this series Set let's get to it I'm Brian and welcome to a MATLAB Tech
1:35 Talk let's start by thinking about the complexity of building a walking robot from the perspective of a traditional controls approach we might use cameras
1:44 to view the environment and then extract image features that can be converted into signals like position or locations of obstacles we could combine those
1:52 observations with other sensors that complete the state estimation which we then use along with a model of the plant and environment to design the control
2:00 system and more than likely this would consist of multiple control loops that all interact with each other for example there would be low-level motor
2:08 controllers and high-level controllers that are managing the leg trajectories or the robot trunk trajectory and maybe a
2:14 higher level controller that is managing the balance or off nominal behavior and everything has to work together in an uncertain environment to generate this
2:23 complex movement of walking which can be really challenging instead of all this complexity let's squeeze it down into a single black box that simply takes in
2:34 observations and outputs the low-level motor commands directly if we were infinitely smart we could sit down and design a function that could get a robot
2:43 to walk without concerning ourselves with all of the internal steps along the way but since we're not that's where machine learning comes in
2:51 broadly speaking machine learning can be subdivided into three categories unsupervised learning supervised learning and reinforcement learning
3:00 unsupervised learning is used to find patterns or hidden structures and datasets that have not been categorized or labeled for example imagine you
3:08 collected information on a hundred thousand animals like a bunch of physical attributes and social tendencies then you can use unsupervised
3:15 learning to group the animals or cluster them into similar features this can be something obvious like grouping them into mammals and birds or to group them
3:24 by patterns that might not be as obvious like finding correlations between physical traits and social behaviors that you didn't know about ahead of time
3:31 supervised learning on the other hand is different in a subtle way in that we train the computer to apply a label to a given input for example let's say that
3:40 one of the columns of our data set of animal features is the species we can then treat species as the label and the rest of the data as inputs into a
3:49 mathematical model then we can use supervised learning to train our model to correctly label each set of animal features by inputting them in one at a
3:57 time letting the model guess the species and then systematically tweaking the model based on whether that guess was correct
4:02 or not and if we had enough training data to get a reliable model we could then send through the input features for a new animal one that we don't have
4:11 labeled and our trained model would apply the most probable species label to it and supervised learning is probably the type of machine learning that most
4:19 people are familiar with because it's what allows computers to recognize pictures of cats or your friends and photos
4:26 and fundamentally applying a label to an image is exactly the same problem as applying a label to a data set of animal features we input a bunch of training
4:33 images into the model and then we tweak the model based on whether it guessed correctly or not until it's accurate the difference is that the input data for an
4:41 image is just a stream of numbers representing pixel intensities so it's not as straightforward to understand how that relates to a cat this is what deep
4:50 learning is good at by representing the model as a deep neural network we have an efficient way to input thousands of numbers and then tweak it during
4:58 training so that it can identify features within an array of pixel intensities that will ultimately allow it to apply the correct label
5:06 reinforcement learning is a different beast altogether unlike the other two learning frameworks which work with a static data set
5:13 RL works with a dynamic environment and the goal is not to cluster data or label data but to find the best sequence of actions that will generate the optimal
5:22 outcome optimal in this sense means to collect the most reward it does this by allowing a piece of software called an agent to explore interact with and learn
5:34 from the environment the agent can take an action which affects the environment changing its state and the environment then produces a reward for that action
5:42 and using this information the agent can adjust which action to take in the future it can learn from this process and although you're presumably not a
5:52 piece of software you learn in essentially the same way a software agent learns with the reinforcement learning framework you can be thought of
5:59 as an agent and the world around you as the environment that you can interact with observe its state and collect rewards you get rewarded by the
6:07 environment by taking actions that are good like you went to college action and you got a job State and that job pays well reward or
6:15 you looked both ways before crossing the street action you got to the other side State and you didn't get run over in the
6:21 process reward alternatively you get low or negative rewards for taking actions that are bad like you stayed up late before an exam action you're tired State
6:32 and you received a poor grade reward within the agent there is a brain that takes in state observations the inputs and maps them to actions the outputs and
6:44 in RL nomenclature this mapping is called the policy given a set of observations the policy decides which action to take and just like with
6:54 supervised learning we can represent the policy as a deep neural network which we'll see later allows our agent to input thousands of states at once and
7:02 still be able to come up with a meaningful action this is where the term deep reinforcement learning comes from in a walking robot example the
7:10 observations might be the state of every joint and the thousands of pixels from a camera sensor the policy would take in all of these observations and output the
7:18 actuator commands and if the robot stays upright and continues walking the environment would generate a reward telling the agent exactly how well that
7:27 very specific combination of actuator commands did of course the policy might not be mapped correctly to take the best actions or the environment might be
7:35 slowly changing and so the mapping is no longer optimal and this is where reinforcement learning algorithms come in they changed the policy based on the
7:44 actions that were taken the observations from the environment and the amount of reward collected in this way the goal of the overall agent is to use
7:52 reinforcement learning algorithms to modify its policy as it interacts with the environment so that eventually given any state it will always take the most
8:00 advantageous action the one that will produce the most reward in the long run for example if you were tired for your exam and you received a bad grade well
8:09 you learn from it and you adjust your policy so that you won't stay up late before the next exam now at its heart reinforcement learning is an
8:18 optimization problem but there are some very interesting concepts that set reinforcement learning apart from other optimization techniques first is the
8:26 idea of value reward is the instantaneous benefit of being in a specific state whereas value is the total reward that an agent can expect to
8:36 collect from that state and onwards into the future assessing the value of a state rather than assessing the reward helps the agent choose the action that
8:45 will collect the most reward over time rather than a short-term benefit for example imagine our agent is in this situation and is trying to collect the
8:54 most reward within three steps if the agent looks directly at the reward for each action then it will step left first to get a higher reward and then right
9:03 and then left again to ultimately collect +1 however if the agent is able to estimate the value of a state then it will see that going right has a higher
9:13 value than going left and will ultimately end up with +8 reward of course often the promise of a high reward in the future still might not
9:22 mean that the action is the best and there's at least two good reasons for this one like with the financial market money in your pocket now can be better
9:31 than a little more money in your pocket a year from now and to your prediction of rewards further into the future become less reliable and therefore that
9:40 high reward might not be there by the time you reach it in both of these cases it's more advantageous to be a little more short-sighted when estimating value
9:50 and in RL we can control this by discounting rewards by a larger amount the further they are in the future another critical aspect of reinforcement
9:59 learning is the trade-off between exploration and exploitation when interacting with the environment this is the trade-off between collecting the
10:07 most rewards that you already know about versus exploring areas of the environment that you haven't visited yet for example let's say the agent only
10:15 knows about the two rewards immediately adjacent to it if it took the greedy approach by exploiting the environment it would only go after the highest
10:22 reward it knows about and so it would go left to collect the +1 however if we occasionally let the agent explore the state space even at the risk of
10:31 collecting fewer rewards it can fill out more of its value function and it opens up the possibility of finding higher rewards that it didn't know about and
10:39 this is part of our normal learning process as humans as well a simple example is deciding on which restaurant you want to eat at do you choose a
10:47 restaurant you know you like and therefore exploit your knowledge or do you venture out and explore a restaurant you've never been to before increasing
10:54 your knowledge now trying a new restaurant gives you the opportunity to find a new favorite place but it also increases your chances of getting a meal
11:02 that you don't like it's tricky to settle on the perfect balance between exploring and exploiting however at the very least our L
11:10 algorithms provide a simple way to set that balance okay it's starting to feel like our L has a completely different goal than what control engineers are
11:18 trying to do when we design control systems but it is pretty much exactly the same problem we're trying to figure out how to design the controller or the
11:27 policy that map's the observed state of the plant or the environment to the best actuator commands the actions and when we design a controller we're basically
11:36 doing a one-time policy update and one of the ways that we can design an optimal controller is by minimizing a cost function like we do with lqr and
11:45 cost is just the negative of reward so by maximizing the reward we're solving the same problem as minimizing cost the difference is that with
11:54 reinforcement learning the computer tries to learn the optimal behavior over time rather than have the designer solved for it explicitly it's like the
12:02 adjustment mechanism in an adaptive controller where it's tweaking the parameters at each sample time in this way we can essentially design a
12:09 controller which is a policy without knowing anything about the system itself and without having to solve any of the traditional control problems we just let
12:18 the computer learn the right parameters on its own through a process that you can think of as fancy trial and error now even with the learning algorithm
12:27 doing most of the work for us we can't enter this process completely ignorant we have to know several things before we start and the first is that we need to
12:35 understand our system that we're trying to control and determine whether it's better to solve the problem with traditional control techniques or with
12:42 reinforcement learning and if we choose the learning path then we need to set up the policy so that it has enough parameters and in the right structure so
12:49 that it can be tweaked successfully it won't do us any good if we're hoping to control a multi-dimensional system but only give it a single parameter we also
12:58 need to know what a successful result would be and reward the controller for doing well this requires crafting a reward function so that the learning
13:06 algorithm understands when it's getting better and ultimately settles on the result that you're actually looking for and third we need to apply an efficient
13:14 algorithm that looks at the reward and the system state and knows how to tweak the parameters so that the process converges with any reasonable amount of
13:23 time this is where we would also set the parameters for exploration and exploitation and the discounting on future rewards in the next few videos
13:32 we'll expand on all of this by looking at the workflow of reinforcement learning in more detail we'll look at the structure of the policy and
13:40 introduce neural networks and we'll talk about how crafting the proper reward function impacts your final result and we'll look at a very high overview of
13:48 some interesting learning algorithms so if you don't want to miss the next Tech Talk video don't forget to subscribe to this channel also if you want to check
13:56 out my channel control system lectures I cover more control theory topics there as well thanks for watching and I'll see you next time