MullOverThings

Useful tips for everyday

When would SARSA likely do better than Q-learning?

When would SARSA likely do better than Q-learning?

If your goal is to train an optimal agent in simulation, or in a low-cost and fast-iterating environment, then Q-learning is a good choice, due to the first point (learning optimal policy directly). If your agent learns online, and you care about rewards gained whilst learning, then SARSA may be a better choice.

Does SARSA converge faster than Q-learning?

… SARSA is an iterative dynamic programming algorithm to find the optimal solution based on a limited environment. It is worth mentioning that SARSA has a faster convergence rate than Q-learning and is less computationally complex than other RL algorithms  .

Is Q-Learning on policy?

For example, Q-learning is an off-policy learner. On-policy methods attempt to evaluate or improve the policy that is used to make decisions. In contrast, off-policy methods evaluate or improve a policy different from that used to generate the data.

How is expected Sarsa transformed to Q-learning?

We see that Expected SARSA takes the weighted sum of all possible next actions with respect to the probability of taking that action. If the Expected Return is greedy with respect to the expected return, then this equation gets transformed to Q-Learning.

When to use Q or a in Sarsa?

According to Sarsa , once the agent gets to s’, it will follow its policy, π. Knowing this information, we can sample an action a’ from π at state s’, and use q (s’, a’) as the estimate of the next state:

How to create expected Sarsa in reinforcement learning?

Code: Python code to create the SARSA Agent. Update the action value function using the SARSA update. Code: Python code to create the Q-Learning Agent. Update the action value function using the Q-Learning update. Code: Python code to create the Expected SARSA Agent.

What’s the difference between expected and expected Sarsa?

Expected Sarsa, on the other hand, reasons that rather than sampling from π to pick an action a’ by, we should just calculate the expected value of s’. This way, the estimate of how good s’ is won’t fluctuate around, like it would when sampling an action from a distribution, but rather remain steady around the “average” outcome for the state.