MullOverThings

Useful tips for everyday

How do you find the optimal policy?

How do you find the optimal policy?

Finding an Optimal policy : We find an optimal policy by maximizing over q*(s, a) i.e. our optimal state-action value function. We solve q*(s,a) and then we pick the action that gives us most optimal state-action value function(q*(s,a)).

Is the optimal policy always deterministic?

For any infinite horizon expected total reward MDP, there always exists a deterministic stationary policy π that is optimal. Theorem 3 (Puterman [1994], Theorem 8.1. Note that an action not only determines the current reward, but also future states and therefore future rewards.

Why do we use Bellman’s equation?

The Bellman equation is important because it gives us the ability to describe the value of a state s, V𝜋(s), with the value of the s’ state, V𝜋(s’), and with an iterative approach that we will present in the next post, we can calculate the values of all states.

How does policy iteration look for value functions?

Does policy iteration simply look for a value function that provides a higher reward than it’s current reward and then update immediately which gives a new distribution of actions for my states (a new policy) and then iteratively does this for every one of its states until convergence?

What does a value function tell you about a policy?

A value function tells you, for a given policy, what the expected cumulative reward of taking action a in state s is. Forget about value iteration and policy iteration for a moment. The two things you should try to understand are policy evaluation and policy improvement.

What do you call a state value function?

State value function A state value function is also called simply a value function. It specifies how good it is for an agent to be in a particular state with a policy π. A value function is often denoted by V(s). It denotes the value of a state following a policy.

How does a value function determine the best course of action?

A value function determines the best course of actions to achieve highest reward. No. A value function tells you, for a given policy, what the expected cumulative reward of taking action a in state s is. Forget about value iteration and policy iteration for a moment.