Markov Decision Processes

Dive into Deep Learning · §14.1

Markov decision processes
states, actions, transitions, and rewards · the Markov assumption · discounting · reward design

Four Objects and One Assumption

Acting changes the data the agent will see next. The model of acting over time:

\textrm{MDP}: \quad (\mathcal{S}, \mathcal{A}, P, r)

  • \mathcal{S}: states. \mathcal{A}: actions.
  • P(s' \mid s, a): transition kernel, a distribution over next states.
  • r(s, a): expected immediate reward, specified by the task designer.

The assumption: given the present state, the past is irrelevant for predicting the future.

The Kernel Is Data

Slippery FrozenLake: a command goes as intended with probability 1/3, and slides perpendicular with probability 1/3 each.

env = gym.make('FrozenLake-v1', is_slippery=True)
env.unwrapped.P[9][1]
[(0.33333333333333337, 8, 0, False),
 (0.3333333333333333, 13, 0, False),
 (0.33333333333333337, 10, 0, False)]
gamma = 0.95
LEFT, DOWN, RIGHT, UP = 0, 1, 2, 3
mdp = TabularMDP.from_gym(env, gamma)
print(mdp.P[9, DOWN])
[0.         0.         0.         0.         0.         0.
 0.         0.         0.33333333 0.         0.33333333 0.
 0.         0.33333333 0.         0.        ]

The commanded move is one outcome among equals.

One Object for the Whole Chapter

class TabularMDP:
    """A finite MDP as dense arrays: P[s, a, s'] and r[s, a]."""
    def __init__(self, P, r, gamma):
        self.P, self.r, self.gamma = P, r, gamma
        self.num_states, self.num_actions = r.shape

    @classmethod
    def from_gym(cls, env, gamma):
        """Read the transition table Gymnasium exposes as env.unwrapped.P."""
        n_s, n_a = env.observation_space.n, env.action_space.n
        P, r = np.zeros((n_s, n_a, n_s)), np.zeros((n_s, n_a))
        for s, actions in env.unwrapped.P.items():
            for a, outcomes in actions.items():
                for p, s_next, reward, _ in outcomes:
                    P[s, a, s_next] += p      # several outcomes may share s'
                    r[s, a] += p * reward     # r(s,a) is the expected reward
        return cls(P, r, gamma)

    def backup(self, V):
        """Q(s, a) = r(s, a) + gamma * sum_{s'} P(s'|s, a) V(s')."""
        return self.r + self.gamma * self.P @ V

backup is one application of r + \gamma P V: the chapter’s algorithms are built out of this line.

Trajectory and Return

\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \ldots), \qquad R(\tau) = \sum_{t=0}^{\infty} \gamma^t r_t

rng = np.random.default_rng(8)
s, _ = env.reset(seed=8)
terminated = truncated = False
ret, t = 0.0, 0
while not (terminated or truncated):
    a = int(rng.integers(4))
    s_next, r, terminated, truncated, _ = env.step(a)
    print(f't={t:>2}  s={s:>2}  a={"<v>^"[a]}  r={r:.0f}')
    ret, s, t = ret + r, s_next, t + 1
print(f'terminated={terminated}, truncated={truncated}, return={ret:.0f}')
t= 0  s= 0  a=>  r=0
t= 1  s= 0  a=v  r=0
t= 2  s= 0  a=<  r=0
t= 3  s= 4  a=^  r=0
t= 4  s= 4  a=<  r=0
t= 5  s= 4  a=v  r=0
t= 6  s= 8  a=>  r=0
t= 7  s= 9  a=^  r=0
t= 8  s=10  a=>  r=0
terminated=True, truncated=False, return=0

FrozenLake gives reward only on reaching the goal; this episode has return zero.

The Discount Is a Horizon

 gamma  horizon 1/(1-gamma)  t: gamma^t < 0.05
   0.5                    2                  5
   0.9                   10                 29
  0.95                   20                 59
  0.99                  100                299

For \gamma = 0.99, the effective horizon is one hundred steps.

Terminated Is Not Truncated

  • terminated: the process entered a terminal state, whose continuation value is zero.
  • truncated: observation stopped, often because of a time limit; the underlying state can still have nonzero continuation value.

An interaction loop may stop for either flag, but value estimation must distinguish them: bootstrapped targets are masked by terminated alone.

Reward Misspecification

A plausible modification of a sparse reward: add 0.3 for a step that moves the agent closer to the goal.

dist = abs(np.arange(16) // 4 - 3) + abs(np.arange(16) % 4 - 3)
closer = dist < dist[:, None, None]    # closer[s, :, s'] = 1 iff d(s') < d(s)
shaped = TabularMDP(mdp.P, mdp.r + 0.3 * (mdp.P * closer).sum(-1), gamma)

def greedy_sweeps(m):                  # repeated backups; named in next section
    V = np.zeros(m.num_states)
    for _ in range(500):
        V = m.backup(V).max(axis=1)
    return m.backup(V).argmax(axis=1), V

def exact_value(pi, s=0):              # V(s) under the true reward, exactly
    i = np.arange(mdp.num_states)
    return np.linalg.solve(np.eye(16) - gamma * mdp.P[i, pi], mdp.r[i, pi])[s]

pi_shaped, V_shaped = greedy_sweeps(shaped)
pi_true, V_true = greedy_sweeps(mdp)
print(f'shaped-optimal: at s=14 goes {"<v>^"[pi_shaped[14]]}, shaped value '
      f'{V_shaped[0]:.2f}, true value {exact_value(pi_shaped):.3f}')
print(f'true-optimal:   at s=14 goes {"<v>^"[pi_true[14]]}, true value '
      f'{exact_value(pi_true):.3f}')
shaped-optimal: at s=14 goes <, shaped value 2.15, true value 0.000
true-optimal:   at s=14 goes v, true value 0.180

At s = 14, the shaped-reward optimum selects left and never reaches the goal. Its shaped value is 2.15, but its true value is 0. Potential-based shaping, r + \gamma \Phi(s') - \Phi(s) with \Phi = 0 at terminal states, preserves the optimal policy.

Recap

  • An MDP is (\mathcal{S}, \mathcal{A}, P, r) with discount \gamma.
  • The kernel is a distribution: on slippery ice, commands are proposals.
  • Effective horizon 1/(1-\gamma): rewards past it are geometrically negligible.
  • Terminal states end the future; truncation only ends the recording.
  • Not Markov? Augment the state (location and velocity; stacked frames).
  • Bandit = one-state MDP; text generation = deterministic, terminal-reward MDP.
  • Optimization follows the specified reward, including any mismatch with the intended task.