Dynamic Programming

Dive into Deep Learning · §14.2

Dynamic programming
value functions · the Bellman equations · a contraction · why the optimal path is not the shortest

Two Value Functions and Their Gap

• V^\pi(s): expected discounted return, following \pi from s.
• Q^\pi(s, a): same, but the first action is pinned to a.
• Linked by averaging: V^\pi(s) = \sum_a \pi(a \mid s)\, Q^\pi(s, a).

The gap is the advantage (:eqref:eq_advantage):

A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s), \qquad E_{a \sim \pi(s)}[A^\pi] = 0

A^\pi(s,a) > 0 identifies actions a whose value exceeds the policy average.

The Bellman Equations

One step now, value thereafter (the Markov assumption at work):

V^\pi(s) = \sum_{a} \pi(a \mid s) \Big[ r(s, a) + \gamma \sum_{s'} P(s' \mid s, a)\, V^\pi(s') \Big]

For the optimal policy the average over actions becomes a max (:eqref:eq_bellman_optimality):

V^*(s) = \max_{a} \Big[ r(s, a) + \gamma \sum_{s'} P(s' \mid s, a)\, V^*(s') \Big]

After the first transition, the continuation is optimal from the resulting state.

Backup Diagrams

The diagrams show one-step backups; the sampled form on the right is used in :numref:sec_qlearning.

Why It Converges

Proposition. \|TV - TV'\|_\infty \leq \gamma\, \|V - V'\|_\infty: the Bellman operator is a \gamma-contraction.

• unique fixed point V^*
• \|V_k - V^*\|_\infty \leq \gamma^k \|V_0 - V^*\|_\infty from any start
• stopping certificate: \|V_k - V^*\|_\infty \leq \frac{\gamma}{1-\gamma} \|V_k - V_{k-1}\|_\infty
• at \gamma = 1, this contraction argument provides no guarantee

Value Iteration

def value_iteration(mdp, num_iters):
    """Sweep V <- max_a backup(V); return the whole history of iterates."""
    V, history = np.zeros(mdp.num_states), []
    for _ in range(num_iters):
        V = mdp.backup(V).max(axis=1)
        history.append(V)
    return np.array(history)

gap = np.abs(np.diff(history, axis=0)).max(axis=1)
true_err = np.abs(history[1:] - V_star).max(axis=1)
certified = gamma / (1 - gamma) * gap
assert (true_err <= certified + 1e-12).all()
for name, e in [('sweep-to-sweep change', gap),
                ('certified error bound', certified),
                ('distance to V*', true_err)]:
    print(f'{name} first below 1e-6 at sweep {np.argmax(e <= 1e-6) + 2}')
k_cert = np.argmax(certified <= 1e-6) + 2

sweep-to-sweep change first below 1e-6 at sweep 128
certified error bound first below 1e-6 at sweep 164
distance to V* first below 1e-6 at sweep 158

The naive test stops at sweep 128, before the target error is reached at 158. The certified rule stops at sweep 164.

Policy Iteration and GPI

Evaluate the policy, act greedily on its values, repeat. The policy-improvement proposition guarantees nondecreasing value.

ok = [(mdp.backup(V).argmax(axis=1) == pi_star).all() for V in history]
print(f'policy iteration: {num_outer} rounds of evaluate-then-improve')
print(f'value iteration: certified at sweep {k_cert}')
print(f'its greedy policy already equals pi* from sweep '
      f'{np.argmax(ok) + 1} on')

policy iteration: 2 rounds of evaluate-then-improve
value iteration: certified at sweep 164
its greedy policy already equals pi* from sweep 14 on

Not the Shortest Path

Compare \pi^* with the shortest-path policy derived for deterministic ice, using 2000 episodes per policy on the stochastic environment.

shortest = np.array([DOWN, RIGHT, DOWN, LEFT,
                     DOWN, LEFT, DOWN, LEFT,
                     RIGHT, DOWN, DOWN, LEFT,
                     LEFT, RIGHT, RIGHT, LEFT])   # optimal on calm ice, by hand
env.reset(seed=0)
for name, p in [('slip-aware optimum', pi_star),
                ('shortest-path policy', shortest)]:
    success = evaluate(env, lambda s, _: int(p[s]), num_episodes=2000)
    print(f'{name}: reaches the goal in {success:.1%} of 2000 episodes')

slip-aware optimum: reaches the goal in 73.6% of 2000 episodes
shortest-path policy: reaches the goal in 4.7% of 2000 episodes

The optimal policy succeeds about sixteen times as often. At four cells it points away from the goal, and at others it points into walls, because :eqref:eq_optimal_policy accounts for the outcomes of stochastic slips.

Recap

• V^\pi, Q^\pi, advantage A^\pi = Q^\pi - V^\pi: defined once, used for two chapters.
• Bellman: expectation form for a policy, optimality form :eqref:eq_bellman_optimality for the best one.
• The operator contracts at rate \gamma: unique V^*, geometric convergence, checkable certificate.
• Value iteration, policy evaluation, policy iteration: one proof, three algorithms.
• Generalized policy iteration alternates approximate evaluation and improvement.
• On ice, the optimal policy is not the shortest path, and we measured the difference: 0.74 vs 0.05.
• Value iteration uses a known model; methods from :numref:sec_qlearning onward use sampled transitions.