The Bellman Equation

The Bellman Equation is a fundamental concept in Reinforcement Learning, developed by Richard Ernest Bellman through his work in dynamic programming. It helps in understanding how to evaluate the value of states and actions based on expected rewards.

Key Concepts

s = State
a = Action
R = Reward
γ = Discount Factor

Example: Navigating a Maze

Imagine a maze where:

Gray Blocks represent walls.
Red Block (Firepit) has a reward of R = -1.
Green Block has a reward of R = +1.
White Blocks are neutral.

The agent can take steps in four directions: left, right, up, or down.

Reward Structure

If the agent steps on the Green Block, it receives a reward of +1.
If it falls into the Red Block, it receives a reward of -1.
When the agent steps on a White Block, it has the potential to lead to a Green Block in the future.

Learning Process

The agent explores the maze and learns from the rewards. Initially, it doesn’t understand which blocks are beneficial or harmful. However, as it navigates, it starts to associate the White Block with the potential reward of stepping onto the Green Block.

For example, if the agent moves from a White Block to a Green Block and receives a reward of +1, it might deduce that stepping on the White Block is a good strategy. Over time, it maps the entire maze, marking the paths that lead to positive outcomes.

Visualization

Here’s a simple diagram to illustrate the maze, with values representing rewards:

[W] [W] [W] [+1]     -> Green Block (+1)
[W] [+1] [W] [W]     
[W] [-1] [W] [W]     -> Red Block (-1)

we dont use this approach,

LaTeX

The Bellman Equation:

V(s) = \max_a \left( R(s, a) + \gamma \sum_{s} P(s'|s, a)V(s') \right)

where:

( V(s) ) is the value of state ( s ).
( a ) is an action.
( R(s, a) ) is the immediate reward after taking action ( a ) in state ( s ).
( \gamma ) is the discount factor.
( P(s'|s, a) ) is the probability of reaching state ( s' ) after taking action ( a ) in state ( s ).

demo.js

The **Bellman Equation**: 
 
$$
V(s) = \max_a \left( R(s, a) + \gamma \sum_{s'} P(s'|s, a)V(s') \right)
$$
 
where:
- \( V(s) \) is the value of state \( s \).
- \( a \) is an action.
- \( R(s, a) \) is the immediate reward after taking action \( a \) in state \( s \).
- \( \gamma \) is the discount factor.
- \( P(s'|s, a) \) is the probability of reaching state \( s' \) after taking action \( a \) in state \( s \).

Temporal Diffrence The Plan