# Bellman Equation Derivation

Using math equations as magic has never been my way. For deep concept understanding, a key point is to first understand all the prior information related with the new concept that you are trying to catch.

In Reinforcement Learning, a important equation is the Bellman Equation. In this video Constantin Bürgi presents a very clear derivation of the equation transforming the original infinite horizon problem into a dynamic programming one.

## Original problem

We want to solve the next infinite horizon optimization problem:

under the constrain $k_{t+1} + C_t = (1 + r) k_t$

being:

• $k_t$ capital invested in t+1
• $C_t$ capital consumption in t
• $k_{t+1}$ capital invested in t
• $r$ interest rate

By definition $C_t$ can also be expressed by $C_t = (1+r)k_t - k_{t+1}$

The infinite horizon problem, akas $L_0$ can be expressed by:

Leavind a part the first element, t=0, $L_1$ can be expressed as:

Now we can express $\mathcal{L_0}$ as the sum of an expression plus $\mathcal{L_1}$:

$\mathcal{L_0}$ is a function that depends only on the value of $k_0$, for its part, $\mathcal{L_1}$ depends solely of $k_1$.

## Bellman equation or value function

Changing the name of variables $V = L$, $k_0 = k$, $k_1 = k'$ we obtain the expression of the Bellman Equation: