In our approach, the dynamic reference policy \(\mu\) has its own distinct off-reward function defined based on the same MDP.
We assume that the series of training reference policies \(\{\mu^n\}_{n=0}^{\infty}\) converge to an optimal policy \(\mu^{*}\) defined as \(\mu^{*} = \arg \max_{\mu} \sum_{t=0}^{\infty} \gamma^t \mathbb{E}_{(s_{t},a_{t})\sim p_{\pi}} [f(\mathcal{M})]\).
Here, \( f(\mathcal{M}) \) represents a function defined over the Markov Decision Process (MDP).
Off-Reward Dynamic Reference Value Iteration
\begin{equation}\label{eq:state_value_function_sequence_def}
\tilde{V}^{n+1}(s_{t}) = (B_{\mu^n} \tilde{V}^n)(s_t) = \alpha \log \sum_{a_{t} \in A} \mu^{n}(a_{t}|s_{t}) \exp \left( \frac{1}{\alpha} (r(s_{t}, a_{t}) + \gamma \mathbb{E}_{s_{t+1} \sim p} [\tilde{V}^{n}(s_{t+1})]) \right) \, .
\end{equation}
Off-Reward Dynamic Reference Policy Iteration
\begin{equation}
\pi_0 \xrightarrow[\mu_1]{\text{E}} Q_{\pi_0/\mu_1} \xrightarrow[\mu_1]{\text{I}} \pi_1 \xrightarrow[\mu_2]{\text{E}} Q_{\pi_1/\mu_2} \xrightarrow[\mu_2]{\text{I}} \pi_2 \xrightarrow[\mu_3]{\text{E}} \dots \xrightarrow[\mu^\infty]{\text{I}} \pi^\infty \xrightarrow[\mu_\infty]{\text{E}} Q_{\pi^\infty/\mu^\infty}.
\end{equation}
MOP reference
The Maximum Occupancy Principle (MOP) is a novel approach to modeling agent's behavior, which diverges from traditional reward-maximization frameworks.
The goal of MOP is to maximize the occupancy of future action-state paths, rather than seeking extrinsic rewards.
This principle posits that agents are intrinsically motivated to explore and visit rare or unoccupied action-states, thus ensuring a broad and diverse range of behaviors over time.
The off-reward function in MOP is the entropy of the paths taken by the agent,
\[
R(\tau) = -\sum_{t=0}^{\infty} \gamma^t \ln \left( \mu_{\text{MOP}}^{\alpha}(a_t | s_t) p^{\beta}(s_{t+1} | s_t, a_t) \right)
\;,
\]
where \(\alpha > 0\) and \(\beta \geq 0\) are weights for actions and states, respectively, and \(\gamma\) is the discount factor.
The agent maximizes this intrinsic reward by preferring low-probability actions and transitions, which encourages exploration and the occupancy of a wide range of action-states.
This intrinsic motivation leads to behaviors that appear goal-directed and complex without the necessity of explicitly defined extrinsic rewards.
Lap reference
We can think of learning a low-dimensional state representation as building a mapping between the original space and a low-dimensional representation such that near/distant points in the original space are near/distant in the representation.
This problem is referred to as graph drawing problem.
In the approach taken here, valid for large and continuous spaces, we want to build a set of features \(\phi(s) = (f_1(s),...,f_d(s)) \in \mathbb{R}^d\) that faithfully represent the input space \(s \in S\).
Once the embedding function \(\phi(s) = [f_1(s), \ldots, f_d(s)]\) has been learned, we ask our off-reward dynamic reference policy to maximize the expected cumulative reward \(\sum_{t=0}^{\infty} \gamma^t R(s_{t}, a_{t})\), where the instantaneous off-reward \(R(s_{t}, a_{t})\) is defined as
\[
R(s_{t}, a_{t}) = \frac{||\phi(s_{t+1})-\phi(s_{t})||_{2}^2}{||\phi(s_{t+1})||_{2}^2 + ||\phi(s_{t})||_{2}^2}
\;.
\]
This novel off-reward function consistently encourages the agent to transition to states that are furthest from the current one, which leads to improved exploration. This approach effectively prevents the agent from wasting time on inefficient transitions that result in minimal changes to the embedded state space, leading to a more sample efficient algorithm.
