Q_\pi(s,a)

重要性采样定理为解决上述问题指明了方向因此理解重要性采样定理是理解离线MC强化学习的关键。

已知随机变量

⎧​Ep​(f)Eq​(g)Ep​(f)∫x​p(x)f(x)dxEq​(g)∫x​q(x)g(x)dx​​​

已知对

S_q\{x_{q,1},x_{q,2},\cdots,x_{q,m}\}

E_p(f)

\hat{E_p}(f)\hat{E_q}(g)\frac{1}{m}\sum_{k1}^m\frac{p(x_{q,k})f(x_{q,k})}{q(x_{q,k})}

\end{align}

Ep​^​(f)Eq​^​(g)m1​k1∑m​q(xq,k​)p(xq,k​)f(xq,k​)​​​

在估计

\hat{E_p}(f)\frac{1}{m}\sum_{k1}^mf(x_{p,k})

E_p(f)

Ep_k\{(s_{k,0},a_{k,0},r_{k,1}),(s_{k,1},a_{k,1},r_{k,2}),\cdots,(s_{k,N_k-1},a_{k,N_k-1},r_{k,N_k}),(s_{k,N_k},a_{k,N_k},r_{k,N_k1})\},k1,2,\cdots,m

Epk​{(sk,0​,ak,0​,rk,1​),(sk,1​,ak,1​,rk,2​),⋯,(sk,Nk​−1​,ak,Nk​−1​,rk,Nk​​),(sk,Nk​​,ak,Nk​​,rk,Nk​1​)},k1,2,⋯,m所有轨迹的

最后一个状态

Gkπ′​(s,a),k1,2,⋯,m2.根据重要性采样公式(2)及

G^{\pi

\hat{Q_\pi}(s,a)\frac{1}{m}\sum_{k1}^m\frac{p_k^{\pi}}{p_k^{{\pi}}}G^{{\pi}}_k(s,a)\\

\frac{1}{m}\sum_{k1}^m\frac{p_k^{\pi}}{p_k^{{\pi}}}G^{{\pi}}_k(s,a)\\

p_k^{\pi}-策略\pi

\rho_k\frac{p_k^{\pi}}{p_k^{\pi}}-重要性采样比例表示待评估策略\pi

下和采样策略\pi

p_k^{\pi}[\pi(a_{k,0}|s_{k,0})P_{s_{k,0}s_{k,1}}^{a_{k,0}}]\times

[\pi(a_{k,1}|s_{k,1})P_{s_{k,1}s_{k,2}}^{a_{k,1}}]\times\cdots

\times[\pi(a_{k,N_k-1}|s_{k,N_k-1})P_{s_{k,N_k-1}s_{k,N_k}}^{a_{k,N_k-1}}]\\

\prod_{i0}^{N_k-1}\pi(a_{k,i}|s_{k,i})P_{s_{k,i}s_{k,i1}}^{a_{k,i}}\\

p_k^{\pi}\prod_{i0}^{N_k-1}\pi(a_{k,i}|s_{k,i})P_{s_{k,i}s_{k,i1}}^{a_{k,i}}\\

\rho_k\frac{\prod_{i0}^{N_k-1}\pi(a_{k,i}|s_{k,i})}{\prod_{i0}^{N_k-1}\pi(a_{k,i}|s_{k,i})}

\end{align*}

Qπ​^​(s,a)pkπ​pkπ′​ρk​pkπ′​pkπ​​pkπ​pkπ′​ρk​​m1​k1∑m​pkπ′​pkπ​​Gkπ​(s,a)m1​k1∑m​pkπ′​pkπ​​Gkπ′​(s,a)−策略π下出现完整轨迹Epk​的概率−策略π′下出现完整轨迹Epk​的概率−重要性采样比例表示待评估策略π下和采样策略π′下获得轨迹Epk​的概率之比[π(ak,0​∣sk,0​)Psk,0​sk,1​ak,0​​]×[π(ak,1​∣sk,1​)Psk,1​sk,2​ak,1​​]×⋯×[π(ak,Nk​−1​∣sk,Nk​−1​)Psk,Nk​−1​sk,Nk​​ak,Nk​−1​​]i0∏Nk​−1​π(ak,i​∣sk,i​)Psk,i​sk,i1​ak,i​​i0∏Nk​−1​π′(ak,i​∣sk,i​)Psk,i​sk,i1​ak,i​​∏i0Nk​−1​π′(ak,i​∣sk,i​)∏i0Nk​−1​π(ak,i​∣sk,i​)​​