Q-learning的核心在于Q表格#xff0c;通过建立Q表格来为行动提供指引#xff0c;但这适用于状态…前言

DQN

Q-learning的核心在于Q表格通过建立Q表格来为行动提供指引但这适用于状态和动作空间是离散且维数不高时当状态和动作空间是高维连续时Q表格将变得十分巨大对于维护Q表格和查找都是不现实的。

DQN

gym.make(env_name)random.seed(0)np.random.seed(0)env.seed(0)torch.manual_seed(0)replay_buffer

env.observation_space.shape[0]action_dim

DQN(state_dim,

agent.take_action(state)next_state,

reward,

env.step(action)replay_buffer.add(state,

action,

replay_buffer.sample(batch_size)transition_dict

{states:

b_d}agent.update(transition_dict)return_list.append(episode_return)if

(i_episode

np.mean(return_list[-10:])})pbar.update(1)episodes_list

list(range(len(return_list)))plt.figure(1)

plt.subplot(1,

fig.1是一个一行两列布局的图且现在画的是左图plt.plot(episodes_list,

return_list,cr)plt.xlabel(Episodes)plt.ylabel(Returns)plt.title(DQN

{}.format(env_name))plt.figure(1)

当前要处理的图为fig.1而且当前图是fig.1的左图plt.subplot(1,

rl_utils.moving_average(return_list,

mv_return,cg)plt.xlabel(Episodes)plt.ylabel(Returns)plt.title(DQN

{}.format(env_name))plt.show()5.2

Agent.py

nn.Sequential(nn.Linear(state_dim,

hidden_dim),nn.Linear(hidden_dim,

action_dim))def

action_dim,learning_rate,discount,

epsilon,

action_dim).to(device)self.target_q_net

QNet(state_dim,

torch.optim.Adam(self.q_net.parameters(),lrlearning_rate)self.gamma

discount

dtypetorch.float).to(self.device)qvalue

actiondef

torch.tensor(data[states],dtypetorch.float).to(self.device)actions

torch.tensor(data[actions]).view(-1,

torch.tensor(data[rewards],dtypetorch.float).view(-1,

torch.tensor(data[next_states],dtypetorch.float).to(self.device)dones

torch.tensor(data[dones],dtypetorch.float).view(-1,

1).to(self.device)#从完整数据中按索引取值[64]#print(\n

actions

self.q_net(states).gather(1,actions)

self.target_q_net(next_states).max(1)[0].view(-1,1)q_targets

rewards

torch.mean(loss)self.optimizer.zero_grad()loss.backward()self.optimizer.step()if

self.count

0:#更新目标网络self.target_q_net.load_state_dict(self.q_net.state_dict())self.count

5.3

capacity):#双向队列可以在队列的两端任意添加或删除元素。

self.buffer

capacity)def

done):#数据加入bufferself.buffer.append((state,action,reward,

next_state,

collections.deque(maxlencapacity)

def

np.cumsum(a[:window_size-1])[::2]

rend

(np.cumsum(a[:-window_size:-1])[::2]

r)[::-1]return

tqdm(totalint(num_episodes/10),

descIteration

range(int(num_episodes/10)):episode_return

0transition_dict

agent.take_action(state)next_state,

reward,

env.step(action)transition_dict[states].append(state)transition_dict[actions].append(action)transition_dict[next_states].append(next_state)transition_dict[rewards].append(reward)transition_dict[dones].append(done)state

rewardreturn_list.append(episode_return)agent.update(transition_dict)if

(i_episode1)

np.mean(return_list[-10:])})pbar.update(1)return

return_listdef

tqdm(totalint(num_episodes/10),

descIteration

range(int(num_episodes/10)):episode_return

0state

agent.take_action(state)next_state,

reward,

env.step(action)replay_buffer.add(state,

action,

replay_buffer.sample(batch_size)transition_dict

{states:

b_d}agent.update(transition_dict)return_list.append(episode_return)if

(i_episode1)

np.mean(return_list[-10:])})pbar.update(1)return

return_listdef

td_delta.detach().numpy()advantage_list

[]advantage

deltaadvantage_list.append(advantage)advantage_list.reverse()return

dtypetorch.float)

https://www.cnblogs.com/xiaohuiduan/p/12993691.html

https://www.cnblogs.com/xiaohuiduan/p/12945449.html

强化学习第五节DQN【个人知识分享】_哔哩哔哩_bilibili

CSDN

3-ε-greedy_ReplayBuffer_FixedQ-targets_哔哩哔哩_bilibili

4-代码实战DQN_Agent和Env整体交互_哔哩哔哩_bilibili

CSDN