Abstract: The traditional optimization algorithm cannot effectively use historical solving experience and is difficult to obtain the optimal solution when solving the type I two-sided assembly line balancing problem.Aiming at this problem,a deep reinforcement learning algorithm named Proximal Policy Optimization with Convolutional Neural Networks(CNN-PPO)was proposed.The deep reinforcement learning agent structure of the CNN-PPO was designed.Based on the Proximal Policy Optimization(PPO),the Convolutional Neural Networks(CNN)was introduced to enhance the data feature extraction capabilities of the agent.According to the characteristics of two-sided assembly line balancing,a state matrix was proposed to describe the two-sided assembly line balancing problem and introduce the mask layer to assist the agent in task decision-making.A reward function was designed according to the optimization goal,the optimal combination behavior strategy was selected for each decision by combining with the reinforcement learning online execution-evaluation(Actor-Critic)mechanism,and the effectiveness and stability of the algorithm were verified through multiple example tests.The experimental results showed that the solution results of the proposed algorithm were better than the current algorithms,of which 57 could reach the lower bound among 59 test cases.

Key words: two-sided assembly line, type I balancing problem, deep reinforcement learning, convolutional neural networks, proximal policy optimization

摘要： 针对传统优化算法求解双边装配线第一类平衡问题时不能有效利用历史求解经验,难以得到最优解,提出一种深度强化学习求解算法CNN-PPO。设计了CNN-PPO强化学习智能体结构,在近端策略优化算法基础上,引入卷积神经网络增强智能体的数据特征提取能力;根据双边装配线问题特征,定义状态矩阵对双边装配线问题进行描述,并引入标记层辅助智能体进行任务决策;根据问题优化目标设计了奖励函数,结合强化学习在线执行—评价机制,为每次决策选择最优的待分配任务,并通过多个案例测试验证了算法的有效性和稳定性。实验结果表明,所提方法的求解结果具有优越性,59个测试案例中有57个可以达到下界。

关键词: 双边装配线, 第一类平衡问题, 深度强化学习, 卷积神经网络, 近端策略优化