关键词: 知识蒸馏, 教师网络, 学生网络, 随机采样, 课程学习, 难度判断

Abstract: With the advent of industry 4.0 era,neural network has been widely used in realizing the automation of the industrial systems.However,large-scale neural networks tend to consume a large amount of storage,memory and computing resource.They are difficult to be applied efficiently in the industrial scenarios where computing is limited.In contrast,lightweight networks have a broader application prospect.Knowledge distillation extracts the knowledge of a large-scale and high-performance teacher network to guide the training of a lightweight student network,which has been successfully verified in improving the performance of lightweight networks.However,the existing knowledge distillation methods all adopt the traditional training data input strategy.The training dataset is shuffled and the sequence of mini-batches is randomly sampled without considering the influence of the sample order on the training of the student network.To solve this problem,the curriculum learning paradigm was introduced into the knowledge distillation scene.Simulating the real education scene,the knowledge distillation was performed by utilizing an easy to hard training sample input strategy,where the difficulty of the samples was evaluated by the teacher and student networks cooperatively to integrate the experience of the teacher network and the demand of the student network,so as to achieve the most reasonable curricula design.Experimental results on CIFAR dataset indicated that the proposed method could greatly surpass traditional knowledge distillation base lines with various network structures.Moreover,the proposed curriculum learning paradigm could also be applied to other mainstream knowledge distillation methods to further improve their performance.The ablation experiments showed that the cooperation between teacher network and student network had obvious advantages over using teacher network or student network alone for difficulty evaluation.It verified the effectiveness of introducing the curriculum learning paradigm into the knowledge distillation scene.Meanwhile a practical and effective algorithm was proposed,which provided some insights for the research of knowledge distillation.

Key words: knowledge distillation, teacher network, student network, random sampling, curriculum learning, difficulty evaluation