摘要：AbstractFor practical applications, deep neural networks need to be deployed with low memory and computing resources. To achieve this goal, we design a lightweight convolutional neural network namely KENet (Knowledge Enhance Network) and propose a knowledge distillation method to improve the performance of KENet. Our proposed KENet is a lightweight convolutional network derived from a wide residual network by replacing the normal convolutions with a hybrid of group convolutions and bottleneck blocks to reduce the number of parameters. However, the use of small kernels and group convolutions loses the information of both spatial and channel-wise dimensions. To solve this problem, we further propose a knowledge distillation method to enhance the information of these two dimensions. We extract both spatial and channel-wise knowledge from a ‘teacher’, and improve the attention transfer features for knowledge distillation. The experiment results on multiple datasets show that KENet is computationally cheap and memory saving with hardly any loss of precision. Moreover, we confirm that KENet can be effectively deployed in the advanced detectors with strong robustness and real-time performance.
关键词：Keywordsmachine learningmodel compressionknowledge distillationattention transfer
