The identification of plant disease is the premise of the prevention of plant disease efficiently and precisely in the complex environment. With the rapid development of the smart farming, the identification of plant disease becomes digitalized and data-driven, enabling advanced decision support, smart analyses, and planning. This paper proposes a mathematical model of plant disease detection and recognition based on deep learning, which improves accuracy, generality, and training efficiency. Firstly, the region proposal network (RPN) is utilized to recognize and localize the leaves in complex surroundings. Then, images segmented based on the results of RPN algorithm contain the feature of symptoms through Chan–Vese (CV) algorithm. Finally, the segmented leaves are input into the transfer learning model and trained by the dataset of diseased leaves under simple background. Furthermore, the model is examined with black rot, bacterial plaque, and rust diseases. The results show that the accuracy of the method is 83.57%, which is better than the traditional method, thus reducing the influence of disease on agricultural production and being favorable to sustainable development of agriculture. Therefore, the deep learning algorithm proposed in the paper is of great significance in intelligent agriculture, ecological protection, and agricultural production.