In the process of the multimedia physical education, the moving athlete detection is the key to tracking and recognizing the athlete. This task contributes to the gesture and action modeling and simulation. However, since the scene of sports events is complicated, the background is commonly affected by the noise and illumination changes, making the background difficult to remove. In order to solve this problem, a novel motion saliency calculation based method is proposed for detecting the moving athlete in sport videos. In our method, four motion features, as the optic flows, direction vectors and frame difference are fused by the Quaternion Fast Fourier Transform. The motion saliency map is generated to identify the area of the moving athlete. The experimental result demonstrated that our method has better ability to detect the moving athletes, contributing to the multimedia physical education.