Because the traditional HMM algorithm has three disadvantages: firstly, the output probability of observed features is irrelevant to its history; secondly, continuous multiplication of the probability values can be easy to cause underflow phenomenon in the Viterbi algorithm; thirdly, the observed values of high dimensional vector will bring about a larger computational burden in the training stage, so a new improved HMM algorithm was proposed. At first, we should separate hands from complex backgrounds by using the deep message of kinect, and reduce the dimensionality of the observed value. Next, we use the angel of adjacent point as trajectory feature of gesture and utilize curvature’s changing of trajectory as the new HMM Model state numbers. Finally, the improved HMM algorithm is used to train and recognize the gesture. Results show that this method of the improved Hidden Markov Model has a low complexity, high efficiency and accuracy of recognition, which also has a good practicability.