In this paper, we address the problem of tracking an object with pose and appearance changes, under possible occlusions by presenting an effective way to embed the texture information provided by the Local Binary Pattern (LBP), Local Ternary Pattern (LTP) and the Complete Local Binary Pattern (CLBP) in the mean shift framework. We combine the information of color distribution with variants of Local Binary Pattern texture for the purpose of robust tracking. Four adaptive scale and orientation mean shift trackers are proposed; the LBP_MS, LTP_MS, CLBP_MS1 and the CLBP_MS2. The last tracker can handle both textured and non-textured objects, and deals with the specific weaknesses of motion trackers, such as failures under specific conditions. As it can exploit more of the image information, we use seven public videos that contain a variety of challenges to illustrate the accuracy of the proposed approaches. The trackers successfully cope with fast moving objects, target scale and orientation changes, and prove to be more stable and less prone to drift away from the target than purely colored or feature-based ones.