Region covariance descriptor recently proposed has been approved robust and elegant to describe a region of interest, which has been applied to visual tracking. We develop a geometric method for visual tracking, in which region covariance is used to model objects appearance; then tracking is led by implementing the particle filter with the constraint that the system state lies in a low dimensional manifold: affine Lie group. The sequential Bayesian updating consists of drawing state samples while moving on the manifold geodesics; the region covariance is updated using a novel approach in a Riemannian space. Our main contribution is developing a general particle filtering-based racking algorithm that explicitly take the geometry of affine Lie groups into consideration in deriving the state equation on Lie groups. Theoretic analysis and experimental evaluations demonstrate the promise and effectiveness of the proposed tracking method.