We study unsupervised and supervised recognition of human actions in video sequences. The videos are represented by probability distributions and then meaningfully compared in a probabilistic framework. We introduce two novel approaches outperforming state-of-the-art algorithms when tested on the KTH and Weizmann public datasets: an unsupervised nonparametric kernel-based method exploiting the Maximum Mean Discrepancy test statistic; and a supervised method based on Support Vector Machine with a characteristic kernel specifically tailored to histogram-based information.