How number is presented? Is it represented difference for different species? We first review behavioural and neuropsychological studies for the representation of different level of mathematical ability for animals, infants, lesion cases and images studies to demonstrate the relationship of mathematics and language. It can be included that 1) both humans and animals share an elemental number quantification system, which is without supporting by language ability; 2) the language ability is crucial to get higher mathematical ability. We then summarize the main research status for each line of studies. Finally, we outline recommendations for future research directions.