Recent years have seen an increase in research articles and reviews exploring mathematical difficulties (MD). Many of these articles have set out to explain the etiology of the problems, the possibility of different subtypes, and potential brain regions that underlie many of the observable behaviors. These articles are very valuable in a research field, which many have noted, falls behind that of reading and language disabilities. Here will provide a perspective on the current understanding of MD from a different angle, by outlining the school curriculum of England and the US and connecting these to the skills needed at different stages of mathematical understanding. We will extend this to explore the cognitive skills which most likely underpin these different stages and whose impairment may thus lead to mathematics difficulties at all stages of mathematics development. To conclude we will briefly explore interventions that are currently available, indicating whether these can be used to aid the different children at different stages of their mathematical development and what their current limitations may be. The principal aim of this review is to establish an explicit connection between the academic discourse, with its research base and concepts, and the developmental trajectory of abstract mathematical skills that is expected (and somewhat dictated) in formal education. This will possibly help to highlight and make sense of the gap between the complexity of the MD range in real life and the state of its academic science.