We try to understand numbers to facilitate the transition to doing algebra. From a mathematical point of view there is no essential difficulty from the counting numbers to learn all the way to the complex numbers without any algebra. The two road blocks theoretically happen when there are infinite numbers to count and again for the continuum of real numbers which are dealt with in a more advanced course. Believe it or not this can and probably should be taught to advanced elementary school students transitioning to middle school. Once the numbers are understood, replacing them by a letter won't cause someone to instantly lose track of everything that the letter represents.