The thing to keep in mind is that medium is our actual starting point, and that we are actually trying to keep things as simple as possible.
Trying to do as you are suggesting would create a lot of complexity, as it would mean tracking and comparing different values for different sizes, or worse yet, calculating adjustments on the fly (like a certain other game, currently under revision, that I am not going to mention).
Now, the Race creation rules DO contain some guidelines on size adjustments (for creating larger races and such), but as I mentioned previously, everything is built around the concept of Medium being the baseline from which you deviate...
Additionally, the whole point of the size comparison pic in the first post was to simply give a visual representation of the sizes of various humanoid races and monsters compared to one another so that players and GMs have some sort of idea of how they might appear to each other (I know that I had not really thought about it until I made the image...)