It's not necessarily about the cpu size. But fitting a small phone with a low power CPU with low graphics capabilities, less sensors, less memory, lower quality cameras modules, less battery, is a different task from cramming everything high quality, high power into a small phone.
They have been doing this for a decade. If anything, the parts have became even smaller and more power efficient during that time, so building a flagship 4" phone today would be orders of magnitude easier than building those flagship 3.5" phones from the mid/late-naughties.