As a former Sony user I can tell you that at certain point of time age plays a roleSony made the exact thing that you wanted with their compact line, a smaller phone with flagship internals, but obviously it's not selling enough to sustain themselves. And have you looked at the offerings from Huawei, Oppo and Xiaomi? Even their cheapest offerings have at least 6.5" screen or larger. These companies cannot afford to cater to the small phone crowd as the return is not enough to justify the cost of R&D, manufacturing, logistic, etc.
So not only there were less people like me that wanted small phone, but most of us are aging to some extent and now our eyes demand something different in terms of size.