The 6/6S/7/8 were great sized phones for me, and to that end the X is just about spot-on. Anything larger and it would be too big for me, just pointless having something you can't use with one hand easily.
The thing is, Apple will make whatever people will pay for. If people feel the need for huge phones that look ridiculous, need two hands to use and need a bag to carry, fair play to them. Not for me though. Some people see the size of phone as a kind of status symbol, "hey, look how big my phone is", going on the basis that bigger costs more and therefore shows wealth. Well no longer, because the smaller X costs more, and I think that's why people are complaining - their once flagship phone is no longer flagship.