I'd personally prefer a 4.7" display in a smaller form over a larger display in the same form. Move to 4.5" in roughly the same form as SE (with 1334×750) and let that replace both the 4" and 4.7" models going forward. Then make the larger one a more pocketable and manageable 5.5" plus model. This allows two models to replace three and sets an inch between them, differentiating them nicely.
I actually would like then to keep the 4" and make it even more compact. I think a lot if the 4" owners just like to have the most compact device possible to have on ourselves at all time. Smaller is better when you're out and about, outdoors, at the gym, jogging, sports, construction type work.
And I don't think it would cannibalize any other form factor but I believe like me, own two sizes. I love my compact SE AND love my 7 Plus. And both for completely different reasons and uses.
We all have different use needs, and that's great to have options because it encourages those that need them to buy two phones instead of just one.