I'm late to the thread, I know, but here's my 2¢.
When the Ultra first came out, I was 100% convinced that is was too big for me. I've got a medium sized wrist but I'm not particularly tall, so I thought pulling off the 45mm was as big as I was going to be able to wear.
When the Ultra 2 came out, I happened to be in an Apple Store and saw a friend with one. I tried it on (for the first time, actually) and liked it. Even my wife says it's not at all too large nor would even get a second look for maybe being too big. Especially when worn with the Ocean band, which kind of fills out the look along the sides, I think it's not too big at all.
So I've completely changed my opinion about the Ultra and likely won't go back to a smaller model unless the next Ultra is substantially larger.
Now, there is still something to be said for my old 45 mm Series 7 Hermes stainless model -- it does fit better under a slim cuffed dress shirt, so the Ultra isn't perfect for all situations. I kept the 45mm Series 7 when I got the Ultra 2, but haven't worn it more than a few times in all the time I've owned the Ultra 2. Besides, the battery on my Series 7 is at 83% health level and drops at a rate of nearly 10-12% per hour, so it barely makes it through a day with anything but the most basic of watch faces. Thus it rarely gets used, and then really only for a few hours at a time (like a formal party I went to about 6 months ago where the Ultra didn't fit well under my suit's shirt cuff).