Hardly. The Rolex Submariner is one of the most ubiquitous luxury watches ever made, and it wears very similar to my 38mm SS on the link bracelet. Both the face and the link bracelet of the 38mm are actually a little larger than on a Submariner.
As mentioned above, if you're a watch guy/girl, and you want something more reminiscent of a classic luxury watch, 38/40 is the way to go. If you want something that looks like a smartwatch, or an oversized department store watch, 42/44 is the way to go.
Here's a pic I took a few years ago with my Series 0 38mm. The 38mm option is what convinced me to even try the Apple Watch in the first place. Even though the screen of the new 40mm version is much nicer, I actually think the more square, old Series 0 38mm version is a little better looking and more masculine when the screen is off.