I should say that, despite being pretty heavily biased towards the 38mm model, in the end, I'll be going with 42mm. Despite posting the photo of my 36mm watch on my wrist, it's key to remember that the width of the rectangle watch, (the distance towards and away from the hand) is less than 36mm, and it's this direction that has the biggest impact on whether a watch looks oversized. In the other direction, the main concern is that the lugs don't overhang and that will not be an issue. Even on the gigantic Moto 360 at 46mm, the lugs don't overhang on my 167mm wrist. And while I've always favored really minimally sized watches, this is not truly a watch. Not honestly. It's something new. It's posing as a watch. It's a Trojan Watch. And that's fine. It's a great time to adjust to a new fit and style as I will bet real money that these two case sizes will stick with us for at least 5 years. In fact, I bet the bands sold next month will work with whatever Apple Watch revision ships in 2020. And sadly, it's still a very small watch compared to the nonsensically large wall clocks that more and more men are strapping to their average sized wrists.
Beyond fit, the 42mm model will offer a little better view, a little better battery life, and the option of the leather loop. I actually think the black modern buckle is suitable for men, and very dressy and nice looking, but, I don't want an extra dangly metal thing at the bottom if I'm going to wear a soft band, and I don't want a classic buckle for being a nuisance to take off and put on. I definitely expect to remove it nightly, and probably sometimes at work when I'm typing heavily. I'm fond of link bracelets for just this reason, but, would rather have a loop (milanese or leather) over a classic buckle. Too bad, then, there is no larger variation of the modern buckle as that clasp design is awesome.
Another factor in going 42mm is the recent testimony from an iOS developer taken in by Apple's little secret watch dev camp. They say Apple focused them on the 42mm first, 38mm by request. That tells me that the 42mm is the canonical target size and the 38mm, while being an excellent option for smaller wrists of any gender, is going to be the most likely model to suffer usability difficulties that might arise from that minor size difference. As a developer, I hope this is not something that materially manifests, but, well, you can't truly target two sizes in exactly the same way and resources are often limited in software development so...
42mm, if you can fit it, is my recommendation.
Watch will be just fine on my wrist.