By nature, I have never liked wearing watches or jewelry. I have 185mm wrists, and decided to go with the 38mm because I figured it would be the least obtrusive and was about the same size as other watches I own (though never wear). Personally, I love it and think the 38mm looks and feels great.
I was originally a little concerned that the 38mm watch might seem like a 'girls' watch - but after doing a lot of research, I found that historically mens watches were much smaller than they are today, and that at one time a 38mm mens watch would be considered huge.
Realistically, though, 4mm is not much difference - but I am very happy that I went with the smaller watch.