I say go for it! I have a small wrist (6.3"), and my primary regular wearer (prior to the S4, which I love so far!) is a 44mm Panerai, which is far bigger and bulkier. It's a matter of personal taste and what you're used to/comfortable with, but I get compliments all the time on my bigger watches.
Here's my 44mm S4 on the leather loop. (worth noting that the silicon sport band it came with is stiffer and shows more band on each end of the watch, which I think makes the fit look perfect. The leather loop fits quite flat, comparatively, so the watch looks a little bigger in relation to the wrist. But it's super comfortable, and doesn't move at all—an advantage of the watch being so light!)