There are small women who like the 44mm, I've seen some guys wearing the 40mm... Mostly a matter of taste and what you're comfortable with, although occasionally the available sizes of a particular band vs. wrist dimension has an impact.
Based on you choice of watch faces, watch face colors, and band colors, you can make a fashion statement that leans towards a particular gender identity regardless of the size of the watch, but overall, this is a unisex product. It's not like the small watch is as small as a traditional woman's "dress" watch (so small you need a jeweler's loupe to read the dial).
I'm a pretty large guy and I do wear the 44mm, but mostly for the size of the display. Overall, I think the whole oversize watch thing (like the divers wrist watches the size of old-fashioned railroad conductors' pocket watches) is a bit much. In comparison to those, the 44mm Watch is petite.