You could say the same thing about wearing a watch or carryng a phone.
That's true. There are some pretty dorky looking watches and phones out there - the difference is that there are a lot of non-dorky looking options when it comes to phones and watches, which can't be said of head-mounted wearable devices.