The thing with hats is that they work best when paired with the sort of clothing that looks well with hats; in modern fashion, that means collared shirts (doesn't have to be a with a tie), and perhaps a waistcoat (vest).
A jacket (or coat) is not necessary, - but will look well - and, proper well cut trousers or slacks should flatter an outfit when wearing a hat. However, if wearing jeans or khakis, they need to be well cut, but will work well. Likewise, shoes: Wear proper shoes, not runners, - loafers are fine - with hats.
For an example of how to match a stylish yet casual American look, one which mixes and matches ideas from both old and new, I recommend that anyone interested take a look at the website of Colonel Littleton, an excellent US company, a company that
@SandboxGeneral may be aware that our mutual friend,
@Shrink, introduced me to.
Best of all is a really nice over-coat with a hat.