Yup, if one wants to look dapper, choose your hat and clothing wisely. For instance, I always wear a collared shirt with my fedora's. But if I happen to go out in a t-shirt (getting rarer and rarer for me) I will grab a baseball cap to go with it.
Exactly.
Anyone who tries on a hat while wearing a t-shirt, or shorts, or casual jeans, or runners will not look right, and thus, will not feel right. The clothes and the style of headgear must somehow work together.
Baseball caps (which I loathe, by the way) go best with shorts, t-shirts, & runners; you wouldn't wear a baseball cap with a shirt and tie and suit and black leather shoes.
However, hats work better with some sense of elegance, & formality - and that means, well cut, classic styling.
The music movement that called themselves "Mods" had - or cultivated - a very distinctive appearance; dark well-cut suits, white collared shirts, ties, black shoes, and porkpie hats. I loved it, and thought it incredibly dapper, elegant & stylish.