1. Wouldn't you agree that color depends on context? For example, I like my clothes muted, but my guitar is surf green, a color I'd probably not wear at the moment, but I love it on my guitar, it looks really cool.
2. Wouldn't you agree that color preference by gender is something that is not programmed biologically inside humans? You were not born as a pastel color hater. In fact, you were raised as one. People around you told you that pastel colors are feminine, so you adapted.
3. Wouldn't you agree that these sort of preferences change over time and depend on cultural context? I don't even need to take a debatable example from the middle ages or something. Think of the 90s: The combination of pastel green, violet and yellow was the thing back then. It was everywhere. Would you say everyone was homosexual back then? Probably not.