I always purchased aluminum. Series 2 then Series 4. When the 5 came out, they were discounting the S4 Stainless models at Best Buy for I think near 40% off. I bought one, I couldn't pass it up. If money is no object, or if you get a good deal, I prefer the Stainless. The watch feels a lot more like a traditional watch in terms of weight. The Stainless black case is beautiful. I wear the Stainless majority of the time, aluminum to work out with and usually to sleep with. Then put that on the charger and throw on the stainless. My perspective is, since I'll likely be replacing it more frequently due to the tech upgrades, I'll stick with Aluminum unless I can find a deal on a stainless model that sings to me.
Regarding durability, many seem to swear by the stainless models but I don't have a nick on my aluminum and nothing wrong with the screen either. I do use screen protection though.
Weight wise, flipping between the two each day, I barely notice much of a difference.