That whole issue is one of the main reason I never got on with link bands. Either too tight or too loose, and since our wrists DO change size depending on ambient temperature or body heat/cold, it's a lesson in frustration.
I did the same put an extra link on the pinky side. For me out of the 2 I like it snug so that's the way I went but really depends. I don't like my watch moving and like how it feels that way. Im also wearing it above the wrist bone so maybe try the tighter option and see if moving it a bit higher helps? Like mine fits perfect in a sweet spot but it took a while to get it. It is too bad we can't do micro adjustments or it would have been really cool if the removable links were the same size across the band but had different widths so taking out different links would allow micro adjustments.