I just got a 32GB 5S and my power button rattles pretty bad. It even rattles when the phone vibrates as a text or something comes in. Not sure what to do with it either, keep it or exchange it?
Please leave feedback here for Apple about the issue.
http://www.apple.com/feedback/iphone.html
Personally I'd return it if you have a remorse period and either reorder a new unit or wait a while if you can.