The 10w & 12w chargers WILL charge your iPad Mini faster, it will draw a full 10w / 12w from them unlike the iPhone that only pulls 5w no matter what charger you use. Apple must ship Mini's with the 5w charger either because they cost less to manufacture or to try and get you to buy the better one separately.
On the Apple website and also on the packaging for the better chargers they list them as compatible with the Mini so there's no way they would do any damage.
Proof thanks to wwwinsanelygreatmac here:
http://www.youtube.com/watch?v=d93yb7kkTFI
So if fast charging is important to you and you don't mind coughing up the money for the 12w charger, I would go get one.
EDIT:
I actually did this myself and now I just use the 5w charger that shipped with my Mini to charge my iPhone 4S, the USB cable on my 4S charger was getting a bit frayed at the ends anyway.