I guess the only damage you could do is decrease the number of remaining re-writes to the solid-state drive. They can only be written to a certain number of times, but I wouldn't worry about it. I've restored my iPod touch countless times and it's still working fine. Most of the time I had to re-write 14GB of music back onto it too!
My advise? Don't worry... Restore whenever you want.