Only if you want to save on batteries. I find the magic mouse eats batteries like there is no tomorrow. Keyboard and trackpad does not use the batteries up that fast.
You are correct. when the iMac is in sleep mode, after a short delay, the mouse, keyboard etc go into sleep mode too. Therefore there is no need to switch the peripherals off when not using your iMac.
The average battery life of the keyboard is 6 months and one month for the Magic Mouse.