In general Apple will only let you install the latest firmware for the device in question. You should be able to restore it to the latest firmware it supports. I am curious as to why you want to downgrade it.
All the forums that I've found suggest to first downgrade it to the version I stated above. I've tried a multitude of versions of which none work.
Have you tried following
iOS: Unable to update or restore and make sure you're trying to install the latest firmware for the device.