Does the headphone output port on the iMac offer "dual outputs" -- that is, does it offer a digital (optical) output as well as analog (ordinary headphone) output?
If it does offer dual outputs, be aware that there is a small "switch" that is a part of the headphone jack, that is designed to switch the digital output on or off, depending on what kind of plug is inserted into the jack.
The plug for a digital connection is slightly "longer" than a regular headphone plug.
It's -possible- that this switch may have gotten activated, and then remained stuck in the digital mode.
This can be ascertained by looking into the headphone port. If you see a "red light", that means that the iMac thinks it is supposed to be supplying a digital/optical output, not an analog output.
If it IS in digital mode, you won't get an analog signal into regular headphones.
In any case -- if the iMac is still under warranty, take it to an Apple Store and have them check it out. Let THEM resolve the problem.