Rogue Amoeba made an app that did this quite a while ago... it was discontinued, and I don't even think they offer it in their freebies section anymore. Also it doesn't function on Intel Macs, I think.
However, you
might be able to get this function via Audio Hijack Pro...
http://www.maclife.com/article/how_can_i_control_the_volume_of_individual_applications_separately
Audio Hijack has an application mixer, and so I think you can also possibly mix the app / adjust their volume, etc., and not just mute them this way.