Okay, let me be absolutely clear on this, so bear with me while I go through it step-by-step.
1. You downloaded a file called mac3dec1.6.3.sit.hqx [note that double file suffix],
2. You dragged it from Downloads to the desktop,
3. You 2x-clicked the mac3dec1.6.3.sit.hqx.
If yes to all the above, then
4. 2x-clicking should have made another blue folder with a little blue symbol on it (I can't replicate it here). Inside that folder should be the application 'mAC3dec' plus a Read Me.rtf file.
That's it!
Now if the answer to any of the 3 steps listed above is 'no', then we need to consider why the download is not right OR why Stuffit Expander is not doing what it should.
mAC3dec is a relatively old application. I take it the audio files that came from the dvd are in AIFF or AC3 format, and you wish to convert them to .mp3 for iTunes. There are other - and I think better - audio file converters. Switch is one, SoundConverter is another. Perhaps you'd care to try them if you can't get mAC3dec to work for you. Keep us posted.