good re-encoding app

I'm trying out some video editing and I export them to apple tv format (for the resolution)
now I've been able to convert these to AVI's but the problem is that a file of 1/10 of its original length is a larger file then the original.
I used to have a program called mencoder or something on windows where i could re-encode the file to a size of my choosing.
If it was to small, it made the resolution smaller so the quality was relatively untouched.
now on mac I'm looking for a program to do the same.
the only programs i found either just converts it to avi and doesn't make the file smaller or just makes the quality really crap without changing the resolution (D-vision 3)

any suggestions on any decent converters? It doesn't have to be free but is prefered.

thanks in advance,
kristalghost

 

heard of one for windows

ive heard that megui is awsome (saw some work of it from a Bleuray disc) but again it is for windows.
anybody know anything like that?

ive tried that ffmpegX but I haven't got the faintest idea how to get the best quality/space ratio with it.
are there any good guides of consorts out there?

thanks in advance,
kristalghost