I have that problem when attempting to convert .avi taken with my Canon Camera to mp4. Because these are short movies taken with the camera, I just use iSquint, settings for TV, h.264 encoding, quality to Go Nuts. I suppose I could try those settings for Handbrake and maybe get it to work, but for this use, iSquint does it easily and well enough.