Remember that under the current DVD standard, the total for all menu-related files must be less than 1 GB. This is true even if you don't have any other content -- the DVD standard can't deal with more than 1 GB of menu info total.
So, while it may be that moving from .avi to MPEG-2 is your main problem, if your total menu info is more than 1 GB, that might be causing the error.
Good luck!