That is because of two things.
1. .avi, .mp4 and most modern video files use a codec that is derived from the MPEG-4 codec, Xvid/Divx or H264 for example.
Video DVDs use the much older MPEG-2 codec, which is not as efficient and advanced as MPEG-4. Thus the video files using the MPEG-2 codec need more storage space (MBs, GBs) for the same length of video than a video file using the MPEG-4 codec.
2. Most movies released on video DVD use a so-called Double Layer (DL) DVD, which can store up to 8.5GB (7.9GiB) of data, in this case video.
Most writable DVD media that can be bought are Single Layer (SL) DVDs, storing only 4.7GB (4.37GiB) of data.
As pointed out in 1., the MPEG-2 codec is not really space efficient, thus video DVDs are using DL-DVDs.
The average length of today's movies is about 100-120 minutes. No problem for MPEG-4 encoded video, and no problem for MPEG-2 encoded video.
But as you probably use SL-DVDs, the video on the resulting video DVD has only 45% of the original capacity to store its information on (as it would be with consumer DVDs), thus the bad image quality.
Again in short:
1. .avi/.mp4 files are more advanced and space efficient than video on a video DVD. Thus the .avi/.mp4 video files look better with less storage space occupied.
2. Consumer video DVDs have 45% more capacity to store video on, therefore the video on a video DVD you burnt yourself (and used an SL-DVD for it) looks crappier than on the original DVD or even the .avi/.mp4 file.