While iMovie can import .MP4 files without encoding, if the .MP4 video file has not been encoded with an Apple friendly codec, then iMovie will need to re-encode the file. This is why importing the clips takes some time. Then iMovie will then need to re-encode the video output when you export. Video encoding is very processor and memory intensive. The time taken to encode on import and export is driven by the settings that you have for your project and Output file settings. So you might have highly compressed small files for import, but if you have widescreen HD as your project setting and you export a HD file then iMovie will try to upscale your source video assets. However as the detail is not in the original files then much of this effort is wasted.
Your Mac does not have a lot of memory and the CPU is getting on a bit now as well. I suspect the long encode time to export is driven by:
1. Small amount of memory (3GB no longer cuts the mustard as far as video editing is concerned).
2. Slow CPU
3. You export / project parameters in iMovie.