I watched the video from ifixit and felt i could damage the mac mini if i do it myself
Two things to note about my experience: The two T10 screws that hold the motherboard down were extremely tight. I thought I was going to break it. I also had a really tough time getting the motherboard out after everything was unscrewed. I actually cut my finger a tiny bit! However, everything worked out and other than those two things, it was pretty easy and quick!
You can spread the clips to remove the memory without removing the rubber covers.
Not really. The covers are there to protect the clips and they actually slide into the clips in a very specific way. Even though they are a little fiddle to take off the whole process still only takes about 10-15mins