There's nothing wrong with upgrading hardware, but just so you know, I have two iMacs that are significantly older than yours (early 2008 w/5 GB RAM, mid-2011 w/6 GB), both running Yosemite, both with conventional HDDs, and neither of them needs anything like 4-5 minutes to boot. As azentropy already suggested, there could be reasons other than RAM and intrinsic HDD speed slowing you down.
If you give yourself a new boot drive, then you (naturally) are starting with a fresh install of OS X - that would "fix" possible issues with HDD disk errors and your current OS X installation (among other things), but it's like replacing your car's engine when it may only need a tune-up or has a bad spark plug wire.
Running Disk Utility and reinstalling OS X cost you nothing but time and effort. When you're done, you may still want the added speed of an SSD, but again, maybe not. (First run Disk Utility > Verify Disk - if the disk needs to be repaired, it's best to do that
before reinstalling OS X - you might not even need to reinstall OS X.)
You may also need to reset NVRAM
https://support.apple.com/en-us/HT204063
When you say it takes 4-5 minutes to boot - does it take that time to get to the system login, or for your desktop and startup items to finish loading? If it's the latter, your login items and/or apps and files that automatically re-open from your previous session may be part of the problem. There are several ways to troubleshoot that.
As to RAM... Maybe you do need more than 8 GB, but relatively few people do. What does Activity Monitor have to say about Memory Pressure and Swap Used (if you're not familiar with what those measurements mean, refer to Activity Monitor's Help pages)? Basically, unless you're seeing a lot of Swap Used and Memory Pressure consistently goes into orange or red, your issues aren't RAM-related. If you do see a lot of Swap Used and Memory Pressure issues, and haven't restarted your system for a while, restart, and see if/how long it takes for those statistics to go bad.
There are other possible causes unrelated to HDD or RAM, but this is probably enough for now.
In the end, you may still benefit by SSD and more RAM, but maybe you'll find a better way to spend your money.