iPhoto became Photos. In many ways, they are quite similar.
What's different is that late iPhoto is when iCloud came to be, which then led to Apple strategies of motivating people to store photos in iCloud... which leads to overloaded iCloud space... which leads to paying more and more to store more pics & videos in iCloud (your current problem). As you anticipate, that will only get worse and worse requiring more monthly rent for more iCloud space.
So you can pull your pics & videos DOWN from iCloud to store locally in Photos (app). When they are all local, you don't need all that space for photos in iCloud anymore... because now you possess them instead of trusting strangers in "the cloud" to possess them for you. This is exactly what we all did before there was an iCloud. We all acted as caretakers of our own media. And we can still do that today.
When you shoot new pics & videos, link camera/iDevice to Mac and import them directly into the Photos app- no iCloud required. If you want a subset of "all" of them on your iDevice, make 1+ Photos albums and then sync those albums back to iDevice. Think of this like the modern equivalent of carrying a subset of all paper photos around in a wallet.
Photos basically creates & manages a "file structure" in the background, organizing your photos and videos in various ways for easy viewing.
Optionally you can break out videos to external storage, which will then free up a LOT of space from the Photos library. The TV app can then take over as index-er of personal video files and it can do all kinds of nice things like stream such videos to all devices around the home (via home sharing), including AppleTVs. This seems like a very good target for you.
I didn't find it difficult. Synology has wizard-based setups. Answer questions in the setup wizards to set it up. I'm fairly technically sharp though so maybe it is not as easy as that reads.
A NAS is basically a simple computer with BIG storage. Synology even allows one to start with an amount of storage and easily add to it when they need more. For example, if you start with maybe 3 hard drive (HDDs) but then need more storage, you can add another drive to expand the pool of storage. They revolve around a storage concept called RAID, which- overly simply- allows you to pool a number of drives together as if they are one larger drive. Different RAID configurations offer various benefits such as some internal backup and faster read/writes.
Here's their storage calculator with which you can "what if."
You can choose NAS boxes with "bays" for HDDs. More bays means you can add more HDDs over time. For example, if you chose an 8-bay and loaded it with 3 HDDs for starters, you still have 5 empty bays for future expansion. You can also replace a drive with a bigger drive to grow the storage pool too. So, for example, let's imagine you start with 3 drives at- say- 6TB each but later replace one of them with a 12TB drive, the net gain is 6 more TBs of storage within the same bay.
NAS is a big concept so you should
read up on it to get your brain around it. However, it's not hugely different than your idea of attaching some drives to the iMac and then the iMac being the brains of that storage. This would simply be a computer especially designed for that kind of thing. It would have a bunch of bays into which you can insert drives for storage. Its own apps can then give you your own free cloud storage, your own cloud photos manager, your own whole home DVR, your own security video monitoring system, etc.