Update: I bought a Synology NAS and have been using the "Photos" app for a couple weeks. I am pretty happy with it and it seems to do what I wanted.
First I exported all the unmodified original photos from Apple Photos to the Synology Photos directory of the NAS. When you setup Photos, you can have a shared folder and then users can have individual photos folders. I decided to export all our existing photos to the shared folder.
Once that finished I then setup my phone and my wife's phone to sync our iPhone camera photos to our own personal photos folder. Synology has an iPhone app to make this simple. Once you sync up your pictures, you also have the option to "free up space", where you can selectively remove from your phone photos using certain criteria such as older than a certain date, but don't delete favorite photos(for example).
Once our phones copied all our pictures to our personal folders, I then moved the good pictures to the shared folder.
The synology photos app scan through the pictures and then categorizes them by location, people, topic, etc. Once that is complete you can search for pictures in a similar way as the Photos app. There is also an Apple TV app that you can use to view your photos on your TV.
Overall, I'm happy with synology photos. It's not quite as polished as an apple product, but does what I want. We would have to do the $10 a month plan w/ apple icloud to store all our photos. The NAS was $600, although that is for 6TB of storage in a RAID 1 configuration. I could have purchased less disk space and probably have spent $400. The break even is a few years, maybe never once if I pay for cloud backup for the NAS. If I find some other application for the NAS, then it will be worth it. Regardless it was a fun project.
I may also try immich once it becomes more mature. It's not in a stable release yet, so I'll probably wait a few months to try it out.