Lots of things are a first world gimmick. Electricity in general is a first world gimmick, we survived just fine with sticks and flames when we lived in caves. But it does enhance our lives. In my moderately filled-out smart home I like the additional control that I have over things I have in my house, not to mention automations, timers, monitoring, etc.
The problem is that not everything needs to be controlled by everyone. Parental controls and security settings exist to prevent unwanted access to other aspects of our digital lives… I don’t understand why HomeKit has to be a free-for-all.