With the same logic, I probably also agreed in the terms that Apple can remote-delete/deactivate my iPhone/Apps for any reason they consider necessary. Doesn't mean they SHOULD.
There isn't a clause about deactivating your device, but there is one about remotely removing apps from it. That ability does exist, but they have yet to use it in over 6 years. Don't forget, in legal terms, the software on that device is still Apple's. You're just leasing it from them. The hardware is the only thing that's actually yours.