Personally, I'd stay where you are unless you really need to upgrade for some specific reason, but not because it's slower - just a personal preference.
But my son has a 4s on 9.3.5 and he's fine with it. I've used it from time to time and there is indeed some slight lag every once in a while, but nothing that would stop me from upgrading if that's what I wanted to do.
Others may tell you it will become slower. That's not been what I have observed with my son's 4s.