iOS 9.3 does make a difference, a big difference for me. It pretty much eliminated most of the lag and stutter on my iPhone 5 and iPad 4 compared to iOS 9.2 and below. My iPhone 5 feels like an iPhone again, fast and smooth. My iPad is a little slower, but still way better than iOS 8 and the previous iOS 9's.
However, I do have an iPad 3 on 7.1.1 and it is faster than my iPad 4 on 9.3 for almost everything. I am curious how the iPad 3 is on 9.3 though.
I'd say, if you just use your iPad for simple stuff like Safari and Youtube, and the current apps make you happy, don't upgrade.