If you're into reading on your iPhone screen, the true tone makes all the difference, changing the white point as the light changes. However, if you want colour accuracy for viewing images or movies, I'd recommend turning it off! This way the temperature of the image you're viewing will be constant and you won't see any colour shifts as the screens colour temperature changes around on you. With true tone turned off, it is balanced at around 6,500K from several reviews that I read, which is fairly close to daylight and pretty good for viewing image colours accurately.
I would say that the 99% probably wouldn't notice it, but if you have a trained eye, you definitely will notice the colour shifts as the light changes around you.