I had the Invisible Shield on the back and front of my White iPhone 3G, and the plastic began to get sort of yellow-ish and looked dirty after about 2 or 3 months of having it on.
I suspect the same thing would happen and you'd be able to notice it on the White iPhone 4.