Here is the simplified explanation:
More than you ever wanted to know about Polarization-
Polarized glasses use a film created from rolls of polyvinyl acetate which has the consistency and thickness of plastic wrap. The plastic is heated and stretched which lengthens the plastic's long chain molecules, causing them to align. It's then dipped in iodine solution which is absorbed into the molecular chains forming long grids of parallel, darkened lines that are not visible to the human eye. The film is then dyed to the color of the desired finished.
For the polarized film to be most effective in sunglasses, it's oriented to reduce the horizontal component of reflected light off a surface. Manufacturers have proprietary formulas for calculating the proper angle.
The iPhone screen is also Polarized- it's supposed to be done to a 45 degree angle allowing the most effective use in both portrait and landscape mode.
But if the angle is off and/or the angle the sunglass manufacturer uses to place their polarization film is set in a certain way, you essentially have two sets of 'invisible' grids that cut out the ambient light and can cause a screen to either darken or undergo a color shift. (take off the glasses and rotate them while looking at the iPhone screen through the lenses and you'll see the color shift and or darkening change).
That's what you are seeing.
And it isn't limited to the iPhone- all products with polarization do it to a certain extent.