Been having this same problem myself as well and it's driving me insane. I thought I was crazy for the longest time, until I was able to capture it on video.
https://vid.me/Vtb
Dims at 4:55, brightens at 6:20
From my experience it seems to happen primarily when playing games or watching videos in Safari. It's quite random when it happens, but after it does the brightness will pop back after around a minute or so
I'll be going to the Genius Bar tomorrow, so I'll be sure to report back.
UPDATE:
Genius insisted I do a clean install of the OS without restoring from a backup. Though this did not work and it's still happening for me.
https://vid.me/H8i
Starts off with the "dimmed" bug, and as you see adjusting the brightness slider doesn't actually change brightness at all when adjusting in the top 20%.
Then at 19 seconds in it pops back up to full brightness, afterwards the brightness slider does work.
Because of this I'm pretty sure it's not a hardware issue, but rather software.
Not sure a replacement device would hugely help out in this situation.