That sounds similar to a the most recent incident I described in this thread:
https://forums.macrumors.com/threads/379654/
Here's a repost of what I wrote there:
I just had a pretty serious incident involving graphics glitches, I'm not sure if it's related. I was just opening a photo to edit in iPhoto, and all of a sudden, everything became unresponsive. The cursor still moved, but nothing responded to cursor movements (dock magnification didn't react) and nothing responded to any clicking or anything. Even trying force quit wouldn't bring up the force quit window, apple-tab wouldn't bring up the app switcher... By accident though, I hit the apple-esc key for Front Row, and that came up just fine. I could navigate through Front Row totally normally, listen to music, look at photos and everything. When I exited Front Row, things still weren't normal though, and I started to notice weird visual glitches. Big black boxes were all over the place, vaguely in the shape of the Force Quit window, or the Shut Down, Sleep etc dialog. Some windows sorta became transparent in sections, and then occasionally the screen would flicker really violently, like seizure-inducing flashing. The mouse cursor was still responding to the trackpad, though nothing else on the screen would respond. I could still go back in to Front Row without any issues there at all. Finally I gave up and held the power button until it shut off. On reboot, I was a little worried to see the screen turning on at maybe a lower resolution setting, and more of a square aspect ratio with black bars down either side. Eventually it started up, everything back to normal.
I found a whole bunch of stuff in the Console logs, especially this message related to NVIDIA that came maybe a 100 times in the space of about 4 minutes, part way into this episode, continuing until I shut the computer down:
10-11-30 7:53:06 PM kernel NVDA(OpenGL): Channel exception! exception type = 0xd = GR: SW Notify Error
10-11-30 7:53:07 PM kernel 0000006e
10-11-30 7:53:07 PM kernel 00200000 00008297 00000474 00000010
10-11-30 7:53:07 PM kernel 0000047e 000015e0 00000000 0000000a
10-11-30 7:53:07 PM kernel 00000000 00000000 00000002
10-11-30 7:53:07 PM kernel 00000010 00000000
10-11-30 7:53:07 PM kernel NVDA(OpenGL): Channel exception! exception type = 0xd = GR: SW Notify Error
10-11-30 7:53:07 PM kernel 0000006e
10-11-30 7:53:07 PM kernel 00200000 00008297 00000474 00000010
10-11-30 7:53:07 PM kernel 0000047e 00001458 0004d801 0000000a
10-11-30 7:53:07 PM kernel 00000000 00000000 00000013
10-11-30 7:53:07 PM kernel 00000010 00000000
