When you double click one of your faces at the bottom and name it, does it not take you to a new window where it asks you to uncheck pictures that are not you?
You still have to do some clicking but it is much faster than clicking each picture at the bottom of the main faces window.
I would love to be able to select multiple faces to ignore, that's for sure. How they could overlook something so basic is beyond me. I keep hoping I'm missing something but I too have tried keys, clicks, menus and can't find a way to do it.