Can't remove app

I installed iOS8 b2 and restored from an iCloud backup. Everything went fine except I had a previously installed app that was installed with Apple Configurator (.ipa). It's an app that's still being developed and isn't in the App Store. The app just stuck on "Waiting..." and if I try to remove it, it gives me the X and allows me to click delete, but nothing happens, the app just stays there. I've tried reinstalling the app with iTunes (Apple Configurator says it's not compatible with iOS8) and the app still just says "Waiting...". I've tried rebooting, removing it from General>Usage (it doesn't show up there actually) as well. Any ideas?