Thank You!!! Tiny Umbrella's "Kick Device Out of Recovery" button did the trick! This was on an iPhone 4 and Tiny Umbrella 4.1.6.Also, there are now several other ways to accomplish this, including recboot and TinyUmbrella.
For the life of me I couldn't get iRecovery even run (I had libusb in the same folder). I just kept getting command not found. Anyway, Tiny Umbrella seems to be the easiest way to get your iDevice out of Recovery mode.