Can you define "pretty much not usable"? Because this is an incredibly misleading argument about larger devices. Can they be a little more cumbersome to use? Yes, I can see that, depending on different people's hand sizes. Are they unusable? Hardly. This is a dramatic (and tiring) argument. Even for small hands, we're talking about screens in the range of 4, 4.3, or 4.6 inches being literally different by millimeters. Even if you want to talk dimensions, we're talking about changes ranging in the ~1 inch vicinity when talking about 4-4.6" screens. This is not that dramatic, and does not render a device anywhere near "unsable".
Like I said, if Apple does release a larger screen/device, I'm sure people will suddenly find it possible to use with one hand (and fit into their pockets, which is the other common argument). They might even come to like it! The trade offs of a larger screen device (easier to read text, easier to type, better for games, pics/videos, browsing, etc.) over a few millimeters is well worth it.
----------
Simple. It was a different time back then. It was before apps really took off (think about how popular gaming is now on iOS devices; they rivals Nintendo DSes), it was before typing became so much more common (more than phone calls now), it was before using a smartphone to watch movies and TV shows (streaming or downloaded from iTunes) became common, etc.
This is like asking, "why are low ppi screens an issue now when it wasn't before?" Standards have changed. Tastes have changed, and so forth.
Maybe not all devices but for me the Samsung galaxy nexus was pretty much unusable one handed. If I held it one handed and tried to have any interaction on the screen with the same hand I felt like I was going to drop it because of the size of the phone. It might be fine for others but not for me so don't try to tell me what my experience with a phone was or was not! I know what my experience was!