Yeah, I didn't think about how you'd know which app was open in an application stack.
But how about this... maybe when you mouse over a stack, if there's an application open inside it, it would appear above the stack? Like how the name of an application appears above the icon when you mouse over. If there's more than one, they could appear side by side, or something.
But that might be confusing.
Hmm.... how about... when you click an app in a stack to open it, it bounces out of the stack, and goes to the right side of the dock, as if you opened an app from the finder that isn't kept in the dock. Or it could even just bounce out and sit beside the stack. And when you close the app, it bounces back into the stack? I think that would be a good way to do it.
Sure, the dock would fill up with open apps, but that way you know what's open at a glance.
And what would happen if you open all apps in a stack? Well the last app to leave the stack would stay put. But let's say you close an app that wasn't the last one to leave... I suppose that app would bounce back to where the stack originally was.
Or maybe once all apps leave a stack, there could be a placeholder for the stack.
Something like that.