I'm mostly curious with how my D90 will stack up against an Olympus PEN EP-3 (this one - link) I am not considering moving down to a P&S
You're lining yourself up to shoot yourself in the foot, you will end up leaving that at home as much as your D90 as even though it is much less bulky it is still beyond being pocket sized so you will find you will leave it at home more often than not.
I found myself in a similar position to yourself recently but kept the D90 and like many others here added a S95 to my arsenal that goes everywhere my man purse goes and often if the murse gets left at home the S95 is slipped into a pocket, I couldn't be happier with that setup.