I can't compare it to the Nexus 4 in real life use since I just turned around and sold the Nexus 4 still sealed in the box. That being said from what I've seen of videos of the Nexus 4, the Note ii may be less buttery.
Anyways, I'm coming from using an iPhone 4 previously. My biggest draw to Android was to get a bigger screen. I had been using a Nexus 7 since September and absolutely loved the experience. I find the Note ii to be much faster and responsive than the Nexus 7. It's taking a bit of time to adjust to Touchwiz but I don't hate it. I would still prefer to have pure Android. I may think about installing ROM's down the road.
The only thing I do hate is the hardware home button and capacitive buttons. The capacitive buttons are in reverse order to pure Android and I find myself hitting the wrong button to go back. Keep in mind that I've only had the device a few days and will soon adjust. When switching from portrait to landscape I am constantly hitting the capacitive buttons unintentionally as well.
The s-pen is a feature I seldom use but it is nice to have when needed. The size isn't an issue for me, I can easily type one handed (having a hand span of 10 & 1/2 inches helps). All that being said, I previously used to carry my Nexus 7 everywhere and tether it to my iPhone. Now that I have the Note ii the only time I've picked up the Nexus 7 is to use in bed while passing out. I'm too paranoid that I'd fling the more expensive Note ii to the floor while sleeping.
If a larger screen and massive battery capacity are important to you, I'd say give it a trial run. Otherwise I think the Nexus 4 is the phone to have.