Not a tough decision. Simply ask yourself, do you want a device that is productive, or do you want to be tied down to a child's toy. Personally speaking, I think the Note 4 blows this out of the water. Design, specs, and hardware are all above par.
Well I just don't like Samsung, I hate Touchwiz and I prefer Motorola. Not to mention that its a Nexus phone. For me that's already a huge advantage. But why this is a hard decision is because I really like the iPhone but the Nexus 6 is the Android device i've always wanted to have. I really like both. Ideally I'd love to have both but that's not happening(or at least not yet haha)