I wasn't planning on buying the EVF either... I was also planning on just doing a lot of manual focusing with an inexpensive adapter to use all of my nikon lenses, but here I am with a couple of primes for the e-mount, and probably $900 less in my wallet for having picked up the primes and the EVF. I'm also having a great time, taking more pictures than ever, and really liking the results, though, so I'm certainly not upset about it.
I don't know exactly what you'd be looking for in terms of whether I think the 5mm is of 'overall higher quality' than the 50mm, so I'm not sure quite how to answer that question. When I first got each of them, I did some side-by-side shooting against the 35mm/1.8 nikon that I tend to love on my Nikon D7000, as well as the 50mm/1.8D that I also tend to like on my Nikons. I found the results to be comparable in both cases, meaning that I could see slight differences, but they were subtle and nothing that would make me choose the nikons vs. the sony's based on IQ alone. When handled properly, all of those lenses will give you fantastic results, and the shallow depth of field and low-ligh capabilities are lovely and helpful, respectively. The Sonys each have IS, so they should technically allow you to shoot in even lower light conditions than the Nikons would.
So, given that I feel the IQ is pretty solid for both the 35 mm and the 50mm, the question is really which focal length you feel is more useful when paired with the APS-C sized crop-sensor of the NEX series. I love the more portrait-oriented 50mm length on that sized sensor, though I find it to be a little too long to just leave on the camera for walking around and not feel like I'm wishing for something a little wider. The 35 mm has, for me, been a pretty good compromise. In a pinch if I'm looking for the same portrait type shot but only have the 35mm, I'll stand at the distance for the perspective I'd want if I had the 50 mm, and then just plan on cropping a bit later. For many uses, there are enough pixels to support this if you take care when shooting.
The 35mm is also a fair bit smaller, and that adds to the feeling of portability. It also seems to hunt a bit more on the autofocus, which is the only downside I see vs. the 50 mm. Not a deal-breaker, or even a big deal, just something I've noticed.
If you're leaning towards the 35 mm focal length, I think you'll like the 35mm sony lens. Another plug for the NEX-5 series vs. the 3 is the articulating screen, which also comes in pretty handy.
Oh, I meant in terms of overall image quality/sharpness/distortion, but it seems like they're very similar in that way?
The 3N does have an articulating screen too, but the 5's is a touch screen. On the 3N, you have to use the control wheel to navigate.