Going back

If you change the gesture to use 3 fingers for back and forwards, all the 3 finger gestures (launchpad etc.) change to four fingers and it does work in finder to go back. The problem is that it is backwards to safari as it doesn't use the "natural" way. So in Safari, you drag the top page to the right to go back but in finder you drag 3 fingers left to go back which is really confusing. This won't be problem if you disable "natural" scrolling, however I quite like it.

 

I only realized this after reading this thread.
Anyway, a solution could be to install BetterTouchTool. It allows you to all sorts of gestures to all sorts of actions, both systemwide and on a per application basis.
So e.g.: you can link the swipe with one finger gesture to go back and forth specifically for the Finder.
That's what I just did anyway