Not sure where you're going wrong on that. Maybe some more info will help...
If you have Show View Options open, and click on the toolbar View buttons, you'll notice that the "Always open in..." line changes to whichever View is active. If the original one is checked, when you change the view, that box will not be checked. And when you go back to the original View, it will still be checked. If you check an unchecked box, then go back to the originally set View, the box will not be checked any more. In other words, it's as if all four were actually listed, but you can only have one checked at a time.
When you click the Use as Defaults button, what happens?
Are folders within the folder you did that from changed to conform to those settings?
Are folders enclosing (containing) the folder you did that from changed also?