If you have opened that screen in the current flow you will eventually be back there. Task switcher doesn't do backtracking.
You'd use a combination of the two. You'd use the back button to go back in the app you're currently in (it would stop going back once you'd reached the highest level of the app) and you'd use the task switcher to switch to another app.