Both are important. Touch response delay is unnatural for our brain which is used to immediate reaction from a real-world environment. If it's too long - relative to our brain's "processing speed" - it will make the device feel unresponsive, no matter how quickly the app opens afterwards. Brain needs to see immediate reaction to user input - something changing on the display in response to the touch, even though it's maybe a simple UI animation and nothing useful to see, but as long as it's presented immediately, brain's happily fooled.
How long it takes to carry out an action is a different matter, one that the user can appreciate on a much higher level of thought, rather than the subconscious low-level sensory/motorics brain processes.
There's a close loop action-reaction servo system between our senses and our motions - that's how manage to manipulate objects in real world with a level of precision. Add an undue delay into the loop and it becomes much harder to even hold something steady or to scroll to the right part of a page etc. This delay simply isn't just a fraction of the app opening time, there's it's much more significant despite reading in tens of miliseconds. It wouldn't be enjoyable to drive a powerful car if the steering had a 100ms lag to any adjustment you made.