Don‘t be silly. Einstein couldn’t remember his own phone number. This sort of nonsense has been said since the dawn of the computer age. They used to say it about calculators, too. Free your brain up for important stuff - making decisions and problem solving. Recall of facts that are easily looked up is something with which one doesn’t need to concern oneself.
Your London taxi driver example is proof: when was the last time anyone needed to know, from memory, the best route to a random address in a big city? Even before GPS we had maps - are maps evil too?
Every engineer I know looks stuff up, yet we seem to do fine designing CPUs containing billions of transistors.
Every lawyer I know looks stuff up, yet we seem to do fine arguing in court, writing persuasive briefs, etc.
We don’t live in an age where the ability to remember every tiny little fact is of much use.