To clear up what might be a misconception, the iPhone always knows its current location as it updates it from various data like the Wifi access points in your area and so on. When the icon appears it just means that the service asked the iPhone to give it the latest location info that's available. That in turn means that if you remove access for one of the services, for example for sensor calibration, that doesn't mean the iPhone stops tracking its (your) location, it just means it won't give out the info to the service you have disabled.
So if you're expecting the iPhone not to track its location, that's an essential part of what it (iOS) does. I assume if you disable location data entirely at the main toggle (at which point navigation breaks obviously) then this stops, but that's not practical for most people.