Mine is on (outline arrow) but I have a location-based reminder set. As the post right above mentioned it is possible to figure out which app is doing it.
When you to to Location Services screen, scroll all the way to the bottom and tap System Services (if you're not looking for it you won't know it's there).
Also, mine was doing the same thing. I noticed that Setting Time Zone was causing the location Arrow to stay on in the status bar. I don't know If that's a big because I updated from iOS 5 instead of doing a fresh install.