Taking it off wouldn't cause problems, they are built to be used without cases obviously. And if somebody else has scratches it dosen't mean yours will magically get scratched. You know well enough that how you treat an item depends on its wear and tear.
Treat it decently and you'll be fine. Treat it poorly and it'll get scratched.
These things don't scratch very easily though, you should be fine. Just enjoy it. I used to be an advocate of the case but I'm pretty much over them. I prefer my naked look : )