The focus rings are part of the Cocoa design standard -- they're supposed to be there.
But, if you're seeing them in more places than you used to, one possibility is that you may get them on more things if you have full keyboard access turned on. You can turn it back off by going to:
System Preferences -> Keyboard and Mouse -> Keyboard Shortcuts
and clicking the "Text boxes and lists only" option at the bottom.