Have you deleted the cache like iGary said?
I wouldn't worry about it too much, unless you're getting beach balls. OSX's RAM management system is pretty sophisticated and confusing, but it works. It's just that it usually says it's using seemingly high chunks of it.