Go to Safari Preferences and click the Privacy tab. From there, click "Remove all website data". This has a side effect of also getting rid of the cookies though.
Alternatively, you can enable the "Develop" menu by going to Safari preferences again and clicking the Advanced tab. From there, click the checkbox that says "Show develop menu in the menu bar". Now inside the develop menu, you can either disable or empty the caches.