I'd recommend using something like OmniDiskSweeper or DaisyDisk, which show you your drive sorted by size, so you can see what's taking up the space; and then make an informed decision about whether it's necessary.
Things like caches will just get refilled (and they generally serve a purpose); but things like iPhone backups, Garageband/Logic samples -- and of course your own files -- are the main contenders.