You mean in terms of speed or security or something? Nah, not really. I don't bother deleting them manually but I reset Safari about once a month or so and that does it for me. I find Safari gets bogged down otherwise. When I reset Safari, I also clear out the URL icon cache folder. 🙂