Apps like Calculator and Contacts come in handy rather often, but the rest, such as Clock, Compass, Notes, Stocks, Voice Memos, and Weather are pretty useless to me. For Notes & Voice Memos, I use Evernote and/or Awesome Note, and The Weather Channel for weather info.
You cant delete them, but other than putting them a folder like others suggested, you can send them past the last page where they cant be seen again. You just need to fill the last page with apps. People were doing stuff like this on OS 3. Search if what I said wasnt the exact procedure, but I know it can be done (I never bothered to do it, since it wasnt a big deal to me). Then if you want to open one of these apps that are no longer visible, you can use Spotlight.