On LTE my phone seems to use less energy, actually, and battery life seems longer.
But if you're not in an LTE area and probably won't be in one soon, turn it off. I've found it actually reduces your battery life with all the searching it does.
I have found the LTE hardware itself is what kills the battery. I had a Motorola Xoom that was originally 3G and later factory upgraded to LTE. There was an obvious hit to the battery even with mobile data turned completely off. I doubt you'd notice much difference with an iPhone unless like mentioned you are in a very weak signal area.