irrespective whether the wifi is on or off or whether the screen brightness is dim or bright. the battery shouldn't be that short. i surf the net while listening to music all the time and get 4 hours at least with the old touch. the new one should have a better battery from what i know. i've also watched movies with wifi turned on and still get to finish a 3 hour movie. and wifi if it is on but you're not using safari or downloading apps shouldn't affect the battery that much. i accidentally left wifi on overnight all the time and the battery only drains slightly if not at all.
apps can't run in the background so just u know and my brightness is full all the time. it's a defect. since it's new, just go claim your warranty. my cousin got the same problem and they changed it, no questions
edit: i thougt your's is a 2g ipod. anyway, the battery shouldn't drain that fast