I'm a grad student in computer engineering. I can say that if you are not paying, a retina MBP is by far and away the better choice; that will serve you well into your college career.
Many math focused apps would run fine under OS X; such as Matlab, runs fine on any version of OS X, with xquartz installed. Macs are prevalent in math departments in most universities, and preferred probably by a majority of math majors. You can also run Windows in VM for the few engineering apps that are not OS X compatible yet.
Taking note on a tablet is awful, not worth wasting your time. Writing on a normal capacitive touch screen is an exercise in frustration, if you want accurate hand writing and not fatiguing your wrist/forearm for extended use. I have tried a variety of tablets, including a couple of version of the ipad for note taking, most of them are invariably bad, since the styli lack the variable friction that you expect from writing on paper, and most tablets have no pressure sensitivity built into the digitizer. And palm rejection is not terribly reliable on any capacitive touch screen these days.
Accurate writing on a screen usually requires a wacom digitizer built into the screen, which exists on very few tablets. The only tablet that I have found tolerable for note-taking is the Entourage edge:
http://www.mobileread.com/forums/forumdisplay.php?f=218
And they don't make them any more; I used this out of necessity for my mathematical graph theory classes (as I frequently had to produce digital copies of my notes, and most of the notes have to be hand drawn).
For most engineering classes, a rMBP will be great for note-taking. Use Daum Equation Editor or similar apps to compose complex formulae (you will get used to them pretty quickly, and it helps to give you some quick intro to Latex, which is a requirement in later engineering classes/research). The only concern would be battery life through a day, if you don't have a place to recharge in the middle of the day. Maybe an external battery pack would be a good investment if you have to go >6 hr on the go.