Macbook overheats when program/s are not responding.

______________________________________________________
Have a look at Activity Monitor (Applications / Utilities /) and select All Processes and sort by CPU to see what the culprit may be.
Also check the "System Memory" tab to see what your "Page ins:", "Page outs:" and "Swap used:" are.

image below uses sorting by CPU as an example

Further reading:

Mac OS X: Reading system memory usage in Activity Monitor
Making The Most of Activity Monitor
What Is Activity Monitor (or How to Take Your Mac's Pulse)​

______________________________________________________​

You could use FORCE QUIT (CMD+OPTION+ESC) to quit the unresponsive application.

PS: The MacBook does not overheat, unless it shuts down to prevent overheating. It just gets hot, due to some process taking up CPU cycles (Activity Monitor will help you finding the culprit) and needing more power.