You have to create an autoreply Rule - there isn't an easy option to turn on/off like there is in Outlook.
Can I do an automatic reply like I could on Outlook 'out of office will return.........'?
Are you using iClould? You can setup a vacation message in this screen from Mail settings at icloud.com.
If you don't use iCloud maybe you can forward your Mail to iCloud and use this feature. This would be better than a Mail app rule because it would work even with your computer off/offline.
I'm not using icloud no.
I would have thought there would be an option for only new messages received but there is not that I can see.
I found the out of office tab in Mac Mail on 10.8 (ML) by right clicking on the "Inbox" selecting "Get Account Info" and the "Out of Office" reply is the second tab along. No need to setup rules etc.
If you use rules in Mail itself you need to run Mail 24/7 for it to work. Best way is to set it serverside (most will allow you to do this via webmail, rules set there are serverside).
In your case that would be Exchange (which also allows to set an internal (everybody in your company) and/or an external (everybody not from your company such as clients) out of office reply).