My sister was 13 when she started to smoke and drink, she was sent to schools for troubled teens to get her to quit, but she would just get back right into it again within a month of being home.
At age 20, my parents decided to kick her out because nothing they were doing was actually helping her, she literally got a breathe at the real world and realized without a education, she was pretty much screwed from finding work.
It wasn't long after that, she started quitting smoking and drinking, (although she enjoys a little wine now) and started to go to night classes to catch up on education.
She now has a Bachelors Degree, and working on her Masters and Bar Exam to become a Lawyer.
Give it time, your brother will probably decide it's doing more harm then good then try methods of quitting.