Being a Democrat with left libertarian social views, I also think all drugs should be legal.
But:
I would be terrified if any kids (or drug addicts) got their hands on my anti-panic attack medication. For me, I can stay alert and function better. But give the same class of drug, whether it's MDMA, Valium, or similar mood altering drug and then you have somebody either getting into a car crash, falling and hitting their head, or having to get their stomach pumped.
I think it's mainly anti-anxiety medications which were partly responsible for the deaths or hospitalizations of Rush Limbaugh, Britney Murphy, Anna Nicole Smith, Heath Ledger, and many others. Add alcohol or pain killers, which may or may not have been the case, and then you have a potentially dangerous drug cocktail. Two rather "safe" drugs added together can have a dangerous, even lethal effect.
That same dosage for somebody who suffers from anxiety and, let's say, serious pain from an injury, is very safe under a doctor's orders. The patient has time to ramp up to that dosage. But give it to a recreational drug user on the street and they can go into a coma or get a heart attack. But even people who are prescribed different drugs have been known to abuse them.
But what is ironic is that many people who get "strong" drugs legally have the big trouble of staying on them, thus the daily MTWThFSaSn pill boxes.