Set it to watch the folder your torrent files down into. Mine's the desktop, so I pointed it that way. Also make sure your browser default opens that program for torrents.
Did you try setting the application through the "Get Info" box? You option-click the file and in the get info panel find where it says "Open With:" after you choose the application(in this case Transmission) make sure to click the button labeled "Change all".