Well you have to make sure you changed your proxy settings in Transmission first. It should be set to HTTP, 127.0.0.1:8118. Beyond that, I don't know how to check if Transmission is actually using Tor.
This is true for the peer to peer connections, but your Bittorrent client still has to make connections with the tracker, thus exposing your IP address. That's why Transmission only has an option for using a proxy to connect with the tracker.