Once you've added the transition, you need to let iMovie render it before it play. If you look in the timeline, you should see a thin red line slowly fill in as the transition renders. Depending on the length of the effect and your Mac's speed, it could take anything from seconds to minutes. Once the red line fills in and disappears, you'll be able to play it. Doing anything else intensive at the same time will slow it down.
Rounded edges mean that the clip starts/finishes at that point. Straight edges mean that you've edited the clip and there's a little more footage beyond the straight edge.