How are you watching the video? My CPU would drop after the end of the video; but some video processes stay alive which would keep the CPU up; and if the CPU is up so will the fan.
Is this quicktime, browser (and if so which), and do you know what kind of video?
Look in the Activity monitor to see what is using the CPU.