The other two answers are correct, for iMovie, Final Cut Express, and Final Cut Pro. For raw video editing, and for applying effects. Processor speed and main system memory are the two big things there. (For example, a beige G3 upgraded with 1.5 GB of RAM and a 1.8 GHz G4 should actually beat out a last-generation PowerBook G4's 1.66 GHz G4 with only 1 GB of RAM in the raw rendering of effects. In reality, I would be surprised if it did, due to other improvements in the PowerBook, but in theory it should win.)
Now if you get into rendering video effects in Shake or Motion, the better the video card, the faster it goes.