+1 on Phrasikleia's post, lots of good info. I especially agree with the starting at the bigger 4x6 filter size, it is really a lot nicer. Several key advantages of the 4x6 size IMHO:
1) Bigger filters means easier to hand hold without getting your fingers in the frame. Hand holding is the fastest and easiest way to use the filters, often I don't even bother with the holder. Also handholding is a lot easier if you are combining a polarizer and an ND filter.
2) If you have a DX system right now, buying a hard edge 4x6 can actually function as a solid ND filter if you move the transition all the way off the frame. This works all the way down to 12mm on DX with my Hitech 4x5" filters, it probably works even better with a Lee or Singh-Ray 4x6" filter that is an inch longer. This means one filter does the job of 2, making it both more cost effective and efficient to carry in the field. Double win!
3) If you ever upgrade to a bigger format like full frame, you can keep using the filters, whereas you may run into problems with the P-sized stuff
However, regarding the Cokin vs. Lee holders, I have to disagree. Having owned both, I vastly prefer the Lee holder. It is all around better built, feels more durable, and easier to work with. The mechanism by which it attaches to the lens is plenty secure (on my holder at least) and a lot faster/easier to work with in the field. That big knob means it's really easy to put on/take off (especially if you are wearing gloves), or adjust the rotation whereas the Cokin holder you are fiddling with the little tabs. Rotation setting on the Cokin doesn't seem as "secure" either (it's way easier to bump the Cokin holder and have it move vs. the Lee which is pretty solidly held in place).
Also in the wide angle department, the Lee holder (when combined with a Lee wide-angle adapter ring) is far superior as it gets the filter a lot closer to the lens, meaning less/no vignetting even with ultrawide angle lenses. The Cokin holder can be easily modified to work more like the Lee, but I still find the WA usage of the Lee to be better.
Oh and P.S. if you end up going with the Lee Big Stopper 10-stop solid ND, it is not compatible with the Cokin filter holder. At least not that I could get it to work. There is a foam lining on the Big Stopper that keeps out extraneous light leakage, and this foam lining interferes with the Cokin holder. On the Lee holder it's light-tight.
The only problem is finding the Lee holder (and those WA adapter rings) in stock. They have chronic supply problems- so if you are in a bind and need a holder right away, you may be forced to go with the Cokin Z-pro. It can sometimes be 3-4 months from placing an order to when you actually get the Lee holder in your hands. I bought mine from
www.2filter.com- if you give them a call they can tell you their current stock situation and give a good estimate of how long the wait is. I haven't followed up on availability too much recently so maybe it's a lot easier to get a hold of now I don't know. That's the whole reason I got the Cokin to start with though, I was going on a trip and needed a holder, and even though I ordered a month in advance, I still couldn't get the Lee so I had to buy the Cokin. When I got my hands on a Lee holder and WA ring though, I sold the Cokin because IMHO the Lee is simply a better designed and better functioning holder.
As for the filters themselves, as stated above I use Hitech. They are the cheapest of the "good" brands (Hitech, Lee, Singh-Ray are all "good" brands and each are fine to use). I bought a total of 4 so far, 2 and 3 stops in both hard and soft edge. Having the selection is pretty nice but you can definitely get away with less. If it was down to one filter I would choose either a 2 or 3 stop hard edge. Reason being that the hard edge can be more easily doubled up as a solid ND, and you can "soften" the edge of the filter manually by feathering the filter handheld during the exposure.
I would also highly recommend looking into the reverse grad filters. I actually think these may be on the whole more useful than the regular grads, I am wanting to pick one of them up as my next filter purchase.
Ruahrc