As with many of you, I use it all the time in iTunes but avoid it in most other applications. With most of them I have window sizes and positions specifically set to my liking and the green button would just mess things up.
In Finder it is also annoying and it's hard enough getting a window size set as default in Finder without chancing messing it up with green.
In TextEdit it does not work properly when an RTF document is viewed at >100% - it won't expand to the appropriate width, much like the problems it used to have in Finder before Panther.
In open/save dialogs I would never ever want the pane maximised and yet that's what green does. Similarly in Software Update, where it seems weird to even have green enabled.
In Microsoft Office apps it gets even more confusing. In Excel, zooming expands the window out underneath where the formatting palette usually hangs out, which is not exactly desirable. In Word it zooms to a width for 100% view - I've just realised - interestingly similar to TextEdit, and moves the window right into the top-left corner which the Office apps normally seem to avoid for some weird reason.
In Calculator it serves more of a purpose similar to in iTunes, actually changing between view modes - by that rule it would actually make more sense for green to switch to spatial no sidebar view in Finder than for the pill to do it.
In summary, I don't think I would miss it.