The round filters should really be avoided. It's not that they don't work, but they will interfere with your compositions by forcing you to position your horizons in the center of the frame.
(On a side note, the round filters would've potentially have been much more useful if the dark portion covered roughly 1/3 of the filter instead... and rotated just like a CPL. I'm somewhat surprised that nobody has thought of that yet!)
I use the rectangular Cokins and the filter holder. As kundalini mentioned, the holder really is a huge pain... but it can be rather useful in some scenarios, so it's nice to have. The problem is that it's such a big production pulling out the holder, screwing it on the lens, positioning the GND, checking the viewfinder, re-positioning... etc, etc. When you're working with changing light, it can feel like an eternity before the rig is set up, even if it really only takes 60 seconds to put it all together.
Nonetheless, they are indispensable for many landscape scenarios. While I agree with kundalini that "digital" GNDs can do impressive things, there are certain times when a real GND is what the situation calls for... such as when the disparity between foreground details and the sky is so great that achieving a properly exposed foreground means pushing the sky too far to be digitally recovered. In those cases, the GND really shines.
As kundalini also mentioned, it's so much easier to hold the filters up the lens rather than seating them in the holder. Just beware: I have destroyed a few Cokins by scratching them when I accidentally held them up to the lens with my metal CPL attached... a stupid mistake, but one that is unforgiving on a plastic GND. Once GNDs gets scratched, they become practically useless, producing all kinds of aberrations and blurs when they are struck by light.