The high end square/rectangular slide-in filters are Lee filters and Singh-Ray filters. They can be fairly pricey.
I don't have the Cokin filters (I have the holder). While I have no first-hand experience with them, a camera store (that also happens to be a Cokin dealer) once told me the only complaint they sometimes hear is that whatever they're made out of seems to like to attract dust so they're harder to keep clean than other filters.
Apart from that, the performance a filter will depend on the specific filter. A good filter will do what it's supposed to do without any unintended side-effects. E.g. if you buy a gradient ND filter then you want to know that what the filter is doing is cutting the light evenly across the spectrum WITHOUT creating a color cast by slightly favoring some color wavelengths.