The square filters slide into a holder. The "Cokin" holder system is probably the most popular "holder" (and these come in sizes). While Cokin also makes filters, the high-end filters are made by Lee or Singh-Ray. The point of the slide in is that you can control how far in you slide the filter. This is for gradient ND filters which are clear on half the filter and tinted on the other half. If you're shooting a landscape and you've got 2/3rds of the frame with foreground and only 1/3rd sky then you want to slide in the filter so that it's just darkening the exposure on the sky (just one example.)
Most people use the round thread-on filters because the whole filter is the same tint (there's no clear half). If you only own 1 ND filter then a 3-stop filter is the most common. Filters might be described based on the number of stops OR they might be described based on their "density" value. If it's described in density then what you need to know is that each "0.1" density is equal to 1/3rd of a stop on the camera. So a 3 stop filter would be listed as an ND 0.9.
A circular polarizer will cut some light but the amount of light it cuts depends on where the light source originated. You twist the filter to tune it. It typically blocks about 2 stops max. That's often not enough to make a stream or waterfall appear smooth. You can stack filters ... you could stack both an ND 0.9 with a Circular Polarizer (CPL) and that would give you up to 5 stops. Don't go crazy with stacking or you'll see vignetting in the corners and it'll degrade the optical quality. It's also possible for stacked filters to cause ghosting and odd reflections.
Circular polarizers dont' work as well in wide-angle lenses. The polarization really depends on the angle of the light source. In a wide lens the light can come from such a wide range of source angles that it'll block some angles more than others and you'll get an add shadowy striped region where blocking was heavy, surrounded by pale regions where it didn't block much at all.
B+W brand is the gold standard. Hoya Pro1 filters are also extremely good. The main thing about a good ND filter is that it should actually be color "neutral". Some budget filters block some wavelengths more than others and can alter the color in the image.
