Polarizer most of the time, unless it is not necessary or causes problems (wide angle sky is a problem with a polarizer).
My polarizer is the only "protection glass" on my lens. Very dusty, blowing sand or salt spray, would be the type of conditions where I would use a skylight or UV filter (if my polarizer were not installed). The conditions that DiploStrat mentioned, are a good exa,ple. Otherwise there is no point in adding another layer of glass.
Split-ND filters can be very useful, in the right situations.
Star filters and SofTar filters can be useful, for special effects. Yes, this can be done in Photoshop, but it is much faster in the field, with a filter. Take a second shot, without the filter, if you want an image that is not modified by that filter.
I own a few other specialty filters, but have not used them in years.
I would like to start working with full-ND filters, to stretch the shutter time, and still have shallow depth-of-field. Using full-ND filters can also allow the removal of moving objects from the image.