I think what most people object to are the pompous artists who think their art doesn't stink.
Believe me; they are in every field of creative endeavor.
And they are actually no better artists, they just think they are.
And they have invested much of their persona in making other people believe they are something special.
Whenever I encounter someone like that, I take a long hard look at their "art" and simply judge for myself whether it is any good or not.