I don't think the term "ginger" is derogatory. It is all about how it is used. For instance:
"Alright ginger, better get in the shade. You're gonna burn if you're in direct sunlight for more than 20 seconds." Perfectly acceptable. Slightly condescending, but in a playful way. Also, completely accurate.
vs.
"That ginger over there is giving me the creeps. Shouldn't she be indoors, resting up for a night of soul snatching?" Kinda crossing the line there.
*shrugs* It really gets under the skin of some. Doesn't bother me, though. It is nearly always meant in playful jest, though occasionally kids are subjected to some pretty nasty bullying for being ginger. I remember reading about "kick the ginger" day at some school. That is rare, though. Most people are secretly fascinated by us. We're like orange dalmations.