I agree with you. I will have to say something to my friend. I asked my mum if I should be upset about it. She told me that my first reaction when I was younger was to be hurt by what someone did or said and then not do anything about it leading to the other person thinking I was fine even though I was frustrated inside by not saying anything. And she said yes I should be upset by it.
If they're a good friend they would ideally be receptive to you (respectfully) saying that you were hurt by their comment (even if they didn't mean it to be hurtful*- pointing that out can help make people less defensive)
whether or not it was something "trivial" or something "worth getting upset over." Everyone has their own peculiar buttons that they don't like pushed.
*Your friend may not have known it would upset you. God only knows what they were thinking to say something like that. They may have been challenging you, but they may have actually thought they were giving you a compliment. (Yes, yes people CAN be that stupid!)