Each of us has a higher or lower level of personal tolerance for diversity depending on the circumstances. Whether we express it openly or not is a different matter. Myself I tend to xenophilia more than xenophobia. But there
have been times when I've felt a distinct fear reaction based
solely on someone's appearance: not the way they were dressed, or the expression on their face, or what they were doing at the time; just their physiognomy. Someone, I think it was Peter, has posted about this phenomenon in more detail from a functional standpoint. He might have some comments to make about this.
... Oh, and I was going to add, the least we can do is try to avoid deliberate rudeness. But the problem with that is that there's little or no consensus on what constitutes rudeness.