The real question is why do you care whichever way people think on the matter. Surely if you know that you are neither racist nor sexist then that would be enough?
LOL, because I'm a human being? And because I see a system where racism persists far beyond what it ought to, despite the presence of affirmative action, political correctness, and all the other social policies designed to eliminate it. So I'm looking for reasons society as a whole can't just get the hell over the issue and move on with their lives.
And one of the reasons I see is that there is actually negative pressure against individuals who have already resolved the issue for themselves. I'm not the only white person I know who is wary of saying this that or the other thing in case it might be misconstrued as racist. And I know how many guys there are who walk on tiptoes about making jokes about women when there's a female within earshot, lest they bring the rabid feminists down on their heads, while women make the same jokes about men with alacrity.
So I think, at least among more liberal people, that there ought to be a more of a trend towards bluntness, instead of all the pussyfooting.