Not sure about the conservatives, but I’m guessing it has nothing to do with mixed races being “bad.”
This is just a guess, but perhaps white people are sick of the black community trying to co-opt him as being their exclusive representative when, in reality, he’s just as white as he is black — he “belongs” to both groups. IMHO, it’s anti-reverse-racism. 😉