I can see how the use of Black & white could be perceived as racist. Firstly, the terms are not simply descriptive; nobody, except perhaps an albino, has completely 'white' skin, and nobody has completely black skin. To turn these terms into a binary opposition somehow implies that the terms are incommensurable.
Secondly, creating binary oppositions in language, some linguistic structuralists or poststructuralists might argue, often leads to the valorization of one of the terms and the minoritization of the other, thus creating a structural bias in our language, and consequently in our perception, experience, behaviour, attitudes and whatever, which degrades the minor term, leading to a cultural bias.
Thirdly, distinguishing between 'black' and 'white' can lead to the exclusion of those who are perceived as mixed race ( this arguably includes everyone to some degree, but in some it is perhaps more noticable than others, that is if they are closer to the centre rather than edges of the spectrum). The binarization of these terms leads some people, caught in the middle, to be excluded from both terms, and possibly in consequence can lead to cultural/social exclusion, that is they may be denied cultural belonging from both 'black' and 'white' cultural groups.
Of course this all relies on the assumption that language affects our perception, experience, attitudes and behaviour within society, rather than language being a purely descriptive, transparent medium for communication. I would argue, however, that the existence of people who do not fit into either of these categories logically proves that language is not purely descriptive, and does contain inherent bias.