The thing is, there is a broad spectrum of beliefs and practices along which 'muslim' communities sit, and they are not all the same. From experience I have found that many traits in British Muslims are social as much as religious.
Now, if you have to deal on a daily basis with people who steal, lie, hide their identities, have multiple identities, subjugate girls and women and who are antagonistic to anyone not of their appearance / community / behaviour.....then I think you should tell it as it is. The fact that they might occasionally worship in a mosque is nether here nor there. But the first argument brought forth by these folk when arrested or challenged about their behaviour is that you are racially prejudiced against them as muslims.
Its natural to protect your own interests, be it ethnicity, disablity, gender or whatever, and the lady Baroness can't speak without a microphone being pushed in her face. She's as entitled to her opinion as anyone.