The ability to remain ‘neutral’ on the oppression of others implies privilege; it’s easier not to bother with oppression when it isn’t happening to you.
An even worse response would be the reactive undermining of someone else’s oppression.
There are no counter protests at breast cancer awareness events, because the words ‘all cancers matter’ are so obvious that they need not be exclaimed.
‘Black Lives Matter’ is not a challenge to the idea that ‘All Lives Matter’, but the latter is a challenge to the former. Our reaction to systematic violence and oppression cannot be to undermine it.