However, in the interest of balance, I can recognize the mirror argument here: we are stating that it is not appropriate to criminalize a behavior simply because someone might take advantage and behave criminally.
Gun control opponents tend to make similar arguments: "we are not criminals, and we do not intend to use our guns to act criminally, therefore it is not appropriate to treat us as criminals simply because someone else might use a gun to commit a crime." It's a fair point, and recognizing and responding to it is the correct response.
I think a valid counter argument is that, while no one becomes trans in the heat of the moment because they got yelled at in a bar or got cut off in traffic or got angry at their spouse, it does seem to happen that otherwise law-abiding, gun-carrying people, otherwise good people seem to have a tendency to turn into threats in those circumstances.
To paraphrase the statement: "the only thing that stops a good trans person from going to the bathroom, is an otherwise good person temporarily blinded by fear."