You can remove yourself from other people's circles. It is a little tedious but I have a few people I wanted to do this too....and so all you have to do it block them and then unblock them.
Apparently you don't want to have too many people you block so that is the reason to unblock them. You'll see if they circle you again and can decided if you want to block them forever but anyway. Use how you want!
Hope you all are happy, healthy and busy!!