Yeah, that definitely irritated me the first time I tried it out. I figured it was a bug, since you can "leave" group huddles. Interesting that it's seemingly intentional.
Okay yeah I've decided the Huddle UI is absurd. AMAZING quote here from Google's Huddle settings help. "there's no way to to leave them." Spooky.