All men being equal I have a hard time seeing that the statement "we should work towards furthering human flurishing" results in some kind of totalitarian state. It just does not follow.
What humanists tend to do is say that we should value human flurishing since it's good for humanity and not waste time trying to get an excuse to force others to fall in line. It's a tautology, human flurishing is good for human flurishing. Wasting time debating groundings for morality, while fun at times, distracts from real problems in the world that we could be solving (or at worst leading people to defend genocide).
You can see it as long as the person agrees that human wellbeing is a good thing then they're welcome to work towards that wellbeing regardless of their motives.