I learned a cool theorem, which translated very loosely into plain English sa...

btw prop 8 is limited to the discrete setting. Not sure if that helps ...

Jul 12, 2012

Yes, there's a big difference between probability distributions on an n-element set and probability distributions on, say, the real line. In the former case the entropy is bounded: it can never exceed log(n). So in this case the claim I heard, that there's no unbiased estimator of entropy with finite standard deviation, sounds strange. But in the latter case the entropy can be arbitrary large and conceivably that extra clause is worth something.

Still, this doesn't touch the fact that Montgomery-Smith and Schuermann's paper claims to have an unbiased estimator of entropy for probability distributions on a finite set! So, I need to figure out what's going on.

+Charley Crissman wrote: "But a distribution can have arbitrarily large entropy!"

Not on a finite set of a fixed size. So I don't think that argument works.

But here's something vaguely related. If you look at Proposition 9 in the appendix of Paninski's paper, you'll see he claims any nonnegatively biased estimator of entropy on an n-element set that always gives an estimate between 0 and log(n), it must always give the maximum possible estimate, log(n).

Jul 13, 2012

John Baez

+Douglas Summers-Stay wrote: "In experiments done with real coins, the heavier side of a coin doesn't land face down more often unless you spin the coin on a surface, in which case it has a big effect."

Interesting. By the way, I'm really disappointed that nobody asked me if there was really an egomaniacal Roman emperor called Pupienus Maximus who put lead on the back of the coins bearing his likeness. Did everyone think I was making this all up? Did everyone think I was not making this all up? Does everyone just use Google these days instead of asking questions?

Jul 13, 2012

Charley Crissman

+John Baez I see; I misunderstood the concept of an "estimator"; I thought it was a process that went from samples from a probability space to estimates of some value, without knowing anything about the probability space.

It actually seems that a lot of the confusion here is stemming from disagreement over what an estimator is: the Harris paper appears to show that if you know the underlying (finite) probability space, then taking a large enough sample lets you estimate the entropy of the distribution with arbitrarily small bias. Harris also notes that "If [the size of the probability space] is known, the bias can be largely removed..." but he doesn't claim the estimator is unbiased.

The Paninski paper demonstrates the impossibility of estimating the entropy given a fixed sample size. This is an estimator in the sense of the Wikipedia definition.

Now, the Montgomery-Smith and Schuermann paper does provide a simple unbiased estimator for entropy, provided one is allowed to continue sampling until all possible outcomes have been seen. But this is not an "estimator" according to the Wikipedia definition. It certainly is a method of estimating your ignorance, though.

I think the conclusion is that your statement should be revised: it actually is possible to estimate how ignorant you are, provided you know all possible outcomes and are given as long as you like to investigate your ignorance. If you don't know all possible outcomes, Montgomery-Smith and Schuermann still provide a method (their Second Estimator for Entropy), but note that it has "unusable confidence limits." I suspect that this is the situation in which your statement is true: you can't estimate your ignorance without knowing the possible outcomes (or else accepting a huge variance in your estimate).

Jul 13, 2012

John Baez+1

+Charley Crissman wrote: "I thought it was a process that went from samples from a probability space to estimates of some value, without knowing anything about the probability space."

It would be challenging to formalize that concept - an element of a not-yet-known set is not something I've heard mathematicians talk about. But it could be worthwhile in the applications to biodiversity, where the set of species is never fully known: you just know what you find. Perhaps using a huge finite set like the set of all possible DNA base pair sequences would suffice: most of these we'll never find, and we never know exactly which ones we will find until we find them.

"Now, the Montgomery-Smith and Schuermann paper does provide a simple unbiased estimator for entropy, provided one is allowed to continue sampling until all possible outcomes have been seen."

Thanks! - I'll have to reread it with focus on that point. This sounds like a subtly different concept than the usual concept of 'estimator'.

"I think the conclusion is that your statement should be revised: it actually is possible to estimate how ignorant you are, provided you know all possible outcomes and are given as long as you like to investigate your ignorance."

I definitely need to revise my statement - that's my big project for today.

I was only considering the case where all possible outcomes are known; I was taking that for granted, since it's built into the usual concept of 'estimator', but I should make it more obvious.

I didn't say there was no way to estimate your ignorance. In fact I said the 'naive' estimator of entropy converges to the right answer as your sample size goes to infinity. I said that there's no unbiased way to estimate your ignorance that doesn't have infinite standard deviation. It seems I can actually strengthen that using Paninski's theorem.... but then I'll need to say a bit more about what notion of 'estimator' is being used here!

Thanks for helping me out.

Jul 13, 2012

John Baez+2

Okay, I completely rewrote the article to fix the big mistake and also add a lot more detail.

Jul 13, 2012