Barbara was interviewed, and the podcast will be up eventually, but so far we’ve heard nothing. I will let her know that you want more information on her book. But, that said, I’m pretty certain that she’s already dropped a copy off for Dennison Library’s collection, and I also found it in Honnold’s on-line catalog here. You could have a student of military history review the book for your magazine, or read the book yourself, or else just quote from the book jacket!

Actually, I want exactly one value! I don’t really want a multi-get, I want a multi-reduce. I want to take those m documents I specify, including the duplicates, and crunch them all through my reduce function, which computes the statistic of interest on the input. I was hoping to be able to use multi-get in this way, since for this application, mapping without reducing isn’t so helpful.

In really simple terms, bootstrapping is sampling with replacement. So for any m randomly chosen values from a set of n, there is a finite chance that some of those choices will be duplicates. I gather that’s the whole point of the bootstrap method. And that is the part I’m having the most difficulty with in CouchDB’ map/reduce.

Why is bootstrapping useful? Well, given my data, I can’t justify computing things like mean and variance in the usual way. The problem is that it is quite likely that I have lots of extreme outliers, and I’m also pretty sure my data is truncated at zero and has a really fat tail, but that’s about it. With bootstrap methods, you can get away with knowing nothing, but the price is that the method is often computer-intesive. So instead of computing the mean and variance once on the entire set once, I have to sample from the data set hundreds or thousands of times, with replacement, and compute the mean and variance for each sample, then use those to compute the bootstrap estimate of the mean and variance.

As to your second suggestion, that just might be a pretty clever idea for tackling the balanced bootstrap, where you just do B copies of the data, randomize the lot, then split it into B subsamples of size n (if I remember correctly). I’m pretty sure as soon as you emit something random, you violate that clause in the view definition that says the map must produce the same outputs given the same inputs. But if I can get around that, say by setting the seed of random each time, that might work pretty well. It also might work well in a temporary view, and frankly given the randomization and multiple sample requirements, it probably doesn’t help at all to have the view cached. But I also don’t want to break my database because some implementation detail relies on that same input same output rule. So I’ll have to test that carefully.

Also, if I gather right, you could try adding a random number in the range [0, 1] and then in have a map function of emit(doc.random_field * M % N, doc.float_value_of_interest) (You pick M and N to give the approximate number of values per group) and then do your statistics on those collections. I have no idea if the sampling math works there or not, but I don’t know that it doesn’t. :)

http://github.com/mmcdanie/erlview/tree/master

would you mind adding this to http://wiki.apache.org/couchdb/View_Snippets? It might be useful to others :)

Thanks for sharing here in any case!

Cheers
Jan
–