I have an idea.  I am going to process the 5 minute aggregates of raw detector data I’ve stored in monthly CouchDB databases using R via Rcurl and RJSONIO.  So, even though my data is split into months physically, I can use Rcurl to pull from each of the databases, and then use RJSONIO to parse the json, then use bootstrap methods to estimate the expected value and confidence bounds, and perhaps more importantly, try to estimate outliers and unusual events.

Update, this works great.  Except it reveals that my JSON structure in CouchDB isn’t so great.  The problem is that I’m dumping JSON objects per line.  For example:

 ["1201044", "00:00:00", "Fri", "12"]:{N:8,O:0.001782, Pct:1, lanes: 5, intrvls: 10}

While that looks great on paper, and logically makes sense if you think about pulling a single record, it doesn’t work so well when you process lots of records. While RJSONIO is pretty darn good, it certainly isn’t a mind reader, and it cannot turn a list of such objects into a matrix or data frame without some help. If you just throw the results of the RCurl fetch at RJSONIO, you get the following:


> demo=fromJSON(data)
> demo$rows[1]
[[1]]
[[1]]$key
[1] "1202024" "17:35:00" "Fri" "12"

[[1]]$value
[[1]]$value$N
[1] 427

[[1]]$value$O
[1] 0.04861833

[[1]]$value$Pct
[1] 1

[[1]]$value$lanes
[1] 6

[[1]]$value$intrvls
[1] 10

In words, what that means is that the CouchDB response of {rows:[...]} is parsed as a labeled list by R, so the response is a list with one element, rows, which contains n elements each with an element key which is a list of character vectors, and another element value, which itself is a list containing several named elements N, O, Pct, lanes, intrvls. I couldn’t figure out a quick way to make R figure out that I wanted a data.frame with named entries for each of the key terms and each of the value terms (9 columns by n rows). Many more gray hairs later, I remembered about unlist and got stuff sorted. Here is my suboptimal R script for the next time I take a long break from using R and can’t remember the syntax anymore.

#parameters: month,id,fivemin
id=1202024  ## randomly chosen
fivemin="17:35"
# get every month in parallel.  RCurl is cool that way
month=c("01","02","03","04","05","06","07","08","09","10","11","12")
couchdb = "http://localhost:5984/"
db = paste("d12_2007_",month,"morehash/_design/summary/_view/fivemin?",sep="")
moreurl = paste("group=true&startkey=[\"",id,"\",\"",fivemin,":00\"]&endkey=[\"",id,"\",\"",fivemin,":01\"]",sep="")
uri=paste(couchdb,db,moreurl,sep="");  ## 12 different URIs to fetch
data = getURL(uri)
## make a list to store data temporarily on the first pass
d1=list()
for(i in 1:length(data)){
  ## parse each month in turn
  jsondata = fromJSON(data[[i]])
  ## unlist flattens the R object
  d1[[i]]=unlist(jsondata$rows)
}
## make the list of flattened R objects into a matrix
## by unlisting again, and specifying that I'm expecting 9 columns
dmatrix = matrix(data=unlist(d1),ncol=9,byrow=TRUE)
## finally, make a dataframe explicitly labeling each column as needed and converting to numeric from text
d2= data.frame(id=dmatrix[,1],
                      tod=dmatrix[,2],
                      dow=dmatrix[,3],
                      dom=as.numeric(dmatrix[,4]),
                      N=as.numeric(dmatrix[,5]),
                      O=as.numeric(dmatrix[,6]),
                      pct=as.numeric(dmatrix[,7]),
                      lanes=as.numeric(dmatrix[,8]),
                      intervals=as.numeric(dmatrix[,9]))

Next up is the actual bootstrapping of interesting statistics.

… except for nukes and bocci.

I can *almost* make bootstrapping work, but not entirely within couchdb.  I am going to have to do external processing.  Which is probably fine. 

Anyway, here’s where I am so far.  I am loading up one database per detector, with documents that look like:

{
   "_id": "40130160",
   "_rev": "1-1446962830",
   "CV_OCC_1": "0.29665",
   "LAG1_OCC_1": "-0.02416",
   "CORR_OCC_1M X LAG1_OCC_R": "0.07549",
   "Severity--PDO": "0.46376",
   "LAG1_OCC_R": "0.15293",
   "left lane accident": "0.055",
   "2 veh accident": "0.22823",
   "vdsid": "1203692",
   "SD_VOL_M": "3.16066",
   "EstimateTime": "2008-02-24T18:00:00-0800",
   "CORR_VOL_1R": "0.45782",
   "month": "02",
   "1 veh accident": "0.0597",
   "day": "Sun",
   "CV_OCC_R": "0.47489",
   "CORR_VOL_1M": "0.4268",
   "MU_VOL_M": "13.60",
   "CV_OCC_M": "0.26119",
   "LAG1_VOL_M": "-0.07773",
   "CORR_OCC_1R": "0.21826",
   "IntervalSeconds": "1200",
   "MU_VOL_R": "11.625",
   "CORR_VOLOCC_1M": "0.22346",
   "VOL_M": "15.00",
   "LAG1_OCC_M": "-0.00962",
   "VOL_1": "20.00",
   "OCC_1": "0.12778",
   "CV_VOLOCC_1 X CORR_VOLOCC_1M": "0.01314",
   "Severity--Injury": "0.10563",
   "OCC_M": "0.11444",
   "fiveminute": "18:00:00",
   "MU_VOL_1": "13.20",
   "CORR_OCC_1M": "0.4936",
   "CV_VOLOCC_1": "0.05882",
   "SumVol": "1,537.00",
   "CORR_VOLOCC_1R": "0.28207",
   "SD_VOL_R": "3.05243",
   "OCC_R": "0.14",
   "CV_VOLOCC_R": "0.21446",
   "off road accident": "0.0949",
   "CORR_OCC_1M X MU_VOL_M": "6.71293",
   "MuVolocc": "352.40435",
   "LAG1_VOL_R": "-0.14296",
   "VOL_R": "11.00",
   "interior lanes accident": "0.14903",
   "CORR_VOL_MR": "0.41195",
   "CORR_VOLOCC_MR": "0.21033",
   "LAG1_VOL_1": "-0.01163",
   "CORR_OCC_MR": "0.31359",
   "3+ veh accident": "0.09185",
   "CORR_OCC_1M X SD_VOL_R": "1.50667",
   "year": "2008",
   "CV_VOLOCC_M": "0.06651",
   "right lane accident": "0.08891",
   "any accident": "0.38626",
   "SD_VOL_1": "3.59629"
}

Then I have a view with the following map

function(doc) {
    if(doc.year){
	var name="any accident";
	emit(doc._id, doc[name] - 0);
    }
}

and a reduce that is more or less  same as the knuthian mean and variance that I wrote up in an earlier post.  My idea was to do bootstrap sampling by just using the POST {“keys”: ["key1", "key2", ...]} call documented on the http view api page.  But it doesn’t work, or rather, it works, but the API requires group=true.  So what I get  out is something like:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=true' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"rows":[
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}},
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}}
]}

If I don’t call it with group=true, I get an error:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=false' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"error":"query_parse_error","reason":"Multi-key fetches for a reduce view must include group=true"}

So I guess that means if I stick with this approach, I will need to ditch the reduce entirely, and do processing in an external program.

I haven’t yet tried my daily average approach, where a single document contains an entire day.  I don’t expect that m out of n sampling will work, at least not with a random number generator in there, as there is that requirement in the couchdb docs that a view always produce the same output given the same input.  But a balanced approach should work, as long as the permutation process is “pseudo-random” and repeatable for the day.  (Pick any normal number and use that).

Closer, but still not yet there using bootstrap sampling in Couchdb.   My prior post was mostly thinking out loud.  I’ve tried some things since, and this post is an attempt to organize my thoughts on the topic.

The first thing I tried was to submit a list of document ids to a view, and see what happened.  This might work, and it might not.  It certainly won’t work as I expected.  That is, I have to do a very flat view—the map has to emit doc._id, value, and the reduce has to compute the statistic of interest over all of the input values.  I haven’t tried this yet, but my guess is that this will simply requery the reduce view with all of the input values.  So not time is saved by CouchDB’s caching of views.

Another approach is to put a random sequence in the view and sample from that.  The problem there is that I need to recompute the view everytime.  Using external programs, I will have to query the db for the list of docids, sample those with replacement to build my bootstrap sample, then create a view and submit it to the one-off view processor.  Given that the view can’t be cached anyway, the performance hit for this approach will always be paid, so no big deal not having a cached view.  Still, it would be nice to not have to rewrite the view every time I want to use it just because the database has grown.

Another approach that I am thinking about now is to save my data differently.  Instead of saving as one document the output of the next observation’s computations, instead collect those results into an entire day’s worth of data, and stuff the db with that.  Unfortunately, I’ll also have to rewrite my java code, as at the moment I am grabbing a few hours of time across all detectors.  Instead I’ll need to grab a day’s data across a single detector.

With this method, I *think* I can implement a bootstrap sampling with replacement inside of the javascript.  The rules of the view engine are preserved—each document is independent of every other, and no reliance on other documents in the database to process a single document.  The map part will sample with replacement from the observations for that day, and then emit as many replicates as I want for the day.

Which brings up another topic I am still unsure about.  Most of the bootstrap references talk about sampling n times from an original sample of size n.  That is, if it is 1,000 observations, each sample has 1,000 observations.  There is some discussion in Chernick’s book on about p 178 or so about using m out of n sampling, that is, drawing a smaller sample than n.  The rule is pretty vague, something about m being on the same order as n, but increasing at a smaller rate than n.  So as n goes to infinity, so does m, but m/n goes to zero.  That is really broad, and I need to get a better source and/or try it out for myself.  Anyway, it seems like log(n) would fit this rule, but would be a terribly small sample.

The point of using the m of n sample is to reduce the impact of outliers or a fat tail.  I do have outliers in my data, so it makes sense to use it.  I guess the best solution is to test it versus, say, the balanced sampling approach (b random permutations of the n observations sampled b times), and then inspect the differences in the resulting bias and variance for both estimates.

Inspired by this post, I am playing around with implementing bootstrapping various statistics as a view in couchdb.  I am not a statistician, so my definition should not be used as gospel, but bootstrapping is a statistical method where one randomly samples from an observed set of data in order to determine some statistics, such as the mean or the median.  Most of the older sources I’ve read talk about using it for small to medium sized data sets, etc., and so the k samples are all of size n.  But I can’t do that—my input data is too big.  So I have to pick a smaller n.  So I’m going with 1,000 for starters, and repeat the draw 10,000 times.

(There’s probably a secondary bootstrap I can do there to decide on the optimal size of the bootstrap sample, but I’m not going to dive into that yet.)

So how to draw 1,000 random samples/documents from a CouchDB database and repeat that process 10,000 times efficiently?    My first inclination was to use the map part of the view, but I suspect that this will fail.  My reading of the docs is that themap must produce a consistent output given the same input. So if the map includes a hard-coded list of samples, great.  But I’m not so sure the map function is allowed to produce this random list on invocation—then the map would not be consistent between runs, which would violate all kinds of assumptions I’m sure.

So the next approach would be to sample all the data, but pull a random subsample with replacement when querying the view.  So in this case, all of the data is used to generate the statistic in question (mean, median, whatever), plus confidence bounds, and then the query does the sampling with replacement.

To do this, my query program (it’d have to be programmatic) would first ping the db and get the docids, then randomly draw from those with replacement k times.  Each combination of ids would then get sent to the db using the post  {“keys”: ["key1", "key2", ...]} api  semantic.

But that might not work for two reasons.  First, I’m not so sure that the post will allow duplicate keys, although I guess this is easy enough to test.  Second, this still requires the complete computation of the map/reduce view function over the entire data set.  I already know this takes a long time and uses a lot of space, so I don’t really want to do that if I don’t have to.  Still, this approach seems pretty good.

The last option I’m thinking about is to do the sampling in the reduce function, not in the map.  There was a response to one of my questions on the mailing list that if a  view is byte-identical to another view, then it is only computed once.  So with that in mind, I could write multiple reduce functions paired to identical map functions that subsampled the available docs in different ways.  But I think this will just turn out to be slower in the end than putting the hardcoded sampling scheme into the view function. The reason I say this is because the map is compiled with every application, whereas the view is just compiled once.  So parsing and loading a long hard-coded list of samples to keep (with possible multiple samples) is going to more time because it is done multiple times.  And I’d need to either keep 10,000 different reduce functions, all with a unique sampling list, or else generate one monster reduce function that simultaneously computes output on 10,000 possible samples.  I see the input set for the reduce job getting sent down each pipe in the matrix, and either being kept, processed, or processed multiple times on its way out the other end, with the final reduce step producing the final 10,000 statistic(s) and confidence bounds.   I’m thinking of a bit-mask index matrix like in R, but maybe it would be more efficient to make it a hash map, with an integer value representing the number of times to include the doc (because resampling means there can be multiples and all that).

The final option that I am *not* going to consider is to use a temporary view.  From the couchdb wiki: “Temporary views are not stored in the database, but rather executed on demand.”   So at first this sounds pretty good, but I think it is misleading.  I’d expect that in a proper bootstrap sampling, most of the observations will get sampled at least once.  In that kind of a situation, I think the second option (building a simple statistic map/reduce view and then sampling it on query) is best, because the computations are cached.  Assuming of course that the map part is non trivial, and the reduce part is relatively fast.

Anyway, plenty to work on this weekend…as always, real tests of these ideas will reveal the truth.

Update:  I just tried some things out, and getting clued in a little bit more.  The keys for the post are keys emitted by the map/reduce output, *not* the input documents.  My current map/reduce view stratifies by time of day, risk prediction as the key.  If I instead want random draws over all observations, my output is going to have to be keyed by the document_ids.  That is pretty unworkable, I think, unless I have separate views for each time period of interest, each risk prediction, and emit keys that are the doc id.  Again, I’ll have to try it and see.  But option 1 is moving up as the most likely anyway, especially as I just read  today about balanced resampling to reduce variance in bootstrap methods in Chernick’s 2008 book, Bootstrap Methods.

It takes a finite amount of time to process loop data into my database, and the results take up a finite amount of space.  So no matter what, if I process and save results, it will take time and space.  We’ve ordered a faster, bigger machine, and that will help speed things up and make space less of an issue, but there are more loop detectors to process.

So the presumption is that it is actually *worth* the time and space to compute and store the data.  This isn’t necessarily the case.  In fact, what I really want access to are the long-term averages of the accident risk values over time.  Going forward, I always want to keep around a little bit of data, but the primary use case is to compare historical averages (sliced and diced in various ways) to the current values.

The problem is that it is difficult to maintain historical trends without keeping the data handy.  As I’ve said in prior postings and in my notes, I really like how CouchDB’s map reduce approach allows the generation of different layers of statistics.  By emitting an array as the key, and a predicted risk quantity as the value, the reduce function that computes mean and variance will be run for a cascading tree of the keys.   So just by writing a map with a key like [loop_id,month,day, 15_minute_period], I can ask for averages over all data, over just a single loop, over a loop for a month, over a loop for a month for a particular Monday, etc etc.

On the other hand, this is limiting.  If I change my mind and want to aggregate over days but without splitting out months, or if I want to put a year field in there to evaluate annual variations, I can’t.  I have to rewrite the map, perhaps using the same view, and the whole shebang has to be recomputed—not trivial when the input set is about 15G per week.

As CouchDB matures, perhaps it will do a faster job computing views.  The approach is certainly there to parallelize the computations, but at the moment I only see a single process thrashing through the calculations.

Finally, if I delete old data, it isn’t clear to me how I would still maintain the running computations of mean and variance.  Technically it is possible—all you have to do is combine partial compuations, knowing the number of observations that fed into each one.  But practically, I have a feeling that when I delete input data, the output will get blown away.

Perhaps the best approach is to maintain couchdb for just a day’s worth of data, and run a separate postgresql process to store the map reduce output.  Then as couchdb matures, I can eventually store longer and longer time periods, but at all times I have a record of past history.

I think a table storing 5 minute-rounded timestamp, loop id, as the key, and all the different mean, variance, and count values for all of the different risk predictions would be good.  This would then feed higher level aggregation tables (like day, year, and so on).  By keeping the 5 minute mean and variance, I can compute any other variance pretty quickly (average across all loops, average for that day, average for a year of that loop and 5 minute period, etc).

I can’t seem to get an efficient setup going for storing loop data in couchdb.  On the surface it seems pretty simple—every loop is independent of every other loop, so every observation can be a document.  But for this application this is more limiting than I first thought.  The problem is that after storing just a few days worth of data, the single couchdb database expands to 35GB.  I tried running my carefully crafted map/reduce to get mean and variance stats, and the process went out to lunch for days before I killed it off.

So one db doesn’t work so well.  There was a posting to the user mailing list asking how many databases people were using.  So I gave that a shot—trying one db per loop, with the same map/reduce.  The downside is that I can’t compute averages across loops as before, but that’s okay because I couldn’t get the view generation to finish at all.

But I still have problems.  The main couchdb process (beam in top) runs up to 80 or 90 percent CPU usage, and there are lots of javascript child processes split off, but the view computation is still super slow, even on just a few days of data (14 GB uncompressed, including view cache)

I’m thinking that the only way to avoid this problem is to keep updating the view with every insert into the database.  But I’m worried that will fall behind real time, let alone allow me to move backwards and process last year’s data too.  Without any super quantitative measures, it seems from my experience that if I get about a gigabyte behind the curve on computing the cached view, I can’t keep up—data loading goes too fast, and index processing never finishes.   Or I get mystery errors like:

at /usr/lib/perl5/site_perl/5.8.8/i486-linux-thread-multi/Coro.pm line 419
[
  {
    Reason => "Connection timed out",
    Status => 599,
    URL => "http://localhost:5984/safetydb1204650/_view/riskstats2/All",
  },
  undef,
] at ping_couchdbs.pl line 61

from my program that pings the views for all of the databases.

So maybe for now I need to go back to postgresql.  I do like the map reduce part of Couchdb, and I do like the unstructured doc format, but perhaps it isn’t so good for massive number crunching yet. But not being able to get a year of data in and a valid annual average out is kind of a show stopper.

But to be fair, I couldn’t do that in Postgresql either.  My use of Couchdb may help there, as I now have a document centric view of the data, rather than a relational view.  And before I get off the couch (sorry), I still need to look into alternate view servers.  Maybe I can make a view server in C that can run faster than the javascript map/reduce process.   But that will have to wait until next weekend probably while I finish other projects.

Started up a new project recently to estimate traffic flows.  Our first question is to extract truck traffic estimates from those estimates.  For something that costs so much money and is such a large part of the economy, it always surprises me how little accurate information is collected about traffic.  While freeways have reasonable coverage in California, streets are, for the most part, only monitored using periodic census counts.  Doing better than that costs lots of money, and collecting better data probably won’t make things a heck of a lot better, and *will* expose more about people’s trips (speed, origin, destination), so nothing will likely happen.

Also, while I’m not at all trolling for blog comments, I find it funny that people get worked up about automatic speed traps and so on.  Speed limits are usually set at prevailing, safe speeds.  If people are exceeding the speed limit, they are breaking the law and should be ticketed.  If they have a problem with that, they should lobby their elected representatives to get the speed limits raised.   As a bicycler, I’m usually put in the most danger not by fast moving cars, but rather by people who are exceeding the speed limit.  Those men and women are moving too fast given the conditions on the road to see and react to my bike.  I cycle pretty defensively, so it hasn’t been a problem yet.  Biking on a road with a fast speed limit usually isn’t a problem because that means the road is wide and sight lines are good.

I’m not saying that it isn’t freaky to bicycle on a road with a speed limit of 50mph.  I’m just saying that it is much safer than riding on a quite side street by my house (speed limits 25 to 30mph) and having some soccer mom come roaring along at 45mph.

I say track all traffic via GPS in license plates, and mail out speeding tickets on an annual basis when you get your license renewed.

A very nice article on Autonet by Anna Lynn from Calit2 just got posted up today.   I guess if we want to get more funding on this project, the time to strike is now.

Trevor’s Autonet paper finally got published, and we’ve gotten a small bit of press.  Funny how that works.  Do research and build a prototype.  Write a paper or two or four, apparently get no interest.  Project mostly trickles off.  Then one paper finally gets published by a slower journal, and hey, everybody is interested.

While the ideas are good, and while Trevor and his team did a great job with the prototype and got a working system running, I think the real barrier to something like Autonet taking off is the difficulty in getting  a local area wireless connection up and running.  Not from a technical, bit/bytes/hand-off/Doppler-shift point of view.  Rather from a non-technical user’s point of view.  It is quite difficult to set up a device so that it both blabs and listens on some open wireless channel without requiring careful attention from the user.  Most wifi links, in contrast, are pretty simple to use because there is a defined server and client. But even then most dialogs ask the user to select which host to access, and some require some sort of password or access code.

In the intervening years between working on that stuff and where we are now, we’ve sort of come to the conclusion that the data channel isn’t as important as just freeing the information from the automobile.  From the person traveling, really.

The primary advantage of a local area wireless connection is that, well, those cars and devices you can talk probably have data that are relevant to you too, because you’re all sitting in the same spot.  The local area wireless link acts like a spatial query on the huge mountain of traffic data that is available.  The disadvantage is the need to configure your wireless device in a secure, user friendly way, and needing to develop some sort of protocol to query distant locations.

On the other hand, a cellular link does not have automatic spatial query on the data.  Of course you can *do* a spatial query, but that costs some cpu cycles, whereas with the Autonet idea, you’re *only* querying geographically proximate neighbors.  You’ve also got the problem that the wide area wireless links cost money to use.  Cellphone companies are known to charge outrageous rates for data transfer, and in fact, AT&T specifically forbids using their data connection in the manner in which we would *like* to use it.  To quote from their service agreement terms and conditions:

Prohibited and Permissible Uses: Except as may otherwise be specifically permitted or prohibited for select data plans, data sessions may be conducted only for the following purposes: (i) Internet browsing; (ii) email; and (iii) intranet access. …[T]here are certain uses that cause extreme network capacity issues and interference with the network and are therefore prohibited. Examples of prohibited uses include, without limitation, the following: (i) server devices or host computer applications, including, but not limited to, Web camera posts or broadcasts, automatic data feeds, automated machine-to-machine connections or peer-to-peer (P2P) file sharing; …

So, an app that automatically uploads location and speed and queries traffic conditions every few seconds is out, but an application that “browses the internet” is okay.   So an application that responds to user input to “browse” the internet with a heartbeat ping is probably okay, but making it a daemon that bleeps every few minutes is not.

Gotta get us some iPhones so we can test this stuff out, I guess.  Which means we have to get funding.

Update:  I posted up a cleaner version of this to the CouchDB wiki at http://wiki.apache.org/couchdb/View_Snippets

So.  I need to compute the standard deviation.  I didn’t trust jchris’ couchdb reduce example, so I decided to dig through google and find (again) the accepted on-line way to compute standard deviation (and other moments).

All in all  a pretty interesting search.  There is a great, free programming book available from MIT.  All about lists.  That didn’t help.  Then I found the above referenced example in the couchdb mailing lists, as well as another that pointed to a java library.  I looked at that, and noted that it cited Knuth’s art of computer programming.  So that was good.  Then I did another google search and eventually looked at the Wikipedia entry (which was typically pretty sloppy) but which atypically had a decent reference or two.  Both of those references were on-line too, so I eventually worked up two algorithms to compute the second moment.

I did two, because the output should match, no?  and it does.  So they must be correct!  (Good old CRA data management strategies coming into play there).

First the raw reduce code, cut from the Futon window:

function (keys, values, rereduce) {

    // algorithm for on-line computation of moments from
//
//    Tony F. Chan, Gene H. Golub, and Randall J. LeVeque: "Updating
//    Formulae and a Pairwise Algorithm for Computing Sample
//    Variances." Technical Report STAN-CS-79-773, Department of
//    Computer Science, Stanford University, November 1979.
    // url: ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/79/773/CS-TR-79-773.pdf
    // so there is some wierdness in that the original was Fortran, index from 1,
    // and lots of arrays (no lists, no hash tables)

    // also consulted http://people.xiph.org/~tterribe/notes/homs.html
    // and http://www.jstor.org/stable/2683386
    // and (ick!) the wikipedia description of Knuth's algorithm
    // to clarify what was going on with http://www.slamb.org/svn/repos/trunk/projects/common/src/java/org/slamb/common/stats/Sample.java

    function combine_S(current,existing,key){
	if(!key){key='risk';}
	var NS=current.S;
	var NSum=current.Sum;
	var M = existing.M;
	if(!M){M=0;}
	if(M>0){
	    var diff =
		((current.M * existing.Sum / existing.M) - current.Sum );

	    NS += existing.S + existing.M*diff*diff/(current.M * (current.M+existing.M) );
	    NSum += existing.Sum ;
	}
	return {'S':NS,'Sum':NSum, 'M': current.M+M };
    }

    function pairwise_update (values, M, Sum, S, key){
	if(!key){key='risk';}
	if(!Sum){Sum = 0; S = 0; M=0;}
	if(!S){Sum = 0; S = 0; M=0;}
	if(!M){Sum = 0; S = 0; M=0;}
	var T;
	var stack_ptr=1;
	var N = values.length;
	var half = Math.floor(N/2);
	var NSum;
	var NS ;
	var SumA=[];
	var SA=[];
	var Terms=[];
	Terms[0]=0;
	if(N == 1){
	    Nsum=values[0][key];
	    Ns=0;
	}else if(N > 1){
	    // loop over the data pairwise
	    for(var i = 0; i < half; i++){
		SumA[stack_ptr]=values[2*i+1][key] + values[2*i][key];
		var diff = values[2*i + 1][key] - values[2*i][key] ;
		SA[stack_ptr]=( diff * diff ) / 2;
		Terms[stack_ptr]=2;
		while( Terms[stack_ptr] == Terms[stack_ptr-1]){
		    // combine the top two elements in storage, as
		    // they have equal numbers of support terms.  this
		    // should happen for powers of two (2, 4, 8, etc).
		    // Everything else gets cleaned up below
		    stack_ptr--;
		    Terms[stack_ptr]*=2;
		    var diff = SumA[stack_ptr] - SumA[stack_ptr+1];
		    SA[stack_ptr]=  SA[stack_ptr] + SA[stack_ptr+1] +
			(diff * diff)/Terms[stack_ptr];
		    SumA[stack_ptr] += SumA[stack_ptr+1];
		} // repeat as needed
		stack_ptr++;
	    }
	    stack_ptr--;
	    // check if N is odd
	    if(N % 2 !=  0){
		// handle that dangling element
		stack_ptr++;
		Terms[stack_ptr]=1;
		SumA[stack_ptr]=values[N-1][key];
		SA[stack_ptr]=0;
	    }
	    T=Terms[stack_ptr];
	    NSum=SumA[stack_ptr];
	    NS= SA[stack_ptr];
	    if(stack_ptr > 1){
		// values.length is not power of two, handle remainders
		for(var i = stack_ptr-1; i>=1 ; i--){
		    var diff = Terms[i]*NSum/T-SumA[i];
		    NS = NS + SA[i] +
			( T * diff * diff )/
			(Terms[i] * (Terms[i] + T));
		    NSum += SumA[i];
		    T += Terms[i];
		}
	    }
	}
	// finally, combine NS and NSum with S and Sum
	return 	combine_S(
	    {'S':NS,'Sum':NSum, 'M': T },
	    {'S':S,'Sum':Sum, 'M': M });
    }

    var output={};
    if(!rereduce)
    {
	output = pairwise_update(values);
      var mean = values[0].risk;
      var min = values[0].risk;
      var max = values[0].risk;
      var M2 = 0;
	for(var i=1 ; i<values.length; i++){
	    var diff = (values[i].risk - mean);
            var newmean = mean +  diff / (i+1);
            M2 += diff * (values[i].risk - newmean);
            mean = newmean;
            min = Math.min(values[i].risk, min);
            max = Math.max(values[i].risk, max);
        }

	output.min=min;
	output.max=max;
	output.mean=mean;
	output.M2=M2;
	output.variance_n=M2/values.length;
	output.variance_nOtherWay=output.S/output.M;
	output.mean_OtherWay = output.Sum/output.M;
        output.n = values.length;

    } else {
	/*
           we have an existing pass, so should have multiple outputs to combine
        */
       var mean = 0;
       var min = Infinity;
       var max = -Infinity;
       var M2 = 0;
       var n = 0;
	for(var v in values){
	    output = combine_S(values[v],output);
            var newn = n + values[v].n;
            var newmean = ( n*mean + values[v].n*values[v].mean)/newn;
            min = Math.min(values[v].min, min);
            max = Math.max(values[v].max, max);
            var diff = values[v].mean - mean;
            newmean2 = mean + diff*(values[v].n/newn)
            M2 += values[v].M2 + (diff * diff * n * values[v].n / newn );
            n=newn;
            mean=newmean2;
	}
	output.min=min;
	output.max=max;
	output.mean=mean;
	output.M2=M2;
	output.variance_n=M2/n;
	output.variance_nOtherWay=output.S/output.M;
	output.mean_OtherWay = output.Sum/output.M;
	output.n = n;

    }
    // and done
    return output;
}

And there you have it.  The input as you might be able to tell from the code, is an object with “risk” as the pertinent bit of information getting reduce.  The above code is very difficult to read, but I wanted to get it up now in case I never get back to it.

My goal is to split the code into two, and time them both on really large sets of data, then use the best one and delete the other.  Another reason to get it up somewhere.

I like the non-Knuth version, even though it is pretty tortured in its use of stacks and so on, because the authors go on at length about how much better it is to use their pairwise algorithm, both numerically and from a storage point of view.  Also it lends itself to tacking on the other code for computing higher order moments.  But I think I will keep the min/max stuff from the java code, as it could be useful.

Oh, to close a loophole, the reason I didn’t trust the code from the couchdb-reduce-example is that I didn’t see immediately what I was looking for.  On closer inspection, it probably does the same thing as my code.  Hmm, yes, for completeness, I am going to plug in this third way and check if it give the same results.  And indeed it does:

{"S": 1276.8988123975391, "Sum": 1257.4497350063907, "M": 955, "min":
0.033031734767263086, "max": 6.011336961717487, "mean":
1.3167012932004087, "M2": 1276.898812397539, "variance_n":
1.3370668192644386, "variance_nOtherWay": 1.3370668192644388,
"mean_OtherWay": 1.3167012932004092, "n": 955, "stdDeviation":
1.1563160550923948, "count": 955, "total": 1257.4497350063905,
"sqrTotal": 2932.5845046149643}

As long as you believe that sqrt(1.3370668192644386) is about 1.1563160550923948.  Oh, and on rereading the rationale behind the  Chan, Golub, and  LeVeque paper they state:

This problem is sometimes avoided by use of the following textbook algorithm, so called because, unfortunately, it is often suggested in statistical textbooks:

S = Sum(x**2)  – (1/N) ( Sum(x) ) **2

[[ the equation used in the github example ]]

This rearrangement allows S to be computed with only one pass through the data, but the computation may be numerically unstable and should almost never be used in practice. This instability is particularly troublesome when S is very small.

For the record, I did not notice any problems so far on my test data, but I certainly have run into numerical stability problems in other cases on the more general data set, so I will probably stick with the other algorithms.