I have an idea.  I am going to process the 5 minute aggregates of raw detector data I’ve stored in monthly CouchDB databases using R via Rcurl and RJSONIO.  So, even though my data is split into months physically, I can use Rcurl to pull from each of the databases, and then use RJSONIO to parse the json, then use bootstrap methods to estimate the expected value and confidence bounds, and perhaps more importantly, try to estimate outliers and unusual events.

Update, this works great.  Except it reveals that my JSON structure in CouchDB isn’t so great.  The problem is that I’m dumping JSON objects per line.  For example:

 ["1201044", "00:00:00", "Fri", "12"]:{N:8,O:0.001782, Pct:1, lanes: 5, intrvls: 10}

While that looks great on paper, and logically makes sense if you think about pulling a single record, it doesn’t work so well when you process lots of records. While RJSONIO is pretty darn good, it certainly isn’t a mind reader, and it cannot turn a list of such objects into a matrix or data frame without some help. If you just throw the results of the RCurl fetch at RJSONIO, you get the following:


> demo=fromJSON(data)
> demo$rows[1]
[[1]]
[[1]]$key
[1] "1202024" "17:35:00" "Fri" "12"

[[1]]$value
[[1]]$value$N
[1] 427

[[1]]$value$O
[1] 0.04861833

[[1]]$value$Pct
[1] 1

[[1]]$value$lanes
[1] 6

[[1]]$value$intrvls
[1] 10

In words, what that means is that the CouchDB response of {rows:[...]} is parsed as a labeled list by R, so the response is a list with one element, rows, which contains n elements each with an element key which is a list of character vectors, and another element value, which itself is a list containing several named elements N, O, Pct, lanes, intrvls. I couldn’t figure out a quick way to make R figure out that I wanted a data.frame with named entries for each of the key terms and each of the value terms (9 columns by n rows). Many more gray hairs later, I remembered about unlist and got stuff sorted. Here is my suboptimal R script for the next time I take a long break from using R and can’t remember the syntax anymore.

#parameters: month,id,fivemin
id=1202024  ## randomly chosen
fivemin="17:35"
# get every month in parallel.  RCurl is cool that way
month=c("01","02","03","04","05","06","07","08","09","10","11","12")
couchdb = "http://localhost:5984/"
db = paste("d12_2007_",month,"morehash/_design/summary/_view/fivemin?",sep="")
moreurl = paste("group=true&startkey=[\"",id,"\",\"",fivemin,":00\"]&endkey=[\"",id,"\",\"",fivemin,":01\"]",sep="")
uri=paste(couchdb,db,moreurl,sep="");  ## 12 different URIs to fetch
data = getURL(uri)
## make a list to store data temporarily on the first pass
d1=list()
for(i in 1:length(data)){
  ## parse each month in turn
  jsondata = fromJSON(data[[i]])
  ## unlist flattens the R object
  d1[[i]]=unlist(jsondata$rows)
}
## make the list of flattened R objects into a matrix
## by unlisting again, and specifying that I'm expecting 9 columns
dmatrix = matrix(data=unlist(d1),ncol=9,byrow=TRUE)
## finally, make a dataframe explicitly labeling each column as needed and converting to numeric from text
d2= data.frame(id=dmatrix[,1],
                      tod=dmatrix[,2],
                      dow=dmatrix[,3],
                      dom=as.numeric(dmatrix[,4]),
                      N=as.numeric(dmatrix[,5]),
                      O=as.numeric(dmatrix[,6]),
                      pct=as.numeric(dmatrix[,7]),
                      lanes=as.numeric(dmatrix[,8]),
                      intervals=as.numeric(dmatrix[,9]))

Next up is the actual bootstrapping of interesting statistics.

Well, last Friday I posted “So, slotting 4 months of data away.  I’ll check it again on Monday and see if it worked.”

It didn’t.  Actually I checked later that same day and all of my jobs had died due to recv errors.  I’ve tried lots of hacky things but nothing seems to do the trick.  From some Google searching, it seems that perhaps it is a timeout issue, but I can’t see how to modify the perl library to allow for a longer timeout.

So, I wrote a little hackity hack thing to stop writing for 5 seconds, make a new connection, and go on writing.  Now it only crashes out of the loop if that new connector also fails to write.  And I also don’t crash until I save my place in the CSV file, so I don’t repeat myself.  So I’m not getting a complete failure, but it is still super slow.

While the documentation for Tokyo Tyrant and Tokyo Cabinet is super great, it seems to be thin on documentation and use cases/examples for stuffing a lot of data into the table db at once.

Interesting probably unrelated fact.  The crashing only started when I recomputed my target bnum, and boosted it from 8 million to 480 million.

Anyway, I had time today to tweak the data load script, and also to finalize my CouchDB loading script.  Having started two jobs each, and with tokyo tyrant started first, it looks like couchdb is going to finish first (The January job is running three days completed to every one in Tokyo Tyrant job;  the March jobs are closer together, but that Tyrant job started about an hour before everything else).

I guess there is still a way for Tokyo Tyrant to win this race.  I am planning to set up a map/reduce type of view on my CouchDB datastore to collect hourly summaries of the data.  It might be that computing that view is slow, and that computing similar summaries on the Tokyo Cabinet table is faster.  We’ll see.

Just to have a recollection of this later, some notes.

setting up tokyo tyrant instances, one per month.  I expect about 4 million records a day, so that is 120 million a month, so I set bnum to 480 million, which seems insane, but worth a shot

One thing I noticed was that in shifting from one day tests to one month populate, and with the bump up of bnum from 8 million (2 times 4 million) to 480 million, I’m noticing a significant speed drop on populating the data from four simultaneous processes (one for each of 4 months).

There is write delay of course, and that may be all of it, since the files are big now.

Perhaps there is a benefit from wider tables, rather than one row per data record?  Like one row per hour of data per sensor, or one row per 5 minutes, etc?

Also, as I wrapped up my initial one-day tests, I got some random crashes on my perl script stuffing data in.  Not sure why.  Could be because I was tweaking parameters and stuff.

One final point, the size of the one day of data in tokyo cabinet is about the same as the size of one day of data in couchdb.  I was hoping to get a much bigger size advantage (smaller file).  The source data is about 100M unzipped csv file, and it balloons to 600 M with bnum set at 8 million in a table database.  Of course, it isn’t strictly the same data… I am splitting the timestamp into parts so I can do more interesting queries without a lot of work (give me an average of data on Mondays in July; Tuesdays all year; 8 am to 9 am last Wednesday, etc.

So, slotting 4 months of data away.  I’ll check it again on Monday and see if it worked.

And by the way, I’m sure I’m not the best at this because I haven’t used it much, but it is orders of magnitude faster to use the COPY command via DBIx::Class to load CSV data into PostgreSQL.  Of course, I don’t want to have all of that data sitting in my relational database, but I’m just saying…

Started testing out TokyoCabinet and TokyoTyrant last Friday, and got my initial test program running this morning.  The documentation is pretty good, but I’m still floundering about a little bit.  Not sure what parameters to pass to the b+ tree database file to make it work well for my data; not sure how to set up multiple databases for sharding; etc etc.  On the plus side, my Perl code that loads the data is running at about 50% CPU, so it is doing something rather than waiting around for writes.  On the down side, now I have to write a small program to check on the progress of those writes to make sure that I am actually writing something!

Update.  I am comparing storing in TokyoTyrant with storing in CouchDB.  CouchDB it turns out is faster for me out of the box because of the way Erlang takes advantage of the multi-core processor.  Tokyo Tyrant server just maxes out one core, and so my loading programs wait around for the server to process the data.  CouchDB, on the other hand, will use up lots more cores (I’ve seen the process go about 400% in top).  So loading a year of data with one data reading process per month simultaneously, TokyoTyrant is only up to day 6 of each month, while my CouchDB loader programs are all up to about day 14 in each month.

I’m sure there is a way to set up TokyoTyrant to use multiple CPUs, but I can’t find it yet.

Just tried out swinger.  It is cool.  But I can’t get authorization to work right using the trunk checkout of couch (0.11.blahblah_git).  Something to hack on

But I’m more interested in playing with Sammy.js.  The two application stack figures on the blog page (and in the Swinger slides) are interesting.  Take away the couchdb bit, add Sakai’s K2, and you’ve got a very similar picture.  Sure couchdb can serve the app with attachments to the _design doc, but that’s not the point.  The point is being able to stick documents into a db and then get them out again in interesting ways without having to bend over backwards on the server side.

But again, I have to play with it for a while and see what it can do.

Digesting Couchapp.

I ran through the documentation at http://wiki.github.com/jchris/couchapp/manual, and set up a test site.  I looked through the test, and saw a buncha stuff I didn’t write.  I like that and don’t like that.  I find app builders lead to cruft laying around—like I noticed jquery 1.2.6 when the latest is 1.3.2 if I remember correctly.  And from the documentation on the wiki, I didn’t really understand what all the files were.  Of course there is no data in the DB, I expected a no-op application, but I didn’t see even *why* all that stuff was there in the lib and vendor and so on.

Then I finally hit upon the actual README at http://github.com/jchris/couchapp/tree/master.  Perhaps it is just years of reading text books, but I found this page to be much more helpful.  I think I get it.  All the library stuff gets shoved into the couchdb as part of the application.  Then the couchapp glue uses macros to leverage the libraries.  Kinda like lots of other programming languages do it, but probably closest to how most HTML templating languages work.  You follow the template construct, for example,

// !json lib.templates.post

and the output is expanded according to what is found at lib.templates.post. In the test app case, I don’t have a lib.templates.post, but I do see a lib.templates.example that shows an html document.  And of course, the thing at the end of the macro rainbow can be useful too, with javascript and queries to the db, etc etc.

Apparently these macros can be used for views (map, and I assume reduce), and it says lists and shows, but I am not yet familiar with those two constructs.

As I said at the start, lots to digest.

And as a quick update, lists and shows are cool too. Very similar to how I process json now in javascript, but on the serverside, and allowing a bit more flexibility. I’m thinking pulling filenames from couchdb based on meta data in the doc, and mapping those to actual image files. I still think I should be wrapping my couch apps in Perl or Java, but there is less and less work to do by the wrapper as this project matures.

… except for nukes and bocci.

I can *almost* make bootstrapping work, but not entirely within couchdb.  I am going to have to do external processing.  Which is probably fine. 

Anyway, here’s where I am so far.  I am loading up one database per detector, with documents that look like:

{
   "_id": "40130160",
   "_rev": "1-1446962830",
   "CV_OCC_1": "0.29665",
   "LAG1_OCC_1": "-0.02416",
   "CORR_OCC_1M X LAG1_OCC_R": "0.07549",
   "Severity--PDO": "0.46376",
   "LAG1_OCC_R": "0.15293",
   "left lane accident": "0.055",
   "2 veh accident": "0.22823",
   "vdsid": "1203692",
   "SD_VOL_M": "3.16066",
   "EstimateTime": "2008-02-24T18:00:00-0800",
   "CORR_VOL_1R": "0.45782",
   "month": "02",
   "1 veh accident": "0.0597",
   "day": "Sun",
   "CV_OCC_R": "0.47489",
   "CORR_VOL_1M": "0.4268",
   "MU_VOL_M": "13.60",
   "CV_OCC_M": "0.26119",
   "LAG1_VOL_M": "-0.07773",
   "CORR_OCC_1R": "0.21826",
   "IntervalSeconds": "1200",
   "MU_VOL_R": "11.625",
   "CORR_VOLOCC_1M": "0.22346",
   "VOL_M": "15.00",
   "LAG1_OCC_M": "-0.00962",
   "VOL_1": "20.00",
   "OCC_1": "0.12778",
   "CV_VOLOCC_1 X CORR_VOLOCC_1M": "0.01314",
   "Severity--Injury": "0.10563",
   "OCC_M": "0.11444",
   "fiveminute": "18:00:00",
   "MU_VOL_1": "13.20",
   "CORR_OCC_1M": "0.4936",
   "CV_VOLOCC_1": "0.05882",
   "SumVol": "1,537.00",
   "CORR_VOLOCC_1R": "0.28207",
   "SD_VOL_R": "3.05243",
   "OCC_R": "0.14",
   "CV_VOLOCC_R": "0.21446",
   "off road accident": "0.0949",
   "CORR_OCC_1M X MU_VOL_M": "6.71293",
   "MuVolocc": "352.40435",
   "LAG1_VOL_R": "-0.14296",
   "VOL_R": "11.00",
   "interior lanes accident": "0.14903",
   "CORR_VOL_MR": "0.41195",
   "CORR_VOLOCC_MR": "0.21033",
   "LAG1_VOL_1": "-0.01163",
   "CORR_OCC_MR": "0.31359",
   "3+ veh accident": "0.09185",
   "CORR_OCC_1M X SD_VOL_R": "1.50667",
   "year": "2008",
   "CV_VOLOCC_M": "0.06651",
   "right lane accident": "0.08891",
   "any accident": "0.38626",
   "SD_VOL_1": "3.59629"
}

Then I have a view with the following map

function(doc) {
    if(doc.year){
	var name="any accident";
	emit(doc._id, doc[name] - 0);
    }
}

and a reduce that is more or less  same as the knuthian mean and variance that I wrote up in an earlier post.  My idea was to do bootstrap sampling by just using the POST {“keys”: ["key1", "key2", ...]} call documented on the http view api page.  But it doesn’t work, or rather, it works, but the API requires group=true.  So what I get  out is something like:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=true' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"rows":[
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}},
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}}
]}

If I don’t call it with group=true, I get an error:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=false' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"error":"query_parse_error","reason":"Multi-key fetches for a reduce view must include group=true"}

So I guess that means if I stick with this approach, I will need to ditch the reduce entirely, and do processing in an external program.

I haven’t yet tried my daily average approach, where a single document contains an entire day.  I don’t expect that m out of n sampling will work, at least not with a random number generator in there, as there is that requirement in the couchdb docs that a view always produce the same output given the same input.  But a balanced approach should work, as long as the permutation process is “pseudo-random” and repeatable for the day.  (Pick any normal number and use that).

Closer, but still not yet there using bootstrap sampling in Couchdb.   My prior post was mostly thinking out loud.  I’ve tried some things since, and this post is an attempt to organize my thoughts on the topic.

The first thing I tried was to submit a list of document ids to a view, and see what happened.  This might work, and it might not.  It certainly won’t work as I expected.  That is, I have to do a very flat view—the map has to emit doc._id, value, and the reduce has to compute the statistic of interest over all of the input values.  I haven’t tried this yet, but my guess is that this will simply requery the reduce view with all of the input values.  So not time is saved by CouchDB’s caching of views.

Another approach is to put a random sequence in the view and sample from that.  The problem there is that I need to recompute the view everytime.  Using external programs, I will have to query the db for the list of docids, sample those with replacement to build my bootstrap sample, then create a view and submit it to the one-off view processor.  Given that the view can’t be cached anyway, the performance hit for this approach will always be paid, so no big deal not having a cached view.  Still, it would be nice to not have to rewrite the view every time I want to use it just because the database has grown.

Another approach that I am thinking about now is to save my data differently.  Instead of saving as one document the output of the next observation’s computations, instead collect those results into an entire day’s worth of data, and stuff the db with that.  Unfortunately, I’ll also have to rewrite my java code, as at the moment I am grabbing a few hours of time across all detectors.  Instead I’ll need to grab a day’s data across a single detector.

With this method, I *think* I can implement a bootstrap sampling with replacement inside of the javascript.  The rules of the view engine are preserved—each document is independent of every other, and no reliance on other documents in the database to process a single document.  The map part will sample with replacement from the observations for that day, and then emit as many replicates as I want for the day.

Which brings up another topic I am still unsure about.  Most of the bootstrap references talk about sampling n times from an original sample of size n.  That is, if it is 1,000 observations, each sample has 1,000 observations.  There is some discussion in Chernick’s book on about p 178 or so about using m out of n sampling, that is, drawing a smaller sample than n.  The rule is pretty vague, something about m being on the same order as n, but increasing at a smaller rate than n.  So as n goes to infinity, so does m, but m/n goes to zero.  That is really broad, and I need to get a better source and/or try it out for myself.  Anyway, it seems like log(n) would fit this rule, but would be a terribly small sample.

The point of using the m of n sample is to reduce the impact of outliers or a fat tail.  I do have outliers in my data, so it makes sense to use it.  I guess the best solution is to test it versus, say, the balanced sampling approach (b random permutations of the n observations sampled b times), and then inspect the differences in the resulting bias and variance for both estimates.

Inspired by this post, I am playing around with implementing bootstrapping various statistics as a view in couchdb.  I am not a statistician, so my definition should not be used as gospel, but bootstrapping is a statistical method where one randomly samples from an observed set of data in order to determine some statistics, such as the mean or the median.  Most of the older sources I’ve read talk about using it for small to medium sized data sets, etc., and so the k samples are all of size n.  But I can’t do that—my input data is too big.  So I have to pick a smaller n.  So I’m going with 1,000 for starters, and repeat the draw 10,000 times.

(There’s probably a secondary bootstrap I can do there to decide on the optimal size of the bootstrap sample, but I’m not going to dive into that yet.)

So how to draw 1,000 random samples/documents from a CouchDB database and repeat that process 10,000 times efficiently?    My first inclination was to use the map part of the view, but I suspect that this will fail.  My reading of the docs is that themap must produce a consistent output given the same input. So if the map includes a hard-coded list of samples, great.  But I’m not so sure the map function is allowed to produce this random list on invocation—then the map would not be consistent between runs, which would violate all kinds of assumptions I’m sure.

So the next approach would be to sample all the data, but pull a random subsample with replacement when querying the view.  So in this case, all of the data is used to generate the statistic in question (mean, median, whatever), plus confidence bounds, and then the query does the sampling with replacement.

To do this, my query program (it’d have to be programmatic) would first ping the db and get the docids, then randomly draw from those with replacement k times.  Each combination of ids would then get sent to the db using the post  {“keys”: ["key1", "key2", ...]} api  semantic.

But that might not work for two reasons.  First, I’m not so sure that the post will allow duplicate keys, although I guess this is easy enough to test.  Second, this still requires the complete computation of the map/reduce view function over the entire data set.  I already know this takes a long time and uses a lot of space, so I don’t really want to do that if I don’t have to.  Still, this approach seems pretty good.

The last option I’m thinking about is to do the sampling in the reduce function, not in the map.  There was a response to one of my questions on the mailing list that if a  view is byte-identical to another view, then it is only computed once.  So with that in mind, I could write multiple reduce functions paired to identical map functions that subsampled the available docs in different ways.  But I think this will just turn out to be slower in the end than putting the hardcoded sampling scheme into the view function. The reason I say this is because the map is compiled with every application, whereas the view is just compiled once.  So parsing and loading a long hard-coded list of samples to keep (with possible multiple samples) is going to more time because it is done multiple times.  And I’d need to either keep 10,000 different reduce functions, all with a unique sampling list, or else generate one monster reduce function that simultaneously computes output on 10,000 possible samples.  I see the input set for the reduce job getting sent down each pipe in the matrix, and either being kept, processed, or processed multiple times on its way out the other end, with the final reduce step producing the final 10,000 statistic(s) and confidence bounds.   I’m thinking of a bit-mask index matrix like in R, but maybe it would be more efficient to make it a hash map, with an integer value representing the number of times to include the doc (because resampling means there can be multiples and all that).

The final option that I am *not* going to consider is to use a temporary view.  From the couchdb wiki: “Temporary views are not stored in the database, but rather executed on demand.”   So at first this sounds pretty good, but I think it is misleading.  I’d expect that in a proper bootstrap sampling, most of the observations will get sampled at least once.  In that kind of a situation, I think the second option (building a simple statistic map/reduce view and then sampling it on query) is best, because the computations are cached.  Assuming of course that the map part is non trivial, and the reduce part is relatively fast.

Anyway, plenty to work on this weekend…as always, real tests of these ideas will reveal the truth.

Update:  I just tried some things out, and getting clued in a little bit more.  The keys for the post are keys emitted by the map/reduce output, *not* the input documents.  My current map/reduce view stratifies by time of day, risk prediction as the key.  If I instead want random draws over all observations, my output is going to have to be keyed by the document_ids.  That is pretty unworkable, I think, unless I have separate views for each time period of interest, each risk prediction, and emit keys that are the doc id.  Again, I’ll have to try it and see.  But option 1 is moving up as the most likely anyway, especially as I just read  today about balanced resampling to reduce variance in bootstrap methods in Chernick’s 2008 book, Bootstrap Methods.

I can’t seem to get an efficient setup going for storing loop data in couchdb.  On the surface it seems pretty simple—every loop is independent of every other loop, so every observation can be a document.  But for this application this is more limiting than I first thought.  The problem is that after storing just a few days worth of data, the single couchdb database expands to 35GB.  I tried running my carefully crafted map/reduce to get mean and variance stats, and the process went out to lunch for days before I killed it off.

So one db doesn’t work so well.  There was a posting to the user mailing list asking how many databases people were using.  So I gave that a shot—trying one db per loop, with the same map/reduce.  The downside is that I can’t compute averages across loops as before, but that’s okay because I couldn’t get the view generation to finish at all.

But I still have problems.  The main couchdb process (beam in top) runs up to 80 or 90 percent CPU usage, and there are lots of javascript child processes split off, but the view computation is still super slow, even on just a few days of data (14 GB uncompressed, including view cache)

I’m thinking that the only way to avoid this problem is to keep updating the view with every insert into the database.  But I’m worried that will fall behind real time, let alone allow me to move backwards and process last year’s data too.  Without any super quantitative measures, it seems from my experience that if I get about a gigabyte behind the curve on computing the cached view, I can’t keep up—data loading goes too fast, and index processing never finishes.   Or I get mystery errors like:

at /usr/lib/perl5/site_perl/5.8.8/i486-linux-thread-multi/Coro.pm line 419
[
  {
    Reason => "Connection timed out",
    Status => 599,
    URL => "http://localhost:5984/safetydb1204650/_view/riskstats2/All",
  },
  undef,
] at ping_couchdbs.pl line 61

from my program that pings the views for all of the databases.

So maybe for now I need to go back to postgresql.  I do like the map reduce part of Couchdb, and I do like the unstructured doc format, but perhaps it isn’t so good for massive number crunching yet. But not being able to get a year of data in and a valid annual average out is kind of a show stopper.

But to be fair, I couldn’t do that in Postgresql either.  My use of Couchdb may help there, as I now have a document centric view of the data, rather than a relational view.  And before I get off the couch (sorry), I still need to look into alternate view servers.  Maybe I can make a view server in C that can run faster than the javascript map/reduce process.   But that will have to wait until next weekend probably while I finish other projects.