Twitter’s insidious influence on my brain has me jotting things down in short phrases.  Postcards to myself.

Back from Hawaii.

Got some knitting done on Emma’s sweater (sleeve 1 is done, sleeve 2 is 80%).

Got some sun.

Got some food poisoning.

Got a clue that I definitely hurt my hip over Easter by swimming.

Got behind on my work.

So Grace is signed up for the kids rockwall class.  Hopefully she has as much fun as she had Sunday.

Strange as it may seem, Maven decided to stop working in Eclipse. I was trying to get an old project up in Eclipse to edit it to use the new Sakai K1 code, and couldn’t import it as an existing Eclipse site.  So I used the Maven import function, but had problems (it kept insisting on making 5 projects instead of one with 4 sub projects).  I was also having problems with my pom.xml files, so I decided to turn off some options with Maven in the Eclipse settings.

That was my mistake.  Something about what I did with the options was very bad, and Maven entirely disappeared from my Eclipse install.  No menu, no Window-> Preferences -> Maven category, no resolution of Maven repository, nothing.  I tried ripping out maven and reinstalling it, but no joy.

So I just deleted .eclipse from my home directory, and have to start over.

Sometimes I wonder if Eclipse is worth it.

update, the sakai app builder might be the culprit here

I remember when the Good Times email virus hoax hit my old company.  Ah those were the days.  And for the record, no one can go to Gabi unless accompanied by a member of the clan.

So stop asking.

Digesting Couchapp.

I ran through the documentation at http://wiki.github.com/jchris/couchapp/manual, and set up a test site.  I looked through the test, and saw a buncha stuff I didn’t write.  I like that and don’t like that.  I find app builders lead to cruft laying around—like I noticed jquery 1.2.6 when the latest is 1.3.2 if I remember correctly.  And from the documentation on the wiki, I didn’t really understand what all the files were.  Of course there is no data in the DB, I expected a no-op application, but I didn’t see even *why* all that stuff was there in the lib and vendor and so on.

Then I finally hit upon the actual README at http://github.com/jchris/couchapp/tree/master.  Perhaps it is just years of reading text books, but I found this page to be much more helpful.  I think I get it.  All the library stuff gets shoved into the couchdb as part of the application.  Then the couchapp glue uses macros to leverage the libraries.  Kinda like lots of other programming languages do it, but probably closest to how most HTML templating languages work.  You follow the template construct, for example,

// !json lib.templates.post

and the output is expanded according to what is found at lib.templates.post. In the test app case, I don’t have a lib.templates.post, but I do see a lib.templates.example that shows an html document.  And of course, the thing at the end of the macro rainbow can be useful too, with javascript and queries to the db, etc etc.

Apparently these macros can be used for views (map, and I assume reduce), and it says lists and shows, but I am not yet familiar with those two constructs.

As I said at the start, lots to digest.

And as a quick update, lists and shows are cool too. Very similar to how I process json now in javascript, but on the serverside, and allowing a bit more flexibility. I’m thinking pulling filenames from couchdb based on meta data in the doc, and mapping those to actual image files. I still think I should be wrapping my couch apps in Perl or Java, but there is less and less work to do by the wrapper as this project matures.

After a long hiatus from programming Sakai tools, I once again find the code base an opaque nest of terms.  Gotta get back into the Sakai way of thinking, so I’m going to write up my thoughts to make it easier the next time I take a break and get back into it.

What I want to do is properly integrate my couch glossary with Sakai.  So what I need is a java wrapper around the couch access.  I want the wrapper to accept simple jsonp calls, and emit json responses, just as the current couchdb-native glossary does.  I’m even up for serving the widget from a doc attached to the design doc, just as in couchdb-native.

So this has to be available everywhere, so it has to be a service.  I think.  Here is where sakai terminology just numbs me.  There is nothing in the Sakai confluence site (nothing recent, that is) describing how to program a simple service.  There is lots of awesome stuff up there to make writing tools easier, but I don’t want a tool.  A tool gets stuck in a site.  A site exists all by itself.  I want a service with a public stub inside Tomcat, I guess like the library?  Except library can be seen always.  I want a real webapp.  Just not a tool.

So I think that is a good start for how to code this up in Sakai.  Use the app builder to make a tool, just get rid of all the tool stuff, and pay close attention to getting in and out of the app from the web.  Make sure all access is mediated by the authorization service, and that should do it.

… except for nukes and bocci.

I can *almost* make bootstrapping work, but not entirely within couchdb.  I am going to have to do external processing.  Which is probably fine. 

Anyway, here’s where I am so far.  I am loading up one database per detector, with documents that look like:

{
   "_id": "40130160",
   "_rev": "1-1446962830",
   "CV_OCC_1": "0.29665",
   "LAG1_OCC_1": "-0.02416",
   "CORR_OCC_1M X LAG1_OCC_R": "0.07549",
   "Severity--PDO": "0.46376",
   "LAG1_OCC_R": "0.15293",
   "left lane accident": "0.055",
   "2 veh accident": "0.22823",
   "vdsid": "1203692",
   "SD_VOL_M": "3.16066",
   "EstimateTime": "2008-02-24T18:00:00-0800",
   "CORR_VOL_1R": "0.45782",
   "month": "02",
   "1 veh accident": "0.0597",
   "day": "Sun",
   "CV_OCC_R": "0.47489",
   "CORR_VOL_1M": "0.4268",
   "MU_VOL_M": "13.60",
   "CV_OCC_M": "0.26119",
   "LAG1_VOL_M": "-0.07773",
   "CORR_OCC_1R": "0.21826",
   "IntervalSeconds": "1200",
   "MU_VOL_R": "11.625",
   "CORR_VOLOCC_1M": "0.22346",
   "VOL_M": "15.00",
   "LAG1_OCC_M": "-0.00962",
   "VOL_1": "20.00",
   "OCC_1": "0.12778",
   "CV_VOLOCC_1 X CORR_VOLOCC_1M": "0.01314",
   "Severity--Injury": "0.10563",
   "OCC_M": "0.11444",
   "fiveminute": "18:00:00",
   "MU_VOL_1": "13.20",
   "CORR_OCC_1M": "0.4936",
   "CV_VOLOCC_1": "0.05882",
   "SumVol": "1,537.00",
   "CORR_VOLOCC_1R": "0.28207",
   "SD_VOL_R": "3.05243",
   "OCC_R": "0.14",
   "CV_VOLOCC_R": "0.21446",
   "off road accident": "0.0949",
   "CORR_OCC_1M X MU_VOL_M": "6.71293",
   "MuVolocc": "352.40435",
   "LAG1_VOL_R": "-0.14296",
   "VOL_R": "11.00",
   "interior lanes accident": "0.14903",
   "CORR_VOL_MR": "0.41195",
   "CORR_VOLOCC_MR": "0.21033",
   "LAG1_VOL_1": "-0.01163",
   "CORR_OCC_MR": "0.31359",
   "3+ veh accident": "0.09185",
   "CORR_OCC_1M X SD_VOL_R": "1.50667",
   "year": "2008",
   "CV_VOLOCC_M": "0.06651",
   "right lane accident": "0.08891",
   "any accident": "0.38626",
   "SD_VOL_1": "3.59629"
}

Then I have a view with the following map

function(doc) {
    if(doc.year){
	var name="any accident";
	emit(doc._id, doc[name] - 0);
    }
}

and a reduce that is more or less  same as the knuthian mean and variance that I wrote up in an earlier post.  My idea was to do bootstrap sampling by just using the POST {“keys”: ["key1", "key2", ...]} call documented on the http view api page.  But it doesn’t work, or rather, it works, but the API requires group=true.  So what I get  out is something like:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=true' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"rows":[
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}},
{"key":"40130190","value":{"M2":0,"n":1,"mean":0.32849,"min":0.32849,"max":0.32849,"variance_n":0}},
{"key":"40130191","value":{"M2":0,"n":1,"mean":0.31275,"min":0.31275,"max":0.31275,"variance_n":0}},
{"key":"40130192","value":{"M2":0,"n":1,"mean":0.31403,"min":0.31403,"max":0.31403,"variance_n":0}},
{"key":"40130193","value":{"M2":0,"n":1,"mean":0.30753,"min":0.30753,"max":0.30753,"variance_n":0}}
]}

If I don’t call it with group=true, I get an error:

curl 'http://localhost:5985/safetydb1213891/_design/Any/_view/bytime?group=false' -d '{"keys":["40130190","40130191","40130192","40130193","40130190","40130191","40130192","40130193"]}'
{"error":"query_parse_error","reason":"Multi-key fetches for a reduce view must include group=true"}

So I guess that means if I stick with this approach, I will need to ditch the reduce entirely, and do processing in an external program.

I haven’t yet tried my daily average approach, where a single document contains an entire day.  I don’t expect that m out of n sampling will work, at least not with a random number generator in there, as there is that requirement in the couchdb docs that a view always produce the same output given the same input.  But a balanced approach should work, as long as the permutation process is “pseudo-random” and repeatable for the day.  (Pick any normal number and use that).

Closer, but still not yet there using bootstrap sampling in Couchdb.   My prior post was mostly thinking out loud.  I’ve tried some things since, and this post is an attempt to organize my thoughts on the topic.

The first thing I tried was to submit a list of document ids to a view, and see what happened.  This might work, and it might not.  It certainly won’t work as I expected.  That is, I have to do a very flat view—the map has to emit doc._id, value, and the reduce has to compute the statistic of interest over all of the input values.  I haven’t tried this yet, but my guess is that this will simply requery the reduce view with all of the input values.  So not time is saved by CouchDB’s caching of views.

Another approach is to put a random sequence in the view and sample from that.  The problem there is that I need to recompute the view everytime.  Using external programs, I will have to query the db for the list of docids, sample those with replacement to build my bootstrap sample, then create a view and submit it to the one-off view processor.  Given that the view can’t be cached anyway, the performance hit for this approach will always be paid, so no big deal not having a cached view.  Still, it would be nice to not have to rewrite the view every time I want to use it just because the database has grown.

Another approach that I am thinking about now is to save my data differently.  Instead of saving as one document the output of the next observation’s computations, instead collect those results into an entire day’s worth of data, and stuff the db with that.  Unfortunately, I’ll also have to rewrite my java code, as at the moment I am grabbing a few hours of time across all detectors.  Instead I’ll need to grab a day’s data across a single detector.

With this method, I *think* I can implement a bootstrap sampling with replacement inside of the javascript.  The rules of the view engine are preserved—each document is independent of every other, and no reliance on other documents in the database to process a single document.  The map part will sample with replacement from the observations for that day, and then emit as many replicates as I want for the day.

Which brings up another topic I am still unsure about.  Most of the bootstrap references talk about sampling n times from an original sample of size n.  That is, if it is 1,000 observations, each sample has 1,000 observations.  There is some discussion in Chernick’s book on about p 178 or so about using m out of n sampling, that is, drawing a smaller sample than n.  The rule is pretty vague, something about m being on the same order as n, but increasing at a smaller rate than n.  So as n goes to infinity, so does m, but m/n goes to zero.  That is really broad, and I need to get a better source and/or try it out for myself.  Anyway, it seems like log(n) would fit this rule, but would be a terribly small sample.

The point of using the m of n sample is to reduce the impact of outliers or a fat tail.  I do have outliers in my data, so it makes sense to use it.  I guess the best solution is to test it versus, say, the balanced sampling approach (b random permutations of the n observations sampled b times), and then inspect the differences in the resulting bias and variance for both estimates.

Trevor’s Autonet paper finally got published, and we’ve gotten a small bit of press.  Funny how that works.  Do research and build a prototype.  Write a paper or two or four, apparently get no interest.  Project mostly trickles off.  Then one paper finally gets published by a slower journal, and hey, everybody is interested.

While the ideas are good, and while Trevor and his team did a great job with the prototype and got a working system running, I think the real barrier to something like Autonet taking off is the difficulty in getting  a local area wireless connection up and running.  Not from a technical, bit/bytes/hand-off/Doppler-shift point of view.  Rather from a non-technical user’s point of view.  It is quite difficult to set up a device so that it both blabs and listens on some open wireless channel without requiring careful attention from the user.  Most wifi links, in contrast, are pretty simple to use because there is a defined server and client. But even then most dialogs ask the user to select which host to access, and some require some sort of password or access code.

In the intervening years between working on that stuff and where we are now, we’ve sort of come to the conclusion that the data channel isn’t as important as just freeing the information from the automobile.  From the person traveling, really.

The primary advantage of a local area wireless connection is that, well, those cars and devices you can talk probably have data that are relevant to you too, because you’re all sitting in the same spot.  The local area wireless link acts like a spatial query on the huge mountain of traffic data that is available.  The disadvantage is the need to configure your wireless device in a secure, user friendly way, and needing to develop some sort of protocol to query distant locations.

On the other hand, a cellular link does not have automatic spatial query on the data.  Of course you can *do* a spatial query, but that costs some cpu cycles, whereas with the Autonet idea, you’re *only* querying geographically proximate neighbors.  You’ve also got the problem that the wide area wireless links cost money to use.  Cellphone companies are known to charge outrageous rates for data transfer, and in fact, AT&T specifically forbids using their data connection in the manner in which we would *like* to use it.  To quote from their service agreement terms and conditions:

Prohibited and Permissible Uses: Except as may otherwise be specifically permitted or prohibited for select data plans, data sessions may be conducted only for the following purposes: (i) Internet browsing; (ii) email; and (iii) intranet access. …[T]here are certain uses that cause extreme network capacity issues and interference with the network and are therefore prohibited. Examples of prohibited uses include, without limitation, the following: (i) server devices or host computer applications, including, but not limited to, Web camera posts or broadcasts, automatic data feeds, automated machine-to-machine connections or peer-to-peer (P2P) file sharing; …

So, an app that automatically uploads location and speed and queries traffic conditions every few seconds is out, but an application that “browses the internet” is okay.   So an application that responds to user input to “browse” the internet with a heartbeat ping is probably okay, but making it a daemon that bleeps every few minutes is not.

Gotta get us some iPhones so we can test this stuff out, I guess.  Which means we have to get funding.

Blog stats are funny.  My incomplete and possibly incorrect posting for my original diamond lace hat is by far the most popular thing I’ve written (popular being a relative term, with only like 300 views).  Looking at the stats, it is mostly one off google searches for diamond lace knitted in the round, etc. etc.  So I really will make an effort to post the actual chart that I used for my second hat, which came off without any glitches and flew off my needles in two evenings.

I’m actually working right now on a cabled hat (v2) and am taking notes on the decreases.  The cables themselves are pretty easy (using Barbara Walker’s second knitting treasury as source, and her advice that fisherman’s sweaters are vertical cable samplers—so this hat is just a cable sampler).

But first I’ve got another monster project  (scarf) to finish.  My goal is 60 rows a night, but I’ve only done 18 and then 24.  At 24 rows a night, I will finish by Christmas, but with no time to spare.