At stage 3 with self-driving cars

I recently wrote that self-driving cars were inevitable and would change nearly everything about our understanding of traffic flow and how the demand for travel (a person wanting to be where he or she is not) will map onto actual trips. We’re planning using the old models, which are sucky and broken, but now they are even more sucktastic and brokeriffic.

Today in the LA Times business section1 an article reports that a “watchdog” group2 is petitioning the DMV to slow down the process of adopting self-driving cars. It struck me that this act is very similar to bargaining, which means we’re at the 3rd stage of grief.

The first stage is denial. “It can never happen.” “Computers will never be able to drive a car in a city street.” Over. Done. Proven wrong.

The second stage is anger. I haven’t seen that personally, but I have seen hyperbole in attacks like “what are you going to do when a robot chooses to kill innocent children on a bus”. A cross between stage one and stage two is probably this article from The Register.

The third stage is bargaining. The linked page above has the example of “just let me see my son graduate”. In this case, we’ve got “slow down to 18 months so we can review the data and make sure it is safe”. While I’m not suggesting we rush to adopt unsafe robot cars, it is interesting to see how quickly the arguments against self-driving cars has moved to stage 3.

I’m keeping an eye out for depression (old gear-heads blaring Springsteen’s Thunder Road while tinkering with their gas guzzling V-8s?) and then acceptance (we’ve got a robot car for quick trips around town, but we also have a driver car for going camping in the mountains).

  1. The link is the best I could find right now, but is exactly the same as the print article 
  2. The group non-ironically calls itself Consumer Watchdog! 

Why is there glitter on the floor?


The light bouncing off the chair leg makes the ugly scratches in the floor sparkle like glitter.

I’ve spent many hours thinking about driverless cars, and have even drafted a few blog posts.  With the announcement the other day from Google, and the subsequent flurry of news coverage, it is time for me to join the party and get my thoughts out there.

A prediction

First, my prediction: Self-driving cars will become standard.

Continue reading

quick tests are great when documentation is thin

I have 14,000 odd items that I want to copy from PostgreSQL into CouchDB. While bulkdocs is great, 14,000 is too much. So I want to group the big array into a lot of smaller arrays.

I thought there was a simple function in [lodash]( that I could use, and remembered having used [groupBy]( in the past.

But the docs are slightly wrong. They imply that the callback function gets passed one argument, the array element, but the usual idiom for these sorts of functions is that they are passed two or three arguments: the array element, the index of the element, and the entire array.

Sure enough that is what is done:

var _ = require('lodash')
var groups = _.groupBy([4.2, 6.1, 6.4], function(num,idx,third) {
                 return idx % 2


Running this (node test.js) produces

4.2 0 [ 4.2, 6.1, 6.4 ]
6.1 1 [ 4.2, 6.1, 6.4 ]
6.4 2 [ 4.2, 6.1, 6.4 ]
{ '0': [ 4.2, 6.4 ], '1': [ 6.1 ] }

So I can group by massive array into smaller arrays by munging the index.

Dante was like Tupac

This post is totally wrong, so there. Disclaimer ahoy.

So the lovely wife came home from some nutty adult education class with some interesting but completely irrelevant facts. One of them was that Dante apparently finished the Inferno just days before he died. I think not. I think more likely he died, and his krew was trying to get up the scratch for a new stable of horses so they put together some almost finished stuff and just *claimed* that Dante finished it. If Dante had died 1996, for sure he would have been on a giant big screen at this year’s Coachella festival.

When in doubt, use async.queue()

As with many other satisfied users, my goto library for handling asynchronous processing in node.js is the excellent async library. But what works in small doses doesn’t always work for larger problems.

Specifically, a common use pattern for me is to use it to handle checking things in CouchDB. Often I’m too lazy to code up a proper bulk docs call, so I’ll just run off a bunch of queries asynchronously. This evening I was testing some such code out and it was working for test cases with 10 and 100 items, but it fell over with “double callback” errors when I loaded up 9,000+ items to the list.

The problem of course is that async really means async. When you have an array with 9,000 items in it, and you use, say, filter on it like so:

var my_array=[...]
                return null
                return null

then what is happening is that filter is firing off as many hits as it can to CouchDB, which in this case is 9000+. This breaks things, with CouchDB shutting down, my SSH tunnels blocking things, etc etc.
The plumbing has gone “higgledly piggedly”, like that old Bloom County punchline.

So instead, use async’s queue:

var filtered_tasks = []
var q = async.queue(function(task,callback){
                    // keep these
                }// drop those
                return callback()
// assign a callback for when the queue drains
q.drain = function() {
    //console.log('all items have been processed');
var tasks =
                      var task = {'options':_.clone(config)}
                      task.cell_id = k
                      task.year = year
                      return task

I chose the concurrency by playing with it. I 10 is too slow (took 25 seconds), 100 takes 9 seconds, and 1000 takes 9 seconds.

From simple examples to complicated real world cases

I have a really irritating use-case for a CouchDB view. I have several hundred million documents representing hourly data for 4km grid cells in California, and I need to group them by areas. For example, grid cell i=100, j=223 is in Mendocino County, and in the “NORTH COAST” air basin. Of course I have the geometry of the grid cells and the geometry of the counties, air basins, and so on, in PostgreSQL/PostGIS, and I usually just shoot off a query to get the relationship and I’m done. This is CouchDB, however, and views cannot rely on external information lest they become idemimpotent (I made that up). Everything that the view needs must be in the view from the start.

Fair enough, I set up the SQL queries and generated my 9,800+ row JavaScript hash lookup table that maps grid cell to various areas of interest. Now I want to mix that into the view without pulling my hair out.

There is a really simple example in the CouchDB wiki. I’ve reproduced it below:

   language: "javascript",
   whatever : {
     stringzone : "exports.string = 'plankton';",
     commonjs : {
       whynot : "exports.test = require('../stringzone')",
       upper : "exports.testing = require('./whynot').test.string.toUpperCase()"
   shows: {
     simple: "function() {return 'ok'};",
     requirey : "function() { var lib = require('whatever/commonjs/upper'); return lib.testing; };"
   views: {
     lib: { 
       foo: " = 42;" 
     test: { 
       map: "function(doc) { emit(doc._id, require('views/lib/foo').bar); }"

So where the above example says foo: " = 42;", I want to add in my massive hashtable. Obviously cutting and pasting so many lines is not the way to go. Instead, I’m using a couchapp tool.

The concept of a couchapp used to get more press that it currently seems to, but the basic idea is to use code to load up your design doc with attachments and views. In my case, I couldn’t care less about the attachments and the notion of a webapp stored and served by CouchDB. I just want to programmatically construct the view document, and push it to CouchDB. I chose to use node.couchapp.js. I could also have "rolled my own", and in fact I probably will this afternoon. I am playing around with grunt, so I used grunt_couchapp (after patching it a bit to use cookie based authentication).

The basic structure of my directory is the following

├── cellmembership.json
└── dump_membership.js
├── ...
└── ...

The config.json file contains my database details, including my username and password. package.json contains the npm dependencies, mostly containing what was pulled in by the grunt_couchapp tool, and the node_modules directory holds all the node modules. I do not have an _attachments directory, so I make sure my design doc has no attachments!

Before getting to app.js, in which the design document is defined, I will first talk about what goes into it. The lookup table is stored as a JSON object in lib/cellmembership.json. The contents looks like:

{ "100_223":{"airbasin":"NORTH COAST","bas":"NC","county":"MENDOCINO","fips":"23","airdistrict":"MENDOCINO COUNTY AQMD","dis":"MEN"},
 "100_224":{"airbasin":"NORTH COAST","bas":"NC","county":"MENDOCINO","fips":"23","airdistrict":"MENDOCINO COUNTY AQMD","dis":"MEN"},
   ... 9,890 more lines like this ...
 "304_48":{"airbasin":"SALTON SEA","bas":"SS","county":"IMPERIAL","fips":"13","airdistrict":"IMPERIAL COUNTY APCD","dis":"IMP"},
 "98_247":{"airbasin":"NORTH COAST","bas":"NC","county":"HUMBOLDT","fips":"12","airdistrict":"NORTH COAST UNIFIED AQMD","dis":"NCU"}

The view code that uses this file is saved to lib/dump_membership.js, and looks like:

module.exports = function(doc){
    var lookup = require('views/lib/cellmembership').lookup
    emit(lookup[doc.cell_id].county, doc.value)

These two pieces are put together in app.js, that looks like this:

var couchapp = require('couchapp')
var cellmembership = require('./lib/cellmembership.json')
var mapfun = require('./lib/dump_membership')

var ddoc = {
    _id: '_design/calvad',
    rewrites: [{
      from: '',
      to: 'index.html',
      method: 'GET',
      query: {}
      from: '/*',
      to: '/*'
    views: {
    lists: {},
    shows: {}

module.exports = ddoc;

So instead of ";", I put in "exports.lookup="+JSON.stringify(...). The key insight that the simple example didn’t really convey is that you want your entire "library" module to be a string. So in this case that means saving my JSON lookup document as a string using JSON.stringify. I probably could have just loaded it directly using fs.readfile(), but I like this way, because it soothes my worries about malformed JSON. If the JSON is screwed up, the app.js won’t run, and the failure happens right away, not in the midst of cranking through hundreds of millions of documents.

The other bit that I didn’t get from the example was how to include an external function in the design document. What I did was pretty simple, and it worked. I just did "map":mapfun. This is exactly the opposite of what needed to be done with the views:lib:cellmembership.. construct. There the exports.lookup= statement needs to be a string inside of the JavaScript, whereas the assignment of the map function needs to be actual JavaScript code, not the string representation of that code.

This is exactly the kind of inconsistency that drives me nuts and that nobody ever thinks to document, because only crazies like me run into those edge cases.

Dream big

Robert Longo was a hot artist the year I graduated from college, with
a show called something like “Dream Jumbo: Working the Absolute” that
included an art exhibit at LACMA and a show at UCLA. We bought
tickets and went and it was great. We copied the idea of jumping
people, not painting them quite so large, but capturing the movements
and shadows nonetheless.

A year later I was in Europe, doing the backpack Eurail thing. I had
worked for a year and saved up a little money, enough to buy a used
Minolta. Once I got into the groove of traveling, life pretty much
revolved around looking for Romanesque churches, finding cheap hotels,
and strategically choosing night trains between cities.

I went to Europe with many rolls of film, some negative, some black
and white, but mostly slides. I shot all of it, and eventually had to
buy more. To guard against disaster, I would occasionally spot a deal
at a shop and would develop a batch of exposed rolls.

My past self is envious of my current self, with digital cameras not
needing the bag full of film canisters. Then I shot and shared my
images with close friends and family; now I can shoot and post to the
internet to theoretically share with everybody. I can “develop”
pictures on my laptop, and even shoot movies with my camera.


My current self is envious of my past self, with no responsibilities
except to myself, able to go wherever and do whatever. I took
pictures, went to museums, and looked at old architecture. I played
harmonica in between cars on night trains. I watched my bank account
drain down, and got a cash advance on my credit card.

I haven’t heard anything about Robert Longo in years. He may still be
doing stuff, but I don’t care, and he’s certainly not as hot as he
once was. I take a lot more photographs now, but I don’t draw nearly
as much and I haven’t aspired to be an artist in years.

Take that, cryptic error message

Sometimes when you have a program that works fine for weeks and weeks, it still has bugs that crop up for no apparent reason. Yesterday I ran into that sort of irritating situation, but I learned some stuff and so I’m writing this up so that there is one more possible solution paired to a cryptic error message for the search engines to suck up.

The situation

I am running a geospatial modeling job to estimate variables in time and space. There are a lot of little grids to process, and each needs a model run for each hour. Continue reading

How I fire off multiple R jobs from node.js

Node.js has become my hammer of choice for most systems programming type jobs. In an earlier post I talked about how to use CouchDB to store the progress and state of jobs that need doing. Here I will demonstrate how I trigger those jobs and update CouchDB using a fairly simple node.js program.

Two key features of node that makes this program possible are spawn and being able to read and manipulate the environment variables.

var spawn = require('child_process').spawn
var env = process.env

Node.js is fully capable of using child processes. One can choose from exec, execFile, spawn, and fork. For my usage, the spawn function does exactly what I want—it creates a child process that reports back when it exits.

The other useful tool is the ability to access the current running environment using the process.env variable. This allows my program to take note of any environment variables that are already set, and to fill in any missing variables that my child process might need.

Concurrency via queue

Using spawn one can fire off as many jobs as desired. Suppose you have a machine with four cores, then calling spawn four times will efficiently use your processing power. Unfortunately it isn’t usually that simple. Instead, what typically happens is that you have a lot of separable data crunching tasks that need to be run, and you want to have four data processing jobs running at all times until the work is all done. To accomplish this, the spawn function will need to be called four times (to fill up the processors) and then will need to spawn a new job whenever one of the existing jobs finishes.

Continue reading

How I use ffmpeg in Linux to record from Macbook Pro iSight

I have an older Macbook Pro (version 5,5) and I recently got screencasting working again after about a year in which nothing worked. There are two steps to getting a usable video output. First I needed to get audio recording working properly, then I needed to get the video to grab without dropping all the frames. Once I got it working I wrote it into a tiny little script that I’ve pasted below.

Continue reading