The basic idea is that planning models are opaque and mysterious, and really buggy and error prone. The problem isn’t the fault of the modelers or the model systems, but rather the lack of input data. Consider that a planning model first tries to model today’s world, and then tries to model the future using that same model with extrapolated conditions. There are two sources of error—the model of the present, and the extrapolation of that model into the future.

In a perfect, totalitarian state, the government would know everywhere you go, and all that information could be loaded into the model of the present. Calibration would be simple, because every vehicle is already in the model, so of course it captures reality. But even in a totalitarian, all-knowing state, predicting the future isn’t possible. Trends reverse themselves, people pick up different habits, and technology happens, changing the way we do things.

We have been watching and participating in the evolution of planning models, in particular pushing for the adoption of activity-based models over trip-based models. The big problem here is the burden of data collection, as well as the increased complexity of the model framework. Activity-based models are being incrementally adopted because they are too complicated and cost too much money to deploy.

Public Planning Models takes a different approach. Rather than trying to come up with better data collection processes and better modeling techniques, we thought it would be better to try to expose the full ugliness of current planning models to the public. This serves three purposes. First, people can see just how weak many of the fundamental assumptions in these models are. Second, everybody can take a look at the model system and suggest corrections and improvements, in the spirit of crowd-sourcing the model calibration step. And third, exposing the models and the applications of those models will give people an incentive to become more involved. That involvement can run the gamut from simply providing a few days worth of travel and activity data to the model’s input data set, to taking the model system itself and playing around with alternate planning scenarios.

Anyway, take a look at our proposal, add comments, and if you know one of the judges, put in a good word for our efforts. There are tons of submissions, and all of the ones I’ve read so far look pretty good.

As far as I understand it, the ability to run Erlang views natively is likely to be removed in the future because it does not offer any sandboxing of the content, and so the view can execute arbitrary commands on the server. So Erlang views are likely to go away.

Problem: big docs crash JSON.parse()

That said, I have a use case for Erlang views. I have a database with two really large documents. They are TopoJSON documents of a 4km grid overlaid on California. One grid is in the original projection, and the other is reprojected as best as I can manage to SRID:4326. Both files are about 3.4MB, and apparently they cause intermittent problems for the JS view engine in CouchDB. I can’t quite nail down the issue enough to file a bug report, but if the docs are in there any view can fail. If I delete the TopoJSON docs and then put them back again, views will sometimes pass, but my suspicion is that the fail is due to a very large block of docs getting sent to the view engine that includes the TopoJSON files…the RAM required to parse the JSON is more than the server has available. My guess is that the reason it works when the TopoJSON files are deleted then re-inserted is that they are processed by themselves, or with a much smaller batch of other docs.

Regardless, the issue definitely comes about due to marshalling the data from Erlang to JS and then breaking it up using JSON.parse (that is what the crash dump says). So, I put together a super simple Erlang view, and it passed just fine, while the equivalent JS view failed. Problem solved (at least until the CouchDB devs pull the trigger on stripping out Erlang views).

But now I have new problems…

While I’m good with JavaScript, I’m about as clueless with Erlang as I am helping my daughters buy ballet leotards. But I found out a couple of things that I want to write down here.

First, due to a few helpful resources on the internet, I was able to get a simple view going. A typical record looks like this:

{
   "_id": "129_160_2007-01-01 00:00",
   "_rev": "2-4f3364efb916a4979dc4efe13bc667d1",
   "geom_id": "129_160",
   "i_cell": 129,
   "j_cell": 160,
   "data": [
       "2007-01-01 00:00",
       "280",
       23153.13,
       184.23,
       458.45,
       0.03,
       73.48,
       46.72,
       5.04,
       61.33,
       13.28,
       2.03,
       52.24,
       13.08,
       52.31,
       2,
       "400338",
       "401512"
   ],
   "aadt_frac": {
       "n": 0.02204350388061379,
       "hh": 0.029391644125345406,
       "not_hh": 0.024323281333117733
   }
}

didn’t have much hope because I didn’t know what would happen if I tried to split a string on a token that wasn’t in the string
I wanted a view that would highlight the documents that had abnormally high values for aadt_frac.n. In JavaScript, that is easy, but as I said, JavaScript failed me. In Erlang, how would one access the internal members of the JSON object? The answer is below.

fun({Doc}) ->
  A = couch_util:get_value(<<"aadt_frac">>, Doc),
  case A of
    undefined ->
          ok;
    _ ->
      G = couch_util:get_value(<<"geom_id">>, Doc),
      {[{<<"n">>, N},{<<"hh">>, Hh},{<<"not_hh">>, Nhh}]} = A,
      if
         N > 0.5 -> Emit(G,N),
                    true;
         true -> true
      end
  end
end.

Apparently CouchDB provides a namespace for some Erlang utilities, and one of those will get a value out of a JSON record. That is what the first line is doing: A = couch_util:get_value(<<"aadt_frac">>, Doc) extracts the ‘aadt_frac’ value from the document and stores it in the variable A. Apparently Erlang variables need to be Capitalized, which was weird and may not be true.

I learned about case statements from the Erlang User’s Guide, with a little more help from the two examples in the CouchDB wiki page. If A is undefined, then skip the doc. If it is defined, then I have to extract its contents. The line {[{<<"n">>, N},{<<"hh">>, Hh},{<<"not_hh">>, Nhh}]} = A does this, following the advice from http://stackoverflow.com/a/2422631. The last if statement simply checks if N is greater than 0.5, and if it is, it will emit the grid’s geometry id and the value of N.

This view worked great, and pointed up about 20 or so locations and times with outrageously high values for n, hh, and not_hh. I dealt with these issues in my code, and created new versions of the documents, and then dove into some modeling in R.

My goal is to build a spatial model of these variables for places where I have no data measurements and save the estimates into the same CouchDB, filling in all the empty grid cells. Unfortunately, my first attempt failed—my code had a bug and I wrote blank entries for the aadt_frac record. Luckily I only messed up on a single month for a very small section. Unfortunately, that was still 200,000+ documents I now needed to delete.

The first step was to find these documents using a view. Javascript was out stil, but because I had learned me some Erlang, I could write the following simple view:

fun({Doc}) ->
        T = couch_util:get_value(<<"ts">>, Doc),
        case T of
            undefined ->  ok;
            _ ->
                D = couch_util:get_value(<<"data">>, Doc),
                case D of
                    undefined ->
                        Emit(1,1);
                    _ ->
                        ok
                end
        end
end.

Simple logic: if the document has a timestamp field, it is a document I want to consider (not a geometry file and not a design document), and if that file with a timestamp is missing a data field, then it needs to be deleted, so flag it by emitting something.

Now I bumped up against another problem with CouchDB…how to bulk delete documents. In SQL, I would write a query, verify that it was returning what I expected, double and triple check, and then drop those records from the table in a big query statement. In contrast, CouchDB speaks HTTP, and you must resort to DELETE requests, or POST to _bulk_docs with a list of document ids, revisions, and _delete:true fields. The view above is the first step. It allows me to gather up all of the document ids and revision numbers that I need to purge. The next step is to write code to bundle up those ids and revs, slap on the delete directive, and send the request to the CouchDB bulk_docs interface.

Ordinarily I just hack a little something out in whatever language I am using to get that job done. But I was working in R, and I hate using R for this sort of thing, so I turned to node.js, my hammer of choice these days. Because I’ve done this before, I decided to just write a self-contained package that will allow the deletion of every document that is emitted by a view. And I also wrote tests, to prove that it worked before I unleashed it on my 6GB database. The code is on github. I ran it and deleted the problem docs. Problem solved.

But now I have new problems…

In the never ending saga of many mistakes that is my life, I fixed the aadt_frac generation code, but failed to notice and fix the fact that the _id values being generated by my R were broken, or rather, non existent. A big part of my work flow is knowing that I can query documents directly by their ids, and I just stored a bunch of documents with randomly generated UUIDs for ids. So I turned to Erlang and was faced with a new problem: I couldn’t simply rely on testing for undefined, because all documents have a valid id! Instead I needed to compare that id value against something. A new skill!

My first attempts, after a bit of searching around, failed miserably. A valid id is supposed to be the concatenation of the geom_id and the timestamp, with an underscore separating the two. What I needed was a way to test if the id looked like that. First I tried a split type approach, using the string:token function:

[Icell,Jcell,Ts] = string:tokens(Id, "_")

but that failed. Actually it failed due to a syntax error higher up, but I abandoned the approach anyway…I was trying to split when I really just wanted to compare a substring, so I went with a different function.

Id = couch_util:get_value(<<"_id">>, Doc),
Gid = couch_util:get_value(<<"geom_id">>, Doc),
case string:str(Id, Gid) of
    0 -> Emit(2,1);
    _ -> Emit(3,1)
end;

This one didn’t have a syntax error, but kept failing no matter how I prodded and poked. The logs said:

** Reason for termination ==
** {function_clause,
       [{string,str,
            [<<"b71476cb16533f8570ca909cad3a02dc">>,<<"197_102">>],
            [{file,"string.erl"},{line,102}]},
        {erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,572}]},
        {erl_eval,expr,5,[{file,"erl_eval.erl"},{line,250}]},
        {couch_native_process,'-run/2-fun-0-',2,
...

“Hmm” I said to myself, “maybe these aren’t strings.” I mean, the data underneath them is a string, but that doesn’t mean Erlang thinks they are strings. So I revisited the CouchDB wiki page, and saw this snippet in the second filter:

Values = binary:split(ValuesParam, <<",">>, [global]),

Armed with that, I went poking around the Erlang docs and found binary:match, and put together this bit that worked:

Id = couch_util:get_value(<<"_id">>, Doc),
Gid = couch_util:get_value(<<"geom_id">>, Doc),
case binary:match(Id, Gid) of
    nomatch -> Emit(2,1);
    _ -> ok
end;

Once again, I had a view with lots of documents in it, so I passed it to my handy dandy node_couch_view_deleter program, and got rid of them all.

Conclusion

I’ve been telling my daughter to always write conclusions that conclude her papers. Of course I also tell her to write paragraphs with at least 5 sentences in them. Apparently I don’t follow my own advice.

The work was interesting and got me thinking whether one could treat “being green” as a life cycle choice rather than as a mode choice. In the usual mode choice context, one considers each trip and weighs the costs and benefits of each mode. In theory, each person will choose the mode that is best for her or him. It is assumed that people are perfectly rational, and any differences in choices given apparently identical attribute sets is assigned to some unobservable error. While this ignores the fact that people are pretty bad at making quick and accurate appraisals of the true costs and benefits of different modes, one could argue that even this fact is taken into account in that different people have different tolerance for spending time and effort searching for a better solution. In the end, the random differences between people are presumed to come from some known distribution, and the modeling exercise estimates the relative weights of the observables (cost of bus, cost of transit, cost of biking, cost of walking, etc). In the end, the analyst has a model that can estimate the answers to questions like how many former drivers would bike if bike travel times (or generalized cost) were decreased 1%.

Applying the idea of mode choice to my trip home today, in the mode choice modeling world my choices would consist of walking, biking, bus, and car, and most models would show that car is the hands-down winner in terms of high benefits and low costs. This is in fact true for me too—I own a car, and I’m already paying taxes and insurance on it, and it is a fast and easy way to get around. The fact that I am going to choose to bicycle home this evening would be attributed to some unobserved randomness in how I personally evaluate those costs and benefits.

This isn’t what happens at all. In reality, I have a beautiful Atlantis Rivendell sitting next to my desk in my office, and I am going to ride it home. I chose a “bicycling lifestyle” quite a while ago, perhaps not even consciously, and so I generally choose bike first unless there is some major constraint which requires me to use a car. Those constraints usually involve my wife and my children, or more accurately, when my lovely wife is out of town and I have run my children around to their after school activities. Which brings me to my real point of this post: bicycle choice is a life cycle stage for me, not a mode choice. This stage or state is not one that my wife and children are in, and so whenever I have to participate in their daily activity patterns (ballet, piano, theater, etc), I am forced out of my bicycle life cycle state, and into a car life cycle state.

At the same TRB conference, I was looking for research in the domain of active transportation, or how and why people choose more active travel modes such as bicycling and walking. This was inspired by my wife, who has a master’s degree in public health and told me that there was a growing interest in trying to use active transportation to combat serious health concerns like obesity and diabetes. In a nutshell, people drive too much, and sit in front of the television too much, and modern humans are becoming fat slugs. We didn’t evolve to be slugs, and so our bodies are not dealing well with this new set of circumstances. One way to end the cycle is to “get” people to walk or bike more when they travel.

The trick is that the car is just so great at providing a low cost, high benefit transportation solution. It is nearly impossible to identify a normal trip example in which driving loses out to walking or biking. Cars have great roads and priority access at intersections and in parking lots, have super-fast top speeds, and plenty of hauling power. So how can you expect someone who is overweight and diabetic to choose to walk to the supermarket (most likely across major roads and through a massive parking lot) to buy groceries?

In some places the “active life style” is most definitely imposed by the urban form, and driving to the market to do some shopping makes no sense at all. For example, when I lived in Boston, the markets were tiny, and even if I did own a car I certainly wouldn’t have given up a parking spot just to spend 20 minutes looking for another one. Instead, I shopped as I walked home each night as I passed different stores. If I needed vegetables, I’d swing through Beacon Hill. If I needed meats or cheeses, I’d go a block out of my way in the North End to the deli I liked. Apparently, there are cities all over the world who are taking steps to try to make their urban environments more walkable, which at the same time will make them at least slightly less drivable. I interpret this movement as trying to “get” people to be more active by trying to improve the benefits to costs ratio of active travel modes versus driving. But the fact is you can’t make every city into Boston, and just a few short miles from where I lived in the North End, driving was the norm, not walking.

After that TRB session, I think there might be a different way to approach persuading people to choose active travel modes. I think it is important to view active travel as a life cycle change. Choosing to walk or bike is going to have a drastic impact on every aspect of a person’s usual travel pattern, not just on a single trip. This means that one who currently drives cannot simply pick any random trip in any random day and choose to perform that trip by bike or on foot. Rather, the person will need to rearrange many activities in order to allow for the time and inconvenience of the non-motorized mode. These secondary impacts are not trivial, and work against the active transportation choices. Perhaps by viewing the choice of an active travel mode as a life cycle stage and by applying some of the same research tools we’ve applied to how more traditional life cycle changes affect travel patterns, we can devise ways to help people to shift to active transportation and lead healthier lives.

In my application, I have a library that converts GeoJSON files into CSV files, dumping the geometry data and just preserving the properties. Because that was one of the design criteria, the GeoJSON documents have buried in their ‘properties’ hash a ‘header’ object that I pull out and use as column headers, and a ‘data’ object that is an array of data, matching the headers.

When I wrote the code, I wanted a streaming JSON parser, but I didn’t like the ones I tested, so instead I just parse the whole doc, find the header, find the data, and pass the result to the csv library.

Last night `npm install` on my laptop upgraded the csv library (http://www.adaltas.com/projects/node-csv) to 0.2.x, which brought with it a increased emphasis on streams along with some minor api changes. I also was working on merging multiple files together, which meant I really wanted to stream the results as I completed each file.

But my problem was that I had no concept of how to write data to the csv stream and get it to work. The examples and test cases all had things like

csv()
.from.path(__dirname+'/columns.in', {
    columns: true
})
.to.stream(process.stdout, {
    columns: ['id', 'name'],
    end: false
})

But I wanted to write stuff over and over again. I tried a minimal hacky fix by just piping one file at a time to the res stream, but somehow my code kept closing the stream. After repeatedly re-reading the same passages over and over again, I finally figured out that what I wanted was the “.write()” method of the stream interface. I want to call “csv.write(row)” when it is available, and then just let the csv do the piping to the res stream. So I rewrote things a bit, and this is what I ended up with.

First, I create the csv writer. Because this is an express web app, I want the output to go to the response object in the function call. I did not set the ‘source’ of the stream, because instead I am going to be writing that line by line.

return function csv_formatter_service(req,res,next){
    var csv_writer = csv()
    csv_writer.pipe(res)
...

Then I do some uninteresting stuff in callbacks to get filenames that I want to concatenate, and then I do the following to actually do the CSV conversion:

function(err,files){
    if(err) return next(err)
    res.writeHead(200, { 'Content-Type': 'text/csv' });
    // create a common builder so that I only dump one header row
    // and can number each record consecutively
    var builder = buildCSV({'columns':column_names}
                          ,function(err,rows){
                               // simple callback for builder, used for debugging mostly
                               if(err){return err;}
                               return null;
                           })
    // handy dandy async call makes sure that each file is processed consecutively
    // so that their output is not jumbled together on the stream
    async.forEachSeries(files
                       ,function(file,cb){
                            // eventually replace the read file with a streaming JSON parser
                            fs.readFile(file,'utf8', function(err,data){
                                // pass the data and csv_writer to the function 
                                // that parses the JSON
                                var err = builder(err,data,csv_writer)
                                // err is the output of the callback defined above (null or error)
                                cb(err);
                            })
                        }
                       ,function(err){
                            // simple placeholder callback for forEachSeries.  end the response
                            res.end(err)
                            return null
                        })
    // emacs complains if I don't return null, and anyway it is good practice
    return null
)

Now for the JSON parser code. Whereas before I would parse the JSON doc, get an array of lines, and then create the csv() call with that array as the source, now I skip a step and just write each line to the csv writer object passed in.

function buildCSV(options,cb){
    var header_row
    var column_names = options.columns ? options.columns : {}
    var total_data_idx = -1

    return function(err,doc,writestream){
        var fcoll = JSON.parse(doc)
        var features = fcoll.features;
        var rows = 0;
        var datacols=[];
        var features_length = features.length;
        // code to figure out the header row, then 
        if(header_row === undefined){
            header_row = _.map(['i','document'].concat(datacols)
                              ,function(key){
                                   return column_names[key] || key
                               })

            // Woo Hoo, just *write* data to the stream, and off it goes to the client!
            writestream.write(header_row)
            
        }
        // now process the data rows
        for (var i = 0; i < features_length; i++){
            var props = features[i].properties;
            var datalen = props.data.length;
            if(props.data !== undefined && Array.isArray(props.data) && datalen){
                for (var dataidx = 0; dataidx < datalen; dataidx++){
                    total_data_idx++
                    var width = props.data[dataidx].length
                    var entry = [total_data_idx,props.document]
                    entry = entry.concat(
                        datacols.map(function(key,idx){
                            if(props.data[dataidx] === undefined
                             || props.data[dataidx][idx] === undefined){
                                return '';
                            }else{
                                return props.data[dataidx][idx] ;
                            }
                        })
                    )

                    // Again, no fuss no muss no temporary arrays.  
                    // Just write to the stream and send the data on its way
                    writestream.write(entry)
                    rows++  // no, I'm not sure why I am tracking rows of data separately but whatever
                            // probably debugging cruft.
                }
            }else{
                // no data
                total_data_idx++  // even in the no data case, I send a row of commas, 
                                  // so increment the counter

               // record of empties, I choose you! 
               writestream.write([total_data_idx,props.docid]
                                                       .concat(
                                                           datacols.map(function(key){
                                                               // this map returns nothing, 
                                                               // to fill out the commas one per column
                                                               return ''; 
                                                           })
                                                       )
                                 );
 
            }
        }
        return cb(null,rows)
    };

And that is it, my minor revelation. With streams, you can just write to them, and the data goes on its way. In this case, the csv library has been set up to handle arrays and strings and such, so I can just write arrays. Specifically, the docs state:

write(data, [preserve])

Implementation of the Writable Stream API with a larger signature. Data may be a string, a buffer, an array or an object.

If data is a string or a buffer, it could span multiple lines. If data is an object or an array, it must represent a single line. Preserve is for line which are not considered as CSV data.

Type code,Expanded full text,Display name abbreviation,Spanish,Translation,Prefix type,Suffix type
103,Academy,Acdmy,,,Y,Y
104,Acueducto,Acueducto,Yes,Aqueduct,Y,N
105,Aeropuerto,Aero,Yes,Airport,Y,N
106,Air Force Base,AFB,,,N,Y
107,Airfield,Airfield,,,N,Y
108,Airpark,Airpark,,,N,Y
109,Airport,Arprt,,,N,Y
110,Airstrip,Airstrip,,,N,Y
112,Alley,Aly,,,N,Y
115,Apartment Building,Apt Bldg,,,N,Y
116,Apartment Complex,Apt Complex,,,N,Y
117,Apartments,Apts,,,N,Y
118,Aqueduct,Aqueduct,,,N,Y
119,Arcade,Arc,,,Y,Y
121,Arroyo,Arroyo,Yes,Stream,Y,N
122,Assisted Living Center,Asstd Liv Ctr,,,N,Y
694,Assisted Living Facility,Asstd Liv Fac,,,N,Y
123,Autopista,Autopista,Yes,Expressway/Freeway,Y,N
124,Avenida,Ave,Yes,Avenue,Y,N
125,Avenue,Ave,,,Y,Y
126,Bahia,Bahía,Yes,Bay,Y,N
127,Bank,Bk,,,Y,Y
704,Base,Base,,,N,Y
128,Basin,Basin,,,N,Y
129,Bay,Bay,,,Y,Y
130,Bayou,Byu,,,Y,Y
131,Beach,Bch,,,N,Y
132,Bed and Breakfast,B and B,,,N,Y
136,Beltway,Beltway,,,N,Y
137,Bend,Bnd,,,N,Y
138,Bluff,Blf,,,N,Y
139,Boarding House,Brdng Hse,,,N,Y
140,Bog,Bog,,,N,Y
141,Bosque,Bosque,Yes,Forest,Y,N
142,Boulevard,Blvd,,,Y,Y
143,Boundary,Boundary,,,N,Y
146,Branch,Br,,,Y,Y
147,Bridge,Brg,,,N,Y
148,Brook,Brk,,,N,Y
149,Building,Bldg,,,Y,Y
150,Bulevar,Bulevar,Yes,Boulevard,Y,N
151,Bureau of Indian Affairs Highway,BIA Hwy,,,Y,N
152,Bureau of Indian Affairs Road,BIA Rd,,,Y,N
153,Bureau of Indian Affairs Route,BIA Rte,,,Y,N
154,Bureau of Land Management Road,BLM Rd,,,Y,N
696,Bypass,Byp,,,Y,Y
156,Calle,Cll,Yes,Street,Y,N
157,Calleja,Calleja,Yes,Narrow Street,Y,N
158,Callejón,Callejón,Yes,Alley,Y,N
159,Caminito,Cmt,Yes,Little Road,Y,N
160,Camino,Cam,Yes,Road/Way,Y,N
161,Camp,Cp,,,Y,Y
163,Campground,Cmpgrnd,,,N,Y
164,Campus,Cmps,,,N,Y
165,Canal,Cnl,,,Y,Y
172,Cano,Caño,Yes,Drain/Sewer,Y,N
166,Cantera,Cantera,Yes,Quarry/Gravel Pit,Y,N
167,Canyon,Cyn,,,Y,Y
168,Capilla,Capilla,Yes,Chapel,Y,N
169,Carretera,Carr,Yes,Road,Y,N
170,Causeway,Cswy,,,N,Y
171,Cayo,Cayo,Yes,Key,Y,N
173,Cementerio,Cem,Yes,Cemetery,Y,N
174,Cemetery,Cmtry,,,N,Y
175,Center,Ctr,,,Y,Y
176,Centro,Centro,Yes,Center,Y,N
177,Cerrada,Cer,Yes,Closed,Y,N
178,Chamber of Commerce,Cham of Com,,,N,Y
179,Channel,Chnnl,,,N,Y
180,Chapel,Cpl,,,Y,Y
181,Childrens Home,Childrens Home,,,N,Y
182,Church,Church,,,Y,Y
183,Circle,Cir,,,N,Y
234,Círculo,Cír,Yes,Circle,Y,N
184,City Hall,City Hall,,,N,Y
185,City Park,City Park,,,N,Y
186,Cliff,Clf,,,N,Y
187,Club,Clb,,,Y,Y
188,Colegio,Colegio,Yes,School,Y,N
189,College,Colg,,,Y,Y
190,Common,Cmn,,,N,Y
191,Commons,Cmns,,,Y,Y
192,Community Center,Community Ctr,,,N,Y
193,Community College,Community Colg,,,Y,Y
194,Community Park,Community Park,,,Y,Y
195,Complex,Complx,,,N,Y
197,Condominios,Condios,Yes,Condominiums,Y,N
198,Condominium,Condo,,,Y,Y
199,Condominiums,Condos,,,N,Y
201,Convent,Cnvnt,,,Y,Y
202,Convention Center,Convention Ctr,,,Y,Y
203,Corners,Cors,,,N,Y
204,Correctional Facility,Corr Faclty,,,N,Y
205,Correctional Institute,Corr Inst,,,N,Y
207,Corte,Corte,Yes,Court,Y,N
679,Cottage,Cottage,,,N,Y
208,Coulee,Coulee,,,N,Y
209,Country Club,Country Club,,,Y,Y
210,County Highway,Co Hwy,,,Y,N
211,County Home,Co Home,,,Y,Y
212,County Lane,Co Ln,,,Y,N
213,County Park,Co Park,,,N,Y
214,County Road,Co Rd,,,Y,N
215,County Route,Co Rte,,,Y,N
216,County State Aid Highway,Co St Aid Hwy,,,Y,N
217,County Trunk Highway,Co Trunk Hwy,,,Y,N
218,County Trunk Road,Co Trunk Rd,,,Y,N
219,Course,Crs,,,N,Y
220,Court,Ct,,,Y,Y
221,Courthouse,Courthouse,,,N,Y
222,Courts,Cts,,,N,Y
223,Cove,Cv,,,N,Y
225,Creek,Crk,,,N,Y
226,Crescent,Cres,,,N,Y
227,Crest,Crst,,,N,Y
228,Crossing,Xing,,,N,Y
229,Crossroads,Xroad,,,Y,Y
233,Cutoff,Cutoff,,,N,Y
235,Dam,Dm,,,N,Y
236,Delta Road,Delta Rd,,,Y,N
237,Department,Dept,,,Y,Y
238,Depot,Dep,,,N,Y
239,Detention Center,Detention Ctr,,,N,Y
240,District of Columbia Highway,DC Hwy,,,Y,N
241,Ditch,Ditch,,,Y,Y
242,Divide,Dv,,,N,Y
243,Dock,Dock,,,N,Y
244,Dormitory,Dormitory,,,N,Y
245,Drain,Drn,,,N,Y
246,Draw,Draw,,,N,Y
247,Drive,Dr,,,N,Y
248,Driveway,Driveway,,,Y,Y
249,Dump,Dump,,,N,Y
251,Edificio,Edif,Yes,Building,Y,N
252,Elementary School,Elem School,,,N,Y
253,Ensenada,Ensenada,Yes,Cove,Y,N
254,Entrada,Ent,Yes,Entrance,Y,N
256,Escuela,Escuela,Yes,School,Y,N
680,Esplanade,Esplanade,Yes,Esplanade,Y,Y
257,Estates,Ests,,,N,Y
260,Estuary,Estuary,,,N,Y
261,Expreso,Expreso,Yes,Expressway,Y,N
262,Expressway,Expy,,,Y,Y
263,Extension,Ext,,,Y,Y
264,Facility,Faclty,,,N,Y
265,Fairgrounds,Fairgrounds,,,N,Y
266,Falls,Fls,,,Y,Y
267,Farm,Frm,,,N,Y
268,Farm Road,Farm Rd,,,Y,N
269,Farm-to-Market Road,FM,,,Y,N
275,Fence Line,Fence Line,,,N,Y
276,Ferry Crossing,Ferry Crossing,,,Y,Y
277,Field,Fld,,,N,Y
278,Fire Control Road,Fire Cntrl Rd,,,Y,N
279,Fire Department,Fire Dept,,,N,Y
280,Fire District Road,Fire Dist Rd,,,Y,N
281,Fire Lane,Fire Ln,,,Y,N
282,Fire Road,Fire Rd,,,Y,N
283,Fire Route,Fire Rte,,,Y,N
284,Fire Station,Fire Sta,,,Y,Y
285,Fire Trail,Fire Trl,,,Y,N
286,Flowage,Flowage,,,N,Y
287,Flume,Flume,,,N,Y
288,Forest,Frst,,,N,Y
289,Forest Highway,Forest Hwy,,,Y,Y
290,Forest Road,Forest Rd,,,Y,N
291,Forest Route,Forest Rte,,,Y,N
292,Forest Service Road,FS Rd,,,Y,N
293,Fork,Frk,,,N,Y
294,Fort,Ft,,,Y,N
295,Four-Wheel Drive Trail,4WD Trl,,,Y,Y
296,Fraternity,Frtrnty,,,N,Y
297,Freeway,Fwy,,,N,Y
298,Garage,Grge,,,N,Y
299,Gardens,Gdns,,,N,Y
303,Glacier,Glacier,,,N,Y
304,Glen,Gln,,,N,Y
305,Golf Club,Golf Club,,,Y,Y
306,Golf Course,Golf Course,,,Y,Y
307,Grade,Grade,,,N,Y
309,Green,Grn,,,N,Y
310,Group Home,Group Home,,,N,Y
311,Gulch,Gulch,,,N,Y
312,Gulf,Gulf,,,Y,Y
313,Gully,Gully,,,N,Y
314,Halfway House,Halfway House,,,N,Y
315,Hall,Hall,,,N,Y
316,Harbor,Hbr,,,N,Y
317,Heights,Hts,,,N,Y
321,High School,High School,,,N,Y
322,Highway,Hwy,,,Y,Y
323,Hill,Hl,,,N,Y
324,Hollow,Holw,,,N,Y
325,Home,Home,,,Y,Y
326,Hospital,Hosp,,,Y,Y
327,Hostel,Hostel,,,N,Y
328,Hotel,Hotel,,,Y,Y
329,House,Hse,,,Y,Y
330,Housing,Hsng,,,Y,Y
332,Iglesia,Iglesia,Yes,Church,Y,N
333,Indian Route,Indian Rte,,,Y,N
334,Indian Service Route,Indian Svc Rte,,,Y,N
336,Industrial Park,Indl Park,,,N,Y
337,Inlet,Inlt,,,N,Y
338,Inn,Inn,,,Y,Y
339,Institute,Inst,,,Y,Y
340,Institution,Instn,,,N,Y
341,Instituto,Instituto,Yes,Institute,Y,N
342,Intermediate School,Inter School,,,N,Y
344,Interstate Highway,I-,,,Y,N
345,Isla,Isla,Yes,Island,Y,N
346,Island,Is,,,N,Y
347,Islands,Iss,,,Y,Y
348,Isle,Isle,,,Y,Y
349,Jail,Jail,,,N,Y
351,Jeep Trail,Jeep Trl,,,Y,Y
352,Junction,Junction,,,N,Y
353,Junior High School,Jr HS,,,N,Y
356,Kill,Kill,,,Y,Y
357,Lago,Lago,Yes,Lake,Y,N
358,Lagoon,Lagoon,,,N,Y
360,Laguna,Laguna,Yes,Lagoon,Y,N
361,Lake,Lk,,,Y,Y
362,Lakes,Lks,,,N,Y
363,Landfill,Lndfll,,,N,Y
364,Landing,Lndg,,,N,Y
365,Landing Area,Landing Area,,,Y,Y
366,Landing Field,Landing Fld,,,Y,Y
367,Landing Strip,Landing Strp,,,Y,Y
368,Lane,Ln,,,Y,Y
369,Lateral,Lateral,,,Y,Y
370,Levee,Levee,,,Y,Y
371,Library,Lbry,,,Y,Y
372,Lift,Lift,,,Y,Y
373,Lighthouse,Lighthouse,,,N,Y
374,Line,Line,,,Y,Y
376,Lodge,Ldg,,,N,Y
377,Logging Road,Logging Rd,,,Y,Y
378,Loop,Loop,,,Y,Y
379,Mall,Mall,,,Y,Y
380,Manor,Mnr,,,N,Y
381,Mar,Mar,Yes,Sea,Y,N
382,Marginal,Marginal,Yes,Service Road,Y,N
383,Marina,Mrna,,,N,Y
384,Marsh,Marsh,,,N,Y
385,Meadows,Mdws,,,N,Y
386,Medical Building,Medical Bldg,,,N,Y
387,Medical Center,Medical Ctr,,,Y,Y
388,Memorial,Meml,,,N,Y
389,Memorial Gardens,Memorial Gnds,,,N,Y
390,Memorial Park,Memorial Pk,,,N,Y
391,Mesa,Mesa,,,Y,Y
392,Middle School,Mid Schl,,,N,Y
393,Military Reservation,Mil Res,,,N,Y
394,Millpond,Millpond,,,N,Y
395,Mine,Mine,,,N,Y
396,Mission,Mssn,,,Y,Y
397,Mobile Home Community,Mobile Hm Cmty,,,Y,Y
398,Mobile Home Estates,Mobile Hm Est,,,Y,Y
399,Mobile Home Park,Mobile Hm Pk,,,Y,Y
400,Monastery,Monstry,,,Y,Y
401,Monument,Mnmt,,,N,Y
403,Mosque,Mosque,,,Y,Y
404,Motel,Mtl,,,Y,Y
405,Motor Lodge,Motor Lodge,,,N,Y
406,Motorway,Mtwy,,,N,Y
407,Mount,Mt,,,Y,Y
408,Mountain,Mtn,,,N,Y
411,Museum,Mus,,,Y,Y
412,National Battlefield,Natl Bfld,,,N,Y
413,National Battlefield Park,Natl Bfld Pk,,,N,Y
414,National Battlefield Site,Natl Bfld Site,,,N,Y
415,National Conservation Area,Natl Cnsv Area,,,N,Y
416,National Forest,Natl Forest,,,N,Y
417,National Forest Development Road,Nat For Dev Rd,,,Y,N
419,National Grasslands,Natl Grsslnds,,,N,Y
420,National Historic Site,Natl Hist Site,,,N,Y
421,National Historical Park,Natl Hist Pk,,,N,Y
422,National Lakeshore,Natl Lkshr,,,N,Y
423,National Memorial,Natl Meml,,,N,Y
424,National Military Park,Natl Mil Pk,,,N,Y
425,National Monument,Natl Mnmt,,,N,Y
426,National Park,Natl Pk,,,N,Y
427,National Preserve,Natl Prsv,,,N,Y
428,National Recreation Area,Natl Rec Area,,,N,Y
429,National Recreational River,Natl Rec Riv,,,N,Y
430,National Reserve,Natl Resv,,,N,Y
431,National River,Natl Riv,,,N,Y
432,National Scenic Area,Natl Sc Area,,,N,Y
433,National Scenic River,Natl Sc Riv,,,N,Y
435,National Scenic Riverways,Natl Sc Rvrwys,,,N,Y
436,National Scenic Trail,Natl Sc Trl,,,N,Y
437,National Seashore,Natl Shr,,,N,Y
438,National Wildlife Refuge,Natl Wld Rfg,,,N,Y
439,Navajo Service Route,Navajo Svc Rte,,,Y,N
440,Naval Air Station,Naval Air Sta,,,N,Y
442,Nursing Home,Nurse Home,,,N,Y
444,Ocean,Ocean,,,N,Y
445,Oceano,Océano,Yes,Ocean,Y,N
446,Office,Ofc,,,Y,Y
447,Office Building,Office Bldg,,,N,Y
449,Office Park,Office Park,,,N,Y
698,Orchard,Orchard,,,N,Y
451,Orchards,Orchrds,,,N,Y
452,Orphanage,Orphanage,,,N,Y
453,Outlet,Outlet,,,N,Y
454,Oval,Oval,,,N,Y
455,Overpass,Opas,,,N,Y
456,Parish Road,Parish Rd,,,Y,N
457,Park,Park,,,N,Y
458,Park and Ride,Park and Ride,,,N,Y
460,Parkway,Pkwy,,,N,Y
706,Parq,Parq,Yes,Park,Y,N
461,Parque,Parque,Yes,Park,Y,N
462,Pasaje,Pasaje,Yes,Passage,Y,N
463,Paseo,Pso,Yes,Path,Y,N
464,Pass,Pass,,,Y,Y
465,Passage,Psge,,,Y,Y
466,Path,Path,,,N,Y
682,Pavilion,Pavilion,,,N,Y
467,Peak,Peak,,,N,Y
705,Penitentiary,Penitentiary,,,N,Y
468,Pier,Pier,,,Y,Y
469,Pike,Pike,,,N,Y
470,Pipeline,Pipeline,,,N,Y
472,Place,Pl,,,N,Y
473,Placita,Pla,Yes,Little Plaza,Y,N
474,Plant,Plnt,,,N,Y
683,Plantation,Plantation,,,N,Y
475,Playa,Playa,Yes,Beach,Y,N
476,Playground,Playground,,,N,Y
477,Plaza,Plz,,,Y,Y
478,Point,Pt,,,Y,Y
479,Pointe,Pointe,,,N,Y
480,Police Department,Police Dept,,,Y,Y
481,Police Station,Police Station,,,Y,Y
482,Pond,Pond,,,Y,Y
483,Ponds,Ponds,,,N,Y
485,Port,Prt,,,Y,Y
486,Post Office,Post Office,,,N,Y
487,Power Line,Power Line,,,N,Y
691,Power Plant,Power Plant,,,N,Y
488,Prairie,Pr,,,N,Y
489,Preserve,Preserve,,,N,Y
491,Prison,Prison,,,N,Y
690,Prison Farm,Prison Farm,,,N,Y
685,Promenade,Promenade,,,N,Y
492,Prong,Prong,,,N,Y
494,Puente,Puente,Yes,Bridge,Y,N
495,Quadrangle,Quadrangle,,,N,Y
496,Quarry,Quar,,,N,Y
686,Quarters,Quarters,,,N,Y
497,Quebrada,Qbda,Yes,Creek,Y,N
499,Race,Race,,,N,Y
501,Rail,Rail,,,N,Y
502,Rail Link,Rail Link,,,Y,Y
504,Railnet,Railnet,,,N,Y
505,Railroad,RR,,,N,Y
506,Railway,Rlwy,,,N,Y
507,Ramal,Ramal,Yes,Short Street,Y,N
508,Ramp,Ramp,,,N,Y
510,Ranch Road,Ranch Rd,,,Y,N
511,Ranch to Market Road,RM,,,Y,N
512,Rancho,Rch,Yes,Ranch/Farm,Y,N
513,Ravine,Ravine,,,N,Y
514,Recreation Area,Rec Area,,,N,Y
515,Reformatory,Reformatory,,,N,Y
516,Refuge,Refuge,,,N,Y
518,Regional Park,Regional Pk,,,N,Y
519,Reservation,Reservation,,,N,Y
520,Reservation Highway,Resvn Hwy,,,Y,N
521,Reserve,Resv,,,N,Y
522,Reservoir,Reservoir,,,Y,Y
524,Residence Hall,Res Hall,,,N,Y
525,Residencial,Residencial,Yes,Public Housing Project,Y,N
526,Resort,Resrt,,,N,Y
688,Rest Home,Rest Home,,,N,Y
527,Retirement Home,Retirement Hme,,,N,Y
528,Retirement Village,Retirement Vlg,,,N,Y
529,Ridge,Rdg,,,N,Y
543,Rio,Río,Yes,River,Y,N
530,River,Riv,,,N,Y
531,Road,Rd,,,Y,Y
533,Roadway,Roadway,,,N,Y
535,Rock,Rock,,,Y,Y
536,Rooming House,Rooming Hse,,,N,Y
537,Route,Rte,,,Y,Y
538,Row,Row,,,Y,Y
539,Rue,Rue,,,Y,Y
540,Run,Run,,,N,Y
541,Runway,Runway,,,Y,Y
542,Ruta,Ruta,Yes,Route,Y,N
498,RV Park,RV Park,,,N,Y
545,Sanitarium,Sanitarium,,,N,Y
546,School,Schl,,,Y,Y
549,Sea,Sea,,,Y,Y
550,Seashore,Seashore,,,N,Y
552,Sector,Sec,Yes,Sector,Y,N
553,Seminary,Smry,,,Y,Y
554,Sendero,Sendero,Yes,Foot Path,Y,N
555,Service Road,Svc Rd,,,Y,Y
556,Shelter,Shelter,,,N,Y
558,Shop,Shop,,,N,Y
699,Shopping Center,Shopping Ctr,,,N,Y
560,Shopping Mall,Shopping Mall,,,N,Y
700,Shopping Plaza,Shopping Plz,,,N,Y
703,Site,Site,,,N,Y
564,Skyway,Skwy,,,Y,Y
565,Slough,Slough,,,N,Y
566,Sonda,Sonda,Yes,Sound,Y,N
567,Sorority,Sorority,,,Y,Y
568,Sound,Snd,,,Y,N
569,Spa,Spa,,,Y,Y
570,Speedway,Speedway,,,Y,Y
571,Spring,Spg,,,N,Y
572,Spur,Spur,,,Y,Y
573,Square,Sq,,,Y,Y
575,State Beach,State Beach,,,N,Y
577,State Forest,State Forest,,,N,Y
578,State Forest Service Road,St FS Rd,,,Y,N
579,State Highway,State Hwy,,,Y,N
580,State Hospital,State Hospital,,,Y,Y
581,State Loop,State Loop,,,Y,N
582,State Park,State Park,,,N,Y
584,State Prison,State Prison,,,N,Y
585,State Road,State Rd,,,Y,N
586,State Route,State Rte,,,Y,N
588,State Spur,State Spur,,,Y,N
589,State Trunk Highway,St Trunk Hwy,,,Y,N
591,Station,Sta,,,N,Y
592,Strait,Strait,,,Y,Y
593,Stravenue,Stra,,,N,Y
594,Stream,Strm,,,N,Y
595,Street,St,,,N,Y
596,Strip,Strip,,,Y,Y
599,Swamp,Swamp,,,N,Y
600,Synagogue,Synagogue,,,Y,Y
601,Tank,Tank,,,N,Y
603,Temple,Tmpl,,,Y,Y
604,Terminal,Trmnl,,,N,Y
605,Terrace,Ter,,,Y,Y
687,Thoroughfare,Thoroughfare,,,N,Y
607,Toll Booth,Toll Booth,,,Y,Y
701,Toll Road,Toll Rd,,,N,Y
610,Tollway,Tollway,,,N,Y
611,Tower,Twr,,,Y,Y
612,Town Center,Town Ctr,,,Y,Y
613,Town Hall,Town Hall,,,N,Y
614,Town Highway,Town Hwy,,,Y,N
615,Town Road,Town Rd,,,Y,N
616,Towne Center,Towne Ctr,,,Y,Y
617,Township Highway,Twp Hwy,,,Y,N
618,Township Road,Twp Rd,,,Y,N
619,Trace,Trce,,,N,Y
620,Track,Trak,,,Y,Y
621,Trafficway,Trfy,,,N,Y
622,Trail,Trl,,,Y,Y
623,Trailer Court,Trailer Ct,,,N,Y
624,Trailer Park,Trailer Pk,,,N,Y
628,Transmission Line,Trans Ln,,,N,Y
702,Treatment Plant,Trmt Plant,,,Y,Y
630,Tribal Road,Tribal Rd,,,Y,N
632,Trolley,Trolley,,,Y,Y
633,Truck Trail,Truck Trl,,,Y,Y
636,Túnel,Túnel,Yes,Tunnel,Y,N
634,Tunnel,Tunl,,,Y,Y
635,Turnpike,Tpke,,,N,Y
637,Underpass,Upas,,,Y,Y
642,Universidad,Universidad,Yes,University/College,Y,N
643,University,Univ,,,Y,Y
638,US Forest Service Highway,USFS Hwy,,,Y,N
639,US Forest Service Road,USFS Rd,,,Y,N
640,US Highway,US Hwy,,,Y,N
641,US Route,US Rte,,,Y,N
644,Valley,Vly,,,N,Y
645,Vereda,Ver,Yes,Path,Y,N
655,Via,Via,Yes,Way,Y,N
646,Viaduct,Viaduct,,,N,Y
647,View,Vw,,,N,Y
648,Villa,Villa,,,Y,Y
649,Village,Vlg,,,Y,Y
650,Village Center,Village Ctr,,,Y,Y
697,Vineyard,Vineyard,,,N,Y
652,Vineyards,Vineyards,,,N,Y
654,Vista,Vis,,,Y,Y
656,Walk,Walk,,,N,Y
657,Walkway,Walkway,,,N,Y
659,Wash,Wash,,,N,Y
660,Waterway,Waterway,,,N,Y
661,Way,Way,,,N,Y
663,Wharf,Wharf,,,N,Y
665,Wild and Scenic River,Wld n Snc Riv,,,N,Y
664,Wild River,Wild River,,,N,Y
666,Wilderness,Wilderness,,,N,Y
667,Wilderness Park,Wilderenss Pk,,,N,Y
668,Wildlife Management Area,Wldlf Mgt Area,,,N,Y
669,Winery,Winery,,,Y,Y
672,Yard,Yard,,,N,Y
673,Yards,Yards,,,Y,Y
670,YMCA,YMCA,,,Y,Y
671,YWCA,YWCA,,,Y,Y
675,Zanja,Zanja,Yes,Ditch,Y,N
676,Zoo,Zoo,,,Y,Y

This post describes how I use the superagent library to test access to restricted resources on my web server. This is something that I found to take a bit more effort than I expected, so I thought I’d write this up for the greater good.

Context

I am running a website in which some resources are open to the internet, while others require authentication versus our CAS server.

I have been logging into the CAS server using request. But in the interest of trying out different libraries and all that, I decided to rewrite my
method using superagent.

I need to log into the CAS server from node.js because I am writing tests that verify that the protected resources are hidden to non-authenticated users, and available to authenticated ones.

Plot development

When I first tried superagent, it didn’t have any way to accept and keep cookies. Or rather, perhaps it did but it wasn’t documented. Now it is documented in the github readme. That code is reproduced below:

var superagent = require('superagent');
var user1 = superagent.agent();
user1
  .post('http://localhost:4000/signin')
  .send({ user: 'hunter@hunterloftis.com', password: 'password' })
  .end(function(err, res) {
    // user1 will manage its own cookies
    // res.redirects contains an Array of redirects
  });

My situation is slightly more complicated. First, the CAS server expects to interact with a human on a web page, so it buries secret messages in the form fields in the generated login page. For example:

<input type="hidden" value="e1s1" name="lt">
<input type="hidden" value="submit" name="_eventId">

Even more irritating, the CAS server implementation that we are running apparently only responds to known user agents. It plays nicely with Firefox and curl, but it pukes and throws an error when
confronted with node.js libraries such as request or superagent. So one has to remember to lie about the user agent.

Rising action

The first step is to get the login page, as performed by the following code block

var ua = superagent.agent()
ua.get(casurl)
  .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')
  .end(_login_handler(ua,function(err,res){
                         if(err) return callback(err)
                         var success_reged=/Log In Successful/i;
                         if(success_regex.test(res.text)){
                             return callback()
                         }else{
                             return callback(new Error('CAS login failed'))
                         }
                      })
      )

The login page itself is passed to the function generated by the call to _login_handler(), which is reproduced below. In this case, I am setting up a test, so I know the user and the user’s password, defined by cas_user and cas_pass.

function _login_handler(ua,callback){
    return function(err,res){
        if(err) return callback(err)
        // parse the body for the form url, with the correct jsessionid
        var form_regex = /id="fm1".*action="(.*)" method="post"/;
        var result = form_regex.exec(res.text)
        var url='https://'+cashost+result[1]

        var form={'username':cas_user
                  ,'password':cas_pass
                  ,'submit':'LOGIN'
                  }

        // scrape hidden input values
        var name_regex = /name="(.*?)"/
            var value_regex = /value="(.*?)"/
            var hidden_regex = /<input.*type="hidden".*?>/g
        while ((result = hidden_regex.exec(res.text)) !== null)
        {
            var n = name_regex.exec(result[0])
            var v = value_regex.exec(result[0])
            form[n[1]]=v[1]
        }
        // ready to post
        ua.post(url)
        .type('form')
        .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')
        .send(form)
        .end(callback)
        return null
    }
}

I glossed over the tricky bits that had me stuck through a bunch of tweak-try-fail iterations. First, every time the agent is used to make a request, the CAS server must be sent the fake user agent header.

ua.get(casurl)
  .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')
// ...
// ...
ua.post(url)
  .type('form')
  .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')

Second, superagent must be told to send the hashref of data as text, not as JSON. This wrinkle I missed entirely until I started reading the superagent docs carefully. The docs clearly state that the default POST transport is JSON:

“Since JSON is undoubtably the most common, it’s the default!”

The next paragraph down in the docs explains the use of the .type('form') options:

To send the data as application/x-www-form-urlencoded simply invoke .type() with “form”, where the default is “json”. This request will POST the body “name=tj&pet=tobi”.

The second call above uses this option to post the login credentials and hidden fields to the CAS server.

With that, if the credentials are correct, the user agent will have an established session with the CAS server that can be verified by each of the web hosts using the CAS authentication system.

Conflict

But merely logging in to the CAS server and establishing a session with it is only half of the story. The main feature that CAS supports is single sign on. This is achieved in practice as follows. First, a user logs on to the CAS system somehow. Then the user agent visits a CAS-protected website. This website redirects the incoming request to the CAS server, with a query parameter called “service”. If the user agent is already logged in to the CAS server, then the server returns the user to the url defined in the “service” query parameter, along with a one-time-use ticket. The CAS-protected webservice then checks this ticket by sending a request directly to the CAS server to verify that the service ticket it just received from the incoming request is valid. If it is valid, then the user agent is considered to be an authenticated user.

Unfortunately, this is when I found a bug in SuperAgent that is not present in request, and to try to fill the open source bargain, I created a test case and opened a bug report. I also tried to fix it myself, but it wasn’t quite as simple as I thought. Merely copying the headers during the redirect doesn’t solve the issue.

Conclusion

In true postmodern fiction style, there is no real end to this story. It works to log in to CAS, but it doesn’t work to establish an authenticated session with a CAS-protected website. As very similar code works fine using Request, I am returning to using that code, and will perhaps be writing another, similar blog post in the future containing that code.

Appendix

For completeness, the chain of functions I use are reproduced below.

function _login_handler(ua,callback){
    return function(err,res){
        if(err) return callback(err)
        // parse the body for the form url, with the correct jsessionid
        var form_regex = /id="fm1".*action="(.*)" method="post"/;
        var result = form_regex.exec(res.text)
        var url='https://'+cashost+result[1]

        var form={'username':cas_user
                  ,'password':cas_pass
                  ,'submit':'LOGIN'
                  }

        // scrape hidden input values
        var name_regex = /name="(.*?)"/
            var value_regex = /value="(.*?)"/
            var hidden_regex = /<input.*type="hidden".*?>/g
        while ((result = hidden_regex.exec(res.text)) !== null)
        {
            var n = name_regex.exec(result[0])
            var v = value_regex.exec(result[0])
            form[n[1]]=v[1]
        }
        // ready to post
        ua.post(url)
        .type('form')
        .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')
        .send(form)
        .end(callback)
        return null
    }
}


function cas_login_function(ua,callback){

             console.log('hitting '+casurl)

             ua.get(casurl)
             .set('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0')
             .end(_login_handler(ua
                              ,function(err,res){
                                   if(err) return callback (err)
                                   var success_regex = /Log In Successful/i;
                                   if(success_regex.test(res.text)){
                                       return callback()
                                   }else{
                                       return callback(new Error('CAS login failed'))
                                   }
                               }) )
             return null
         }

// ...

describe('get files with login',function(){
    var agent =  superagent.agent()
    before(function(done){
        cas_login_function(agent,function(err){
            if(err) throw new Error(err)
            done()
        })
    })
    // in all of the following tests, agent is logged in to CAS
    it('should get index.html from /protected_resource'
      ,function(done){
           agent
           .get(rooturl+'/protected_resource')
           .end(function(err,res){
               if(err) return done(err)
               res.ok.should.be.true;
               ...
               return done()
           })
    })
})

I want to store data and time together, like in the description of Datomic. In face I *have* been doing this for some time, but I want to be more formal about it.

First, all my attempts within PostgreSQL suck. I end up with massive tables that need to be reduced after the fact and that don’t really do what I want.

CouchDB’s document approach is interesting, in that you can store time
stamp in the doc, and/or use the timestamp as part of the doc id, but
it is also annoying…for a case like vds, I just want to record that
the detector started at some point and stopped at some other point,
but since the stop isn’t announce, I need to record every time the
detector

Suppose I have a batch of VDS detectors, announced by a file that says that as of that day, all of those detectors are on. So the next time a new file is received, only the detectors in the file are still on, all the ones not in the file are implicitly off.

But if I am also storing data from all of those detectors at the same time, then in theory at least I should know whether or not the detector is on for any given day.

Okay, but what about the other metadata, like the segment length, that might change based on other changes in the system but that leave the core facts about the detector (it is on the freeway, at a point in space, collecting data for a bucha lanes) intact? Those facts need to be written down each time they change, and if there isn’t a new entry in the new detector listing, then that means that the detector is off and the last state is the penultimate state, and the last day that the detector might have been active and with characteristics is the last day it collected data.

I’m trying to talk myself out of having to write a date stamp to “all the detectors not in this file that do not yet have a final active date”, which seems stupid and inefficient. And I don’t think I have to do that. The facts as of the last update are true until those facts are changed. The fact that the detector is not in the new detector list is irrelevant because I know that it isn’t collecting data anymore.

To get the descriptive facts about a detector for a particular day’s data, all I need to do is get the document from the detector state db that is less than or equal to the day in question.

I bicycle to work every day, except for the occasional day here or there, and I fully realize how limiting bicycling is. In a car I would think nothing of making a side trip to the store on the way home, or performing other household errands. But on a bike, even with an excellent Atlantis with racks and bags, running to the store will add 30 minutes of travel time, plus add extra weight that my legs have to propel. Cars are better, faster, and cheaper for commuting to work, but I prefer to get exercise (and basically I’m stubborn and have a high tolerance for pain).

For short trips near our house, the bike should be excellent. My kids go to school about a mile and a half from our house, and there are restaurants and shops well within a mile of us. But where I live (Costa Mesa, in Orange County, CA), the parking lots and streets are so heavily car-oriented that my kids are afraid to cross the two major streets that close off our subdivision.

For example, Google Maps informs me that just 1.6 miles from my house there is a super awesome coffee bar (Portola Coffee) right next to a gelato place and some of the best pastries in Orange County. The problem is that the 1.6 mile route requires cycling on Harbor Boulevard for most of the way. There is a handy bike path along a flood control channel that we can use, but then the trip becomes 3.1 miles! Double the distance on a bike is difficult, and a strong disincentive when compared to the fast, easy, cheap, and efficient automobile. Even bribery with ice cream or pastries won’t work more than once a month.

Reducing parking requirements is a good baby step towards reducing the relative superiority of the automobile. I’m pretty happy that is finally happening, but not much is going to change with respect to emissions and parking lot oriented development without a lot more parallel measures and a lot more citizen led initiatives. If people are happy in their cars, plain old traffic and difficult parking conditions here and there isn’t going to get them to walk or bike. I’m interested to see how OCTA, SCAG, and so on, will increase development density in the coming years.

(And I’d still like to do some bike touring closer to home than Amsterdam.)