Contour Line » code

Using psql copy from DBIx::Class

jmarca — Wed, 14 Oct 2009 00:04:46 +0000

I am loading up lots and lots of data, and need to track what is going on, but I really don’t need all of the stuff that DBIx::Class brings with it. So I got a clue today and decided I was just going to use copy directly, picking off the file, gunzip-ping it, and using system to execute a psql copy call.

But, when I went to edit my code, I realized that I forgot about stuff like passwords and ports and hosts and all that junk that is nice to have in a portable perl script.

CPAN docs to the rescue! First I found out that indeed DBD::Pg can do copy() calls. Then I scanned around in the DBIx::Class docs and found a code snippet in the Storage docs that said how to use dbh_do.

My final script is pretty simple compared to the line by line code that I was trying to optimize for speed.
#!/usr/bin/perl -w use strict; use warnings; use Carp; use Data::Dumper; use Getopt::Long; use Pod::Usage; use IO::Uncompress::Gunzip qw(gunzip $GunzipError);


use English qw(-no_match_vars);
use DB::CouchDB;

use VDS::Schema;    # my DBIx::Class database schema
use version; our $VERSION = qv('0.0.4.5');
#insert options processing stuff here
#make the postgres connector

my $vdb = VDS::Schema->connect( "dbi:Pg:dbname=$dbname;host=$host;port=$port",

    $user, $pass, {}, { disable_sth_caching => 1 } );
# some other utility subroutines go here
# main file processing loop

while ( my $input = shift @files ) {
    # pick off the actual filename for tracking the file

    my $currentfile;

    if ( $input =~ /.*\/(.*gz)$/sxm ) {

        $currentfile = $1;

    }
    # use CouchDB as state machine to track doc processing

    my $seekpos = check_if_doc_processed(

        {

            'filename' => $currentfile,

            'create'   => 1,

        }

    );

    if ( $seekpos  0 && !$reparse ) {

        carp

"skipping $currentfile, some other process is probably already parsing it";

        next;    # skip this document, go onto the next one

    }
    # okay lets process the file

    # fire off psql copy command

    my $z = IO::Uncompress::Gunzip->new($input)

      or croak "IO::Uncompress::Gunzip failed: $GunzipError\n";

    my @stuff = $vdb->storage->dbh_do(

        sub {

            my ( $storage, $dbh, $file ) = @_;

            $dbh->do(

'COPY pdata (ts,vds_id, n1,o1,s1,n2,o2,s2,n3,o3,s3,n4,o4,s4,n5,o5,s5,n6,o6,s6,n7,o7,s7,n8,o8,s8) from STDIN with csv'

            );
            while ( my $line = $z->getline() ) {

                $dbh->pg_putcopydata($line);

            }

            $dbh->pg_putcopyend();

        },

        $z

    );
    # post to the CouchDB the fact that we're done, and how many lines we read

    track_doc_processing(

        {

            'filename' => $currentfile,

            'row'      => $z->input_line_number(),

            'done'     => 1

        }

    );

    $z->close();

}

Washed out gray is the new black

jmarca — Wed, 10 Dec 2008 20:33:27 +0000

I was working on Sakai help yesterday and decided that using CouchDB might be the way to go if I ever got some time to try to replace Sakai’s default help system. Why replace? The biggest problem I ran into is that changing the help contents seems to require a server reload. And Sakai is a monster to reload (well, more than a minute is pretty darn slow for a reload). A better help system would be to do as it currently does (parse loaded jar files for help contents) but then create or update existing help in a couchdb database, with versioning and what not so that reloads of old data don’t obliterate more recent changes.

That way, you can hop on the site and edit the help contents, rather than having to edit the help when you write the java code and load the help when you do the “mvn install sakai:deploy && /etc/init.d/tomcat-5.5 restart” incantation.

BUT I’d rather spend time working on a transportation glossary app for the site, rather than updating the help system. The two can eventually be the same, with some backend tweaks, but the glossary is more immediately useful to us.

slow progress on the couchdb front

jmarca — Thu, 20 Nov 2008 00:44:35 +0000

Some small, slow progress today, in and amongst other things. Got a db up, put in two simple glossary entries, set up two views, and was able to ping from perl using AnyEvent::CouchDB, which seems more up-to-date than Net::CouchDb, and anyway, I couldn’t get Net::CouchDb to pass along a key=”blahblah” parameter on a get request.

Tonight maybe I will modify the upload script to put the glossary in MediaWiki to instead populate the couchdb. But I first have to solve a method for embedding links into the entries. Right now my glossary app intercepts and rewrites the media wiki -generated links as needed. I can make my life somewhat easier if I use couchdb directly, perhaps.

And then I have ideas for an editor, so that I can use ajax dialogs to pull in existing terms to make creating and editing entries somewhat easier. Say an entry is being edited, query the db for all terms that are related to the text in the current definition, post those in a growing list on the right for drag and drop.

Which raises another question I’ll have to answer, how do you drag and drop stuff into an text box that is being edited and have the drop land at the point of edit. When as I recall, the jquery editable type of widget thingee reverts when focus is lost. Perhaps that is something that is settable.

couchdb

jmarca — Wed, 19 Nov 2008 00:50:19 +0000

Stumbled across couchdb thanks to the Sakai devel mailing list. Looks cool, but I need to use it before I can get my head around what it might be able to do. I think a good toy application that will also be useful is to code up Mike’s glossary using it. That would be good because the limitation of a wiki is that when you want to add links to existing pages, you have to guess or whatnot, and anyway a wiki is not a glossary. I just want a dynamic, editable, cross-linked glossary. That isn’t hand-generated!

javascript thing

jmarca — Wed, 29 Oct 2008 04:27:22 +0000

I can’t find the bookmark to the original source for this on my home computer, and anyway the bookmark that I thought worked on my work computer seems to be out of date. So, here is the shell of how I start javascript programs. Your mileage may vary.

// closure to hide the prototype
var MyObject = (function(){
    /* private static (class) members */

    /* private static methods. */

    /*class constructor.*/
    function constructorFn(id){
        this.id = id;
        var self = this;
        /* call private static (class)  methods, etc */

        /* privileged instance methods. */

    };

    /* privileged static (class) methods   (a property of the constructor) */
    //constructorFn.getNoOfInsts = function(){
    //    return counter;
    //};

    /* public instance method privaliged at the  class level */
    //constructorFn.prototype.pubInstMethod = function(){
    //   ...
    //};

    /*return the constructor.*/
    return constructorFn;
})(); //simultaneously define and call (one-off)!

/* public static  members */
// MyObject.pubStatic = "anything"

I even tried searching for that characteristic “define and call (one-off)” line that I know was in the original source, but the original didn’t turn up.

Ah, found this link, and clicking on the link in that page turns up a dead end.

Salasaga frustration

jmarca — Thu, 09 Oct 2008 23:17:08 +0000

Super frustrating day trying to build Salasaga, with the kicker coming at about 3:30 when ./configure finally complained that it needs libgnome. I don’t run gnome. Most of the time that isn’t a problem. How annoying that they don’t list Gnome as one of their project prerequisites. I guess that is one of the evils of having Ubuntu become so popular—perhaps developers don’t even realize that running Linux on a desktop doesn’t “require” a gnome environment.

I think I’ll wait for the ebuild to come out.

concrete 5 looks like a winner

jmarca — Tue, 07 Oct 2008 21:59:25 +0000

I’ve been looking for something for Brooke to use for a while now to bang out websites. I tried rolling my own, tried a wiki, tried setting up a blog (not in that order) but nothing worked. Concrete5 looks like “the bomb” as my old friend Bob used to say. I used their demo (until it reset) and it looked like it could do what I think Brooke wants to do with a site (format, arrange, write content, get automatic listings/directories, etc). The FAQ says it has community tools too. Finally, I did a control-u and saw jquery, etc etc, so I’m downloading it and going to fire it up. I just emerged drupal yesterday, but I got all bogged down in plugins and add ons and so on. If I can’t set it up fast, it isn’t going to get done.

We’ll see how C5 works out.

Update: C5 is indeed cool, but two problems. First my upgrade from Firefox 2.0 to 3.0 was done poorly. I didn’t do a clean restart. So once I figured that out and got a new .mozilla/firefox directory, the flash bits started working for me. But the second problem is that uploading files causes an immediate session fail. I haven’t solved that yet. Still, I see enough potential to keep plugging at it.

Update Update: Turns out suhosin is the problem, or rather, it encrypts the session and flash then trips up on the encryption and the session is then invalidated. I think. Anyway, turning off encryption in suhosin config solved the problem, as I wrote here