The Eclectic Engineer

VGA Pixelclock in Verilog

2009-06-20T14:32:00.000-07:00

I've been teaching myself Verilog and just recently got VGA working. More info and full source soon.

Four line program for historical dividend and split data

2008-08-14T20:32:00.000-07:00

I spent some time reverse engineering how the charts in Google Finance work. I was mostly interested in the historical dividend and stock split information. I finally found it, it's stored as a JavaScript variable in the page. I wrote a 4 line Python program which will pull the page from Google, find the variable and import it as a list of dictionaries.

import urllib2, re
url = "http://finance.google.com/finance?q=GE"
eventReg = re.compile("_chartConfigObject\.corpActionsArray = eval\('\((.*)\)'\);")
eventData = eval(eventReg.search(urllib2.urlopen(url).read()).group(1))

Small update to div-data

2008-08-11T22:47:00.000-07:00

There was a bug in the div-update application having to do with the growth calculations. I had just hard coded the division of the sum of dividends regardless of whether or not their was dividend data for the entire period. Instead if there isn't data for the entire time period I respond with "--" for those columns.

I'm back and brought financial goodies.

2008-08-05T17:29:00.000-07:00

So I'm back after a rather long unplanned hiatus. In the intervening time I've moved, hosted several sets of out-of-towners, began a career change and blew the disc between vertebrates S1 and L5. Blowing my disc was the big one, I've been out of work and flat on my ass for over three weeks. The meds they give you have really intense side effects, namely mania and psychosis. The mania manifested itself as an obsession with CD rates and dividend producing equities.

While doing my research I stumbled upon a blog post which provides a tutorial on using Google Docs as an investment calculator. This inspired me to attempt to create my own spreadsheet to help me do research on prospective investments. I was quickly disappointed by Google Docs' GoogleFinance function, it was just way too limiting. Primarily I wanted to be able to get things like dividend yield and historical dividend growth. Yahoo provides historical dividend data exportable as a CSV but I couldn't figure out how to import it into Google Docs and aggregate the data cleanly. Frustrated but not defeated I cranked out a little web app yesterday which does exactly that and deployed it to Google AppEngine.

The URL has one parameter, "t" which is the ticker symbol for the equity that you are interested in. It returns 5 columns of CSV formatted data: the cash value of the last dividend payment, the date of the last payment, the dividend frequency, the average dividend growth for the previous 5 years, the average dividend growth for the previous 10 years. If the stock doesn't produce a dividend then the first three fields have a value of "--" and the last two will have a value of 0.0. In order to use this app in conjunction with GoogleDocs use the formula below replacing A2 with the cell that contains your ticker value.

If you do use this application and have suggestions, bug reports or feature requests please contact me. In addition to me providing you with better data it will help me learn what other people think is important when doing research.

Example Formula:

=importData("http://dividend-data.appspot.com/?t=" & $A2)

URL Example:

http://dividend-data.appspot.com/?t=IBM

Source can be found here.

ISight Screen Saver

2008-06-09T14:47:00.000-07:00

A while ago I wrote a screen saver for OSX as a learning exercise. I wanted to learn how to access the camera built into my Macbook Pro and get my feet wet with OSX. I was going to integrate face recognition using the OpenCV or MPT libraries but never got around to it. I'm posting it now because some of the guys at work saw it and wanted a copy. I added the source to my SVN repository and made a binary available via the download site. The screen saver is really as minimalistic as it gets. It uses the new QTKit API for accessing the camera and sets the capture view to the frame size of the inherited ScreenSaverVeiw class. The QTKit API came out around the time of Leopard's release and is much simpler and has better performance then the old streamgrabber API which had been around since the OS 9 days. To install the binary just download the zip file, unzip and double click. It will automatically install itself and become available as a screen saver in the system preferences.

Jarvis March Convex Hull Algorithm in F#

2008-06-08T16:57:00.000-07:00

I recently obtained the "Dutch" computational geometry book to supplement the other computational geometry book that I have. Instead of doing what I normally do which is skip directly to the topic that I'm interested in, I've resolved to work through the entire book systematically implementing each of the algorithms in F# or some other functional language. The purpose is two fold; I know almost nothing about the topic and it's hard to find anything about computational geometry implementations in a functional language. Who knows, someone may actually find my floundering helpful.

Chapter one is all about convex hulls and after doing a little bit of reading I determined that the "Jarvis March" algorithm looked like a good place to start. I've only included core part of the algorithm here in the blog post. The entire source including the visualization can be found here.

The algorithm is pretty simple I start first by sorting the point list by the Y coordinate and use the minimum as the starting point for my hull. I start working through the list trying to find the left most point in relation to the previous point added to the hull; this is the fold_left bit of the _jarvis function. Once I've found the left most point I append it to the head of the hull list and make a recursive call to _jarvis. When I reach the point with the maximum Y value I know that I have successfully built the left side of the hull. I then repeat the entire process but this time instead of searching for the left most point I search for the right most point building up the right side of the hull. When done I append these two lists together to form the complete hull.

type Point = {
   x :double;
   y :double;
   } with
  
   static member ( = ) a b =
       a.x = b.x && a.y = b.y;
    
   static member ( <> ) a b =
       not(a = b);
  
end;;
  

type Edge = {
   orig:Point;
   dest:Point;
   } with

   static member ( = ) a b =
       a.orig = b.orig && a.dest = b.dest;
    
   static member ( <> ) a b =
       not(a = b);
      
end;;
  

let sort point_list =
   List.sort( fun p1 p2 ->
              match Pervasives.compare p1.y p2.y with
              | 0 -> Pervasives.compare p1.x p2.x;
              | _ as v -> v;
             
   ) point_list;;


//    Input:  three points P0, P1, and P2
//    Return: >0 for P2 left of the line through P0 and P1
//            =0 for P2 on the line
//            <0 for P2 right of the line
let testSide p0 p1 p2 =
   (p1.x - p0.x) * (p2.y - p0.y) - (p2.x - p0.x) * (p1.y - p0.y);;


let isLeft p0 p1 p2 =
   (testSide p0 p1 p2) > 0.;;


let isRight p0 p1 p2 =
   (testSide p0 p1 p2) < 0.;;
  

let rec _jarvis cmp max_point (hull_list:Point list) (input_list:Point list) =      
   match input_list with
   | p::tl ->
  
       let last = List.hd hull_list in
       if last = max_point then hull_list
       else (
      
         let best = List.fold_left(fun p1 p2 ->
                                      if (cmp last p1 p2) then p2
                                      else p1;  
                                   ) p input_list in
                                  
         _jarvis cmp max_point (best::hull_list) tl
        
       )
      
   | [] -> hull_list;;


let jarvis point_list =
   if List.length point_list < 3 then failwith "3 points are required for a polygon";
  
   let point_list = sort point_list in
   let miny = point_list |> List.hd in       
   let maxy = point_list |> List.rev |> List.hd in
      
   let left = _jarvis (isLeft) maxy [miny] point_list in
   let right = _jarvis (isRight) maxy [miny] point_list in
   right @ left |> List.rev;;

XNA, Boids and Buffalo! Oh My!

2008-05-18T10:40:00.000-07:00

So I've spent my time since my last post basically re-writing the entire Boids project. I've thrown out the GDI UI and have replaced it with an XNA based one. I also rewrote my Boid type to be a class which inherits from the Body type in the Farseer 2D physics engine, making the simulation much more realistic. I’ve now got mass, rotation, torque, collisions, drag and friction to play with.

I've also refined and rewritten most of my rules, fixing the stationary swarm problem, and have added more random noise making things much more interesting. Random Boids will peel off from the group, wander around and then later rejoin. Sometimes the herd separates into two or more sub-groups then later coalesces back into a single herd. There is definitely some additional tweaking of the rules to be done, but I’m definitely headed in the right direction.

Oh, and this is one of my favorite parts, I replaced the dots that represented the Boids in the old simulations with little buffalo which rotate to reflect the actual heading of the Boid. So I have a little herd of buffalo roaming around in my computer, for some reason that makes me very happy.

This being a learning exercise, I separated the F# rules and logic into multiple files so that I could play with modules and namespaces, and have now tackled almost every subject in the Expert F# book. I don’t quite live in F# yet, but I’m getting much faster and am programming with fewer mistakes which is a good sign.

Playing with XNA was also quite fun. I am by no means utilizing all of the functionality provided, but from the limited area that I have touched it appears that Microsoft did a pretty good job. I played with OpenGL a couple of years ago, creating a planetary body orbit/gravity simulation with my roommate, and I remember it being painful. With XNA this was definitely not the case, it was very simple to get something up and running quickly. I am by no means a Microsoft fan-boy but you have to give them credit, they do make nice development tools.

So where do I go next? I need to do some mundane things like tweak the existing rules, and port all of the XNA code from C# to F#. I also need to go through and clean up some of the code, I know I’m not doing several things very efficiently and as a result my laptop has trouble with 100 or more Boids. I also want to implement several more rules: herd leaders, predator/prey behavior, obstacle avoidance etc… I still want to rewrite the nearest neighbor search portion of the code, but that is slower going. One of the problems of being a college drop-out is that I never took discreet math or linear algebra, so understanding some of the algorithms takes a little bit more time than I anticipated. It’s not exactly hard, just requires more research on my part but it’s good for me. Also now that I’ve been reading computational geometry books I see solutions to problems that I encounter all the time, I just wasn’t aware of the body of work that existed. I don’t feel too guilty though, I don’t think any of these algorithms were taught at an undergraduate level anyways at the U of Arizona. Who knows, when I was there 2001-2004 the program was in a real state of decline and was rife with in-fighting and I was all too eager to leave.

FYI: The project has gotten too big to keep posting code in the blog, so I created a project on Google Code.

Boids: Revision Two

2008-05-11T22:07:00.000-07:00

A friend of mine was in town this week, and haven't had much spare time to play with my side projects. When I finally got down to it I was able to redo my boids simulation to use operator overloading, boundary rules, and improved some of the rendering. I based my implementation off of a couple of different sources: a pseudo-code explanation of the original boids paper, a Python implementation, and this Java implementation. There is one remaining problem that I have yet to figure out. I originally used the rules laid out in the Python implementation, but the animation was very jerky when rule 3 took affect, and once the dots swarmed together they just sit their stationary. Finding no difference between his implementation and my own I started looking around for other examples that might give me an insight. That is when I found the previously mentioned Java implementation which resulted in a much more smooth trajectory. Unfortunately it still did not fix the problem where after swarming together the boids don't go anywhere. I'm going to continue to work on it but wanted to post my results thus far.

open System;;
open System.Windows.Forms;;
open Microsoft.FSharp.Collections;;

let BOID_COUNT = 40;;

let HEIGHT = 700.;;
let WIDTH = 1000.;;

let WALL = 50.;;
let WALL_FORCE = 30.;;

let MAX_VEL = 800.;;

let R1_CONST = 0.2;;
let R2_CONST = 0.2;;
let R2_RANGE = 100.;;
let R3_CONST = 0.2;;

let FPS = 24;;
let RECT_HW = 3;;


type vector = {
  x : double;
  y : double;
} with   

  member v.mag() = 
    (v.x ** 2.0) + (v.y ** 2.0) |> Math.Sqrt;

  static member ( + )(a:vector, b:vector) = 
    {x = a.x + b.x;
     y = a.y + b.y};

  static member ( - )(a:vector, b:vector) = 
    {x = a.x - b.x;
     y = a.y - b.y};

  []
  static member ( * ) (a:vector, b:int) = 
    let bd = Convert.ToDouble(b) in
    {x = a.x * bd;
     y = a.y * bd};

  []
  static member ( * ) (a:vector, b:double) = 
    {x = a.x * b;
     y = a.y * b};

  []
  static member ( ** ) (a:vector, b:int) = 
    let bd = Convert.ToDouble(b) in
    {x = a.x ** bd;
     y = a.y ** bd};

  []
  static member ( ** ) (a:vector, b:double) = 
    {x = a.x ** b;
     y = a.y ** b};

  []
  static member ( / ) (a:vector, b:int) = 
    let bd = Convert.ToDouble(b) in
    {x = a.x / bd;
     y = a.y / bd};

  []
  static member ( / ) (a:vector, b:double) = 
    {x = a.x / b;
     y = a.y / b};

  static member ( > ) (a:vector, b:double) = 
    a.mag() > b;

  static member ( < ) (a:vector, b:double) = 
    a.mag() < b;

end;;

type boid = {
  pos : vector;
  vel : vector;
} with   

  static member ( + )(a:boid, b:boid) = 
    {pos = a.pos + b.pos;
     vel = a.vel + b.vel};

  static member ( - )(a:boid, b:boid) = 
    {pos = a.pos - b.pos;
     vel = a.vel - b.vel};

  member b.distance b2 =
    Math.Sqrt(Math.Abs(((b.pos.x - b2.pos.x) ** 2.0) - ((b.pos.y - b2.pos.y) ** 2.0)));

  member b.update_pos() = 
    {b with
       pos = b.pos + b.vel};

  member b.neighbors(r, bl) = 
    [for n in bl -> 
       (b.distance(n), n)] |> 
        List.filter(fun (d:double, _) -> d < r) |> 
            List.sort(fun (d1:double, _) (d2:double, _) -> d1.CompareTo(d2)) |> 
                List.tl |> 
                    List.unzip;
    
end;;


let rand = new Random();;


let rand_double(max:double) = 
  Convert.ToDouble(rand.Next(Convert.ToInt32(max) - 1)) + rand.NextDouble();;


let vector_init = 
  {x  = 0.;
   y  = 0.};;


let boid_init = 
  {pos  = vector_init;
   vel  = vector_init};;


let boid_sum_list bl = 
  List.fold_left(+) boid_init bl;;


let boid_adjust_v_to_fps b = 
 { b with
      vel = b.vel / (Convert.ToDouble(FPS))};; 
 
  
let boid_limit_vel b = 
  let v = b.vel.mag() in
  match v with
    | _ when v > MAX_VEL ->
        let vl = v / MAX_VEL in
          { b with
              vel = b.vel / vl}

    | _ -> b;;



let boid_check_x_bounds b =
  match b with
    | _ when b.pos.x < WALL ->
        {b with 
           vel = {b.vel with 
                    x = b.vel.x + WALL_FORCE}};

    | _ when b.pos.x > (WIDTH - WALL) ->
        {b with 
           vel = {b.vel with 
                    x = b.vel.x - WALL_FORCE}};

    | _ -> b;;


let boid_check_y_bounds b =
  match b with
    | _ when b.pos.y < WALL ->
        {b with 
           vel = {b.vel with 
                    y = b.vel.y + WALL_FORCE}};

    | _ when b.pos.y > (WIDTH - WALL) ->
        {b with 
           vel = {b.vel with 
                    y = b.vel.y - WALL_FORCE}};

    | _ -> b;;


let boid_rule1 b_sum count =
  fun (b) ->
    {b with
       vel = b.vel + ((((b_sum.pos - b.pos) * (1. / (count - 1.))) - b.pos) * R1_CONST) };;


let boid_rule2 bl count = 
  fun (b:boid) ->
    let (_, nbl) = b.neighbors(R2_RANGE, bl) in
    let r2b = List.fold_left(fun b1 b2 -> b1 - (b2 - b)) boid_init nbl in
    { b with
        vel = b.vel + ((r2b.pos * (1. / (count - 1.))) * R2_CONST) };;


let boid_rule3 b_sum count =
  fun b ->
    {b with
       vel = b.vel + ((((b_sum.vel - b.vel) * (1. / (count - 1.))) - b.vel) * R3_CONST) };;


let boid_apply_rules b bl b_sum count = 
    b |> boid_check_y_bounds |> boid_check_x_bounds |> boid_rule1 b_sum count |> boid_rule2 bl count |>  
        boid_rule3 b_sum count |> boid_limit_vel |>  boid_adjust_v_to_fps |> fun b -> b.update_pos();;


let get_wall() = 
  let height = Convert.ToInt32(HEIGHT) in
  let width = Convert.ToInt32(WIDTH) in
  let wl = Convert.ToInt32(WALL) in
  [|new Drawing.Point(wl, wl);
    new Drawing.Point(wl, height - wl);
    new Drawing.Point(width - wl, height - wl);
    new Drawing.Point(width - wl, wl) 
  |];;


type BoidControl = class
    inherit UserControl as base
    
    val mutable boidList:(boid list);
    val brap:(Drawing.Point array);
    val count:double;
    val timer:Timer;
    
    new(bl:(boid list)) as this = {                                               
        boidList = bl;                                               
        count = bl |> List.length |> Convert.ToDouble;
        timer = new Timer();
        brap = get_wall();
    } then this.init();
    
    member this.init() = 
        let height = Convert.ToInt32(HEIGHT) in
        let width = Convert.ToInt32(WIDTH) in
        let size = new Drawing.Size(width, height) in

        base.Width <- width;
        base.Height <- height;
        base.MinimumSize <- size;
        base.MaximumSize <- size;
        base.BackColor <- Drawing.Color.White;
        base.SetStyle (ControlStyles.UserPaint, true);
        base.SetStyle (ControlStyles.DoubleBuffer, true);
        base.SetStyle (ControlStyles.AllPaintingInWmPaint, true);

        this.timer.Interval <- FPS;
        this.timer.Tick.Add(fun _ ->
                            this.tick(););

        this.timer.Start();

    member this.boids_update() =
        let b_sum = boid_sum_list this.boidList in
        let al = [for b in this.boidList -> async { return boid_apply_rules b this.boidList b_sum this.count }] |> Async.Parallel in 
        this.boidList <- (Async.Run(al) |> Array.to_list);
        ();
    
    override this.OnPaint e =
        let g = e.Graphics in
        List.iter(fun b -> 
            let x = Convert.ToInt32(b.pos.x) in
            let y = Convert.ToInt32(b.pos.y) in
            let bp1 = new Drawing.Point(x - RECT_HW, y - RECT_HW) in 
            let bp2 = new Drawing.Point(x - RECT_HW, y + RECT_HW) in 
            let bp3 = new Drawing.Point(x + RECT_HW, y + RECT_HW) in 
            let bp4 = new Drawing.Point(x + RECT_HW, y - RECT_HW) in 
            g.FillPolygon(Drawing.Brushes.Red, [|bp1; bp2; bp3; bp4|]);) this.boidList;
        
        g.DrawPolygon(Drawing.Pens.Blue, this.brap);
    
    member this.tick() =
        this.boids_update();
        this.Invalidate();
                
    
end;;


[]
do

  let height = Convert.ToInt32(HEIGHT) in
  let width = Convert.ToInt32(WIDTH) in
  let size = new Drawing.Size(width, height) in

  let bl = [for x in 1..BOID_COUNT -> {pos = {x = 50. + rand_double(WIDTH - 50.); 
                                              y = 50. + rand_double(HEIGHT - 50.)};
                                       vel = vector_init }] in
  let mw = new Form() in
  let bc = new BoidControl(bl) in
  mw.Controls.Add(bc);
  mw.ClientSize <- size;
  mw.MaximumSize <- mw.Size;
  mw.MinimumSize <- mw.Size;
  mw.Text <- "Boids";
  mw.Show();
  Application.Run(mw);;

Boids!

2008-05-05T19:12:00.000-07:00

I've got revision one of my boids simulation done. I'll post more details and code later, but for now here is screen capture.

Asynchronous workflows in F#

2008-04-30T11:58:00.001-07:00

Work has been really busy and I haven't had time to work on any of my side projects until last night, when I had chance to play with asynchronous workflows in F#. I've been wanting to explore to a greater extent what is out there in the world of concurrent and distributed computing, but in order to get started I needed to find some project that would be the catalyst. Not that having a project is absolutely necessary but it helps keep me on task and focused towards an end goal. Some of the ideas I've been tossing around were creating an engine to decrypt/encrypt PGP MIME messages, or build a better load/stress engine, both of which require a large up-front investment of time.

After about a week of going nowhere I stumbled upon the idea of doing swarm animations, and possibly throw in some additional AI behavior if I had time. The thing about doing swarm animations is that each entity in the swarm needs to move in relation to each of it's neighbors which involves a nearest neighbor search, one of those computationally costly and hard to tackle problems. I'm going to start with a really naive brute force approach and evolve it over time, into something more efficient. One of the things that I've been looking at is the Stolfi and Guibas method for creating Voronoi diagrams in order to speed up the nearest neighbor search. It's a divide an conquer algorithm so it should lend itself to parallelization, but that comes later, today I start with asynchronous workflows. The bit of code below had the following runtime on my Dell D620 dual-core laptop when executed using fsi:

Async time: 00:00:34.7309424
Sync time: 00:00:40.1761800

let largest = 800;;

type point = {
    x : double;
    y : double;
}

let rand = new Random();;

let rand_double(max) = 
  System.Convert.ToDouble(rand.Next(max - 1)) + rand.NextDouble();;

let rec quick_sort(x) =
  match x with
  | pivot :: rest ->
      let left, right = List.partition (( > ) pivot) rest in
      quick_sort left @ pivot :: quick_sort right
  | [] -> [];;

let distance(p1, p2) =
  Math.Sqrt(Math.Abs(((p1.x - p2.x) ** 2.0) - ((p1.y - p2.y) ** 2.0)));;

let time(f) =
  let start = DateTime.Now in
  f();
  DateTime.Now - start;;

let neighbors(p, pl) = 
  [for n in pl -> (distance(n, p), n)];;

let async_neighbors(p, pl) = 
  let a = Async.Parallel[for n in pl -> async {return (distance(n, p), n)}] in
  Async.Run(a);;

let sync_test() =
  let pl = [for x in 1..5000 -> {x=rand_double(largest);y=rand_double(largest)}] in
  [for p in pl -> neighbors(p, pl)] |> ignore;;

let async_test() = 
  let pl = [for x in 1..5000 -> {x=rand_double(largest);y=rand_double(largest)}] in
  let a = Async.Parallel[for p in pl -> async { return neighbors(p, pl) }] in
  Async.Run(a) |> ignore;;

let async_test2() = 
  let pl = [for x in 1..5000 -> {x=rand_double(largest);y=rand_double(largest)}] in
  let a = Async.Parallel[for p in pl -> async { return async_neighbors(p, pl) }] in
  Async.Run(a) |> ignore;;  

let cleanup() = 
  GC.Collect();
  GC.WaitForPendingFinalizers();;

do Console.WriteLine("Async time: " + any_to_string(time(async_test)));;  
do cleanup();;
do Console.WriteLine("Sync time: " + any_to_string(time(sync_test)));;

Calling asynchronous Twisted Python code from a blocking function

2008-04-03T21:28:00.000-07:00

So I had to write a VAPI-XP test for Mercury/HP Quality Center, and the way that this particular test type works is that you write a script in Python, VBScript, or Perl. This script must have a TestMain function that is called and as soon as that function exits the test is over and everything exits. This test had to SSH into one of the machines in our virtualized test bed, and tell the machine to netboot in order to pick up the newest build of our server product.

I could have just used a call to os.popen() to some SSH client on our intermediary testing host but I had have bad experiences with that in the past, mostly weird behavior with OpenSSH and plink. I had written some SSH code with the Twisted module Conch before so I decided to just go ahead and use that. So got the SSH code up and running no problem, but how was I to block execution until the event was triggered letting me know that the session was complete? Furthermore how was I going to get my data back without doing something dirty like a global variable?

I talked to some guys on IRC and came up with the solution below. The SSH code itself isn't that interesting and I've only really included it for completeness the important bits to note is the call to "threads.blockingCallFromThread" in RunSSHCommand and note that I'm passing the deferred that I create in "_runSSHCommand" through each of the SSH classes (usually 'd' or 'self._d') and that it is finally being called in the method "CommandChannel.closed".

#!/usr/bin/python

class ClientTransport(transport.SSHClientTransport):

   def __init__(self, user, command, d):
       self._user = user
       self._command = command
       self._d = d

   def verifyHostKey(self, pubKey, fingerprint):
       #we don't care accept any host key
       return defer.succeed(1)

   def connectionSecure(self):
       #connection made, instiantiate the class that will handle authentication
       #and pass an instance of the class that embodies the user behavior as a param
       self.requestService(ClientUserAuth(self._user, ClientConnection(self._command, self._d)))


class ClientUserAuth(userauth.SSHUserAuthClient):

   def getPassword(self, prompt = None):
       # this says we won't do password authentication        
       return

   def getPublicKey(self):
       #return the public key which is defined up top as a string
       return keys.getPublicKeyString(data=SSH_PUB_KEY)

   def getPrivateKey(self):
       #return the pricate key which is also defined up top as a string
       return defer.succeed(keys.getPrivateKeyObject(data=SSH_PRIV_KEY))


class ClientConnection(connection.SSHConnection):

   def __init__(self, cmd, d, *args, **kwargs):
       connection.SSHConnection.__init__(self)
       self._command = cmd
       self._d = d


   def serviceStarted(self):
       self.openChannel(CommandChannel(self._command, self._d, conn=self))


class CommandChannel(channel.SSHChannel):
   name = 'session'

   def __init__(self, command, d, *args, **kwargs):
       channel.SSHChannel.__init__(self, *args, **kwargs)
       self.command = command
       self.d = d
       self.data = ""

   def channelOpen(self, data):
       #send an execute request passing the user supplied string.
       self.conn.sendRequest(self,
                             'exec',
                             common.NS(self.command),
                             wantReply=True
                             ).addCallback(self._gotResponse)

   def _gotResponse(self, _):
       #command has returned, send an EOF in order to terminate the connection
       self.conn.sendEOF(self)

   def dataReceived(self, data):
       #append the output data to the string
       self.data = self.data + data

   def closed(self):
       #connection closed, execute the callback on the deferred
       #that we have been passing around
       self.d.callback(self.data)


class ClientCommandFactory(protocol.ClientFactory):
   #factory class for our SSH protocol
  
   def __init__(self, user, command, d):
       self._user = user
       self._command = command
       self._d = d

   def buildProtocol(self, addr):
       protocol = ClientTransport(self._user, self._command, self._d)
       return protocol


def RunSSHCommand(hostname, port, user, cmd):
   #Run an SSH command as a user on a given host, we only use key authentication
   thread.start_new_thread(reactor.run, (False,))
   try:
       return threads.blockingCallFromThread(reactor, _runSSHCommand, hostname, port, user, cmd)
   except:
       return None
  
   reactor.stop()

def _runSSHCommand(hostname, port, user, cmd):
   d = defer.Deferred()
   factory = ClientCommandFactory(user, cmd, d)   
   reactor.connectTCP(hostname, 22, factory)
   return d
  

if __name__ == "__main__":
   RunSSHCommand(hostname, port, user, "register_netboot")

Formatting code for HTML

2008-04-03T15:46:00.001-07:00

This being my first blog I didn't have any tricks in my bag for formatting code. After some Google searching I only found two projects that looked useful. The first being this project which wasn't language aware and couldn't do any syntax highlighting. The second more promising project was this one and fortunately he posted the C# source for his library. Only problem is that it didn't support F# or Python. I went ahead and added support for those two languages, created a front end for the thing and added support for user specified CSS documents. In addition to exposing most of the built-in functionality my app has a preview mode which will display the formatted code in an embedded IE pane. I've posted my version of the library here, and I've also made available a pre-compiled binary.

The no-frills guide to .Net COM interop

2008-03-31T17:11:00.000-07:00

So there a couple of guides out there that already cover some aspects of this post, but none of them cover all of the topics and many of them are convoluted. When I publish something to COM I usually end up consuming it via a scripting language like Python, Perl or VBScript so this little howto will reflect what I have found to be best practices. So here is the no-frills guide to exposing a C# class to COM.

Create an interface
Create a class that inherits from the interface
Add GUIDs to both interface and class using the "Guid" attribute define in "System.Runtime.InteropServices". You can use the GUID generation tool found in the "Tools" menu of Visual Studio.

using System;
using System.Runtime.InteropServices;

namespace MyNamespace{

    [Guid("EFB47A26-D84F-4092-927D-232E92FB77E8")]
    public interface IMyComponent{
        [DispId(1)]
        void DoStuff();
    }

    [Guid("8F3DFCDB-45F7-461d-9B19-B140CD6F05FA")]
    public class MyComponent : IMyComponent{
        public void DoStuff() { }
    }
}

Add the interface type to the interface, you don't need to know all the details but you should probably use the Dispatch type if this component is being consumed by a dynamically typed language such as Python.

Add the DispId to each method exposed by the interface. Again this helps when being consumed by Python. A note of caution though, that once these values are set and you register the component they should not change. It's not the end of the world if they do, but you will cause headaches for yourself.

[Guid("EFB47A26-D84F-4092-927D-232E92FB77E8")]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
public interface IMyComponent {
    [DispId(1)]
    void DoStuff();
}

Add the ProgId attribute to the class, set the value to Namespace.ClassName. This is again one of those things that may not be necessary but helps when using Python, Perl etc...
Add the ClassInterface attribute to the class setting the value to "ClassInterfaceType.AutoDispatch"
Add the ComDefaultInterfaceAttribute setting the value to the type of your interface. Again, not neccessarily required but Python has all sorts of trouble generating the wrapper classes if it isn't present.

[ProgId("MyNamespace.MyComponent")]
[ClassInterface(ClassInterfaceType.AutoDispatch)]
[ComDefaultInterfaceAttribute(typeof(IMyComponent))]
[Guid("8F3DFCDB-45F7-461d-9B19-B140CD6F05FA")]
public class MyComponent : IMyComponent {
  public void DoStuff() { }
}

You need to enable two options in Visual Studio for your project. Open the properties for your project, and go the the application tab. Click on the "Assembly Information" button, and enable the make COM visible option. On the build tab you will also need to enable the register for com interop option at the bottom. You will also need to sign your assembly; you can use a self generated key through the signing tab of your project's properties. Once compiled Visual Studio will auto-register the assembly with COM so that you can test the interface with a script or OLEView. You can use regasm or build a Wix project to have it installed permanently.

using System;
using System.Runtime.InteropServices;

namespace MyNamespace
{

    [Guid("EFB47A26-D84F-4092-927D-232E92FB77E8")]
    [InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
    public interface IMyComponent{
        [DispId(1)]
        void DoStuff();
    }

    [ProgId("MyNamespace.MyComponent")]
    [ClassInterface(ClassInterfaceType.AutoDispatch)]
    [ComDefaultInterfaceAttribute(typeof(IMyComponent))]
    [Guid("8F3DFCDB-45F7-461d-9B19-B140CD6F05FA")]
    public class MyComponent : IMyComponent{
        public void DoStuff() { }
    }
}

Playing around in WMI with F#

2008-03-28T10:49:00.000-07:00

Started playing around with WMI in F# in order to update DNS records. After several hours I finally got it working. I ran into some problems on the last line of the function below. Apparently function used with iter() must return "unit" which is the F# equivalent of null. My anonymous function was returning an obj, appending the "; ()" to the end of the function fixed the return type. "()" in F# is the notation for the unit type. The guys from F# hub said this could also be done by using "|> ignore".

let UpdateCNameDNSRecord(record:string, value:string) =
    let query = String.Format("SELECT * FROM MicrosoftDNS_CNAMEType WHERE OwnerName='{0}'", record) in
    let scope = (@"\\.\root\MicrosoftDNS") in //new ManagementScope
    let searcher = new ManagementObjectSearcher(scope, query) in
    if (not (value.EndsWith("."))) then
        let value = (value + ".") in
    
    let inputP = [|null; box(value)|] in 
    IEnumerable.iter(
        
            fun (x:ManagementObject) ->
                if (x.Item("PrimaryName").ToString() <> value) then
                   x.InvokeMethod("Modify", inputP) |> ignore
            
       ) (IEnumerable.untyped_to_typed(searcher.Get()));;

An example win32 service in F#

2008-03-26T10:51:00.000-07:00

I needed a small single purpose TCP socket server for work, and I thought it might be an opportune time to try a new programming language. My choices were Scala, F# or Erlang. The target platform was a win2k box and I wanted to be able to register it as a service so F# was the obvious choice. In the code below I modified the echo server example from the MIT F# site to run as a service.

// This sample code is provided "as is" without warranty of any kind. 
// We disclaim all warranties, either express or implied, including the 
// warranties of merchantability and fitness for a particular purpose. 

#light
#r "System.ServiceProcess.dll";;
#r "System.Configuration.Install.dll";;


// Echo Server: opens a service on the localhost which bounces data straight back  

open System.Net
open System.Net.Sockets
open System.Threading
open System.ServiceProcess
open System.ComponentModel
open System.Configuration.Install
open Microsoft.FSharp.Compatibility.CompatArray

let spawn f = (new Thread(new ThreadStart(f))).Start()

type EchoServer = class
    inherit ServiceBase as base
    
    new() as this = {} then base.ServiceName <- "SimpleEchoService"
    
    override this.OnStart(args:string[]) = 
        spawn(this.listen)
            
    member this.listen() =
        let sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.IP) 
        sock.Bind(new IPEndPoint(IPAddress.Any, 5120))

        let backlog = 2 // number of incoming connections that can be queued for acceptance
        sock.Listen(backlog)

        let port = (sock.LocalEndPoint :?> IPEndPoint).Port 
        Printf.printf "server accepting on port %d...\n" port

        while (true) do
            let csock = sock.Accept() 
            Printf.printf "the server has accepted a client on port %d...\n" port;
            spawn(fun () -> 
                let total = ref 0 
                try 
                    let buf = Bytearray.create 64
                    let stream = new NetworkStream(csock) 
                    while true do
                        Printf.printf "server reading...\n";
                        let nread = stream.Read(buf,0,64) 
                        Printf.printf "server read %d...\n" nread;
                        stream.Write(buf,0,nread);
                        total := !total + nread;
                    done;
                with 
                    | :? Sockets.SocketException 
                    | :? System.IO.IOException -> 
                        Printf.printf "client gone: server read %d bytes from that client\n" !total;
                        flush stdout
            )
        done

end