I've been playing around with SparkleShare, Amazon AMI Linux and S3 which required building a bunch of RPMs. I'm posting them here so that others don't have to.
For S3FS:
Figured out how to get all of the Cassandra key/value pairs for a column using the thrift API. The documentation on top of the horrible naming scheme definitely doesn't make it easy to figure things out quickly. Below is an example in Python
#/usr/bin/env python
from thrift import Thrift
from thrift.transport import TTransport
from thrift.transport import TSocket
from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated
from cassandra import Cassandra
from cassandra.ttypes import *
import time
socket = TSocket.TSocket("localhost", 9160)
transport = TTransport.TBufferedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)
client = Cassandra.Client(protocol)
keyspace = "MyUberSite"
user_uuid = "0ad503dd-2642-4a1e-9113-a75bfd183c34"
try:
transport.open()
print "UUID: ", user_uuid, "\n"
# Create a column (user record) who's name is a UUID.
# Add two key/value pairs: one for email, one for username
column_path = ColumnPath(column_family="Users", column=user_uuid)
client.insert(
keyspace,
"email",
column_path,
"email@example.com",
time.time(),
ConsistencyLevel.ZERO)
client.insert(
keyspace,
"username",
column_path,
"user_account_name",
time.time(),
ConsistencyLevel.ZERO)
# Which column family are we interested in querying.
column_parent = ColumnParent(column_family="Users")
# A slice dictates our start and stop for column names we are interested in.
# We only want records for one column (user) so we are going to set the
# start and stop to the same values which is the UUID of the user we
# created above.
slice_range = SliceRange(
start=user_uuid,
finish=user_uuid)
# Create our predicate using the range instantiated above.
predicate = SlicePredicate(slice_range=slice_range)
# We want all of the column's (user's) keys so we are going to specify an
# empty key range. If we wanted a subset of the columns keys we could
# specify that subset here. The range is from start to stop using the
# sort we specified when we declared our column family in storage-conf.xml.
key_range = KeyRange("", "")
# Perform the query and pring the results.
print "\n\n".join(
map(
repr,
client.get_range_slices(
keyspace,
column_parent,
predicate,
key_range,
ConsistencyLevel.ONE)))
except Thrift.TException, tx:
print 'Thrift: %s' % tx.message
finally:
transport.close()
When troubleshooting a multi-threaded server it's sometimes nice to be able to split up the resulting log file by process making it a little bit easier to figure out what's going on. In this instance I'm splitting the log files from PostgreSQL by process ID which happens to be field 4.
# input format: 2010-01-25 15:41:46 PST [6534]: [30900-1] LOG: duration: 0.243 ms statement: SELECT ...
awk '{print > "substr($4, 2, length($4) - 3)"}' enroll.log
This is an AWK one liner that comes in handy. It will strip newlines from it's input replacing them with a single space. I often use it in conjunction with things such as svn, cut, sed and grep.
awk -v RS="\n" -v ORS=" " {print}
These are my notes for installing Thrift on OSX Snow Leopard. The two pre-requisites are XCode with the 10.4 SDK, which is not installed by default and MacPorts. This is a mashup of three similar guides, none of which contained all of the necessary information to get up and running.
export LOG4J_PATH='/opt/local/share/java/jakarta-log4j.jar'
export THRIFT_URL='http://gitweb.thrift-rpc.org/'
export THRIFT_PARAMS='?p=thrift.git;a=snapshot;h=HEAD;sf=tgz'
sudo port install boost
sudo port install jakarta-log4j
sudo port install pkgconfig
sudo port install libtool
echo "thrift.extra.cpath = ${LOG4J_PATH}" > ~/.thrift-build.properties
curl "${THRIFT_URL}${THRIFT_PARAMS}" > thrift.tgz
tar xzf thrift.tgz
cd thrift
export CFLAGS="-arch i386 -m32"
export CXXFLAGS="$CFLAGS"
cp /opt/local/share/aclocal/pkg.m4 aclocal/
./bootstrap.sh
./configure --prefix=/opt/local
make
sudo make install
I've been teaching myself Verilog and just recently got VGA working. More info and full source soon.
A while ago I wrote a screen saver for OSX as a learning exercise. I wanted to learn how to access the camera built into my Macbook Pro and get my feet wet with OSX. I was going to integrate face recognition using the OpenCV or MPT libraries but never got around to it. I'm posting it now because some of the guys at work saw it and wanted a copy. I added the source to my SVN repository and made a binary available via the download site. The screen saver is really as minimalistic as it gets. It uses the new QTKit API for accessing the camera and sets the capture view to the frame size of the inherited ScreenSaverVeiw class. The QTKit API came out around the time of Leopard's release and is much simpler and has better performance then the old streamgrabber API which had been around since the OS 9 days. To install the binary just download the zip file, unzip and double click. It will automatically install itself and become available as a screen saver in the system preferences.
I recently obtained the "Dutch" computational geometry book to supplement the other computational geometry book that I have. Instead of doing what I normally do which is skip directly to the topic that I'm interested in, I've resolved to work through the entire book systematically implementing each of the algorithms in F# or some other functional language. The purpose is two fold; I know almost nothing about the topic and it's hard to find anything about computational geometry implementations in a functional language. Who knows, someone may actually find my floundering helpful.
Chapter one is all about convex hulls and after doing a little bit of reading I determined that the "Jarvis March" algorithm looked like a good place to start. I've only included core part of the algorithm here in the blog post. The entire source including the visualization can be found here.
The algorithm is pretty simple I start first by sorting the point list by the Y coordinate and use the minimum as the starting point for my hull. I start working through the list trying to find the left most point in relation to the previous point added to the hull; this is the fold_left bit of the _jarvis function. Once I've found the left most point I append it to the head of the hull list and make a recursive call to _jarvis. When I reach the point with the maximum Y value I know that I have successfully built the left side of the hull. I then repeat the entire process but this time instead of searching for the left most point I search for the right most point building up the right side of the hull. When done I append these two lists together to form the complete hull.
type Point = {
x :double;
y :double;
} with
static member ( = ) a b =
a.x = b.x && a.y = b.y;
static member ( <> ) a b =
not(a = b);
end;;
type Edge = {
orig:Point;
dest:Point;
} with
static member ( = ) a b =
a.orig = b.orig && a.dest = b.dest;
static member ( <> ) a b =
not(a = b);
end;;
let sort point_list =
List.sort( fun p1 p2 ->
match Pervasives.compare p1.y p2.y with
| 0 -> Pervasives.compare p1.x p2.x;
| _ as v -> v;
) point_list;;
// Input: three points P0, P1, and P2
// Return: >0 for P2 left of the line through P0 and P1
// =0 for P2 on the line
// <0 for P2 right of the line
let testSide p0 p1 p2 =
(p1.x - p0.x) * (p2.y - p0.y) - (p2.x - p0.x) * (p1.y - p0.y);;
let isLeft p0 p1 p2 =
(testSide p0 p1 p2) > 0.;;
let isRight p0 p1 p2 =
(testSide p0 p1 p2) < 0.;;
let rec _jarvis cmp max_point (hull_list:Point list) (input_list:Point list) =
match input_list with
| p::tl ->
let last = List.hd hull_list in
if last = max_point then hull_list
else (
let best = List.fold_left(fun p1 p2 ->
if (cmp last p1 p2) then p2
else p1;
) p input_list in
_jarvis cmp max_point (best::hull_list) tl
)
| [] -> hull_list;;
let jarvis point_list =
if List.length point_list < 3 then failwith "3 points are required for a polygon";
let point_list = sort point_list in
let miny = point_list |> List.hd in
let maxy = point_list |> List.rev |> List.hd in
let left = _jarvis (isLeft) maxy [miny] point_list in
let right = _jarvis (isRight) maxy [miny] point_list in
right @ left |> List.rev;;
So I've spent my time since my last post basically re-writing the entire Boids project. I've thrown out the GDI UI and have replaced it with an XNA based one. I also rewrote my Boid type to be a class which inherits from the Body type in the Farseer 2D physics engine, making the simulation much more realistic. I’ve now got mass, rotation, torque, collisions, drag and friction to play with.
I've also refined and rewritten most of my rules, fixing the stationary swarm problem, and have added more random noise making things much more interesting. Random Boids will peel off from the group, wander around and then later rejoin. Sometimes the herd separates into two or more sub-groups then later coalesces back into a single herd. There is definitely some additional tweaking of the rules to be done, but I’m definitely headed in the right direction.
Oh, and this is one of my favorite parts, I replaced the dots that represented the Boids in the old simulations with little buffalo which rotate to reflect the actual heading of the Boid. So I have a little herd of buffalo roaming around in my computer, for some reason that makes me very happy.
This being a learning exercise, I separated the F# rules and logic into multiple files so that I could play with modules and namespaces, and have now tackled almost every subject in the Expert F# book. I don’t quite live in F# yet, but I’m getting much faster and am programming with fewer mistakes which is a good sign.
Playing with XNA was also quite fun. I am by no means utilizing all of the functionality provided, but from the limited area that I have touched it appears that Microsoft did a pretty good job. I played with OpenGL a couple of years ago, creating a planetary body orbit/gravity simulation with my roommate, and I remember it being painful. With XNA this was definitely not the case, it was very simple to get something up and running quickly. I am by no means a Microsoft fan-boy but you have to give them credit, they do make nice development tools.
So where do I go next? I need to do some mundane things like tweak the existing rules, and port all of the XNA code from C# to F#. I also need to go through and clean up some of the code, I know I’m not doing several things very efficiently and as a result my laptop has trouble with 100 or more Boids. I also want to implement several more rules: herd leaders, predator/prey behavior, obstacle avoidance etc… I still want to rewrite the nearest neighbor search portion of the code, but that is slower going. One of the problems of being a college drop-out is that I never took discreet math or linear algebra, so understanding some of the algorithms takes a little bit more time than I anticipated. It’s not exactly hard, just requires more research on my part but it’s good for me. Also now that I’ve been reading computational geometry books I see solutions to problems that I encounter all the time, I just wasn’t aware of the body of work that existed. I don’t feel too guilty though, I don’t think any of these algorithms were taught at an undergraduate level anyways at the U of Arizona. Who knows, when I was there 2001-2004 the program was in a real state of decline and was rife with in-fighting and I was all too eager to leave.
FYI: The project has gotten too big to keep posting code in the blog, so I created a project on Google Code.
A friend of mine was in town this week, and haven't had much spare time to play with my side projects. When I finally got down to it I was able to redo my boids simulation to use operator overloading, boundary rules, and improved some of the rendering. I based my implementation off of a couple of different sources: a pseudo-code explanation of the original boids paper, a Python implementation, and this Java implementation. There is one remaining problem that I have yet to figure out. I originally used the rules laid out in the Python implementation, but the animation was very jerky when rule 3 took affect, and once the dots swarmed together they just sit their stationary. Finding no difference between his implementation and my own I started looking around for other examples that might give me an insight. That is when I found the previously mentioned Java implementation which resulted in a much more smooth trajectory. Unfortunately it still did not fix the problem where after swarming together the boids don't go anywhere. I'm going to continue to work on it but wanted to post my results thus far.
open System;;
open System.Windows.Forms;;
open Microsoft.FSharp.Collections;;
let BOID_COUNT = 40;;
let HEIGHT = 700.;;
let WIDTH = 1000.;;
let WALL = 50.;;
let WALL_FORCE = 30.;;
let MAX_VEL = 800.;;
let R1_CONST = 0.2;;
let R2_CONST = 0.2;;
let R2_RANGE = 100.;;
let R3_CONST = 0.2;;
let FPS = 24;;
let RECT_HW = 3;;
type vector = {
x : double;
y : double;
} with
member v.mag() =
(v.x ** 2.0) + (v.y ** 2.0) |> Math.Sqrt;
static member ( + )(a:vector, b:vector) =
{x = a.x + b.x;
y = a.y + b.y};
static member ( - )(a:vector, b:vector) =
{x = a.x - b.x;
y = a.y - b.y};
[]
static member ( * ) (a:vector, b:int) =
let bd = Convert.ToDouble(b) in
{x = a.x * bd;
y = a.y * bd};
[]
static member ( * ) (a:vector, b:double) =
{x = a.x * b;
y = a.y * b};
[]
static member ( ** ) (a:vector, b:int) =
let bd = Convert.ToDouble(b) in
{x = a.x ** bd;
y = a.y ** bd};
[]
static member ( ** ) (a:vector, b:double) =
{x = a.x ** b;
y = a.y ** b};
[]
static member ( / ) (a:vector, b:int) =
let bd = Convert.ToDouble(b) in
{x = a.x / bd;
y = a.y / bd};
[]
static member ( / ) (a:vector, b:double) =
{x = a.x / b;
y = a.y / b};
static member ( > ) (a:vector, b:double) =
a.mag() > b;
static member ( < ) (a:vector, b:double) =
a.mag() < b;
end;;
type boid = {
pos : vector;
vel : vector;
} with
static member ( + )(a:boid, b:boid) =
{pos = a.pos + b.pos;
vel = a.vel + b.vel};
static member ( - )(a:boid, b:boid) =
{pos = a.pos - b.pos;
vel = a.vel - b.vel};
member b.distance b2 =
Math.Sqrt(Math.Abs(((b.pos.x - b2.pos.x) ** 2.0) - ((b.pos.y - b2.pos.y) ** 2.0)));
member b.update_pos() =
{b with
pos = b.pos + b.vel};
member b.neighbors(r, bl) =
[for n in bl ->
(b.distance(n), n)] |>
List.filter(fun (d:double, _) -> d < r) |>
List.sort(fun (d1:double, _) (d2:double, _) -> d1.CompareTo(d2)) |>
List.tl |>
List.unzip;
end;;
let rand = new Random();;
let rand_double(max:double) =
Convert.ToDouble(rand.Next(Convert.ToInt32(max) - 1)) + rand.NextDouble();;
let vector_init =
{x = 0.;
y = 0.};;
let boid_init =
{pos = vector_init;
vel = vector_init};;
let boid_sum_list bl =
List.fold_left(+) boid_init bl;;
let boid_adjust_v_to_fps b =
{ b with
vel = b.vel / (Convert.ToDouble(FPS))};;
let boid_limit_vel b =
let v = b.vel.mag() in
match v with
| _ when v > MAX_VEL ->
let vl = v / MAX_VEL in
{ b with
vel = b.vel / vl}
| _ -> b;;
let boid_check_x_bounds b =
match b with
| _ when b.pos.x < WALL ->
{b with
vel = {b.vel with
x = b.vel.x + WALL_FORCE}};
| _ when b.pos.x > (WIDTH - WALL) ->
{b with
vel = {b.vel with
x = b.vel.x - WALL_FORCE}};
| _ -> b;;
let boid_check_y_bounds b =
match b with
| _ when b.pos.y < WALL ->
{b with
vel = {b.vel with
y = b.vel.y + WALL_FORCE}};
| _ when b.pos.y > (WIDTH - WALL) ->
{b with
vel = {b.vel with
y = b.vel.y - WALL_FORCE}};
| _ -> b;;
let boid_rule1 b_sum count =
fun (b) ->
{b with
vel = b.vel + ((((b_sum.pos - b.pos) * (1. / (count - 1.))) - b.pos) * R1_CONST) };;
let boid_rule2 bl count =
fun (b:boid) ->
let (_, nbl) = b.neighbors(R2_RANGE, bl) in
let r2b = List.fold_left(fun b1 b2 -> b1 - (b2 - b)) boid_init nbl in
{ b with
vel = b.vel + ((r2b.pos * (1. / (count - 1.))) * R2_CONST) };;
let boid_rule3 b_sum count =
fun b ->
{b with
vel = b.vel + ((((b_sum.vel - b.vel) * (1. / (count - 1.))) - b.vel) * R3_CONST) };;
let boid_apply_rules b bl b_sum count =
b |> boid_check_y_bounds |> boid_check_x_bounds |> boid_rule1 b_sum count |> boid_rule2 bl count |>
boid_rule3 b_sum count |> boid_limit_vel |> boid_adjust_v_to_fps |> fun b -> b.update_pos();;
let get_wall() =
let height = Convert.ToInt32(HEIGHT) in
let width = Convert.ToInt32(WIDTH) in
let wl = Convert.ToInt32(WALL) in
[|new Drawing.Point(wl, wl);
new Drawing.Point(wl, height - wl);
new Drawing.Point(width - wl, height - wl);
new Drawing.Point(width - wl, wl)
|];;
type BoidControl = class
inherit UserControl as base
val mutable boidList:(boid list);
val brap:(Drawing.Point array);
val count:double;
val timer:Timer;
new(bl:(boid list)) as this = {
boidList = bl;
count = bl |> List.length |> Convert.ToDouble;
timer = new Timer();
brap = get_wall();
} then this.init();
member this.init() =
let height = Convert.ToInt32(HEIGHT) in
let width = Convert.ToInt32(WIDTH) in
let size = new Drawing.Size(width, height) in
base.Width <- width;
base.Height <- height;
base.MinimumSize <- size;
base.MaximumSize <- size;
base.BackColor <- Drawing.Color.White;
base.SetStyle (ControlStyles.UserPaint, true);
base.SetStyle (ControlStyles.DoubleBuffer, true);
base.SetStyle (ControlStyles.AllPaintingInWmPaint, true);
this.timer.Interval <- FPS;
this.timer.Tick.Add(fun _ ->
this.tick(););
this.timer.Start();
member this.boids_update() =
let b_sum = boid_sum_list this.boidList in
let al = [for b in this.boidList -> async { return boid_apply_rules b this.boidList b_sum this.count }] |> Async.Parallel in
this.boidList <- (Async.Run(al) |> Array.to_list);
();
override this.OnPaint e =
let g = e.Graphics in
List.iter(fun b ->
let x = Convert.ToInt32(b.pos.x) in
let y = Convert.ToInt32(b.pos.y) in
let bp1 = new Drawing.Point(x - RECT_HW, y - RECT_HW) in
let bp2 = new Drawing.Point(x - RECT_HW, y + RECT_HW) in
let bp3 = new Drawing.Point(x + RECT_HW, y + RECT_HW) in
let bp4 = new Drawing.Point(x + RECT_HW, y - RECT_HW) in
g.FillPolygon(Drawing.Brushes.Red, [|bp1; bp2; bp3; bp4|]);) this.boidList;
g.DrawPolygon(Drawing.Pens.Blue, this.brap);
member this.tick() =
this.boids_update();
this.Invalidate();
end;;
[]
do
let height = Convert.ToInt32(HEIGHT) in
let width = Convert.ToInt32(WIDTH) in
let size = new Drawing.Size(width, height) in
let bl = [for x in 1..BOID_COUNT -> {pos = {x = 50. + rand_double(WIDTH - 50.);
y = 50. + rand_double(HEIGHT - 50.)};
vel = vector_init }] in
let mw = new Form() in
let bc = new BoidControl(bl) in
mw.Controls.Add(bc);
mw.ClientSize <- size;
mw.MaximumSize <- mw.Size;
mw.MinimumSize <- mw.Size;
mw.Text <- "Boids";
mw.Show();
Application.Run(mw);;