bvanderveen.com

`this` Is Object-Oriented Programming

2012-02-08T00:00:00-08:00

A method is a function whose first argument is this. Methods hanging from this are functions with this a curried into them. this contains data which is sometimes unneeded by the function into which it is curried. The information that this conveys is mutable, and could be in any state.

In programming, we make functions which operate on vectors of high-level datatypes. Perhaps the old-beards wanted objects so they could more easily operate on structs, being as they are the highest of high-level C data types. C programs tend to contain functions that operate on structs, and they’re often grouped by the type of their first argument. We don’t have to squint hard to see a simple class-based object system.

Now, it’s so easy to add fields and methods to objects. We don’t have to type out all those typedefs and pointer dereferences and allocations and deallocations and header files. We have ninjas maneuvering heroically in our codebases. The gravity of our C structs does not imbue our classes.

Testing a method involves more than just passing arguments to it and checking to see if its return value is correct; it is necessary to prepare this before testing a method and verify the state of this afterwards. The widely-adopted convention of keeping this private to its methods makes this difficult–since the test cannot directly read or mutate this, the object’s other methods must be used to set up this before the test, and used again after the test to ensure that this has been mutated as expected. No longer is a single function under test; the whole set of functions—the object’s behavior—is under test.

Writing object-oriented code requires constant vigilance. Think twice about adding or modifying an instance variable. Every method on an object should depend on every field on the object. Make a clear distinction between what is absolutely necessary and what can be calculated. Ensure that every callee can be, in whatever way is most idiomatic, curried into its caller.

In .NET, open source does not beget open source

2012-01-09T00:00:00-08:00

Ruby, Python, Perl, Node, and Clojure all have, to varying degrees, a unified story for defining a project, translating its source code into runnable code, running a test suite, building distributable packages, and resolving and installing a project or package’s dependencies. These stories just work across all supported platforms.

Setting up a cross-platform open source .NET project is non-trivial.

.NET open source developers jump through crazy hoops to try to get their projects to a point where anyone can check out the code from source control, build, and test the software on both .NET and Mono. What follows is a description of how I approached this task. It was painful to devise this solution and it continues to be painful to maintain.

Step 1: Build

We want to build our source code into .DLLs. We’ve got MSBuild, and Mono maintains it’s own quirky clone of MSBuild, called xbuild. They both read in .sln files, which are totally opaque and cannot be generated by hand, and .csproj files, which are XML, but hey they’re hand-editable. Okay. Ignore for a second that MonoDevelop and Visual Studio will fight over the formatting of those files, stomp on each other changes, and add garbage that’s meaningless to the other. A VS- or MD-generated .csproj will be default import some target definitions called Microsoft.CSharp.targets. Uh oh, what’s that? Where does that live on my system? What does it do? What package do I have to install to get that?

Now we’re in IDE-vs-IDE-vs-source-control hell, just so we can get a build running with two separate implementations of an XML-based project specification. But it’s okay, it’s cool. Let’s assume we’ve got that figured out and we’ve got a cross-platform source-code-on-disk to DLLs-on-disk transformation. With the caveat that we’re using the msbuild command on Windows and xbuild everywhere else.

Step 2: Test

Now I’m ready to run tests. How am I supposed to do that again? NUnit? XUnit? Something else? What’s the difference between all of those? Does Microsoft have their own test framework/runner? Is there a Mono clone?

Okay, I picked a test framework and runner. It doesn’t matter which one because they’re all the same. Now my code has a dependency on the DLL that defines the test framework. Well, I don’t want to include that DLL in source control because checking binaries into source control is a no-no. I guess I’ll just tell my contributors that they need to have TestFrameworkX-versionY.Z installed in the GAC to hack, and I’ll just bite the bullet and field the inevitable build problems and complaints about a GAC dependency on the mailing list.

Okay so I’m building my tests (and all my contributors are too). Now I want to run them. Hopefully I picked something that’s portable between .NET and Mono and I can easily run those tests on both platforms. Hopefully myself and my contributors can have a great test debugging experience whether we’re using MonoDevelop or Visual Studio. Hopefully.

Step 3: Repeat

I can build my code and I can run tests. Sweet! Now I want to set up continuous integration so I can have nightly builds and maybe some benchmarks to track the performance of my codebase over time. I’d like the system to email my devs if they check in broken code. It want to do automated builds on both .NET and Mono to ensure that my codebase works on both.

So I set about writing a build script that my CI system can run. Hm. I guess I could write an MSBuild file, but do I really want to maintain an XML build file? Will it be able to everything I need? Will I run into platform-dependent hiccups?

Oh hey, Rake is pretty nice, it’s well-supported on both Windows and Unix, and someone put together a nice suite of .NET-specific Rake tasks. Okay, I write a Ruby script to build my .NET projects. Pretty nice! I have separate commands for msbuild and xbuild to build the .csproj files that my IDE understands, but hey, just an implementation detail, right?

I also want to test that the Mono-compiled DLLs work on .NET and vice-versa. Gonna have to add some plumbing for that…

Step 4: Ship

I’ve got nightly builds that are tested on both .NET and Mono, so I have a pretty solid codebase! Okay, let’s get it into the hands of users as easily as possible. How are users getting my code these days? Should I tell them to check out and build the source with my Ruby script? What if they build from their IDE, I guess I should make sure that story works too. But ideally I should also ship binaries so users don’t have to compile the project themselves

Oh hey, NuGet sounds like pretty hot way to distribute binaries! Let’s investigate it since it’s all new and fancy. Maybe I can use this to resolve my dependency on the test framework DLL while I’m at it. Okay, now I’ve changed my Ruby build script to use NuGet to go download the test framework package before it builds my tests. Ah crap, NuGet blows up on Mono. Well, guess that test framework dependency stays in the GAC, and I must change my Rake script so it only uses NuGet to create the NuGet package for my project if it’s running on Windows.

Ah dang, you know, this also means that only Windows user are going to be able to use my project’s NuGet package. I’ll have to make my Ruby build script generate a zip file containing my project’s DLLs. Doing it in Ruby is a heck of a lot easier than using the ZIP task in MSBuild and I don’t have to worry about cross-platform issues.

So, my .NET users can use Visual Studio and NuGet to take a dependency on my project, and my Mono users can have the ZIP file.

Step 5: Go forth and innovate!

Not so fast. Nancy, FubuMVC, Kayak, and Gate all use Rake to perform their builds, and as far as I recall those Rake scripts all branch on whether the current platform is Mono or .NET to accomplish that task. We are all using slightly different set-ups, and there’s a ton of duplication of effort.

If any of our users want to create an open source project that depends on our open source project, they’re going to have to figure out and reimplement everything described above. The tooling to allow developers to rapidly create, distribute, and collaborate on portable modules doesn’t exist in the .NET ecosystem.

Open source does not beget more open source in .NET, because the barriers to standing on each other shoulders are so high.

These are not unsolvable problems, but as far as I can tell, no one in the community has both resources and incentive to solve them.

Sequence, UITableView

2011-12-19T00:00:00-08:00

A great usecase for Sequence is to declaratively construct content for UITableView. As the user scrolls the table view, it pulls row data from a UITableViewDataSource for the row visible on the screen. In this post, I discuss how Sequence supports this usecase and demonstrate how to use Sequence for this purpose.

The seeking problem

Our implementation of itemAtIndex: supports short-circuit partial materialization of the sequence, but it doesn’t help us if we’re seeking to arbitary positions within the sequence. Suppose we had a table view with 1,000,000 rows of content represented by a sequence. If we call itemAtIndex: on that sequence to retrieve row information while we’re scrolling around the bottom of that list, we are calling itemAtIndex: with an argument of ~1,000,000 every few frames of animation. Our implementation of itemAtIndex: runs in O(n) time, so our table view animation grinds to a halt.

The best way to seek within a sequence is if you can jump to an easily-calculated offset. Arrays are great for this—you take a pointer to the start of the array, add the product of the desired index and the array element width, and you’ve calculated a pointer to the item at that index in O(1) time.

In order to support our table view scenario, we’ll make a time-space trade-off—we’ll copy the values of the sequence into a contiguous array, so we can easily seek to arbitrary positions within our array. We can do this by calling array on a sequence.

Table view

Let’s take a look at an abbreviated controller which specifies its contents in a declarative fashion. First, let’s introduce a construct which represents the content and behavior of a table row:

// an object which represents a single row of table content
@interface TableItem : NSObject {
    UITableViewCell *(^cell)();
    void (^action)();
}

// returns a cell to display the row
@property (nonatomic, copy) UITableViewCell *(^cell)(); 

// returns an action to be called when the row is selected
@property (nonatomic, copy) void (^action)(); 

@end

Now, a trivial table view implementation using that construct:

@implementation SequenceTableController

- (id)init {

    ...

        self.content = [[[[Seq rangeWithStart:1 end:500] // a sequence of numbers, 1..500
        filter: ^ BOOL (id i) { return [i intValue] % 2 == 0 }] // filter out the odd numbers
        map:^ id (id n) { // return a TableItem to represent the row
            NSString *s = [n description]; // turn the NSNumber into a string
            TableItem *item = [[TableItem new] autorelease];
            item.cell = [[^ id () { return [self cellWithLabelText:s]; } copy] autorelease];
            item.action = [[^ void () { [self alertText:s]; } copy] autorelease];
            return item;
        }] array]; // materialize the sequence into an array

    ...

}

...

- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {
    return content.length;
}

- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath {
    TableItem *item = [content objectAtIndex:indexPath.row];
    return item.cell();
}

- (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath {
    TableItem *item = [content objectAtIndex:indexPath.row];
    item.action();
}

@end

Not too bad. I’m imagining all the ways I could use this to avoid ever implementing UITableViewDelegate or UITableViewDataSource again.

Check out the full source code on GitHub.

Sequence, push, pull

2011-12-18T00:00:00-08:00

When I wrote last week about Sequence, I was not satisfied with the implementation of operations like itemAtIndex:, any:, or all:. These operations fully materialized the entire sequence, but technically didn’t need to do so. This was because I hadn’t decided whether the canonical representation of a sequence should be push-style (items returned by producer to consumer) or pull-style (items passed to consumer by producer), and because the implementations of those operations were agnostic to whether the sequence was a push- or pull-form, they couldn’t provide partial materialization of the sequence.

Consider itemAtIndex:. To support retrieval of a single item from a sequence by index, the code performing the look-up needs to materialize a sequence up until the requested index, but does not need to materialize it any further to provide the correct value. This partial materialization can be accomplished provided the consumer of sequence values can give some of indication to the producer of sequence values instructing the latter to abort.

This could be done with a pull-style sequence if the value-receiving consumer function returned a BOOL indicating if it wanted more. This strikes me as a bit more conceptually tricky than the equivalent solution for the pull-style sequence form: the consumer simply stops calling the value-returning producer function when it no longer wants values.

Settled.

I decided to go ahead implementing itemAtIndex: using a pull-style form. It looks like this:

- (id)itemAtIndex:(NSUInteger)index {
    id (^next)() = impl(); 
    id item = nil;

    // iterates over items
    for (NSUInteger i = 0; (item = next()); i++) {
        if (i == index)
            return item;
    }
    return nil;
}

Here, we have an impl function which returns an id (^)() block called next. That block is called index times to get the corresponding element. If the the next function returns nil, the sequence has ended, and the result of itemAtIndex: is nil.

Other operations which can short-circuit work similarly:

- (BOOL)any:(BOOL(^)(id))predicate {
    id item = nil;
    
    for (id (^next)() = impl(); (item = next());)
        if (predicate(item))
            return YES;
    
    return NO;
}

- (BOOL)all:(BOOL(^)(id))predicate {    
    id item = nil;
    BOOL evaluatedItem = NO;

    for (id (^next)() = impl(); (item = next());) {
        evaluatedItem = YES;
        if (!predicate(item))
            return NO;
    }
    
    return evaluatedItem;
}

In both these cases, the impl function is called at the start of the operation to retrieve the item-producing function. That function is called until the result of the operation can be determined.

Node does IO.

2011-12-15T00:00:00-08:00

Consider these few lines from Mark McGranaghan’s post ClojureScript and Node.js:

(defn handler [_ res]
  (.writeHead res 200 (.strobj {"Content-Type" "text/plain"}))
  (.end res "Hello World!\n"))

Here is the equivalent JavaScript:

handler = function (_, res) {
    res.writeHead(200, { "Content-Type" : "text/plain" });
    res.end("Hello World!\n")
}

This is a Node.js HTTP request handler. When a request comes in, this handler function is called. It writes an HTTP response header, sends a bit of body, and calls it a day. Nothing fancy.

The variable res is unnecessary. This code doesn’t care about res-the-object; that’s just syntactical noise. What this code actually cares about is writeHead and end, the behaviors res-the-object provides.

The devils and angels bicker:

That isn’t fair, because what we have here is a writer pattern, which necessarily implies state—both writeHead and end return void. There is IO going on here, and functional programs don’t do IO! We need OOP!

or,

Implement a writer monad! Then abstract that idea it so the request handler returns a data structure which represents the response! This parallels the request data structure nicely and it’s so beautiful this way.

Who cares. Node does IO. Abstract it, that’s what it’s there for.

Goodbye, .NET

2011-12-11T00:00:00-08:00

I’ve been trying not to write this blog post for months, rage-quitting .NET being as it is almost a running joke in the web development community. I hope this isn’t a rage-quit. The fact is that today I can no longer reason away my misgivings about the situation in which I’ve found myself. Today I say goodbye to .NET.

I’ve met a number of awesome people during this journey, many of them extraordinarily talented engineers and forward-thinking, creative, and intelligent people. I am very glad to have had to opportunity to work with and learn from you. Keep working to improve your tools, whatever they may be.

I’ll spare this post my conspiracy theories. The bottom line is that .NET is not a technology that anyone uses for Getting Shit Done in my world. For me it was a toy. I mastered a lot of advanced topics whilst playing with .NET. But I have to be honest with myself: the .NET platform doesn’t pay me, and it no longer aligns with my personal creative goals. I would be a fool to spend another minute working with it.

I’ve seen how mixing up personnel in any project or organization can be a very positive thing, and I hope that my departure benefits the projects with which I was involved. Best of luck to everyone!

How to run Mono on Heroku

2011-12-10T00:00:00-08:00

What follows is an account of how I got Mono running under Heroku.

Part 1 – Compiling Mono

Get an account on Heroku.
Install the Heroku Toolbelt and follow the quickstart instructions.
Install vulcan. Make sure you’re using the latest Ruby.

gem install vulcan
Create a Vulcan build server on Heroku.

vulcan create mono-build
Vulcan needs a CouchDB instance to stuff code into. It’s hard-coded to use Cloudant, which is a free CouchDB provider. Sign up for an account and configure the URL of your database in your Heroku environment variables.

heroku config:add CLOUDANT_URL=https://USERNAME:PASSWORD@USERNAME.cloudant.com --app mono-build
Downloaded the Mono sources. I used the tarball but a clone of the Git repo will work too (you might want to check out a specific release tag).
Decide how you want to build Mono. I settled on this:

./configure --with-moonlight=no --enable-nls=no
Now we use Vulcan to build Mono on Heroku’s cloud. Important: you must tell Vulcan where the build artifacts are going to end up using the -p flag so that it can archive them and make them available for download. This location corresponds to the --prefix flag passed to the configure script. So make sure those match:

vulcan build -s ~/code/mono/mono-2.10.7/ -p /tmp/my-mono-build-output -c "./configure --with-moonlight=no --enable-nls=no --prefix=/tmp/my-mono-build-output && make install"
Go drink a beer (or two) whilst the beast is compiled.
You should have the build artifacts (which were deposited into the prefix dir by the make install command on Heroku) downloaded to your local system at /tmp/mono-2.10.7.tgz

Part 2 – Make a Heroku buildpack for Mono.

Sequence: a preview

2011-12-05T00:00:00-08:00

I’m cooking up a utility library for Objective-C called Sequence. It allows you to manipulate sequences of Objective-C objects using functional constructs such as map, reduce, filter, and the like. Check out Sequence.h for the full list of supported operations.

How to use it

Sequence declares category methods on NSObject. When you call one of these methods on a collection object such as NSArray, NSSet, or NSDictionary, those are treated like sequences. If you call one of the methods on a non-collection object such as NSNumber or NSNull, it is treated like a single-element sequence containing that object.

Example: map

NSArray *input = [NSArray arrayWithObjects:@"a", @"b", @"c", nil];

// all sequence-returning methods return a sequence typed as id
id mapped = [input map:^ id (id i) { return [i stringByAppendingString:i]; }];

// to materialize a sequence into an array, send it the -[ array] message.
NSArray *output = [mapped array]; // contains @"aa", @"bb", @"cc"

Example: reduce

NSArray *input = [NSArray arrayWithObjects:@"a", @"b", @"c", nil];
NSString *output = [input reduce:
	^ id (id acc, id i) { return [acc stringByAppendingString:i]; } seed:@""];
// output is @"abc"

Example: any

NSArray *input = [NSArray arrayWithObjects:[NSNull null], @"c", [NSNumber numberWithInt:0], nil];
BOOL containsString = [input any:^ BOOL (id i) { return [i isKindOfClass:[NSString class]]; }];
// containsString is YES

How it works

The implementation works with sequences of form (item -> unit) -> unit (called PushSeq) and unit -> (unit -> item) (called PullSeq). PullSeq ends by nil-termination, just like -[NSEnumerator nextObject].

For my first go at it, I made a conscious decision to keep most of the implementation agnostic of which sequence form is used. The consequence of this is that operations like any, all, itemAtIndex, etc. can’t “short-circuit” (return before materializing the whole collection) because their implementation can’t control how many items are enumerated.

Caveats

Operations that could short-circuit currently do not. This is a problem, for example, if you were use this to implement UITableViewDelegate or UITableViewDataSource—you wouldn’t want to have an intermediate array or two being allocated for every call to cellForRowAtIndexPath: when you only want to query a single item out of a sequence.

I’m still thinking about how to best rectify that. My goal is to allow users to define complex declarative structures and query them without allocating intermediate storage beyond the heap objects that make up the query itself.

As a next step, I would like to figure out how to both call and implement NSFastEnumeration. It’s rather obtuse and the docs are terse and highly general, but I’ve a hunch that Sequence could benefit from interacting closely with NSFastEnumeration.

Offline this week

2011-11-21T00:00:00-08:00

I will be taking a break from the internet over the Thanksgiving holiday. I’ll resume correspondence next week.

Testing is not a substitute for good design.

2011-11-17T00:00:00-08:00

On the compiler

2011-11-06T00:00:00-07:00

In every language that I’ve used, writing code is a war against the compiler or interpreter or the language grammar or whatever. It’s the thing allows you to talk to a CPU. The compiler is what stands between your intent and what the computer actually ends up doing. It is your enemy.

In the future, we won’t have compilers. We’ll have Siri.

New Gate Developments

2011-11-06T00:00:00-07:00

Writing Gate middleware is so damn easy. You just write some test case method names which, in plain English, describe the intent of the test. Then, you fill out the implementation of the tests. Then, you implement the middleware itself. It’s so damn easy. You just write out the implementation. You don’t even have to think. I don’t know why I didn’t do this ages ago.

Actually, I do, but that’s a story for another time.

I’ve written some middleware. I hate the word middleware.

HTTP Basic Authentication
Method Override
Content Length
Chunked Response

Those were built with two primitives: Predicate and Transform.

I want to make these others modules:

Header dictionary fixer-upper thing, ensures mutability and perhaps normalizes case.
Handle X-Sendfile
Static file server/directory index
Digest auth
Multipart form parsing + files
Caching/304/ETag
Deflate/Gzip
Lint (to verify OWIN compliance)

These already exist in Gate/Gate.Helpers and I want to move them to Gate.Middleware:

ContentType
RewindableBody
ShowExceptions
Map

Stuff that could exist but which hella exists way far out of the scope of Gate:

Session
Logging
Perhaps some kind of Compiler-as-a-service source reloading sort of thing which would probably be hosting dependent but I’m not sure.

I’m basically ripping these ideas straight off of Rack, but the foundation is compatible with both event-loop thread models like that of Node and Nginx and traditional thread-per-connection models like IIS and Apache.

What other sorts of server stuff would you like to see implemented?

Bank Transfer Day

2011-11-06T00:00:00-07:00

I spoke with bravado of walking into Chase and demanding that they dispense an amount of cash to me equal to the amount of money contained within my account. As I drove past the branch, there was one woman standing on the corner with a sing that said “Stop funding corporate greed, withdraw your funds.” I went to roll my window down and shout at her with my support, but I had a plastic fork in my hand, which I was holding afer having puchase a big-size Whole Bowl (vegan style), and was unable to simultaneously make a left turn and roll down my window and shout all in one fluid motion.

Needless to say, I failed to actually go to the bank and transfer my funds.

But, I wrote some middleware.

On jobs

2011-10-09T00:00:00-07:00

I’ve been doing poorly at my resolution to start doing more interesting things. I was out of town on business for a week and a half, and since then I’ve been very busy getting ready for user testing of my project at work.

In lieu of finding time or motivation or headspace to do visceral arty things, I am redoubling my efforts to make sure that the job I work at is a worthwhile pursuit. If I’m going to spend a third of my time at my job, I better be working on an awesome project.

I’m over coding. It’s a solved problem. Anything can be cross-compiled to anything else and then JITed or statically precompiled or whatever you insist is the flavor of the week. No problem. I want to operate in the domain of the bigger picture.

The days following the passing of Steve Jobs were interesting. I see now that I must act immediately.

On engagement

2011-09-15T00:00:00-07:00

I’ve been experimenting with a secret Twitter account with the privacy setting enabled. The idea was that I could tweet about things I might not want to have associated with my professional reputation or software community persona.

In practice, I don’t feel more at liberty to make spur-of-the-moment, drunken, revealing, profane, arty, or otherwise inappropriate tweets to my private followers than to my public followers. It started as an attempt to be more spontaneous, edgy, and entertaining, but has led me back to the same interactions I was trying to shake off: at best boring, and more often awkward, self-conscious, and anxious.

Google+ explicitly attempts to solve the different-subject-matter-for-different-contacts problem, but somehow it falls short of that goal. There are a lot of problems with Google+, most of which have been discussed in more depth and at greater length on Google+ itself than I will discuss here. But I will say this: the format is rather boring. You post, a few people might +1 and someone might even attempt witty banter, which inevitably becomes awkward and peeters out.

We’re left with the same problem we’ve been having: we desire to communicate a message, and do so through the channels to which we’ve become accustomed, and expect to get some utility out of having done so. Perhaps there is the momentary gratification of having received a notification of engagement from a contact, and we strive to keep up the volley for as long as it suits us.

This ad-hoc usage social media is naïve and mundane. We should take a more measured approach to engaging these channels and work to communicate a coherent message with a specific goal in mind. For me, the secret Twitter represents a fragmentation of personal goals—my public and private accounts both represent me, but choosing to engage one or the other ends up being an arbitrary decision made in the moment, and neither representation is improved.

My public Twitter account has value because it provides a somewhat coherent message about specific topic—I have about 350 followers, which have come very organically through my engagement with the open source software community. I’ve met some awesome people and had amazing experiences as a direct result of engagement through Twitter.

I can see that there is value to be created through that channel, but on the other hand, both my public Twitter persona and the projects with which I’ve built it seem to have reached a certain maxima in terms of audience and mindshare, despite a large investments of time and energy on my part. There are myriad technical, organizational, historical, and social reasons why this might be the case, but it’s clear to me that I’m not optimally managing the time and energy I devote to my public persona and projects.

With this in mind, and with the help of my Comcast-imposed exile from the internet, I am actively re-evaluating my method of engagement by seeking out collaborations, affectations, and technologies through subterfuge, appropriation, obscenity, and other classical methods of research.

Been offline

2011-09-14T00:00:00-07:00

I’ve been offline for the past few weeks. I moved to a new place at the end of August, and haven’t had the internet connection set up yet. I got round to calling Comcast a week ago. They made me sign a contract but can’t seem to give me a date for the installation. It’s pretty rad.

I’ve got lots of project ideas on my mind for the winter. I need to work on something with a visceral quality. I currently deal in the realm of abstractions perhaps more than I ought to.

I need collaborators. All this going it alone I’ve been doing seems insane. It would be easiest for me to break out of this habit in the context of an art project and/or business. To that end I am actively recruiting:

engineers and creatives
beautiful people
strategic partners

Hip-hop provides a model for this endeavor. The hip-hop industry makes awesome, visceral stuff that people love, and they get rich as hell doing it. They have a mind for the whole picture—creative, business, marketing, cross-promotion, everything. They’re not about to starve. They sell out before they even start.

Step one: buy an MPC.

Gate 0.1.4 Released

2011-08-19T00:00:00-07:00

Gate 0.1.4 is on NuGet. This release addresses an issue pointed out and tracked down by Miguel Jimenez—thanks Miguel!

Gate 0.1.4

When using the RescheduleCallbacks middleware, the request body onNext callback gets a properly rescheduled continuation if and only if the wrapping app (i.e., Kayak hosting) provides a non-null continuation.

Gate 0.1.2, Kayak 0.7.2

2011-08-15T00:00:00-07:00

There are some new releases of Gate and Kayak up on NuGet this week. The Gate release is a bug-fix release that addresses a few items that showed up since the first bits of Kayak/Gate/Nancy stack went out. The Kayak release adds some new APIs for getting request information and drops the explicit HttpMachine dependency.

Gate 0.1.2

Environment class no longer deep-copies the dictionary passed to its constructor, it simply wraps it (Louis DeJardin)
Fixed missing dependency in Gate.Wcf and Gate.AspNet packages.
Fixed missing query string OWIN environment variable in Gate.Kayak
Fixed method, path, and request header OWIN environment variables in Gate.Kayak to never be null.

Many thanks to Rob Horvath and Håkan Canberger for spotting these bugs!

Kayak 0.7.2

HttpMachine is no longer a dependency; the HttpMachine source is compiled directly into the Kayak DLL.
Added Path, QueryString and Fragment fields to HttpRequestHead struct

Again, the best way to start using Nancy on Kayak is to create a new console application project, and pull in the Nancy.Hosting.Owin and Gate.Kayak NuGet packages. These two packages will bring all the necessary dependencies with them and you can get started. See the Gate 0.1.0 release post for details.

Update: Many thanks to Miguel Jimenez for quickly spotting a bug when using Gate 0.1.2 with Kayak and Razor. I’ve just pushed 0.1.3 to address the issue.

Gate 0.1.3

RescheduleCallbacks middleware re-buffers response data before posting it to the scheduler so that the writing thread does not stomp previously written data in-flight.

Gate 0.1.0 Released

2011-08-04T00:00:00-07:00

Today I’m very pleased to announce that Gate is available on NuGet.

Gate is a utility for locating, manufacturing, and hosting OWIN web applications. Hosting adapters are available for Kayak, ASP.NET, and WCF.

Make an OWIN application with Nancy and Kayak

Nancy currently ships a NuGet package which provides OWIN support, so we’ll make a simple demo of running Nancy on Kayak. We’ll need to pull in the Kayak, Gate, and Nancy packages, as well as two adapter packages to glue them all together: Gate.Kayak and Nancy.Hosting.Owin.

Start by creating a new console application project. Grab the aforementioned NuGet packages—the easiest thing to do would be to pull in Gate.Kayak since that will bring Gate and Kayak with it, then Nancy.Hosting.Owin, which will bring Nancy along.

Then, punch these codes into your terminal:

using System;
using System.Net;
using Gate;
using Gate.Kayak;
using Kayak;
using Nancy.Hosting.Owin;
using Nancy;

namespace KayakGateNancyDemo
{
    public class Program
    {
        // plain-ol' entry point.
        public static void Main(string[] args)
        {
            var ep = new IPEndPoint(IPAddress.Any, 5500);
            Console.WriteLine("Listening on " + ep);
            KayakGate.Start(new SchedulerDelegate(), ep, Startup.Configuration);
        }
    }

    public class Startup
    {
        // called automatically when Kayak starts up.
        public static void Configuration(IAppBuilder builder)
        {
            // we'll create a very simple pipeline:

            builder
                // insert a middleware between Kayak and Nancy
                // to ensure Kayak objects are only accessed 
                // on Kayak's worker thread.
                .RescheduleCallbacks()

                // cap off the pipeline with Nancy
                .RunNancy();
        }
    }

    public class SchedulerDelegate : ISchedulerDelegate
    {
        public void OnException(IScheduler scheduler, Exception e)
        {
            // called whenever an exception occurs on Kayak's event loop.
            // this is good place for logging. here's a start:
            Console.WriteLine("Exception on scheduler");
            Console.Out.WriteStackTrace(e);
        }

        public void OnStop(IScheduler scheduler)
        {
            // called when Kayak's run loop is about to exit.
            // this is a good place for doing clean-up or other chores.
            Console.WriteLine("Scheduler is stopping.");
        }
    }

    // Your Nancy application.
    // Check out Nancy's docs for more info on how it works.
    public class MyNancyModule : NancyModule
    {
        public MyNancyModule()
        {
            Get["/"] = x =>
            {
                return "This is the root.";
            };
        }
    }

    // a bit of sugar for running Nancy ;)
    public static class NancyExtensions
    {
        public static IAppBuilder RunNancy(this IAppBuilder builder)
        {
            return builder.Run(Delegates.ToDelegate(new NancyOwinHost().ProcessRequest));
        }
    }
}

The KayakGate.Start method invokes the Startup.Configuration method, uses the user-configured IAppBuilder instance to manufacture an OWIN application delegate, and hosts it on port 5500. Visit http://localhost:5500 in your web browser and bask in the glory.

With this bit of boilerplate, it should be possible to use Kayak to host almost any application developed with Nancy.

Voilà, a completely open source .NET web application stack using an open interface brought to you by hackers in the community. Have fun, and let us know your thoughts on the .NET HTTP Abstractions list.

Kayak 0.7.1 Released

2011-08-01T00:00:00-07:00

Last weekend, I met Dru Sellers at Monospace. At some point, I complained to him that my projects didn’t have proper cross-platform build scripts because blah blah blah. He called me a whiny baby and kicked me in the balls.

Kayak 0.7.1 is out this evening. It is available on NuGet. This is mostly a bugfix release, but it’s a rather significant milestone for me because for the first time, Kayak has a proper build script. Many thanks to Dru for inflicting such acute abuse, and also to the FubuMVC and Nancy guys—I got quite a bit of, er, inspiration from FubuMVC’s build script and Nancy’s build script. Thanks also to Derick Bailey for his work on Albacore without which none of this would be possible.

Anyway, the changes:

Turns out a proper 100-continue response is prefixed with “HTTP/1.1”. So I added that.
BeginConnect seems to be a pile of fail on Mono 2.10. Work-around by invoking the synchronous Connect method on the thread pool.
Rake-based build script!

If you have problems, challenge me to a game of roshambo on the Kayak Google Group.

Kayak 0.7.0 Released

2011-07-20T00:00:00-07:00

I just pushed Kayak 0.7.0 up to NuGet. Here’s what changed:

Support for 100-continue. This will speed up most PUT and POST requests quite a bit since clients won’t wait for the 100 Continue status (usually about 200ms) before sending the body anyway.
IHttpRequestDelegate implementations will get null for the request body producer if the request headers do not contain Content-Length or Transfer-Encoding: chunked.
Implementations of IScheduler, IServer, and ISocket are now private and cannot be directly instantiated using the new operator. Use KayakScheduler.Factory, KayakServer.Factory, and KayakSocket.Factory instead.
Fixed a ~~typo~~ bug in socket implementation so that if BeginReceive synchronously throws an exception the error-on-read logic is invoked (previously error-on-write logic was invoked).
Several small optimizations to socket implementation.
Improved test coverage of HTTP transaction logic.

As always, play around, stress it, and report your experiences to the Kayak Google Group.

How Are You Managing Test Permutations?

2011-05-17T00:00:00-07:00

I’m going to bite the bullet and admit that I’m rather new to the whole unit-testing thing. For a long time, I didn’t see the point in doing it, but it will suffice to say that I have recognized the err of my ways.

It turns out that writing good, pretty tests is an art form and is every bit as difficult and subtle as writing good, pretty code. In every introduction to unit testing I’ve ever read on the internet, the author gives the whole philosophical run down about testing, which is nice. Then, he provides an example where he tests that a function can add two and two. As a long-time coder trying to learn about unit testing, this is completely useless to me.

Today, I’m testing a rather complicated bit of code in Kayak. It exposes a number of functions which can be called at lot of different times and in different orders, and it calls out to a number of functions a lot of different times and in different orders.

Here, I’ve modeled just three areas which can differ. They’re broken right now because I’m attempting to be “disciplined” and write the tests first as much as possible. I’ve got some very verbose method names to describe what exactly the test is doing. Each dimension of variation is separated with a double-underscore, and each variation on a dimension must be combined with all the other variations of the other dimensions. So far, each area of variation has only two possible scenarios, so each time a new permutation is added, I’ve doubled the number of tests. (Save for a trivial base case, which is green.)

But I’m not done writing tests. There are three or four more dimensions that can vary, which, if my math is right, means I’m going to end up with 8-16x as many tests as above. Factor in error conditions and I’m looking at possibly over 100 20-line tests for a component that’s about 100 lines long.

If I ever change the interface for that component, I’m going to have a lot of refactoring to do. And If I ever decide that the job that component performs doesn’t solve the over-arching problem in a very good way, and I want to solve the problem in a different way, I throw out not just 100 lines of code, but almost 2000.

No one ever talks about this aspect of testing. To me, it seems dishonest not to test at this level of granularity. So my question is, how would you manage this sort of thing? Would you write and maintain all of these tests by hand? Do you automate the permutations? If so, how? Or do you just fudge it, tweaking your system-under-test in the debugger under real-world-ish use-cases until it stops crashing?

OWIN as C# Interfaces

2011-03-17T00:00:00-07:00

There’s been some talk about the apparent complexity of the OWIN signature, due to the fact that the signature is expressed in the spec as a nested delegate structure. I’ll admit it looks a bit hairy at first blush.

Consider this excerpt from the specification:

In this document, the C# Action/Func syntax is used to notate the delegate structure. However, the delegate structure could be equivalently represented with F# native functions, CLR interfaces, or named delegates. This is by design; when implementing OWIN, choose a delegate representation that works for you and your stack.

I thought it might be instructive to demonstrate how one could leverage this property of OWIN to create an equivalent structure expressed using C# interfaces.

Break it down

Let’s start with the environment dictionary.

public interface IRequest : IDictionary<string, object>
{
    string Method { get; set; }
    string Path { get; set; }
    string PathBase { get; set; }
    string QueryString { get; set; }
    IDictionary<string, string> Headers { get; set; }
    IProducer<ArraySegment<byte>> Body { get; set; }
    string Scheme { get; set; }
    string Version { get; set; }
}

This is just a helper that adds strongly-typed properties for each of the environment items the spec defines.

Next, take a look at the application delegate.

Action<
    // env
    IDictionary<string, object>, 

    // response callback
    Action<
        string, // response status, e.g., "200 OK"
        IDictionary<string, string>, // response headers
        [a body delegate] // response body, represented by a body delegate
    >, 

    // error callback
    Action<Exception>
>

This delegate becomes the following interface:

public interface IApplication
{
    void Call(IRequest env, IResponse response);
}

Here, we’ve replaced the environment dictionary with our strongly-typed request helper, and the response and error callbacks with an interface which holds those two functions:

public interface IResponse
{
    void OnComplete(string status, IDictionary<string, string> headers, IProducer<ArraySegment<byte>> body);
    void OnError(Exception e);
}

Now for the body delegate.

Func<

    // on next
    Func<
        ArraySegment<byte>, // data
        Action, // continuation
        bool // will invoke continuation
    >,

    // on error
    Action<Exception>,

    // on complete
    Action,

    // cancel 
    Action 
>

This is a function which takes three functions as arguments and returns another function. Just like we did for the response and error callbacks of the application delegate, we’re going to group these three arguments into an interface.

public interface IProducer<T>
{
    Action Subscribe(IConsumer<T> consumer);
}

public interface IConsumer<T>
{
    bool OnNext(T next, Action continuation);
    void OnError(Exception e);
    void OnComplete();
}

Voilà, OWIN as C# interfaces.

Why not just use interfaces?

We designed OWIN so that it could be expressed as function signatures because we didn’t want to introduce concrete types and ship an Owin.dll assembly. Further, we didn’t want to be biased towards any particular CLR language.

If you want to implement an OWIN stack entirely in F#, use FSharpFunc rather than Action and Func. If you’re a big fan of the C# delegate keyword, use named delegates. It’s your stack, it’s up to you. By following the OWIN call structure, you can trivially interoperate with other OWIN-aware components.

A neat side-effect of this is that OWIN could be implemented in a straight-forward manner even in non-CLR languages. A Python version is left as an exercise to the reader.

SMWebRequest

2011-03-14T00:00:00-07:00

A couple of weeks ago, Nick Farina emailed to tell me he was going to post a bit of code I cooked up at Spotlight Mobile to GitHub: none other than the venerable SMWebRequest, an iOS/Mac class for talking to HTTP servers.

I am very excited that this is now open-source. For data sets which fit in memory—which covers approximately 100% of API calls made by modern iOS apps—this is the best HTTP request class there is. Short, sweet, and gets the job done with less code and complexity than the others.

I wrote a simple Hacker News reader to demonstrate using the class and how it lends itself to a clean model-controller architecture in an iOS app. Nick has a great write-up of SMWebRequest which covers the motivation for the class and gives an overview of the features.

Thoughts on OWIN, Buffering, and Async

2011-02-02T00:00:00-08:00

Early on, we decided that OWIN should be an asynchronous interface. There are many possible approaches to this design goal, and ultimately it is important that whatever we come up with accommodates performant host implementations across platforms, and provides application and framework developers a sane programming model which takes advantage of upcoming and current “first-class” async features of C# and F#.

Buffering Streaming Data

Consider the situation where data is coming at an application from a backend server faster than the client connected to the OWIN host can receive it—perhaps as when streaming data from a backend service to a mobile device. At some point, the OWIN host streaming data to the client will need to communicate to the OWIN app which is providing the data to the host that it ought to back off, to slow down the rate at which it provides data to the host. Of course, this means that the app will need to communicate this to the backend service.

Ultimately, the app is talking to the backend service over TCP, and it can slow down the data coming over the connection by temporarily ceasing to send TCP ACK packets to the backend service to which it is connected. Once the slow client empties the OWIN host’s outgoing buffer, and the host then must communicate to the OWIN app that it ought to resume sending ACK packets to the backend service, thereby triggering the service to send more data. I refer to this concept as “back-pressure”. Back-pressure is key to prevent the hosts outgoing buffers from growing indefinitely as the backend service puts data in faster than the slow client takes it out. Of course, this problem can work in the opposite direction: a client connected to an OWIN host might be sending data faster than the app can deal with it.

This buffering problem adds a special twist to the basic goal of making an interface whereby a producer can huck data at a consumer. The consumer of the data must be able to somehow apply back-pressure to the producer—to communicate that it should “back off”.

In our discussions, we have identified two primary approaches to implementing an asynchronous interface. What follows is a discussion of each mechanism and how back-pressure could be implemented within it.

The Pull Approach

In this model, calling code asks that a value be retrieved or an operation be carried out, and later, that code is called back (by way of an interface or delegate) with a means of accessing the return value (or exception) of the operation. The callback is guaranteed to be invoked exactly once, with one value.

The compiler takes care of the details, and what results is a construct which mirrors a traditional function—the user calls a function with some arguments, gets a single value back, and his code continues where it left off. This form of async is a readily grok-able conceptual model.

async void ShuttleData()
{
	while (true)
	{
		var data = await inputSocket.Read();

		if (data.Count == 0)
		{
			inputSocket.Close();
			outputSocket.Close();
			return;
		}

		await outputSocket.Write(data);
	}
}

The back-pressure mechanism is implicit, just like in a traditional synchronous model. The incoming data is (for all intents and purposes) not “ACK’d” until the user reads the input socket. Thus, until the intervening write operation succeeds and read is called again, no more data is read into user-space. The outgoing write mechanism can wait to return until its buffer has been emptied, thereby providing back-pressure by delaying further reads.

In .NET, the designers of C# and F# first took the old Win32 APIs and wrapped the .NET APM patterns in the BCL, then later designed the Task Parallel Library, and now they’re working on C# 5’s async/await syntax; F#’s async syntax is equivalent and available today. These language features make using this “pull” model very easy.

The Push Approach

This approach more closely mirrors that of the Reactive Framework. For an example implementation, we look to Node. Node’s approach to asynchronicity is different the pull model—the user must wire up callbacks and deal with incoming values passed to the callbacks “immediately”. The callbacks cannot block on IO; this is enforced by the APIs exposed by the Node libraries to user code.

In Node, whenever you write outgoing data using a write function, it may be copied into to the kernel’s outgoing network buffer immediately, or placed into a user-space buffer to be sent later. The write function returns immediately. If the user-space buffer is larger than some configured maximum, the write operation will return false, otherwise true.

inputSocket.on("data", function (data) {
	if (data.length == 0)
	{
		inputSocket.close();
		outputSocket.close();
	}
	else if (outputSocket.write(data) === false)
		inputSocket.pause();
});
outputSocket.on("drain", function () {
	inputSocket.resume();
});

Based on this flag, the user can apply back-pressure to the source of the data by “pausing” it. Later, the outgoing buffer will fire a “drain” event, informing the user that the size of the user-space buffer has fallen to a configured minimum and that data may be added to it without causing undue memory pressure in the process. In response to this “drain” event, the user “resumes” the incoming data source, and continues to push data to the client with the write function, perhaps until it returns false again.

Because JavaScript lacks the expressive power of C# and F# (which will, with the release of C# 5, both have first-class asynchronous programming primitives), Node has to introduce these additional “pause” and “resume” functions.

Why I’m Leaning Toward the Pull Approach

It may be possible to construct some an adapter from a push-style interface into a pull-style interface, and it may be the case that a push-style interface is the way to go for OWIN. But ultimately, it is important that OWIN’s asynchronous interface be easy and intuitive to use with the asynchronous language features provided be C# and F#, and these language features map cleanly onto a pull-style interface.

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 3

2011-01-02T00:00:00-08:00

Note: This is the last in a series of three articles exploring differences between Rx and the TPL. The series assumes familiarity with the .NET Asynchronous Programming Model (APM).

As you may imagine, threading is very important to asynchronous programming. Where are all those callbacks coming from? What thread are they executed on and when? The answers to these questions are determined by a mechanism called a scheduler.

A scheduler?

Both the Reactive Framework and the Task Parallel Library define their own concept of a scheduler. A scheduler is like a gatekeeper to the CPU. Schedulers expose a method which allows outside code to provide a piece of work which should be performed. In many scheduler implementations, the piece of work is put into a queue, and the scheduler performs the work at some point in the future. The thread on which the work is performed and when is decided by the scheduler, and is important to consider if you’re writing a program with a user interface, or if you’re writing a server which manages special worker threads.

Scheduling in Rx

In the Reactive Framework, there are three places where the calling thread is important: the call to IObservable.Subscribe, the call to the Dispose method of the IDisposable returned by IObservable.Subscribe, and the calls to the methods of IObserver. Because a subscription or a disposal of a subscription may cause the observable to start or stop doing something, its important to consider what thread those calls are coming in on when implementing an observable. Similarly, when subscribing to an observable, you need to be sure about the thread on which your observer is notified by the observable.

The Reactive Framework defines an interface called IScheduler, which allows consumers of the interface to provide an Action which should be invoked by the scheduler instance.

public interface IScheduler
{
    IDisposable Schedule(Action action);
    IDisposable Schedule(Action action, TimeSpan dueTime);
    DateTimeOffset Now { get; }
}

The Reactive Framework provides two extension methods for working with schedulers, both defined on IObservable.

public static class Observable
{
    public static IObservable<TSource> ObserveOn<TSource>(this IObservable<TSource> source, IScheduler scheduler);
    public static IObservable<TSource> SubscribeOn<TSource>(this IObservable<TSource> source, IScheduler scheduler);
}

SubscribeOn returns an observable which, when subscribed to or unsubscribed from, causes subscription to and unsubscription from the source observable to occur as scheduled by the scheduler. Essentially, it creates and returns a proxy observable which, rather than immediately performing the actions, delegates them to the given scheduler.

The ObserverOn method does the same for notifications. It creates and returns a proxy observable which simply forwards subscription to the source observable, but whenever the source observable yields a value, an exception, or completes, the proxy schedules an action on the scheduler which in turn invokes the appropriate callback on any observer which subscribes to the proxy observable. The effect is that when you subscribe to the resulting observable, you will always get callbacks on a thread determined by the scheduler you provided to ObserverOn.

Scheduling in the TPL

Since the TPL is all about running user delegates, it provides a mechanism to easily configure the thread on which those delegates are run.

The scheduler interface in the TPL is an abstract class called TaskScheduler. The fact that it is an abstract class makes it a bit more difficult to sum up neatly in the same way that the Reactive Framework’s IScheduler interface, but it largely performs the same function—it accepts work in the form of Task instances, and eventually performs that work.

To control the execution of tasks, several APIs in the TPL accept a TaskScheduler as an argument. The most important and commonly used ones are Task.ContinueWith, Task.Start, and TaskFactory.StartNew. The behavior of Task.Start, and TaskFactory.StartNew can be considered equivalent—TaskFactory.StartNew just saves you from having to allocate a new Task yourself and then call the Start method on it.

All of these APIs create a new Task, and a TaskScheduler may be provided to control the execution of the newly-created task. If a TaskScheduler is not provided, the TPL will execute the task on the current TaskScheduler. The current scheduler is available through the static method TaskScheduler.Current. In the context of an executing Task, the current scheduler is, as you might expect, the TaskScheduler which is executing the current task. Outside the context of an executing task, the current TaskScheduler is the default TaskScheduler, which queues tasks for execution on the thread pool.

When completing a task using TaskCompletionSource, the thread which calls SetResult or SetException does not matter. Any tasks consequent to the Task managed by TaskCompletionSource will execute on the scheduler provided to the call to ContinueWith which created the consequent task.

What does it all mean?

For most users of these frameworks, schedulers are a relatively opaque mechanism which addresses threading concerns. For example, if you’re writing a program with a GUI, all you have to do when you want to perform a background operation is make sure it is performed on a scheduler which works in the background. When you want to retrieve the results of that operation, just make sure that the callbacks which receive the results are performed on a scheduler which runs on the UI thread.

void RxButton_Click(...) 
{            
    Observable.Create<Unit>(o => {
        try
        {
            longRunningOperation();
        }
        catch (Exception e)
        {
            o.OnError(e);
        }
        o.OnCompleted();
        return () => { }; // no-op disposable
    })
    .SubscribeOn(Scheduler.ThreadPool) // subscribe operation will kick off on a thread pool thread
    .ObserveOn(Scheduler.Dispatcher) // results will come in on the UI thread
    .Subscribe(
        _ => { }, 
        e => Console.WriteLine("Got error."), 
        () => Console.WriteLine("Success! Back on UI thread."));
}

void TplButton_Click(...)
{
    var uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();

    var task = new Task.Factory.StartNew(() => longRunningOperation());
    task.ContinueWith(t => {
         if (t.IsFaulted) Console.WriteLine("Got error.");
         else Console.WriteLine("Success! Back on UI thread."); 
    }, uiScheduler);
}

An important difference between the TPL and Rx is that tasks are usually passed around already-started, so stringing up callbacks with ContinueWith happens after the task has already started and possibly already completed. With Rx, observables can be created but are generally not active until callbacks are provided through a subscription.

In the previous articles, we saw that the asynchronous mechanisms of the TPL, Rx, and the APM pattern have been roughly equivalent; that it is possible to implement adapters between all of them. However, the APM doesn’t provide any explicit concept of a scheduler. The thread on which the callback from an APM operation is called is up to the implementor of the operation. Most APM operations provided by the .NET BCL invoke their callbacks on thread pool threads, and in the callback you must manually marshal the result or exception of the APM End method using the SynchronizationContext class or a similar mechanism. Both the TPL and Rx make this much easier to do by providing a scheduling mechanism and exposing APIs which allow the user to easily provide a scheduler to indicate what thread should be used to perform work.

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 2

2010-12-24T00:00:00-08:00

Note: This is the second in a series of three articles exploring differences between Rx and the TPL. The series assumes familiarity with the .NET Asynchronous Programming Model (APM).

Part 1: Wrapping APM Operations
Part 2: Implementing APM Operations
Part 3: Schedulers

In the Part 1, we discussed how to wrap APM asynchronous operations in Task<T> and IObservable<T>. In this article we’ll take a look at doing the opposite: how to implement an APM-style asynchronous operation using the Task Parallel Library and the Reactive Framework. The first draft of the OWIN specification defined APM operations for invoking a web application and reading the body of an HTTP request. Because Kayak had used both Rx and the TPL for all of its internal asynchronous logic, I needed to be able to expose that asynchronous logic as an APM operation in order to implement the spec.

A Bit About `IAsyncResult`

One of the nastiest things about implementing the APM pattern is the need to implement IAsyncResult. The idea behind IAsyncResult is that it should provide code that initiates an APM operation information about the identity and state of the operation. A full discussion of IAsyncResult is beyond the scope of this article, but it will suffice to say that in modern C#, IAsyncResult is little more than an opaque token used to obtain the result of an asynchronous operation from an APM End method.

Quite curiously and very unfortunately, the .NET BCL did not historically provide an implementation of IAsyncResult. For our APM pattern implementation using the Reactive Framework, we will use the implementation of IAsyncResult described in this MSDN Magazine article by Jeffrey Richter.

Implementing the APM Pattern with the Reactive Framework

We will create a simple wrapper class that adapts a Func<TArg, IObservable<TResult>> delegate into an APM pattern Begin/End pair. When the APM Begin method is called, our wrapper will call the delegate to retrieve an observable which represents the asynchronous operation. When the observable yields a result or a value, our wrapper class will invoke the AsyncCallback, and the wrapper’s APM End method will provide that result or value.

public class RxAsyncOperation<TArg, TResult> 
{
    Func<TArg, IObservable<TResult>> operation;
    IDisposable disposable;

    public RxAsyncOperation(Func<TArg, IObservable<TResult>> operation)
    {
        this.operation = operation;
    }

    public IAsyncResult BeginInvoke(TArg arg, AsyncCallback callback, object state)
    {
        AsyncResult<TResult> asyncResult = new AsyncResult<TResult>(callback, state);
        disposable = operation(arg).Subscribe(
            // the boolean arguments to SetAsCompleted indicate whether the operation 
            // completed synchronously. IObservable doesn't relay that information,
            // so we assume false.
            r =>
            {
                UnsubscribeIfPossible();
                asyncResult.SetAsCompleted(r, false);
            },
            e =>
            {
                UnsubscribeIfPossible();
                asyncResult.SetAsCompleted(e, false);
            },
            () => { });

        // if the operation completed during the call to subscribe, we won't have 
        // unsubscribed yet, so do it now.
        if (asyncResult.IsCompleted)
            UnsubscribeIfPossible();

        return asyncResult;
    }

    public TResult EndInvoke(IAsyncResult asyncResult)
    {
        // the AsyncResult<T> implementation takes care of waiting and throwing exceptions for us.
        return ((AsyncResult<TResult>)asyncResult).EndInvoke();
    }

    void UnsubscribeIfPossible()
    {
        if (disposable != null)
        {
            disposable.Dispose();
            disposable = null;
        }
    }
}

In the APM Begin method, we first construct an instance of AsyncResult<T>, the implementation of IAsyncResult we’re using. We provide the APM AsyncCallback and state object to the constructor of the AsyncResult<T>—it will take care of invoking the AsyncCallback for us.

Next, we invoke the delegate to retrieve an observable, and subscribe to that observable. Notice that we are keeping a reference to the IDisposable returned by the Subscribe method of the observable. This is necessary because after the observable yields a value or an exception, we need to unsubscribe from it—inform it that we no longer want to receive notifications. This is accomplished by calling the Dispose method of the IDisposable object returned from the Subscribe method of the observable.

In the OnNext and OnError callbacks, the first thing we do is unsubscribe from the observable to prevent further callbacks. It’s important to note that an observable may yield a value or an exception before Subscribe returns, which means that it’s possible that the observable callbacks might get called before the value of the disposable instance variable has set! Thus, unsubscribing from the observable might not be possible from within the OnNext and OnError callbacks. For this reason, after the Subscribe method returns, we check to see if the operation has already completed, and if so, attempt again to unsubscribe.

After unsubscribing in the OnNext and OnError callbacks, we inform the AsyncResult<T> implementation that the operation has completed with a value or error by calling the SetAsCompleted method. In response to SetAsCompleted, the AsyncResult<T> object will call the AsyncCallback we provided to its constructor, and the IsCompleted property of the will AsyncResult become true.

The final step in the implementation of the APM Begin method is to return the IAsyncResult instance.

The implementation of the APM End method is fairly trivial. Its job is to wait for the operation to complete, and return the result of the APM operation, or, if the operation resulted in an exception, re-throw that exception. Fortunately, the AsyncResult<T> class takes care of all of this for us with its EndInvoke method.

Implementing the APM Pattern with the Task Parallel Library

It is quite a bit simpler to implement the APM pattern with the TPL due to one convenient fact: Task<T> implements the IAsyncResult interface! In this example, we’re going to make a wrapper class which is very similar to the one we created for the Rx example—it will adapt a Func<TArg, Task<TResult>> delegate to an APM operation. When the APM Begin method is called, our wrapper will call the delegate to obtain a Task<T> which represents the operation, and when the operation completes, our wrapper will invoke the AsyncCallback delegate provided by the user, and the APM End method will provide the result value or exception.

public class TplAsyncOperation<TArg, TResult>
{
    Func<TArg, Task<TResult>> operation;

    public TplAsyncOperation(Func<TArg, Task<TResult>> operation)
    {
        this.operation = operation;
    }

    public IAsyncResult BeginInvoke(TArg arg, AsyncCallback callback, object state)
    {
        var task = operation(arg);
        task.ContinueWith(_ => callback(task));
        return task;
    }

    public TResult EndInvoke(IAsyncResult result)
    {
        return ((Task<TResult>)result).Result;
    }
}

As you can see, the TPL version is quite a bit simpler than the Rx version! We simply retrieve the underlying Task<T>, register a callback to be notified of its completion using ContinueWith, and return the Task<T>! When that task completes, the delegate we provided to ContinueWith is invoked, which in turn invokes the AsyncCallback provided by the client code.

Like the implementation of AsyncResult.EndInvoke which we used for the Rx example, the implementation of Task<T>.Result takes care of waiting and throwing exceptions for us, so we can just return that value, and everything will work as expected.

A winner?

The TPL version of the code for wrapping and implementing the APM pattern is much shorter and sweeter than the equivalent Rx code. This is not to say that the TPL is superior to the Reactive Framework, but it is clear that the TPL onto the APM pattern more cleanly than does the Reactive Framework. Both the APM and TPL share single-value-or-exception semantics, with only the producer side defining when a value will be produced; the semantics of the Reactive Framework are more general, allowing many-values-maybe-one-exception, with either the producer or consumer choosing when how many values are transmitted and when transmission should start or stop.

In the previous article, we wrapped APM operations with the Task Parallel Library and the Reactive Framework; this time, we’ve gone the other way and wrapped the Task Parallel Library and the Reactive Framework with APM operations. The complete source code for this article is available in a Gist. In the next article, we will consider threading and the how-when-where of asynchronous callbacks in both the TPL and Rx.

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 1

2010-12-21T00:00:00-08:00

Note: This is the first in a series of three articles exploring differences between Rx and the TPL. The series assumes familiarity with the .NET Asynchronous Programming Model (APM).

Part 1: Wrapping APM Operations
Part 2: Implementing APM Operations
Part 3: Schedulers

On the .NET HTTP Abstractions list we decided early on that we wanted to allow OWIN applications to provide some sort of asynchronous primitive through the response enumerable. The rationale is that sometimes an OWIN host might attempt to enumerate the next item in the enumerable sequence, but the application might not be ready to provide the item immediately. We considered the options .NET had built-in: IObservable<T> (the Reactive Framework) and Task<T> (the Task Parallel Library).

I had already built Kayak around the Reactive Framework, and I was pretty happy with that mechanism. The one perceived drawback was that an observable could contain multiple values, but everywhere I used it throughout my system, it was axiomatically required that every observable yield a single value and complete, or yield an exception. The fact that this wasn’t enforced by the compiler nagged at me.

I was leaning toward Task<T> for OWIN because, by definition, it yields a single value or an exception. However, I hadn’t played with the Task Parallel Library yet, so I didn’t press the issue too hard initially. Last week, I spent some time familiarizing myself with the Task Parallel Library by porting Kayak’s asynchronous architecture from the Reactive Framework to the TPL.

A Brief Introduction to the Reactive Framework

In the Reactive Framework, there are two primary interfaces: IObservable<T> and IObserver<T>.

public interface IObservable<T>
{
    IDisposable Subscribe(IObserver<T> observer);
}

public interface IObserver<T>
{
    void OnNext(T value);
    void OnException(Exception e);
    void OnCompleted();
}

IObservable<T> defines a single Subscribe method which takes an IObserver<T>. After Subscribe is called, the observable may call the methods of the observer with values and exceptions at some point in the future. The Subscribe method returns an IDisposable which, when disposed, informs the observable that the observer provided to Subscribe is no longer interested in receiving values from the observable.

Depending on the semantics of the observable, Subscribe may cause the observable to “start” whatever process generates the values that make up the observable sequence—for instance, if the observable represents a TCP listener, its values would be sockets representing accepted connections. Upon a call to Subscribe, the TCP listener would cause a new listening socket to be created and bound, and whenever a connection is accepted, it would call the OnNext method of the observer with the socket object which represents the connection.

Again, depending on the semantics of the observable, disposing the disposable returned from the Subscribe method may cause the observable to “stop” doing whatever it was doing to generate values. In our TCP listener example, the listener would shut down the listening socket, preventing any more connections from being accepted.

Wrapping an APM operation with `IObservable<T>`

In the Kayak’s asynchronous architecture, I originally used IObservable<T> to represent and wrap up asynchronous operations following the .NET Asynchronous Programming Model (APM) pattern. The Reactive Framework provides Observable.FromAsyncPattern which creates an function which, when invoked, returns observable representing an APM operation. When an observer subscribes to this observable, the Begin method of the APM operation is invoked, and the operation is started. When it completes, the OnNext method of the observer is invoked with the result of the operation. Unsubscribing from this observable has no effect, because APM operations cannot be cancelled.

public static partial class Extensions
{
    public static Func<IObservable<int>> ReadAsyncRx_FromAsync(this Stream stream, byte[] buffer, int offset, int count)
    {
        return Observable.FromAsyncPattern<int>(
            (c, s) => stream.BeginRead(buffer, offset, count, c, s),
            iasr => stream.EndRead(iasr));
    }
}

However, this method was not adequate for Kayak. Kayak supports a “green threaded” model, so I need to very carefully control when and on what threads Kayak code gets executed, and I achieve this by making sure that the AsyncCallback of any APM operation is invoked on a special thread managed by Kayak (or more specifically, the underlying IO driver). The observables created by FromAsyncPattern do not guarantee that the OnNext, OnException, and OnCompleted methods of subscribed observers are called on the same thread on which the underlying APM calls the AsyncCallback provided to it. Thus, I wrote a special observable which issued callbacks in a predictable way.

public static partial class Extensions
{
        public static IObservable<int> ReadAsyncRx_ApmObservable(this Stream stream, byte[] buffer, int offset, int count)
        {
            return new ApmObservable<int>(
                (c, s) => stream.BeginRead(buffer, offset, count, c, s),
                iasr => stream.EndRead(iasr));
        }

}

// Observable.FromAsyncPattern reschedules the callback on the ThreadPoolScheduler. Lame!
// We need to allow the underlying APM operation to determine what thread the callback 
// comes in on.
// 
// And so we roll our own.
public class ApmObservable<T> : IObservable<T>
{
    Func<AsyncCallback, object, IAsyncResult> begin;
    Func<IAsyncResult, T> end;

    public ApmObservable(Func<AsyncCallback, object, IAsyncResult> begin, Func<IAsyncResult, T> end)
    {
        this.begin = begin;
        this.end = end;
    }

    public IDisposable Subscribe(IObserver<T> observer)
    {
        IAsyncResult asyncResult = null;

        AsyncCallback callback = (iasr) =>
        {
            T value = default(T);
            try
            {
                value = end(iasr);
            }
            catch (Exception e)
            {
                observer.OnError(e);
                return;
            }
            observer.OnNext(value);
            observer.OnCompleted();
        };

        try
        {
            asyncResult = begin(callback, null);
        }
        catch (Exception e)
        {
            observer.OnError(e);
        }

        return Disposable.Empty;
    }
}

Upon subscription, the asynchronous operation is initiated by a call to the APM Begin method, and an AsyncCallback delegate is provided. In typical APM fashion, the implementation of the AsyncCallback calls the APM End method to retrieve the result of the operation, and catches the exception, if any. The implementation of the AsyncCallback then calls the observer’s OnNext method with the result of the operation followed by OnCompleted, or, if the APM End method threw an exception, it calls OnError with the exception. Again, because APM operations in .NET cannot be cancelled, the IDisposable returned from Subscribe is Disposable.Empty.

A Brief Introduction to the Task Parallel Library

The core interfaces of the Reactive Framework roughly correspond to TPL classes Task<T> and TaskCompletionSource<T>. Task is a bit like IObservable<T>, in that it, by itself, represents an asynchronous operation. TaskCompletionSource<T> is a bit like IObserver<T>, in that it defines SetResult and SetException methods which accept values and exceptions.

Tasks provide values and exceptions though their Result and Exception properties, but these properties are a bit freaky—if the task hasn’t completed yet, these properties will block the calling thread until the task completes before returning a value. It’s a very, very rare case where this behavior is what you want—if you’re doing asynchronous programming, threads blocking is often what you’re trying to avoid in the first place! Instead, you want to get a callback when the task is completed, so that you can safely access the Result or Exception properties without blocking. Task objects provide the ContinueWith method for this purpose. There are a lot of overloads of ContinueWith which introduce slight variations, but the basic idea is that it takes a delegate which is executed when the target (or in MSDN parlance, “antecedent”) task completes, and returns a new Task which represents the execution of the delegate.

Think about that for a second. Task.ContinueWith doesn’t take a task, it creates one from a delegate. Now take a look at the constructors for Task—they all take delegates and create new Tasks from them. If a delegate is sufficient to create a task, then one might infer that the task completes when the delegate is finished executing. So constructing a task with a delegate is just a fancy way of saying ThreadPool.QueueUserWorkItem, with the added plumbing of handling a return value and any exceptions and making them available through some properties.

So how can a Task represent an asynchronous operation which isn’t backed by a thread pool thread, such as a read or write to or from a network socket?

Enter TaskCompletionSource<T>. TaskCompletionSource<T> is kind of like IObserver<T> and IObservable<T> wrapped into a single package. (In the Reactive Framework, there is a type called ISubject<T> which is just that—it implements both IObservable<T> and IObserver<T>). When you construct a TaskCompletionSource<T>, it internally constructs a special task, accessible through the TaskCompletionSource’s Task property, which doesn’t require a delegate. Instead, TaskCompletionSource<T> defines two methods, SetResult and SetException, which when called, cause the TaskCompletionSource’s special Task to complete with the given result or exception.

Wrapping an APM operation with `Task<T>`

I’m now using Task<T> to wrap up APM operations in Kayak. The Task used to represent these operations is provided by a TaskCompletionSource<T>, since in Kayak, every asynchronous operation may be performing asynchronous IO somewhere and therefore cannot be expressed as a single delegate.

public static partial class Extensions
{
    public static Task<int> ReadAsyncTpl(this Stream stream, byte[] buffer, int offset, int count)
    {
        var tcs = new TaskCompletionSource<int>();

        stream.BeginRead(buffer, offset, count, iasr =>
        {
            try
            {
                tcs.SetResult(stream.EndRead(iasr));
            }
            catch (Exception e)
            {
                tcs.SetException(e);
            }
        }, null);

        return tcs.Task;
    }
}

So the first step in wrapping up an asynchronous operation with Task<T> is to construct a TaskCompletionSource<T>, whose task will ultimately be returned. Next, the wrapper method invokes the APM Begin method, and provides an AsyncCallback whose implementation attempts to call the APM End method. The AsyncCallback implementation calls TaskCompletionSource<T>.SetResult with the return value of the End method, or, if the End method throws an exception, the AsyncCallback calls TaskCompletionSource<T>.SetException with the exception. When either SetResult or SetException are called, the TaskCompletionSource’s special task completes. Any threads waiting on the Result or Exception properties of the task will continue, and any tasks to which the TaskCompletionSource’s task is antecedent (i.e., those tasks created by calling ContinueWith on the special TaskCompletionSource task, which is returned from this wrapper method) will be executed.

The TPL does actually provide an adapter from the APM pattern out of the box in the form of the method TaskFactory.FromAsync. This method is heavily overloaded, but there are two basic variations:

public Task<TResult> FromAsync<TResult>(Func<AsyncCallback, object, IAsyncResult> beginMethod, Func<IAsyncResult, TResult> endMethod, object state);
public Task<TResult> FromAsync<TResult>(IAsyncResult asyncResult, Func<IAsyncResult, TResult> endMethod);

Our example is equivalent to the first. The second however has a markedly different behavior. Notice that it takes IAsyncResult as an argument—when using this overload, the calling code must invoke the APM Begin method to provide an IAsyncResult, which means the FromAsync method cannot provide an AsyncCallback to the begin method. Therefore, it cannot be notified asynchronously when the APM operation is completed. Instead, it must wait on the AsyncWaitHandle property of the IAsyncResult. Beware when using overloads of FromAsync which take IAsyncResult—threads waiting is rarely the behavior you want! This subtle difference has very large implications for the flow of your program.

What else?

An important difference between the TPL and the Reactive Framework versions of these wrapper method is that the APM Begin method is called when the TPL wrapper method is called, whereas the Rx version defers calling the APM Begin method until the observable gets a subscription. This has important ramifications for your programs which will be covered in a subsequent article.

We seen how to invoke existing APM operations using both the Reactive Framework and the Task Parallel Library. The source code for these examples is available in a Gist.

In the next article, we will implement new APM operations in terms of both the Reactive Framework and the Task Parallel Library.

Clojure Served This.

2010-12-02T00:00:00-08:00

For 8 years I have been on a quest for the perfect website generation tool. Along the way, I got side-tracked on the web server, and in the process of writing that, I felt myself bumping up against the limits of imperative programming languages. A few weeks ago, I woke up, went to a coffee shop, and realized that I didn’t want to think about web servers for a while, and that today would be the day I finally broke down and set about learning a functional programming language. I chose Clojure largely because it seemed all new-age and hip, and I’d heard Haskell might be too scary.

I figured my first project would be to make that website generation tool, so today I’m happy to put up this here fancy new Clojure-powered web site. Very impressed with Clojure overall. The language is fun to use and the community is top-notch. There are lots of fresh open source utilities for doing all sorts of cool things with Clojure. I’ve been very happy with Ring, Compojure, Enlive, clojure-couchdb, and, in particular, Leiningen. Leiningen is probably the single most important Clojure tool. It’s an all-in-one project generating-running-testing-packaging-dependency-managing apparatus that makes starting new Clojure projects feel very light and inviting, and basically solves all of the problems related to maintaining a nice source tree. Just put the folder on GitHub and you’ve got an open source project.

By far the coolest hack involved in the making of this site is the Markdown support, courtesy of Brian Carper, which uses Rhino, Mozilla’s JavaScript interpreter for Java, to parse Markdown with the Showdown.js Markdown parser. I found this solution particularly impressive; ease of interoperability with Java a huge win for Clojure. Say what you may about its LISPy roots, I find it to be extremely pragmatic.

I’ve got a feeling this website tool is going to go somewhere.

.NET HTTP Abstractions

2010-12-01T00:00:00-08:00

Note: OWIN has evolved significantly since this post. Please consult the latest specification. – bvanderveen, 6 January 2011

For the past few days, I’ve been furiously posting on the .NET HTTP Abstractions Google Group, where a .NET “clone” of WSGI is taking shape. Although I started working on something like WSGI toward the end of October, I had all but given up due to perceived lack of interest. Naturally, I was very excited to see the discussion on .NET HTTP Abstractions taking shape. Ryan Riley, Sebastien Lambla, and myself have been particularly actively hashing out the details of a new specification we’re tentatively calling Open Web Interface for .NET, or OWIN. Ryan has been keeping a GitHub repo in sync with the discussion.

The interface is really nice and minimal. A few details might change, but here’s where we are now:

public interface IApplication 
{
    IAsyncResult BeginInvoke(IRequest request, 
        AsyncCallback callback, object state);
    IResponse EndInvoke(IAsyncResult result);
}

public interface IRequest 
{
    IDictionary<string, object> Items { get; }
    
    string Method { get; }
    string Uri { get; }
    IDictionary<string, IEnumerable<string>> Headers { get; }
    
    IAsyncResult BeginReadBody(byte[] buffer, int offset, int count, 
        AsyncCallback callback, object state);
    int EndReadBody(IAsyncResult result);
}

public interface IResponse
{
    string Status { get; }
    IDictionary<string, IEnumerable<string>> Headers { get; }
    IEnumerable GetBody();
}

Architecturally, this is a pretty significant departure from WSGI and Rack, due to the statically-typed nature of C#. Personally, I think it’s an improvement. The request is represented explicitly by a concise interface rather than a big dictionary, and gone are the ugly CGI-style variables required to access request headers. Response objects are simply returned by applications, which is a lot cleaner than WSGI’s approach, which requires applications to call start_response for the header data and the return an enumerable for the body data. Additionally, the whole thing is fully asynchronous, which we’re hoping will bolster performance, and saves on some of the awkward implementation consequences of WSGI surrounding threading and response buffering.

A few quick points:

Environment-specific data (probably including base path, though this is still being discussed) is stored in IRequest.Items.
Header dictionary values are IEnumerable<string> to support headers which may appear multiple times, such as Set-Cookie.
The IEnumerable returned by IResponse.GetBody() can contain different types of objects that host implementation can handle in different ways. None of these are set in stone, but for example, a FileInfo might be served using the OS’ file-sending support, or a Task<byte[]> might represent an asynchronously-fetched buffer of bytes. The possibilities here are exciting.

We’ve opted to keep the interface compatible all the way back to .NET 2.0, but it wouldn’t be difficult to write extensions which would make it easy to use even in something as bleeding-edge as the C# Async CTP.

In the next few days, I hope to start drafting a more formal specification as we nail down remaining details. Many thanks to everyone who has contributed to the discussion! We’ve got a hell of a lot of firepower behind this thing and I’m very excited to see where it goes.

bvanderveen.com

this Is Object-Oriented Programming

In .NET, open source does not beget open source

Setting up a cross-platform open source .NET project is non-trivial.

Step 1: Build

Step 2: Test

Step 3: Repeat

Step 4: Ship

Step 5: Go forth and innovate!

Open source does not beget more open source in .NET, because the barriers to standing on each other shoulders are so high.

Sequence, UITableView

The seeking problem

Table view

Sequence, push, pull

Settled.

Node does IO.

Goodbye, .NET

How to run Mono on Heroku

Part 1 – Compiling Mono

Part 2 – Make a Heroku buildpack for Mono.

Sequence: a preview

How to use it

Example: map

Example: reduce

Example: any

How it works

Caveats

Offline this week

Testing is not a substitute for good design.

On the compiler

New Gate Developments

Bank Transfer Day

On jobs

On engagement

Been offline

Gate 0.1.4 Released

Gate 0.1.4

Gate 0.1.2, Kayak 0.7.2

Gate 0.1.2

Kayak 0.7.2

Gate 0.1.3

Gate 0.1.0 Released

Make an OWIN application with Nancy and Kayak

Kayak 0.7.1 Released

Kayak 0.7.0 Released

How Are You Managing Test Permutations?

OWIN as C# Interfaces

Break it down

Why not just use interfaces?

SMWebRequest

Thoughts on OWIN, Buffering, and Async

Buffering Streaming Data

The Pull Approach

The Push Approach

Why I’m Leaning Toward the Pull Approach

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 3

A scheduler?

Scheduling in Rx

Scheduling in the TPL

What does it all mean?

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 2

A Bit About IAsyncResult

Implementing the APM Pattern with the Reactive Framework

Implementing the APM Pattern with the Task Parallel Library

A winner?

Asynchronous Programming with the Reactive Framework and the Task Parallel Library — Part 1

A Brief Introduction to the Reactive Framework

Wrapping an APM operation with IObservable<T>

A Brief Introduction to the Task Parallel Library

Wrapping an APM operation with Task<T>

What else?

Clojure Served This.

.NET HTTP Abstractions

`this` Is Object-Oriented Programming

A Bit About `IAsyncResult`

Wrapping an APM operation with `IObservable<T>`

Wrapping an APM operation with `Task<T>`