A Big, Programmable Event Loop in the Cloud


roller coaster loop

An event loop is a message dispatcher. The loop runs, waiting for events, and responding to them. This is how I've come to think of the Kynetx Network Service (KNS): a big, programmable event loop that runs in the cloud. We haven't always thought of KNS as an event loop, we used to think of it as a ruleset evaluation engine. These ruleset evaluations were tied to users visiting a page. We now view that as just one kind of event (a pageview event, to be specific). We also promoted some other events in the system (like Web clicks and form element changes) to first class events. Thinking of KNS as an event loop that is programmed via rules rather than simply as a ruleset evaluation engine, doesn't change the underlying system very much, but it does change the way we think about how it can be used.

When I wrote about the the addition event expressions to KRL last month, I introduced the idea of KNS being an event loop and explained our event expression language for describing complex event scenarios. These scenarios combine primitive events in various ways. Like any language however, the power of this concept is based on both the expressiveness of the operators for combining primitives and the breadth and depth of primitives that can be combined. Last month, there were just four primitives and they all worked only in the Web event domain: pageview, submit, click, and change. I hinted that more were coming someday; today is that day.

But wait! I'm not just going to introduce a few more event domains and event types within those domains. The changes we made in today's release will allow an infinite number of event domains and types because they're now completely definable by developers writing KRL. In addition, we've introduced the concept of "directives" which are ways for a KRL ruleset to create a set of instructions to send back to the endpoint that has raised a particular event. Here's how it works.

Rules are selected when it's event expression is satisfied. For primitive events, that means that the event is raised and the conditions around that event are met. The syntax for a select statement looks like this:

select when <event_domain> <event_type> {<param_name> <regexp>}* 
  [setting (<var>*)]

For example consider the following select statement from a KRL rule:

select when mail received

In this event expression, mail is the event domain and received is the event type. Any rule containing this statement will be selected when events matching the event domain and type are raised. Of course event expressions can be more complicated:

select when mail received from "(.*)@windley.com" 
  setting(user_id)

This statement adds a parameter (from) and regular expression ("(.*)@windley.com") along with a setting clause to name the captured variable. You can have as many parameter checks as you want. The select statement shown above will only be satisfied when the event domain and type match and there is a parameter called from that has a value that matches the given regular expression.

Of course, defining your own events is only half of the game. You also want to respond to them. For Web events, KNS responds with Javascript. We expect that there will be many endpoints besides browsers that choose to accept Javascript because of it's flexibility, but simple endpoints will want something easier to consume than a complete Javascript program. We've introduced a send_directive action that provides for this. For example the following action will send a directive named say with a parameter named something that has the value "Hello World".

send_directive("say") with
    something = "Hello World";

A rule or ruleset can send zero or more directives to the endpoint as the result of a single event being raised. The endpoint will see them as JSON and can interpret them any way it wants. The developer's job is to

  • Design the events that the endpoint will raise
  • Design the directives that the endpoint will respond to
  • Create an endpoint that does these things
  • Write rulesets that the endpoint uses.

Of course, its possible that a single endpoint might have multiple rulesets that use it from multiple developers trying to do different things. I suspect that will be the case for the IMAP endpoint we're working on right now.

Defining events that an endpoint can raise and directives that it can consume is similar to creating a protocol. Since the quintessential introductory example for a protocol is an echo server, I've built one in KRL. We'll define the event domain to be echo and define two event types: hello and message. That means we need at least two rules, one for each event type. We could have more if we want to respond to different parameters. Here's the rules I defined:

rule hello_world is active {
  select when echo hello
  send_directive("say") with
    something = "Hello World";
}
  
rule echo is active {
  select when echo message input "(.*)" setting(m)
  send_directive("say") with
    something = m;
}

The rule hello_world responds to the hello event by sending the directive named say with the parameter something set to "Hello World". The rule echo responds to an echo event with a parameter called input. That entire value of the input is caputured and bound to the variable m. The echo rule send a directive named say with the parameter something set to the value of m.

It's critical to note that the underlying KNS engine doesn't know anything about the event domain echo or the event types hello and message. We could define these to be anything we wanted and the example would work the same.

Of course, this ruleset with it's understanding of the echo events and directives is useless without a corresponding endpoint to raise these events and consume the directives. Endpoint can be created in any number of different ways. I've written a simple Perl program to do the job. Here it is:

 
#!/usr/bin/perl -w
use strict;

use Getopt::Std;
use LWP::Simple;
use JSON::XS;

use Kynetx::Raise;

# global options
use vars qw/ %opt /;
my $opt_string = 'h?e:m:';
getopts( "$opt_string", \\%opt ); # or &usage();

my $event_type = $opt{'e'} || 'hello';
my $message = $opt{'m'} || '';

my $event = Kynetx::Raise->new('echo',
\t\t\t       $event_type,
\t\t\t       'a16x66',
\t\t\t       {'host' => '127.0.0.1'}
\t\t\t       );

my $response = $event->raise({'input' => $message});

foreach my $d (@{$response->{'directives'}}) {
  if ($d->{'name'} eq 'say') {
    print $d->{'options'}->{'something'}, "\
";
  }
}

This simple script uses a module called Kynetx::Raise that I wrote to take the relevant information about the event, create the right URL for the Kynetx event API, raise the event by calling the URL and process the response. You can see that it has the possibility of taking the event type from the command line with the -e switch. If none is given, the event type defaults to hello. Consequently running this program with no arguments results in the hello_world rule we defined above firing and the directive say "Hello World" being sent to the program which prints the message.

Running the program with the -e switch like so:

./echo.pl -e message -m 'KRL programs the Internet!'

results in the string KRL programs the Internet! being printed in the terminal. W00t!

The addition of support for generalized primitive event specifications and directives is a significant increase in the power of KRL to program the Internet. Now any connected device can serve as a KRL endpoint and received instructions from cloud-based KRL programs. Who will be the first to mashup their sprinkler system with Google calendar?