Static Queries, Dynamic Data: Enabling the Real Time Web


Notice

If you're not up on event systems and event programming, no worries. Most programmers aren't. But I'm here to tell you that they're cool, they're accessible, and they're not just for enterprise kinds of problems anymore. Event processing enables the real-time Web. Most of the cool things we want the Web to do will be easier when we no longer rely exclusively on time and request-response based architectures and look to event based architectures to solve more problems.

One of the unique things about the Kynetx architecture is that there's no database in the traditional sense. We have a ruleset repository and a session store for persistent context information, but the architecture is very different from what people consider a traditional Web service architecture that, almost always, has a database at its core. When I created the architecture I wanted to stay away from traditional databases for scaling reasons and to ease the operations burden.

What I've never had a good answer for was "why can you get away with that?" Yesterday it hit me. In a traditional database, the data is (relatively) static and the queries are dynamic. An event processing system is the dual of that. In an event processing system, the queries are static and the data is dynamic. Since KNS is an event processing network, or a system of programmable event loops, and KRL is an event processing language, it has provision for expressing static queries that it continuously applies to streams of events.

There are several well-known examples of this concept. When you create a Google alert, for example, you're giving Google a standing, static query that it continuously runs over the stream of new stuff that it sees. When there's match, you get an email. Twitter searches are similar. Once you've done the search, Twitter continues to apply it to the stream of tweets looking for matches. Static queries, dynamic data.

Here's a KRL example. In KRL, queries on an event stream are specified using the select statement in a rule, like so:

select when pageview "www.google.com/calendar"
       then mail received subject "schedule change" 

This query, or event expression, says "select this rule when the user has been on Google calendar and then receives an email with a subject that contains the words 'schedule change'". When a rule containing this event expression is active, the system continuously watches the stream of event data looking for this pattern. Whenever the pattern is seen, the rule engine selects the rule and evaluates it. Static query, dynamic data.

Making the real-time Web real will require more systems to turn the traditional Web service inside-out and handle dynamic data. Events are the way to do that because they enable loosely coupled systems that are always watching for scenarios that matter to the user. Without events, this is hard and expensive to pull off.