Contextually Correlating Events: The Power of Emergent Behavior in Loosely Coupled Systems


One of the real powers of event-based programming is the ability to contextually correlate events. That's a pretty dense phrase. I'm writing this blog post to unpack it a little in my mind. Hopefully it will do you some good as well.

Single events, like web:pageview, mail:received, or phone:inbound_call can be interesting. We've made money on Kynetx apps that rely exclusively on these kinds of primitive events.

In spite of that, we can create even more interesting applications when we start to correlate one event with another. This is analogous to building Web applications. While there are plenty of Web applications that don't require that a Web site maintain any kind of state or context between service calls, most interesting Web applications do. In the case of the Web site, we correlate requests to the server in order to group those requests and relate them to each other--think "shopping cart" for an example.

When we speak of correlating things, we're talking about finding the relationship between them. When we correlate data in statistics, we're looking to see how strongly two or more data values (like price and volume sold) are related and even the nature of the relationship.

For events, we're looking for events in a stream that have a meaningful relationship. In KRL, events can be correlated in at least three ways:

  1. Events can be correlated by entity -- all of the events for a particular person or thing can be related to each other.
  2. Events can be correlated explicitly through event expressions -- KRL allows developers to create complex event expressions, or eventexs, that declare specific relationships between their constituent events.
  3. Events can be correlated implicitly through shared data -- When a user installs multiple rulesets that have data in common (say a TODO list, a Google calendar, or an email acccount), the events that those rulesets respond to become implicitly linked through the shared data.

Entity Correlation

The first kind of correlation is the most obvious and matches the Web application example I used above. Correlating events by entity requires that we identify, at least pseudonymously, on whose behalf the events are being raised (since entities rarely raise events for themselves, but use an endpoint to do that). One of the primary responsibilities of an endpoint is managing the identity context.

Correlating events for an entity enables personalization of the interaction and provides a thread upon which otherwise disconnected events can be strung. But there might be thousands of events raised in day for a given entity. Consequently, we usually look for stronger correlation than mere entity-correlation.

Explicit Eventex Correlation

An eventex describes an explicit relationship between two or more events. Whether those events are correlated through an entity, an eventex established further conditions that are required for a rule to consider those events correlated. For example, consider the following eventex

select when web pageview "support/main.html"
     before mail received subject "[help]"

The intent of this eventex is to select the rule if the user has visited the main support page before sending an email that has [help] in the subject. These events, to be meaningful should be correlated by entity. After all, it's hard to think of why you'd want to know when one person emails support after someone else has visited the Web page. Consequently, this eventex further correlates the web:pageview event with the mail:received event temporally, stating that one must come before the other.

As another example, consider the following eventex1:

select when web pageview "/archives/(\\w+).html" setting(pagename)
     before web pageview "/archives/#{pagename}.html"

This eventex not only correlates the two pageviews by entity (default behavior) and temporally through the eventex operator before, but it also correlates the two page views by requiring that they be to the same page. The eventex doesn't care which page is visited, only that it's visited twice.

Contextual Correlation

Implicit event correlation occurs when events are related by context. In many ways, implicit event correlation is the most powerful because it has the power to allow serendipitous interactions. Serendipitous interactions provide for uses that designers didn't anticipate--perhaps couldn't anticipate. Serendipitous interaction is enabled by the inherent loose coupling of event-driven systems and their resilience in the face of error.

Context is information that is not explicitly bound to a specific process but is relevant to multiple processes through implicit relationships. Knowing, for example, the time of day, the weather, the prime rate, commodity prices, or that it's Super Bowl Sunday provides information that can be used to correlate events. These correlations will be made in the logic of the rules that respond to the events.

One of the significant benefits of evented systems is that they exhibit extreme loose coupling. To understand this consider the situation where you want your DVD player to pause when you get a phone call. Assuming each has an API, programming your phone to send a "pause" command to the DVD is easy enough to do. Now suppose you decide you want to also raise the room lights when the phone call comes in; you have to program the phone to also send a "raise lights" command to the room lights. Each new interaction requires an explicit command from one system to another. Such point-to-point connections make changes difficult. Change out the DVD player and every device sending it commands has to be reprogrammed.

Now, imagine instead that the phone merely raises a phone:inbound_call event. The DVD and lights can both listen for such events and do the right thing. Add something else to the mix and it can easily listen for the phone:inbound_call event and do whatever is right for it. Nothing else needs to be reprogrammed or even told its there as long as it can listen for events of interest. Each actor in the event-driven system interprets the event according to its own context and purpose. Loose coupling leads to better resilience in the face of errors and configuration mistakes.

In an event-driven architecture, the responsibility for flow control shifts from the event source or sender to the listener. Event-driven architectures distribute and delegate flow control to event listeners. This is a new way to create programmed systems and represents a substantial shift away from the traditional way that developers have scripted systems. In fact, the word "scripted" is the wrong way to think about event systems. Any given ruleset is scripted to a particular set of use cases, anticipated by the developer. But the interaction of rulesets in an event network isn't scripted; rather it emerges from the individual behavior of the many actors.

Rulesets don't execute in a vacuum. Rather, rulesets operate in an environment with other rulesets, endpoints, APIs, and datasources. These actors consume and propagate events according to their own programmed functionality. Thus, an event can become entangled--correlated--with other events through the interaction of the various actors in the environment. Changing the environment by replacing one component or adding another has an impact on the overall behavior and thus the correlation of events.

At first this may seem like a recipe for disaster, but some of that trepidation is born of our experience with tightly coupled systems. Creating event-driven systems that respond in a certain way is a talent more akin to throwing a really good party, than that of directing a play. You have less control over what happens at the party, but you also have less chance of things going completely off the rails and crashing or falling flat. An event-driven system where something isn't quite right is more likely to do most things well, some things OK, and a few things poorly. A request-response system where something isn't quite right is more likely to just not work at all. Event-driven systems are more resilient. Their failures are more linear--that is, small mistakes are likely to result in small, not large, failures.

An Example

Consider the event-driven system shown in the following diagram. The system comprises two rulesets, one for managing a TODO list and one for watching emails, and the TODO list itself, a cloud-based service with an API as well as the ability to raise events.

Event-Driven TODO List System

The TODO list management ruleset is fairly straightforward, offering ways to add items to the TODO list and cross them off or reschedule them. Most of the events it processes are related to a Web interface that the user manipulates to manage the list (not shown).

The TODO list system isn't a Web application, just a Web service with a management API and the ability to raise events when certain things happen in the TODO list (todo:added_item, todo:deleted_item, etc.).

The email assistant ruleset listens for mail:received events. When the message contains a TODO item2 it raises an event todo:new_item that contains the TODO item and other relevant information (like the message ID) whenever it sees one.

The TODO list management ruleset sees the todo:new_item event and adds it to the TODO list via the API. As that is complete, the TODO list service raises the todo:added_item event. The email assistant ruleset has been programmed to look for those events and responds by creating a draft email responding to the original email. The user is also notified (using yet another event) that the draft has been created and needs approval. Once approved, the draft will be automatically sent once the email assistant sees the todo:deleted_item event for that TODO item.

Now, imagine that the user adds a TODO item "Send an email to Bob Jones about the company party." Of course, the email assistant can listen for the todo:new_item event and automatically create a draft to Bob with the right subject and have it waiting for the user in the drafts folder when she's ready to work on this task. The email assistant can also know, via the event's parameters, which TODO item is associated with the task and, when the email is sent, send a todo:done_item event. The TODO list management ruleset sees the event and automatically marks the TODO item as "done" via the API.

In this example, the TODO list management ruleset and TODO list service function perfectly well with or without the email assistant ruleset. In fact, they never know that the email assistant exists or that it's listening to events from the TODO list. If it fails, the TODO list system keeps working.

On the other hand, the email assistant knows that there may be a TODO list available and uses events to listen for it's workings and to interact with it. The email assistant doesn't know about the TODO list service, or it's API, only the events. The email assistant won't be as useful without a TODO list ruleset installed, but it will keep working because any TODO events it raises will simply be ignored and any events it's listening for will never be raised.

When both rulesets are installed, events that are otherwise only entity-correlated, such as inbound_email and todo:new_item, are implicitly correlated by the interaction of the two rulesets and the TODO service. The correlation occurs because of the particular rulesets that the user has installed and how they interact.

Conclusion

The use of events allows us to create systems that are more loosely coupled and more robust than with the request-response architectures we're used to. While explicit event correlation by entity or event expressions can create interesting applications, the implicit correlations that emerge from the interaction of the actors in the event-driven system are the most interesting and compelling.

Building event-driven systems is a new skill for most developers because they cannot script the emergent behavior. The most interesting behaviors will emerge from interactions between actors that the develop can't necessarily anticipate. One of the most important lessons of the World Wide Web is that there is real power and value in the emergent behavior of loosely coupled actors. Contextually correlated events bring that power to applications.


1Variable correlation in eventexs is not supported in KRL yet, but will be part of a future release.

2The email parsing ruleset is using a microformat, RDFa, or some other semantic markup technology to recognize TODO items in emails.