Phil Windley's Technometria

« March 2011 | Main | May 2011 »

April 30, 2011

On Strict Constitutionalism

Yesterday I had an interaction on Twitter with a few folks about strict constitutionalism. The context was the Utah County Republican Organizing Convention happening today. In response to a tweet I made about being leery of strict constitutionalists, Connor Boyack said:

You oppose strictly adhering to the party's constitution? Seems odd. :)

The point isn't that I oppose strictly adhering to the constitution. The point is that I distrust the motives of people who make strict constitutionalism a point of their platform for office (in the party or otherwise).

I don't doubt that there are many sincere people who are seriously distressed at what they see as departures from the rules that have been clearly established. Certainly there is danger in such departures. Nevertheless, I fear that many people use the argument of strict constitutionalism as a way of gaining power without clearly stating why they want power.

There is nothing humans get more worked up about than when someone else isn't following the rules. We see it all the time from the playground to instances of road rage. People hate when someone else seems to be getting away with something. I'm sure there's some social science theory about how that instinct is critical to societal cohesion or something. Still, we don't need to understand why to know that it is a powerful force.

Given that, making a case--valid or otherwise--that someone in power isn't following the rules is an easy way of gaining support to "throw the bum out." Not only is it an easy play, but it has the advantage of not requiring that the person making the charge be for anything other than following the rules.

I distrust the motives of people who want to be elected because "they'll follow the rules." I want to know what they will do and what they stand for. I'm not interested in playground politics and constant arguments about rules. Such arguments make me tired of party politics. Today at the convention, I'll be voting for people who will run the party to get things done, not those who make their platform about strictly following the rules.

10:36 AM | Comments () | Recommend This | Print This

April 26, 2011

The Value of Browser Add Ons

Today a question about the size of the browser add-on market came up and more specifically, what kinds of things people used browser addons for. I spent a little time at AMO to try and find some answers--at least for Firefox.

First the AMO statistics site shows that there are almost 200 million addons in use and that number continues to show healthy growth for the last few months. AMO lists over 5000 add ons. The most popular is Adblock Plus with almost a million weekly downloads.

I classified the top 100 addons on AMO according to the following categories:

• context - addons that mash pages together, pull information from one place and put it in another, or otherwise modify the in-page browsing experience.
• download helpers - addons that manage the downloading process or allow downloading of videos, MP3, and photos from sites like YouTube.
• chrome - addons that modify the browser chrome in some way such as adding color to the tabs or modifying the browser's search field
• convenience - addons that make using the browser more convenience such as making bookmarking simpler or adding screen capture capabilities.
• programming and Web design - addons that make debugging Javascript and creating Web sites easier.
• security, privacy, and safety - addons that aim to make the browsing experience safer
• cross browser - addons that support using more than one browser (such as bookmark syncing)
• performance - addons that enhance Firefox performance

The following table shows the percentage of the top 100 addons that fall into each category:

Category	Percent
context	31
download helpers	20
chrome	13
convenience	13
programming and Web design	10
security, privacy, and safety	6
cross-browser	3
performance	2
other	2

The least popular of the 100 most poplar addons was ReminderFox and it gets 17,500 weekly downloads. This is an impressive list and speaks volumes to the power of Mozilla's distribution through the Firefox browser and the willingness of people to install addons to customize their browser.

Many of these addons have a "contribute" button that allows people to donate to their favorite addons. I wish I knew what kind of revenue was being generated by that, but I couldn't find any statistics on it.

The "other" category had two special entries: Greasemonkey (240,000 weekly downloads) and iMacros (28,000 weekly downloads). Greasemonkey allows you to run scripts written by other developers and consequently functions as a universal browser extension. iMacros allows you to record and play back oft-repeated interactions with the browser.

The Greasemonkey script repository lists 66,000 Greasemonkey scripts. Many of the top 100 seem to be in Russian (I'm assuming they are associated with a game). Many of the other top scripts have something to do with enhancing Facebook and other Web pages or gaining an advantage at Farmville and other games. The total downloads for each of these GM scripts are impressive ranging from 2.5 million for the top GM script to around 300,000 for the 100th.

I think the numbers along make a clear case that people value being able to customize their browser experience and willing to overcome some amount of friction to accomplish it. The question of what monetary value they might attach to this is unproven. I don't think it's a stretch to think you could get people to pay to get an advantage in a game--people do that all the time. Would they pay for a better Facebook? I don't know.

Of course getting people to pay directly is only one way to create value from a browser addon. You can also present people with an addon that gives them a better experience (say the opportunity to save money) and get someone else to pay for it (the retailer) or one that offers in-app sales of something the user will pay for (like music).

Meanwhile, if you'd like to try out our version of the universal browser extension, head over to apps.kynetx.com and install one. Then install a second. I think you'll like the experience and they're all free.

4:04 PM | Comments () | Recommend This | Print This

April 25, 2011

The Trends Driving KRL and Kynetx

In my last post, I outlined a few benefits that developers could gain from using KRL. Well and good, but so what? Are those benefits that matter? How do they relate to the other reasons programmers learn new languages like money and opportunity? In this post, I will talk about some of the big trends driving my thinking and informing the decisions around what KRL is and why it enables the abstractions that it does.

There are three trends I'd like to examine:

1. The rise of real-time data and the evented Web
2. The rise of personal data and user-centricity
3. The rise of context as an organizing and extending framework for content

We'll examine these in turn.

Real-time Data and Events

In their book The Power of Pull, John Hagel, John Seely Brown, and Lang Davison describe some of the incredible changes impacting businesses. In of these, they describe how the world is moving from valuing "stocks" of knowledge to "flows" of knowledge. Anyone spending more than a few minutes on the Web can see how this is true. We don't amass information and hoard it, rather we search for it when we need it.

More and more companies are putting the data and services that drive their Web sites online using an API. Euphemistically the result of this move has been called "the cloud." There are good reasons why cloud-based data and services is gaining traction: Cloud-based services are more accessible, more convenient, and cheaper than equivalent services delivered using more traditional means.

As I write this in April 2011, Programmableweb.com, a directory of online APIs, lists over 2300 APIs in its index. This number is certain to grow. The list includes APIs for searching, financial services, blogging, ad networks, dating, email, government, security shopping, and so on. Some are free and others charge money. Some are personalized (like my Twitter friend feed) and others are general information (like the Google news feed). APIs are the unit of programming on the Web---similar to libraries in traditional applications.

But mere APIs aren't enough. To see why, imagine two scenarios. In the first, your teenage daughter is out with friends. Her curfew is midnight but it's 12:20am and you haven't heard a word. You're worried, imagining an accident on the freeway, or worse. You're calling her cell, her friends, and trying to keep calm.

In the second scenario, your daughter is again out with friends, but this time, a few minutes before twelve, you get a call that goes something like "Hi Mom. I'm going to be 15 minutes late...the movie ran long."

One scenario is filled with hassle and anxiety---the other with convenience and tranquility. There's no doubt which scenario we'd rather be in. And yet, online, we're rarely in a situation where a service anticipates our needs and meets them without prodding on our part. The location metaphor of the static Web puts in the mode of "seek and find" or "go and get."

When an API merely responds to requests, it's like a program that only accepts input, but can never send its output to another application unless it's asked first. We call such APIs half-duplex APIs and readers familiar with the Web and the underlying client-server model will recognize the roots of half-duplex APIs in the foundational technologies of the Web.

In the early days of the Web, all that mattered were domain names and Web pages---brochure-ware, I called it. Later the name Web 2.0 embodied the idea of interactive Web sites where users could actually do something beyond filling out simple forms. The earliest interactive Web services were ecommerce tools in the late 1990's. Later the idea of interactive Web sites extended to all kinds of services from finance to document editing.

Interactive Web services have several problems. First, they tend to be silos that interact with other sites and services in only the most rudimentary ways. If the service doesn't do everything that the user wants, there's almost never a way to combine two services together to solve a unique problem. Second, and more problematically, as we've seen, they are built on a request-response model that requires the user to initiate each interaction.

In contrast, some web applications are beginning to push information to users. This is not only more convenient for users, but creates data streams that other tools can combine to solve problems that require mashing up information from multiple sources. We call the set of technologies and practices that enable users to receive information as soon as it is published by its authors the real-time Web.

You don't have go any further than the Facebook, Twitter, or Foursquare to see the real-time Web in action. These services aren't just interactive Web sites, but are creating streams of data about the people you follow. The stream of tweets from my friends is available to me in a variety of places without me needing to visit any particular Web site. RSS, SMS, and even plain old email are all ways of notifying that information is available now.

The real-time Web won't replace the interactive Web---we'll always need Web sites---but the real-time Web will augment it in important ways. Real-time is a trend that is so natural when we think of what people want and what they pay for that its hard to imagine the world any other way once you experience it. People want to be connected all the time. But more than that, they want information to come to them when its useful or interesting.

Personal Data and User Centricity

One of the hottest trends of the last several years is user-controlled identity. Just as you'd expect, user-controlled identity places the user structurally inside decisions regarding the user's identity information. This is a sharp contrast to how enterprise systems usually talk about the user behind her back.

User-controlled identity systems are coming along just in time. Organizations are increasingly finding that storing and managing personal data is expensive, prone to error and inaccuracy, and undermines, rather than strengthens, the relationship of trust they want with their customers. People resent having data about them used, without their permission, by companies for profit.

There have been some significant advances in the technology of identity and personal data as well. OpenID, OAuth, Webfinger and the like create opportunities for applications to user personal data without having to store and be responsible for it.

With the advent of user-controlled identity services, we can envision a future where users more freely link together their personal data, stored in online services. Personal data has the power to make our online interactions more meaningful (as a simple example, imagine a world where online ads are random vs. a world where they are based on things that interest you). Laying aside privacy concerns most people would rather see relevant ads than random ones. What's more, user-controlled identity will make such sharing more private and secure.

People are already becoming socialized to the idea of keeping data about themselves online through services like Facebook and LinkedIn. If you want someone's current email or phone, chances are better that they've kept their Facebook page up to date than that they've contacted you so you can update you personal list. The idea that we'll have one place for canonical versions of personal data is gaining momentum.

OAuth and more sophisticated protocols like UMA allow people to assemble virtual stores of personal data from the various services they use around the Web. These virtual stores will also allow you to augment the attributes that various services keep about you to assemble as complete a picture as you'd like. Because it's all under your control.

We usually speak of identity in the singular, but in fact, we all have multiple identities. I view all of my accounts as linked together and they seem like different facets of my overall identity. The problem is that the only thing linking them together is me, the person. From the outside looking in, they appear disjointed and incomplete. My bank sees one set of attributes and my employer sees another. Occasionally they link up, such as when I give my employer my checking account number for direct deposit, but they mostly exist in utter, frustrating isolation.

This may seem like a good thing from a privacy standpoint, but we've all suffered from the hassle of having data in one place that we can't use somewhere else. The message of user-controlled identity is we don't have to sacrifice privacy for convenience. We can link all of this data through an online agent under our control.

We tend to think about the potential of better access to personal data from the perspective of what's happening now, but it's also instructive to examine what could be. For example, in his book, Pull, David Siegel describes a golfing scenario where your clubs, bags and even the balls and cups communicate with each other and transmit information about your game to a personal data store in the cloud.

In this scenario, as you play golf, the score is calculated as data comes in. You see your score on your phone. You don't calculate the strokes; rather your strokes and even how far your ball is from the hole are calculated for you . Your phone can even help you find your ball in the tall grass. After the game, you can replay it on a map of the course and get analysis about how you could have played better. While I can't guarantee that your game will improve, this system will keep your fellow golfers honest about their game.

As this example illustrates, even the most mundane data about us is likely to find its way online. Endomondo records information about my bike rides and puts it online. The Withings Bodyscale has WiFi built in and pushes your weight to the Web every time you weigh yourself. A FitBit will record your daily activity and push it to an online account. As the world becomes increasingly instrumented more and more of this kind of data will find its way into the cloud.

Even greater instrumentation is only as far away as your phone. Most of today's smart phones are really sophisticated mobile sensing platforms that also happen to make calls. App stores are full of software that uses those sensors to collect data on our behalf and push it into the cloud. This "exhaust data," as it's called, will create new opportunities for people and the companies that serve them.

Context

In Storytime on the Interwebs, Venkatesh Rao makes an important connection between "plural, interconnected and dynamic 'mesh' experiences," what I'd call Live Web applications, and stories. The relationship is more than analogy. Every good Web experience is a story and, knowing that, we can use the techniques of story telling to help us better understand them.

Every story has certain components: Stories have a purpose or intent---the thing that defines them. Stories have a context that creates a model of what information is relevant. They need a tempo to move them along and define the timeframe. Stories have a narrative structure that connects things together through the beginning, middle and end as well as connecting the past, present and future. And stories have characters---antagonists, protagonists, and supporting cast.

Building applications is, at its best, story telling. The best application developers are storytellers. That's why techniques like storyboarding work as well in designing software as they do in creating books or movies.

Understanding what information is relevant to a particular story is critical to creating a story that is compelling and interesting. Context isn't just about what you include, it's also about what you leave out. In Rao's article, he says:

"When my sister and I were kids, she used to ask apparently odd questions about scenes in movies. Say two people are chatting over tea on the screen. Cut to a scene where one of them is no longer holding a tea-cup. My sister would ask, 'What happened to the teacup?'"

"The answer is, 'Who cares?'"

"Presumably the character put it down, and somebody later washed it. It doesn't matter. If the teacup were significant, the storyteller would usually foreshadow events to come by noting it without explanation (such as a close-up shot of the cup after the people have left the room; perhaps signaling a potential poisoning)."

"So context is not merely about retaining developing momentum, it is about doing so selectively. You have to decide what is significant in the story."

Context is the model we use to make sense of the overabundance of information that any system faces. In the world of the real-time Web, no one wants to be constantly interrupted by events and notifications. Filters are the common answer, but filters need to be more than mere sieves. They need to be smart. They need context.

Context is related to intent. For example, if I show you a handful of URLs, you will interpret them differently if I tell you my intent is to plan a trip or if I tell you my intent is to write a research report for my international relations class. The context changes and with it, the relevance of other information we might run across.

When we tell stories on the Web, we use context to relate events to each other in meaningful ways. One of the real powers of event-based programming is the ability to contextually correlate events. Correlation goes beyond filtering, because it allows us to look for patterns in the event stream that are meaningful for a particular purpose. Contextual correlation can use patterns to explicitly link events across time and event domains.

Context can also be used to correlate events implicitly. In many ways, implicit event correlation is the most powerful because it has the power to allow serendipitous interactions. Serendipitous interactions provide for uses that designers didn't anticipate---perhaps couldn't anticipate. The inherent loose coupling of event-based systems and their resilience in the face of error enables serendipitous interaction and is one of the primary ways to create Web applications that respond dynamically and flexibly to user intent.

Context is not explicitly bound to a specific process or event stream but is relevant to multiple processes through implicit relationships. Knowing, for example, the time of day, the weather, the prime rate, commodity prices, or that it's Super Bowl Sunday provides information that can be used to correlate events. These correlations will be made in the logic of the Live Web applications respond to the events.

Content is more valuable when it's in context. And every user has their own context. Only a system designed to respond to events important to the user on a personal level can deliver the value associated with content in context. That's what KRL is designed to enable programmers to do.

Whither KRL?

These trends are significant and real. I believe they offer important and valuable benefits to people and will result in a Web that we can hardly recognize today.

In my last post, I offered up three key benefits for developers using KRL. The trends I've described above are related to each of those benefits and give clear reasons why I believe developers will care about these benefits in building future Web applications.

KRL's event expression language provides an abstraction that is as important and powerful for handling dynamic data streams as SQL was in dealing with databases. Event expressions (or eventexs) provide a common language for reacting to and talking about events that people care about. I cannot overstate the importance of eventexs to what KRL is and why it's different from every other programming language you know.

KRL's built-in support for personalization goes well beyond merely saving developers from having to build a few login screens and the associated logic. KRL recognizes at its foundation that personal data is important and makes it easy to keep track of the separate entities using an application. More importantly, KRL also makes it simple for developers to link in other data from other cloud-based services. The system underlying KRL, the Kinetic Rule Engine or KRE, is built from the ground up to support user-centricity and the interactions that it requires.

Having a lot of personal data hanging around provides little value in and of itself. Facebook isn't valuable because they have lots of data about you, but because of what they allow you to do with that data. Similarly, the personal data trend demands applications that use the data to help people accomplish valuable tasks. People value interactions--the data in motion--not the mere stockpiling of it. KRL is positioned to build applications that use personal data.

On the Web, creating contextual applications often means breaking down the silos of individual Web applications. People don't go online to visit Web sites. They go online to get something done--achieve some purpose. The browser is the perfect platform for deploying applications that cross organizational boundaries and the artificial barriers they impose.

Everyone I know can relate to the idea that they often have to cut and paste data out of one Web site and into another. They keep the context of their interactions in their head and flip between browser windows or tabs to create an experience that gets the job done. KRL's ability to link data from multiple Web applications and use it in context is a core value that the language and it's underlying system provide. Using KRL, building cross-domain Web applications is simple and quick.

You might think it silly to design a language around trends, but frankly that's where many new languages come from. PHP became popular because it supported the need for data-driven Web sites. Rails (a domain specific language in Ruby) became popular because it made creating interactive, Web 2.0-style application easier. If I'm right and real-time, event-driven, personalized, contextual Web applications are going to rule the future, then a language that supports building them isn't a bad idea--it's essential. Without the abstractions that a language provides, the job is just too complicated otherwise.

The ideas within KRL are important and the abstractions that it supports are timely. Install a few KRL applications to try them out yourself. Write a KRL program if you're a developer. We're anxious to get your feedback.

5:57 PM | Comments () | Recommend This | Print This

Why KRL?

A lot of people ask me why I invented Kinetic Rule Language (KRL) instead of just using an existing language. There are a lot of answers to that question, but I want to get to some specifics.

Tools shape how we think and work. I learned long ago that the best way to think differently about a problem is to create a nomenclature that describes and illuminates the new domain. KRL is a rule-based language that is custom built for the domain of event-based applications that operate on real-time data in the cloud. KRL was designed from the ground up with events and the cloud in mind. KRL provides a number of familiar touch-points for users already accustomed to Web programming and JavaScript but provides a framework for making the most of an evented Web.

Let's start by exploring why anyone learns any new language. Programmers learn new languages for a variety of reasons:

1. The cool factor - some languages are just cool and people learn them because there is a cachet in them. They like the people who program in them. They like the conferences around them. Or they simply are the latest thing.
2. The money factor - some programmers learn a new language because they want to use it to solve a particular problem that the they'll get paid for. Often, your job requires using a particular language.
3. The opportunity factor - some languages are seen as important components of the future. The HTML markup language certainly seemed that way in 1993. Java was viewed similarly about the same time frame.
4. The platform factor - some languages are better for a particular purpose or platform. Lots of people learned BASIC back in the day (way back, in fact) because it was the only language available (other than machine language) on the PC, the TRS-80, and so on. More recently, people learned JavaScript because it is the language of the browser. Learn it or forget working on the browser.
5. The creative expression factor - some languages are more efficient for certain tasks. Some languages are more flexible. Programmers also learn new languages because they like how they look. They enjoy the shear beauty of the expression and the elegance of the code they can write.

Even though I think items (1-3) are important, I'm going to ignore them for now. I certainly think KRL is cool, that there is money to be made with it, and that it's an important language to know, but I want to focus on aspects of KRL that tie into points (4) and (5).

KRL offers developers three primary benefits over other languages:

1. KRL is uniquely positioned to support dynamic data streams and cloud-based service APIs. Web 2.0 was characterized by interactive Web sites that felt like applications. Behind those Web applications were big databases. Relatively static, these pools of data underly the functionality of any sophisticated Web site. We make queries against them when ever we interact with the site.

   In contrast, many modern Web sites--think Twitter, Facebook, and Foursquare--are characterized by streams of data. Real-time data is what makes these sites interesting. Sure there are databases behind them, but the functionality that users value is based on information in motion. RSS was the harbiger of this trend and is still a key piece of infrastructure. With protocols like PubSubHubBub, it participates fully in the real-time information movement.

   KRL is based on events--a notification or message that something happened--and has a sophisticated and robust event expression language built in. Think of this like regular expressions, but for information in motion. KRL rules are seleted based on patterns of events and represent a small chunk of functionality that the programmer has determined should take place when that event expression is satisfied (i.e. when a particular sequence of events have arrived).

   For example, the following event expression is satisfied when an RSS feed contains a story that includes a stock ticker symbol and the price of that same stock goes up by more than 2% within 10 minutes.

   select when rss item content #Stock Symbol: (\w+)# 
                setting (symbol)
        before stocks price_rise ticker eq symbol && 
                                 percent > 2
     within 10 minutes
   

   This presumes that event streams for the RSS feed and stock data are available, but that's not hard to set up.

   This benefit falls into category (5) from above. KRL is an efficient and flexible way of handling real-time data streams and events. For a more complete explanation of event expressions, see Chapter 3 of my upcoming book The Live Web. I'm happy to share an early draft with you if you like--just contact me.

2. KRL makes building peronalized applications easy. Any application that supports more than rudimentary personalization has to allow for multiple tenants. This usually means creating an identity system, figuring out accounts, setting up login screens, and then keeping track of the entity information for each user. All of that is built into KRL. Creating an app that stores data for each user and behaves differently based on the user is trivial in KRL because there's nothing to do. It's all done lower down in the system.

   This benefit falls into category (5) from above. Building personalized applications in KRL is simply more efficient than in other languages. To see an example of a personalize browser app, install Personal Block List (PBL). PBL lets each user block search results on Google, Yahoo! and Bing. My list is different than your list and regardless of what computer I'm using I see my list. And yet, Ed Orcutt, who wrote PBL didn't have to create any of the infrastructure of code that makes that personalization possible. KRL took care of all of those details for him and even provided built-in persistent variables so that for this application no database was necessary.

3. KRL gives developers the ability to bust out of silos and work across multiple Web sites. More than that, KRL allows developers to bring other information and systems into the mix: voice, SMS, email, and almost anything else that can create a notification of some kind. Go install TomatoFlix which works on IMDB.com, Netflix.com, Redbox.com, and Fandango.com, to see an example of an application that has broken down the walls of the silo to give the user useful information in context.

   This benefit falls into category (4) from above. Writing applications that run on someone else's Web sites requires cross-browser support for the Javascript that runs on those sites and an extension for every browser you want to support. Writing cross-browser JavaScript requires picking the right libraries and using them correctly. KRL makes that problem go away by generating the right JavaScript for the job. More importantly, KBX, the univeral Kynetx Browser Extension works in Chrome, Firefox, and Safari (with support for IE on the way) without the developer writing a line of code.

I believe KRL solves an important class of problems better, more easily, and more flexibly than other languages. What better reasons could there be to learn something new?

7:58 AM | Comments () | Recommend This | Print This

April 13, 2011

Building a Personal Commenting System Using Twitter and Kynetx

Earlier today I was talking to Britt Blaser about making the Web friendlier for people who want to organize around political issues. Of course, I immediately imagined a Kynetx application for doing. Doing this as a browser application would have a few advantages:

1. A browser application operates anywhere people wanted to comment and yet still bring the comments together in various, meaningful ways.
2. A browser application can pull information from various contexts and make it available where people are.
3. A browser application would be easily updated as new features were added.

What I envision is something that combines the functionality of the UtahPolitics.org app I built a while back (and yes, it needs to be updated) and Aaron Frost's TweetPlus app. The idea is that you could Tweet on any page and see what others have tweeted on that page.

The next step would be to filter the tweets on the page according to people in my social graph. So get people I follow from Twitter and my friends from Facebook (using Qwerly to find their Twitter handle) and filter the tweets on a page so I only see the ones from my Twitter and Facebook friends.

Final step would be to let me register hash tags with the app and then I'd see tweets from anyone on pages as long as they had those hash tags.

Now, I've moved tweets out of Twitter, showing them on pages I visit, as long as their from people I follow on Twitter or are my friends on Facebook. I can follow issues by hash tag and see relevant tweets on those issues as well. We'd probably want more, but this is a good start and pretty easy to do.

8:29 PM | Comments () | Recommend This | Print This

April 8, 2011

Tweeting from KBlog: An Experiment in Loose Coupling

Last week I posted about creating a blog using KRL as an example of an application that used multiple rulesets and has a more complicated event hierarchy than typical KRL applications. I followed that up with a post about making the back button work in the resulting blog as an example of using external JavaScript libraries and emitting raw JavaScript from rules. We even added a new event type (web:hash_change) using JavaScript. You can see the resulting blog here.

Another idea I've wanted to explore is how event-driven applications can be extended in a loosely coupled way. The argument is that because event-driven applications don't rely (as much) on request-response (remote RPC) interactions, the should be more loosely coupled. Functionality can be layered onto event-drive applications in ways that's difficult to imagine in more tightly coupled architectural styles.

If you're familiar with hooks in programming languages, then you're part-way to understanding this. I'm most familiar with them from Emacs Lisp. Hooks provide places for intercepting control flow in the program. For example, when you load a major mode in Emacs Lisp, there is a hook that will call a user-provided function, allowing the mode to be configured and customized. Hooks usually cause functions to execute. Events, on the other hand, provide a level of indirection so that what runs can be dynamically determined.

Tweeting a Post

To put this into action in KRL, I expanded the blog application by layering on functionality that gave the author the option of automatically tweeting the blog post. If you recall from the previous design, when the form is submitted, a rule, handle_submit raises the event new_article_available. We can post the content of the post to Twitter by listening for that event and doing what's necessary to format and send the information to Twitter:

rule send_tweet {
    select when explicit new_article_available
    pre {
      author = event:param("postauthor");
      title = event:param("posttitle");
      tweet = <<
New blog post from #{author}: #{title}
http://www.windley.com/kblog 
>>;
    }  
    if (twitter:authorized()) then {
      twitter:update(tweet);
    }
  }

The rule send_tweet listens for the new_article_available event and uses the built-in KRL Twitter module to check the keys (stored in the key pragma of the meta section of the ruleset) and update the associated Twitter stream. This was pretty easy to put into place because the event was already being raised.

Making Tweeting Optional

The next step--making tweeting the post optional--however, revealed the mistakes of my earlier design that got in the way of modifying it by simple adding new rules rather than modifying the original rules.

Making Twitter update optional requires us to give the author the option of tweeting or not. Adding a checkbox to the form give the author this option and is relatively easy if an event reveals when the form is there. In my original design, the place_form rule was a terminal in the event hierarchy, meaning that it raised no events. Without an event, there's nothing that the rule adding the checkbox can watch for. So, I modified the place_form rule to raise the explicit event post_form_ready. Now the place_checkbox rule can listen for that event and add a checkbox to the form:

rule place_checkbox {
    select when explicit post_form_ready
    pre {
      form_id = event:param("form_id");
      checkbox_html = <<
<p class="checkbox">
  <label for="posttitle">Post to Twitter?</label>
  <input id="tweet" value="1" 
   type="checkbox" checked="checked">
</p>
      >>;
    }
    after(".text-area", checkbox_html);
  }

This adds a checkbox to the form without modifying the original ruleset (besides the fix to raise the event). Other changes to the form for other features could also be added. The form now becomes a canvas that rules can paint as they modify the functionality of the underlying application.

(click to enlarge)

There's just one more tiny problem. The handle_submit rule raises the new_article_available event, but it places the form values in individual parameters. In other words, it's tightly coupled to the original structure of the form and thus won't pass on the value of the checkbox so that we can test it in the rule conditional. The answer is to be less specific in the form parameters we pass with the event. Here's the new handle_submit:

rule handle_submit {
  select when submit "#blogform"
  always {
    raise explicit event new_article_available 
      for ["a16x89", "a16x91"] with
        post = event:params();
  }
}

Notice that where before, the explicit event had a separate parameter for each form element, this version just passes all of the parameters onto any listening rules. Now send_tweet can pick out what it needs:

rule send_tweet {
  select when explicit new_article_available
  pre {
    post = event:param("post");
    author = post.pick("$..postauthor");
    title = post.pick("$..posttitle");
    tweet = <<
New blog post from #{author}: #{title}
http://www.windley.com/kblog 
>>;
  }  
  if (post.pick("$..tweet",true).length() && 
      twitter:authorized()) then {
    twitter:update(tweet);
  }
}

This rule is largely the same except that it has to pick the author and title out of the post parameters. The rule condition has been changes to check the value of the tweet parameter in the form. A bit of explanation: the second, true argument to pick causes it to always return an array. We check it's length since a checkbox doesn't return anything if it's not checked and thus we get an empty array out of pick. The length of an empty array is zero, which is interpreted as false in KRL. We don't actually care about it's value if not zero.

Lessons Learned

The new ruleset consists of two rules: place_checkbox for putting the checkbox in the form and send_tweet for composing and sending the tweet if the checkbox is checked. The new rules overlay the existing application nicely. If they're not present then the functionality isn't either. The application keeps on working just as it did before. Adding the functionality requires no configuration of the original application or wiring the new functionality in place. Everything fits nicely together and comes apart just as easily.

Of course, that assumes that original application is written in a way that supports loose coupling. We ran into two problems:

1. Terminal rules reduce chances for loose coupling Because the place_form rule was terminal--didn't raise any events to indicate what it had done--there was no way for another ruleset to extend it's functionality. The lesson here is that terminal rules should be avoided if your goal is to create extensible, loosely coupled applications that don't require code modifications. That said, as long as you have access to the code, adding explicit events to rules when you need them isn't difficult and is unlikely to break anything, so it's a low-cost, low-risk code mod.
2. Specificity in parameters leads to tight coupling When events have parameters, we should send them on as a map (hash). Picking out individual parameters and sending them on by name means that only those parameters will ever be available to other rules. That doesn't mean you can't filter parameter maps to remove data you don't want available downstream. But the result ought to be everything but the data you filter, not just the data you choose to pass on. Pass parameters as a structure rather than by name for support the greatest flexibility.

These problems are easy to find and a simple test application showed where they existed. I believe that writing applications as collections of rulesets in such a way that they support extending functionality through accretion in a loosely coupled manner is practical, but it does require some planning and design effort.

Event Hierarchy

The modification to the event hierarchy is fairly modest since only two new rules were added. This reflects the loosely coupled nature of the application.

(click to enlarge)

There is a new ruleset for the tweet rules. The place_form rule now raises the post_form_ready event for which the place_checkbox rule is listening. The send_tweet rule listens for the new_article_available event. As they should, the new rules are accretive to the overall event hierarchy, not requiring changes to it, but merely adding to it.

4:48 PM | Comments () | Recommend This | Print This

April 6, 2011

The Cost of Saving Money

This morning I had an interesting experience. I want to get a new bike from REI using rebate money I earned from me REI Visa card. REI also does a thing where you get 20% off one full priced item during "rebate season." I stopped at REI on the way to the airport to look at the bike and give it a test drive. I liked it. But I wasn't really ready to buy yet. But the thought of the 20% discount ending on Saturday was pushing me to buy today. Then I found out that the 20% discount doesn't apply to bikes (except the house brand). Suddenly the pressure to buy was gone. The odd thing is that I was relieved to find out that I was not going to save hundreds of dollars.

This illustrates something I too often fail to realize: discounts, coupons, rebates, and points--indeed the entire apparatus of loyalty in modern retail--are a source of untold stress and inconvenience. We'll do almost anything to save a little money without counting the cost of that reward. We all know people who have elaborate systems for keeping track of coupons, discounts, and point programs. I have a few such programs that I succumb to: frequent flyer miles and REI rebates. Anyone who has ever used frequent flyer miles can testify to the inconvenience and frustration they cause. And yet we let these programs drive our behavior in unnatural ways.

Would we all be happier if we just spent a little more money and gave up coupons and discounts? To that end, would we all be happier if we just had fewer things? Probably. I'm not sure where this goes personally, but the experience was so defining that I'll be thinking about it for a good while.

9:49 PM | Comments () | Recommend This | Print This

KBlog: Making the Back Button Work

Friday, I posted a long blog article that discussed using KRL to create a simple blogging application. the application writes multiple pages, manages a navigation bar, and allows new posts. (Try it here.) One problem with the implementation is that because it paints the entire app in a single page (sometimes called a single page interface or SPI), the back and forward buttons don't work.

The problem is that the browser doesn't know there's a new page and put it on the history unless the URL changes. The back and forward buttons are just indexes on the browser history. The trick that SPI Web applications use to get around this to write URL fragments (the stuff after the # symbol). Adding a fragment doesn't cause the browser to reload the page because fragments are designed for interpage navigation.

Change the URL Fragment

Rewriting the fragment on the page just takes a little JavaScript. And using JavaScript from within a KRL ruleset is easy with the emit action. We add the emit action to the rules that write pages. Here's the show_contact rule that controls the Contact page:

rule show_contact {
  select when web click "#siteNavContact"
  pre {
    contact_html = <<
    <div id="leftcontainer">
      <h2 class="mainheading">Contact</h2>
      <article class="post">...</article>
    </div>
    >>;
    title = <<
<title>#{blogtitle} - Contact</title>
    >>;
  }
  {
  paint_container(title, contact_html);
  emit <<
    self.document.location.hash='!/contact';
   >>;
  }
}

Adding the emit ensures that whenever the Contact page is displayed, the URL will have #!/contact appended:

http://www.windley.com/kblog/#!/contact

We make similar changes to the Home page so that it has #!/ appended to the URL whenever we visit that page.

Note: technically, I shouldn't be putting the ! in the fragment since that indicates to search engines that the page is available at the non-fragmented URL (with #! removed) for search engine crawlers.

Updating the Page When the Fragment Changes

After we've correctly modified all the rules that control a page, the pages will all be identified with the right fragment as you navigate from page to page using the links in the navigation bar at the top of the blog. The problem is that the back and forward buttons are still broken. If you use them the URL will change, but there's nothing to change the content as the URL changes. As we said before, the browser does not alert the server when the fragment changes and this is as we want it. We want to decide what happens when the URL fragment changes on our blog.

To remedy this problem, we'll use a jQuery plugin called hashchange. Since the KRL Web endpoint has jQuery built in, using jQuery plugins is easy.

First we put the plugin on our server and change the jQuery variable in the final line so that we pass in $KOBJ instead:

...
// was --> })(jQuery);
})($KOBJ);

The plugin must call the KRL jQuery library, named $KOBJ. The Web endpoint uses jQuery in extreme compatibility mode to ensure that it doesn't interfere with Web pages that have already loaded jQuery.

Second, we have to load the plugin in our ruleset. KRL provides a facility for loading external JavaScript resources as a pragma in the meta section of the ruleset:

use javascript resource "http:/.../kblog/jquery.hashchange.js"

This loads the library once (and only once).

Third, we need to deploy the hashchange watcher so that it can see when the fragment changes and run a function that will raise an appropriate event to the KRL engine. You'll recall that we had a few other watchers in the code that initialized the page. We want to add an emit action to that same rule that watches for the fragment change and raises an event:

rule init_html {
  select when pageview ".*" setting ()
  {
    replace_html("#about", about_text);
    watch("#siteNavHome",  "click");
    watch("#siteNavContact", "click");
    emit <<  
      self.document.location.hash='!/';
      $KOBJ(window).hashchange(function() { 
        if(KOBJ.a16x88.previous == undefined || 
           KOBJ.a16x88.previous != self.document.location.hash) {
         var app = KOBJ.get_application("a16x88");
         app.raise_event("hash_change",
                         {"newhash": self.document.location.hash});
         KOBJ.a16x88.previous = self.document.location.hash;
        }
      });
    >>;
  }
  always {
    raise explicit event blog_ready
  }
}

The only thing new is the emit action. The JavaScript in that action sets to fragment to the default page fragment and attaches a hashchange JavaScript listener to the window. The function that it calls uses the Web endpoint runtime to get the app object associated with the current ruleset and raise and event called hash_change to the KRL engine with the new fragment as a parameter. The whole thing is wrapped in an if statement to ensure it only runs once per fragment change.

You might be tempted to put the $KOBJ(window).hashchange() call in the global block. As programmers we're used to "global" blocks being evaluated once. And that's true for KRL as well, but in KRL it's once per event. Note that any given interaction with the blog causes multiple events. The init_html rule is only run once per blog interaction--on the initial pageview--so that's the place to put any watchers. After that all the events are the result of clicks, submits, or explicit raises.

Fourth, and last, we need to modify the rules that paint the pages to respond to the hash_change event. Here's the show_contact rule again, with the changes that do that:

rule show_contact {
  select when web click "#siteNavContact"
           or web hash_change newhash "/contact$"
  pre {
    contact_html = <<
    <div id="leftcontainer">
      <h2 class="mainheading">Contact</h2>
      <article class="post">...</article>
    </div>
    >>;
    title = <<
<title>#{blogtitle} - Contact</title>
    >>;
  }
  {
  paint_container(title, contact_html);
  emit <<
    KOBJ.a16x88.previous = '#!/contact';
    self.document.location.hash='!/contact';
   >>;
  }
}

There are two changes:

1. We added another clause to the select statement that looks for a hash_change event that has a parameter named newhash with the value matched by the appropriate regular expression to ensure we're looking at a contact page.
2. The emit has an additional line of JavaScript to set the previous variable to ensure that this event only happens once.

Note: the JavaScript location.hash variable has an inconsistent interface. When you set it, you don't include the #. But when you read it, the string you get back has the # prepended. That took more than a few minutes to figure out.

Event Hierarchy

The event hierarchy diagram is only slightly more complicated (here's the original):

(click to enlarge)

A new Web event, hash_change has been added. The appropriate rules are selected based on the event parameters passed in. These are used to fire existing rules.

With these changes navigating the blog causes the URL to update and, conversely changing the URL changes the blog. Now the back and forward buttons work as we'd expect.

9:01 AM | Comments () | Recommend This | Print This

April 1, 2011

Event Hierarchies: A Blog Written in KRL

I've mentioned several times that I'm writing a book called The Live Web that discusses how events and strong user identity combine to produce new ways of creating Web applications that are significantly different than those in use today. Chapter 12 is about creating event-driven applications that are based on multiple rulesets. At the recent Kynetx Impact developers conference, Ed Orcutt built a blog application using KRL. Actually, he did it twice: once with one ruleset and another time with three.

If you're like to read the book in draft form, contact me and I'll invite you to the Dropbox.

Ed's example intrigued me for several reasons:

1. I saw DHH do a tutorial once that involved building a blog application in real-time as a method of teaching the audience about Rails. At the time I thought it was cool how easily you could build an interesting application like a blog in a short amount of time and little code.
2. I believe that KRL applications have to "grow up" beyond the current crop of simple Web augmentations to become full fledged applications. There's nothing wrong with Web augmentations--I have half a dozen running in the KBX (go install it now). But I think evented architectures have much more to offer us.
3. I've been looking for multi-ruleset application examples to explore event normalization and hierarchies.

I rewrote Ed's multi-ruleset application to make it my own in terms of understanding it and to simplify the HTML a bit. You can explore the resulting blog. In my design, I have three rulesets:

1. KBlog - the main ruleset that controls the HTML assets and presentation. This ruleset functions like the presentation layer of a traditional Web application, responding to user input and displaying results.
2. KData - the ruleset that manages the blog data. I wanted to experiment with putting all the data assets and their access rules in one place. In the first instance I use application variables to store the blog data. In a later instance, I'll use an online storage pool.
3. KBlog Posting - I wanted posting to be separated from the other parts of the application so that only people who had access this ruleset can post. This is not a good solution to blog posting access, but it works for this demo.

Let's explore the highlights of each of these.

Building the Blog

The primary job of the KBlog ruleset is presenting the blog. The ruleset does this by preparing the container for the blog articles and then filling it with the articles. The blog is AJAX enabled, meaning that the framework of the blog including the basic HTML content and CSS files are downloaded once and then everything else is painted on that framework in response to user actions. This is a style of Web site construction that has been made popular by sites like Google, Twitter, Gawker, and others. The idea is that it saves bandwidth by not requiring that things like the HTML and CSS be downloaded each time. KRL is a natural way to build sites that use this style of construction.

The following shows the basic layout of the site.

(click to enlarge)

A navigation bar at the top contains links that can be used to navigate between different parts of the site. At present, just the home page and contact page. The DOM for that portion of the page looks like this:

<nav id="sitenav">
  <ul id="navlist">
   <li><a href="javascript:..." id="siteNavHome">Home</a></li>
   <li><a href="javascript:..." id="siteNavContact">Contact</a></li>
  </ul>
</nav>

We'll use this later in rules that modify the navigation buttons.

The primary section of the blog--the left container--is where page content is written. Initially it is empty and has the following structure:

<div id="leftcontainer"></div>

The information section of the blog--the about section on the right--is also initially empty:

<div id="sidebarwarp">
  <h2>About KBlog</h2>
  <p id="about"></p>
</div>

The init_html rule is selected when a pageview event occurs. The selector is very general because it should fire whenever the blog is viewed. This rule is responsible for painting the "about" text on the right side and setting watchers on the navigation buttons.

rule init_html {
  select when pageview ".*" setting ()
  {
    replace_html("#about", about_text);
    watch("#siteNavHome",  "click");
    watch("#siteNavContact", "click");
  }
  always {
    raise explicit event blog_ready
  }
}

When this rule is done it raises the explicit event blog_ready because--get ready for this--the blog is ready to be populated with the page content. In this blog there are two pages: the home page with the blog articles and the contact page that has contact information.

The show_home rule is selected when one of two events occurs: someone clicks the home link on the navigation bar or there is an explicit blog_ready event (note this makes the home page the default). The show_home rule sets up the container to receive blog posts and sets the title:

rule show_home {
  select when web click "#siteNavHome"
           or explicit blog_ready 
  pre {
    container = <<
<div id="leftcontainer">
 <h2 class="mainheading">Latest from the blog</h2>
 <div id="blogarticles">Code Monkey was here :)</div>
</div>
   >>;
   title = <<
<title>#{blogtitle}</title>
   >>;
  }
  paint_container(title, container);
  always {
    raise explicit event container_ready;
    raise explicit event need_blog_data for a16x89
  }
}

The show_home rule makes use of a user defined action, paint_container:

paint_container = defaction(title, container) {
  {replace_html("title", title);
   replace_html("#leftcontainer", container);
  }
}

The paint_container action is also used by the show_contact rule that puts up the contact page when someone clicks on that link. A user defined action ensures that we do this the same way both times.

The show_home rule also raises two explicit events, one that indicates that the container is ready and another that says that data is needed. We'll discuss the rules that respond to the latter event in the next section, but ultimately, the result of that process is that the explicit blog_data_ready event is raised.

Putting the actual blog articles in the container is the job of a rule called show_articles. This rule fires when the container_ready and blog_data_ready events are raised. The rule loops over each member of the hash representing the blog data, formats the correct HTML, and inserts it into the page:

rule show_articles {
  select when explicit container_ready
          and explicit blog_data_ready
   foreach event:param("blogdata") setting (postKey,postHash)
    pre {
      postAuthor = postHash.pick("$.author");
      postTitle  = postHash.pick("$.title");
      postBody   = postHash.pick("$.body");
      postTime   = postHash.pick("$.time");
      postArticle = <<
<article class="post">
  <header>
  <h3>#{postTitle}</h3>
  <span class="author">by #{postAuthor}</span>
  </header>
  <p>#{postBody}</p>
  <footer>
   <p class="postinfo">Published on <time>#{postTime}</time></p>
  </footer>
</article>
      >>;
    }
    prepend("div#blogarticles", postArticle);
}

This flow allows the blog to be built in pieces and for navigation actions to simply update the portion of the structure that requires it.

Handling Data

One of the things I wanted to do in this exercise is to separate out handling the data from the presentation pieces of the application. I did that by creating a ruleset that uses it's app variables to store the blog posts. This is not the most efficient way to handle the data, but it raises some awareness of event design, so bear with me. In a later post, I'll show how to use another method to handle the blog data.

Using an app variable to store the blog data is easy enough. We define a function that formats the map representing a single blog post like so:

mk_article = function (author, title, body) {
  postTime = time:now({"tz":"America/Denver"});
  { postTime : {
     "author" : author,
     "title"  : title,
     "body"   : body,
     "time"   : postTime
    }
  }
}

Now we simply use a rule, add_article, to watch for new articles, format the data with the mk_article function, and add it to the app variable BlogArticles:

rule add_article {
  select when explicit new_article_available
  pre {
    postHash = mk_article(event:param("postauthor"),
                          event:param("posttitle"),
                          event:param("postbody"));
    BlogArticles = app:BlogArticles || {};
  }
  always {
    set app:BlogArticles BlogArticles.put(postHash);
    raise explicit event new_article_added for a16x88;
  }
}

Note that this rule takes no action. The benefit is all in the effects. When the rule is fired, it raises an explicit exception to indicate a new article has been added. The KBlog ruleset contains an intermediary rule, show_new_article that transforms this event into a blog_ready event. Of course, the show_home rule is listening for the blog_ready event and this causes the screen to be repainted.

rule show_new_article {
  select when explicit new_article_added
  always {
    raise explicit event blog_ready
  }
}

Reading the data out of the ruleset is also event-driven. The retrieve_data rule selects on the explicit event need_blog_data and raises the explicit event blog_data_ready, attaching the blog data from the app variable as an event parameter:

rule retrieve_data {
  select when explicit need_blog_data
  always {
    raise explicit event blog_data_ready for a16x88 with
    blogdata = app:BlogArticles || []
  }
}

Posting

All that's left to complete the application is the ability to post. As explained earlier, this functionality is in a separate ruleset and layered on top of the basic functionality of the blog. Most users will never need the functionality and won't see it. Only posters need access the rules that control it.

The first order of business is to add a navigation button to the bar at the top of the page exposing the functionality. The place_button rule fires on a pageview event and puts the button in place and makes it active by assigning a watcher to it:

rule place_button {
  select when pageview "kblog"
  pre {
    button = <<
<li><a href="javascript:void(0);" id="siteNavPost">Post</a></li>
    >>;
  }
  {
   prepend("#navlist", button);
   watch("#siteNavPost",  "click");
  }
}

When someone clicks on this button, the Web endpoint will raise a click event and the place_form rule will fire. Place_form creates a form, places it on the page by replacing the left container, and watches for the submit button (some of the HTML in the form has been removed for brevity):

rule place_form {
  select when web click "#siteNavPost"
  pre {
    form = <<
<div id="leftcontainer">
<h2 class="mainheading">Post</h2> 
<article class="post">
 <form onsubmit="return false" method="post" id="blogform">
   <label for="postauthor"><small>Name</small></label>
   <input name="postauthor" id="postauthor" value="" type="text">
   ...
   <input name="submit" ... type="image" src="images/submit.png">
  </form>
 </article>
 </div>
     >>;
  }
  {replace_html("#leftcontainer",form);
   watch("#blogform", "submit");
  }
}

The following screenshot shows the blog after the Post link has been clicked, the form painted on the page, and filled out by the user:

(click to enlarge)

When the submit link is clicked, the Web endpoint raises a submit event. The handle_submit rule handles the event and raises the new_article_available event with the data from the form. As we've seen, the new_article_event is handled by the add_article rule from the data ruleset.

rule handle_submit {
  select when submit "#blogform"
  always {
    raise explicit event new_article_available for a16x89 with
      postauthor = event:param("postauthor") and
      posttitle = event:param("posttitle") and
      postbody = event:param("postbody");
  }
}

We now have a complete blog application, albeit simple.

Event Hierarchies

Building this application taught me a few things. First, getting the names of events and what they mean right is important. This is called "event normalization." Second, understanding what rules were listening to what events and how they played together was the key design space. Once that was done, writing the rules was pretty easy.

I put together the following event hierarchy graph to represent what I'd designed--mostly in my head--as I went along.

(click to enlarge)

In the graph, rectangular boxes are rules. They are named as verbs. The ovals represent events. They are nouns. The Web events are the entry point for any path through the graph. Rules that don't produce events are terminals in the graph. I was tempted at first to think of this as a kind of state diagram, but that's not quite right--the transitions don't happen because of some input. The events themselves are the input. Events are most useful when they are moving and so there's no notion of "stopping" at some point in the diagram. Once you enter at the top, you flow through to one of the terminal rules.

The idea of events as nouns and rules as verbs is a useful way to keep your design straight. If you find yourself giving events names that are verbs, you're probably not really creating an event-driven application. Rather you're using events to do a request.

Conclusion

Writing this blog application in KRL was instructive. The rulesets separated different functionality cleanly and allowed special functionality--like the ability to post--to be layered onto the other rules easily.

One objection to the design might be that the controller logic that is not clearly delineated. The controller is in the in the various select statements. The controller logic for the application could be brought together by using more intermediary rules in a controller ruleset.

As I've worked through this demo, I've wondered if good rule design ought not to require that every rule raise an event when it's done to indicate what is true at that point. For example, the show_articles rule is terminal in the event hierarchy I show above. But what is true at that point, the event, is the articles are showing. Nothing in the rulesets I've created here uses that information, but what if someone wants to layer functionality onto this application later in a loosely coupled way? They might want to know when the articles are there so they can take the next step--whatever that is. Any given rule, terminal now, may not always be so as new functionality is created.

I have several things I want to do from here. As I've aluded, I want to replace the ruleset data management with a module that interfaces to an online data repository and test performance differences. This considerably simplified the event hierarchy and introduces the idea of hybrid event and request drive applications. Also, I want to show how jQuery plugins can be added to make things fancier on the UX side from within KRL.

The complete code for the three sample rulesets is available on Github.

2:06 PM | Comments () | Recommend This | Print This