Phil Windley's Technometria

« December 2010 | Main | February 2011 »

January 25, 2011

My LinkedIn Map: Visualizing Your Connections

LinkedIn Labs has released a visualization tool called an InMap that shows your connections and how they're related to each other. Mine is linked from the thumbnail on the right. A few things I noticed:

• The two large lobes that look like the upper wings of the butterfly are my Utah tech links on the left and my Non-Utah (mostly identity) links on the right. Very little cross-over. At least for the people I know, I'm the link between many people in Utah and many people in the Bay Area and the Identity Community.
• The Utah tech community represented by my links is the largest pool of connections I've got and they're quite well connected to each other.
• Steve Fulling, my partner at Kynetx and colleague in lots of other things is very well connected to the people I'm connected to.
• My former Excite@Home colleagues (and a few others) are the lower right wing of the butterfly. interesting that they're mostly disconnected from the tech friends I've made in the Bay Area since the late 90's.
• Colleagues from when I was writing at InfoWorld sit in between the Excite@Home community and the Identity community. Not surprising there.
• Former iMall colleagues don't seem to be well connected to the larger Utah Tech community.
• IT Leaders and CIOs are connected to each other and spread throughout the four main lobes.
• There's a strong enough connection between Kynetx, iMall, Utah, and Sento that Kynetx people sit smack dab in the middle of the other three.
• I know techies from church and they split into two distinct groups along geographic boundaries and neither are well connected to the greater Utah tech community.
• BYU colleagues seem to be VERY disconnected from the greater Utah tech community (at least the people I know). Not too surprising.

This kind of analysis is fascinating. I could imagine asking someone to see their LinkedIn map as part of a job interview for certain jobs (PR, evangelist, etc.) as it's concrete evidence of their connectedness. Visualization is going to play a bigger and bigger role in our daily decision making as we get more and more connected data like LinkedIn.

10:29 AM | Comments () | Recommend This | Print This

January 24, 2011

Reading for January 24, 2011

I'm supposed to be posting these weekly, but the book writing keeps getting in the way. I guess that's better than the other way around.

• Why Google and Demand Media Are Headed for a Showdown - Interesting play out here since DM is headed to an IPO soon. They're in a quiet period and can't respond. This would be a good way for a 3rd party to damage the IPO to their own ends.
• Is Law School a Losing Game? - I hear this more and more. I wouldn't advise my kids to go to law school even though I think that learning legal thinking would be fun.
• Kindle Comes To The Mac App Store, Already No. 5 Free App - Mac apps are just regular applications with some special files in the resource fork. I was worried at first that they'd break the consistency of application installs and management.
• A Matter of Life and Death: How important is economics? - "Although it may not be possible to say definitively, Mises's and other market economists' abstract theories probably laid the intellectual foundation of the Thatcher and Reagan revolutions in the 1980s that, again, significantly improved well-being..."
• Bump Raises $16 Million Round Led By Andreessen Horowitz - I don't get this. I've never had anyone mention Bump to me. Not a phenomenon as far as I know.
• Words with Friends - "I don't mind paying but [Apple] made it about paying. Uck."
• Google Code Blog: Google URL Shortener gets an API - I wish we didn't need shorteners in the first place. Think of all the links that are going to wink out in the future when Bit.ly or someone else goes under and takes their database with them.
• Location-tagged events in elmcity hubs - I love the practical exploration that Jon's doing in the Elm City project. The semantic Web needs this kind of work as much as it does a careful study of ontologies. Maybe more.
• VLC for iOS vanishes 2 months after eruption of GPL dispute - Interesting read about how the GPL licensing terms are incompatible with Apple's app store. It's not clear they could be compatible with any commercial app marketplace. There's a showdown coming if app marketplaces really take off.
• Facebook Wins Relatively Few Friends in Japan - This surprises me.
• Facebook will NOT end on March 15th. Hoax news story spreads like wildfire - I hadn't heard this from anyone--usually a few technically unsophistacted friends will ask about things like this.
• The New New Telco - "All just in time for a generation who cannot remember a time before the web, a time before the mobile phone. All just in time for a generation for whom 'rent' means more than 'buy', for whom 'share' means more than 'own'."
• Price and adoption timeline of gadgets - Just look at faxes and land line phones (corded or not)
• HTML to lose the version number - Now we'll be saying "new, new HTML" instead
• What did Google do? - Schmidt's geeky sense of humor was not grokked by media. (When he set off a tempest in the news teapot saying we should all be able to change our names at age 21 and start over with youthful indiscretions left behind us, he was joking, folks. Really, he was.) Page is even less show-bizzy.
• 24 Gigabytes of Memory Ought to be Enough for Anybody - That's a lot of memory...until its not.
• Utah Teen Creates Most Popular Free iPhone App - This is Steve Nay's (my CS462 TA's) little brother Robert. My kids love the game.
• The Quora For X - This kind of shorthand is what a lot of people look for in describing their ideas. Describing the "next thing" is hard.
• Death in the Corn: Part II of III - Amazing descriptions of battles from the inside out. I love reading Michael Yon.
• Bureaucrats Keep Us Safe? - "The homeless people fed by the Herrings are not made safer by the ruling, nor are they suddenly less susceptible to food-borne illness. But it will be more difficult for them to find a meal, and when they do, most likely the food will be inferior to what they had. All in the name of safety."
• Google search and search engine spam - The first salvo in the "Google vs Demand Media" wars?

8:01 AM | Comments () | Recommend This | Print This

January 19, 2011

Contextually Correlating Events: The Power of Emergent Behavior in Loosely Coupled Systems

One of the real powers of event-based programming is the ability to contextually correlate events. That's a pretty dense phrase. I'm writing this blog post to unpack it a little in my mind. Hopefully it will do you some good as well.

Single events, like web:pageview, mail:received, or phone:inbound_call can be interesting. We've made money on Kynetx apps that rely exclusively on these kinds of primitive events.

In spite of that, we can create even more interesting applications when we start to correlate one event with another. This is analogous to building Web applications. While there are plenty of Web applications that don't require that a Web site maintain any kind of state or context between service calls, most interesting Web applications do. In the case of the Web site, we correlate requests to the server in order to group those requests and relate them to each other--think "shopping cart" for an example.

When we speak of correlating things, we're talking about finding the relationship between them. When we correlate data in statistics, we're looking to see how strongly two or more data values (like price and volume sold) are related and even the nature of the relationship.

For events, we're looking for events in a stream that have a meaningful relationship. In KRL, events can be correlated in at least three ways:

1. Events can be correlated by entity -- all of the events for a particular person or thing can be related to each other.
2. Events can be correlated explicitly through event expressions -- KRL allows developers to create complex event expressions, or eventexs, that declare specific relationships between their constituent events.
3. Events can be correlated implicitly through shared data -- When a user installs multiple rulesets that have data in common (say a TODO list, a Google calendar, or an email acccount), the events that those rulesets respond to become implicitly linked through the shared data.

Entity Correlation

The first kind of correlation is the most obvious and matches the Web application example I used above. Correlating events by entity requires that we identify, at least pseudonymously, on whose behalf the events are being raised (since entities rarely raise events for themselves, but use an endpoint to do that). One of the primary responsibilities of an endpoint is managing the identity context.

Correlating events for an entity enables personalization of the interaction and provides a thread upon which otherwise disconnected events can be strung. But there might be thousands of events raised in day for a given entity. Consequently, we usually look for stronger correlation than mere entity-correlation.

Explicit Eventex Correlation

An eventex describes an explicit relationship between two or more events. Whether those events are correlated through an entity, an eventex established further conditions that are required for a rule to consider those events correlated. For example, consider the following eventex

select when web pageview "support/main.html"
     before mail received subject "[help]"

The intent of this eventex is to select the rule if the user has visited the main support page before sending an email that has [help] in the subject. These events, to be meaningful should be correlated by entity. After all, it's hard to think of why you'd want to know when one person emails support after someone else has visited the Web page. Consequently, this eventex further correlates the web:pageview event with the mail:received event temporally, stating that one must come before the other.

As another example, consider the following eventex¹:

select when web pageview "/archives/(\w+).html" setting(pagename)
     before web pageview "/archives/#{pagename}.html"

This eventex not only correlates the two pageviews by entity (default behavior) and temporally through the eventex operator before, but it also correlates the two page views by requiring that they be to the same page. The eventex doesn't care which page is visited, only that it's visited twice.

Contextual Correlation

Implicit event correlation occurs when events are related by context. In many ways, implicit event correlation is the most powerful because it has the power to allow serendipitous interactions. Serendipitous interactions provide for uses that designers didn't anticipate--perhaps couldn't anticipate. Serendipitous interaction is enabled by the inherent loose coupling of event-driven systems and their resilience in the face of error.

Context is information that is not explicitly bound to a specific process but is relevant to multiple processes through implicit relationships. Knowing, for example, the time of day, the weather, the prime rate, commodity prices, or that it's Super Bowl Sunday provides information that can be used to correlate events. These correlations will be made in the logic of the rules that respond to the events.

One of the significant benefits of evented systems is that they exhibit extreme loose coupling. To understand this consider the situation where you want your DVD player to pause when you get a phone call. Assuming each has an API, programming your phone to send a "pause" command to the DVD is easy enough to do. Now suppose you decide you want to also raise the room lights when the phone call comes in; you have to program the phone to also send a "raise lights" command to the room lights. Each new interaction requires an explicit command from one system to another. Such point-to-point connections make changes difficult. Change out the DVD player and every device sending it commands has to be reprogrammed.

Now, imagine instead that the phone merely raises a phone:inbound_call event. The DVD and lights can both listen for such events and do the right thing. Add something else to the mix and it can easily listen for the phone:inbound_call event and do whatever is right for it. Nothing else needs to be reprogrammed or even told its there as long as it can listen for events of interest. Each actor in the event-driven system interprets the event according to its own context and purpose. Loose coupling leads to better resilience in the face of errors and configuration mistakes.

In an event-driven architecture, the responsibility for flow control shifts from the event source or sender to the listener. Event-driven architectures distribute and delegate flow control to event listeners. This is a new way to create programmed systems and represents a substantial shift away from the traditional way that developers have scripted systems. In fact, the word "scripted" is the wrong way to think about event systems. Any given ruleset is scripted to a particular set of use cases, anticipated by the developer. But the interaction of rulesets in an event network isn't scripted; rather it emerges from the individual behavior of the many actors.

Rulesets don't execute in a vacuum. Rather, rulesets operate in an environment with other rulesets, endpoints, APIs, and datasources. These actors consume and propagate events according to their own programmed functionality. Thus, an event can become entangled--correlated--with other events through the interaction of the various actors in the environment. Changing the environment by replacing one component or adding another has an impact on the overall behavior and thus the correlation of events.

At first this may seem like a recipe for disaster, but some of that trepidation is born of our experience with tightly coupled systems. Creating event-driven systems that respond in a certain way is a talent more akin to throwing a really good party, than that of directing a play. You have less control over what happens at the party, but you also have less chance of things going completely off the rails and crashing or falling flat. An event-driven system where something isn't quite right is more likely to do most things well, some things OK, and a few things poorly. A request-response system where something isn't quite right is more likely to just not work at all. Event-driven systems are more resilient. Their failures are more linear--that is, small mistakes are likely to result in small, not large, failures.

An Example

Consider the event-driven system shown in the following diagram. The system comprises two rulesets, one for managing a TODO list and one for watching emails, and the TODO list itself, a cloud-based service with an API as well as the ability to raise events.

The TODO list management ruleset is fairly straightforward, offering ways to add items to the TODO list and cross them off or reschedule them. Most of the events it processes are related to a Web interface that the user manipulates to manage the list (not shown).

The TODO list system isn't a Web application, just a Web service with a management API and the ability to raise events when certain things happen in the TODO list (todo:added_item, todo:deleted_item, etc.).

The email assistant ruleset listens for mail:received events. When the message contains a TODO item² it raises an event todo:new_item that contains the TODO item and other relevant information (like the message ID) whenever it sees one.

The TODO list management ruleset sees the todo:new_item event and adds it to the TODO list via the API. As that is complete, the TODO list service raises the todo:added_item event. The email assistant ruleset has been programmed to look for those events and responds by creating a draft email responding to the original email. The user is also notified (using yet another event) that the draft has been created and needs approval. Once approved, the draft will be automatically sent once the email assistant sees the todo:deleted_item event for that TODO item.

Now, imagine that the user adds a TODO item "Send an email to Bob Jones about the company party." Of course, the email assistant can listen for the todo:new_item event and automatically create a draft to Bob with the right subject and have it waiting for the user in the drafts folder when she's ready to work on this task. The email assistant can also know, via the event's parameters, which TODO item is associated with the task and, when the email is sent, send a todo:done_item event. The TODO list management ruleset sees the event and automatically marks the TODO item as "done" via the API.

In this example, the TODO list management ruleset and TODO list service function perfectly well with or without the email assistant ruleset. In fact, they never know that the email assistant exists or that it's listening to events from the TODO list. If it fails, the TODO list system keeps working.

On the other hand, the email assistant knows that there may be a TODO list available and uses events to listen for it's workings and to interact with it. The email assistant doesn't know about the TODO list service, or it's API, only the events. The email assistant won't be as useful without a TODO list ruleset installed, but it will keep working because any TODO events it raises will simply be ignored and any events it's listening for will never be raised.

When both rulesets are installed, events that are otherwise only entity-correlated, such as inbound_email and todo:new_item, are implicitly correlated by the interaction of the two rulesets and the TODO service. The correlation occurs because of the particular rulesets that the user has installed and how they interact.

Conclusion

The use of events allows us to create systems that are more loosely coupled and more robust than with the request-response architectures we're used to. While explicit event correlation by entity or event expressions can create interesting applications, the implicit correlations that emerge from the interaction of the actors in the event-driven system are the most interesting and compelling.

Building event-driven systems is a new skill for most developers because they cannot script the emergent behavior. The most interesting behaviors will emerge from interactions between actors that the develop can't necessarily anticipate. One of the most important lessons of the World Wide Web is that there is real power and value in the emergent behavior of loosely coupled actors. Contextually correlated events bring that power to applications.

---

¹Variable correlation in eventexs is not supported in KRL yet, but will be part of a future release.

²The email parsing ruleset is using a microformat, RDFa, or some other semantic markup technology to recognize TODO items in emails.

4:42 PM | Comments () | Recommend This | Print This

VARs, Clouds, and Silver Linings

Hunter Richards has a nice article on the challenges facing value-added resellers with the ongoing move to the cloud. He makes some excellent points about what parts of the VAR business are going away and which are expanding, giving five moves VARs can make to gain a foothold in the new cloud computing market:

1. Specialize
2. Develop competancy in PaaS poducts
3. Make the cloud's efficiency work for you
4. Offer technology-enabled services
5. Promote the cloud to your existing customer base

One of the things I like about Hunter's thinking is that he's not focusing, as many do, on the IaaS plays. There's little money there after the Amazons and Rackspaces sop up everything they can with their huge economies of scale. He also recognizes that SaaS plays are "consumerizing" enterprise software purchases. There's not going to be very many ways to insert yourself between the enterprise and a consumer-style SaaS product.

Hunter points to the "mama-bear" strategy of landing right in between those two and concentrating on PaaS offerings. This will challenge VARs and ask them to change because there will be more programming on platforms that others provide. Becoming experts on those platforms and solving problems for customers using them will be the play VARs use to move past this transition.

We've seen this with Kynetx. We have several partners who are using Kynetx' PaaS offering to create rich, compelling applications for their customers. I think it's a viable model.

9:46 AM | Comments () | Recommend This | Print This

January 15, 2011

KRL App A Day

Mike Grace, probably the most experienced KRL programmer and without a doubt the most prolific, has been doing a Kynetx App a Day and publishing them on a special blog. When he started we teased hik about doing the Twelve Days of Kynetx, but he's up to 33 now.

Most of these are small useful little apps designed to teach a particular feature of KRL or technique for programming in KRL. Here's the list so far:

• Day 1--Basics and Notify
• Day 2--Dual Execution Environment part 1
• Day 3--Web events and debugging
• Day 4--Explicit Events
• Day 5--MD5 and SHA1 hashing
• Day 6--Conditional Action Blocks and Else Postludes
• Day 7--Looping Using Foreach
• Day 8--Raise Web Events From JavaScript
• Day 9--Exclusive Selection Expressions
• Day 10--Capturing Selection Expressions
• Day 11--Global Block
• Day 12--Persistant Variables
• Day 13--Use External Resource Files
• Day 14--Posting Data to Google Spreadsheet
• Day 15--Ternary Operators or Conditional Expressions
• Day 16--Delay, Redirect, and Replace inner
• Day 17--Detecting Recent Run of Kynetx App Within Timeframe
• Day 18--Actions Before and After Foreach Looping
• Day 19--Regular Expression Matching In KRL
• Day 20--Datasets and Datasources--Remembering Which To Use
• Day 21--Modifying Facebook Stream With Kynetx
• Day 22--Already In Friend List--Working With Arrays
• Day 23--Decode Function--Convert JSON To Objects
• Day 24--Selection Expression Filtering On Variable Value
• Day 25--Merging Two Hashes
• Day 26--Modify Other Browser Extensions Behavior
• Day 27--Building Hashes Using Beestings
• Day 28--Updating User's List When User Joins App
• Day 29--URL Encode Strings
• Day 30--Detecting Empty Pick
• Day 31--Load Stuff Inline In Twitter Stream
• Day 32--Old School Twitter Retweet
• Day 33--First Time App Is Run For The Day

You could do a lot worse if you're trying to learn KRL than to work your way through these and understand what Mike is doing. Most of these simple examples come from real apps he's written.

12:06 PM | Comments () | Recommend This | Print This

January 14, 2011

Alerts and Peace of Mind

I've been on blood presure medication for years--ever since the iMall days. Recently, with the holidays and all, I'd let my prescription get too close to needing refilling. On top of that, I was due to get the prescription renewed from my doctor. And on top of that, I was headed out of towm. I hurried to the doctor Wednesday afternoon but he wasn't there to just take care of it. They said they'd leave him a note to call in the prescription the next day. Uh huh. I anticipated a day of missed calls and the hassle of coordinating it all so I could pick up the prescription on Saturday.

Thursday morning, I got a text from Wal-Mart telling me my prescription was ready to be picked up. Wow! I'd forgotten I'd signed up for their alerts.

My feeling of being hassled trying to get my prescription refilled in time contrasted sharply with the peace of mind I felt upon getting that text. They were night and day. That text let me know, in as long as it took to read it, that everything was fine and I could relax.

This is a small example, but it illustrates with great clarity the power of event-based interactions over a request-response model. Constantly checking to see if something has happened or has changed is a pain. A simple alert system fixes all of that. Request-response systems aren't going away--they're too valuable, but evented models fit so much more cleanly with how our lives really work.

9:52 AM | Comments () | Recommend This | Print This

January 12, 2011

Agent or Endpoint?

Before the New Year, I put up a poll asking what the word "endpoint" meant to people. Overall there were 68 responses. The following table summarizes them:

Question	Answers	Percentage
Other answer...	18	26.47%
an entry point for an online service	13	19.12%
any node on the Internet	13	19.12%
a node on a communication bus	11	16.18%
a URL	8	11.76%
a participant in an event network	5	7.35%

While other is the largest category, some of the answers, could be grouped broadly with ones I gave to slightly shift the answers:

Question	Answers	Percentage
Other answer...	15	22.05%
an entry point for an online service	13	19.12%
any node on the Internet	13	19.12%
a node on a communication bus	13	19.12%
a URL	9	13.23%
a participant in an event network	5	7.35%

The rest of the "others" chose multiple of the preceding choices, gave something independent of all of them, or were non-responsive. So, the word "endpoint" doesn't seem to have a clear consensus on meaning. For my purposes, that might be good since it would seem to indicate that the meaning is fairly maleable.

I want it to mean, in the context of Kynetx, "a participant in an event network." That response got few votes because it's not a common usage and I asked people associated with Kynetx like our employees to not vote for fear of skewing the results.

So, I'd like to ask for your help one more time, please read the following two descriptions. They are identical except that in one I use "endpoint" and in the other I use "agent." Then take the poll to give me your feedback.

As we've discussed, to be useful, an event has to be observed and communicated. In Kynetx, events are generated by endpoints. An endpoint that generates an event may be observing the some activity directly and reporting salient state changes or it might just be reporting or transforming event data from another source.

Endpoints are responsible for

1. maintaining state to link separate interactions with the event processor together in meaningful ways to create context,
2. raising relevant events to the event processor, and
3. responding to directives from the event processor.

Not every endpoint will do all three. Some endpoints will not be entity specific and thus won't play a role in context. Some endpoints may just raise events. Others may only take directives. All endpoints will at least raise an event or respond to directives.

Here's the second:

As we've discussed, to be useful, an event has to be observed and communicated. In Kynetx, events are generated by agents. An agent that generates an event may be observing the some activity directly and reporting salient state changes or it might just be reporting or transforming event data from another source.

Agents are responsible for

1. maintaining state to link separate interactions with the event processor together in meaningful ways to create context,
2. raising relevant events to the event processor, and
3. responding to directives from the event processor.

Not every agent will do all three. Some agents will not be entity specific and thus won't play a role in context. Some agents may just raise events. Others may only take directives. All agents will at least raise an event or respond to directives.

Please take this poll to say which you like best (again, Kynetx employees, please abstain):

10:29 AM | Comments () | Recommend This | Print This

January 7, 2011

Engineers Do for $1.00 What Any Fool Can Do for $2.00

Guest post by Wade Billings, VP Operations, Kynetx

When we started Kynetx, we had a very limited IT budget due to fact that the company was bootstrapped with the proceeds of selling the founders Piper Turbo Arrow airplane. From the beginning, Kynetx was built upon the conviction to be initially self funded and very strong beliefs regarding system performance and stability, so from day one, we lived inside a paradox: how do you maintain a high level of reliability, accessibility and stability when you have little in the way of resources or capital to expend.

Had we started Kynetx a scant five years ago, this paradox may well have been an unassailable barrier to our success as we would not be able to afford all of the infrastructure required to build out a scalable and stable platform. Just take a moment to think about all the equipment required to build highly scalable, available and performant systems these days. There are routers, switches, firewalls, IPS and IDS systems all of which are in a redundant mesh. Then there are the compute farm servers, network operating systems and backend storage units, all of which need to be redundant and configured to fail over in the event of a failure. All of these elements must be able to exist along two distinct dimensions represented by the X and the Y axes. The X is the horizontal axis which signifies the number of "boxes" you must scale out in order to meet your load projections. The Y axis, or the vertical axis, signifies the ability to scale up a single machine. Each dimension has unique considerations and challenges, and as a seasoned capacity planner, I take all of them into account with forming capacity plans.

To provide a point of comparison to Kynetx, when I worked for Lowermybills.com, I had the opportunity to re-architect the production platform from scratch. It was a much different experience as the capital budget I had to work with was just over 2 million dollars. When the project was complete, my team had built out over ten racks of gear and two hundred square feet of data center cage space to meet the capacity and scaling projections for the next three years. Not only did this build out provide ample headroom, resiliency and availability, it still serves as a production site for the company with very little in the way of upgrades, even seven years later.

Given our reality at Kynetx, we had to think and plan differently but end up at the same place which was a platform which would provide 99.9% (less than 8 hours of downtime a year) uptime and would be performant for our users who are spread out across the entire country. Thankfully today there exist options which allowed us to build out infrastructure without spending large amounts of capital or man hours. I am speaking specifically of the cloud computing services that are springing up all over the Internet.

Our platform, called the Kynetx Network Services or KNS platform, makes use of multiple open source virtualization technologies as well as the the Amazon EC2 cloud platform. We went virtual in order to ensure we are able to scale along both the X and Y axes with little delay or investment. Being a cloud computing based company allows us to meet demand on an "as needed" basis without raiding the company coffers. It also allows me to provide very detailed usage metrics and a quick return on investment (ROI), which makes our CFO and CTO very happy.

As a practical example of why we choose a virtual over physical infrastructure, I offer the following real life scenario. Recently we had a Kynetx platform based application, Hover.me, "go viral" after it was featured in TechCrunch. This produced an unexpected increase in demand on our platform in a very short amount of time. Had we not made the planning and development investment in an virtual and elastic platform, this development would have meant long hours spent scrambling to keep up with the demand. Instead, it turned out to be a "non event" as our platform auto scaled to meet the demand just as it was designed to do.

Had the author of the application been required to build out his own physical infrastructure, he would not have been able to provide the experience he wanted his users to have at a reasonable cost. In a word, he would have been screwed. In comparison, the overall KNS cloud based platform usage was just under 50% of available resources, which provided plenty of headroom to handle spikes. Meanwhile, the Kynetx staff was able to enjoy the holidays with family, safe in the knowledge that the platform was performing as designed.

At the end of the day, in a successful startup you have to make the best of what you have been given to work with. As the title states, engineers do for $1.00 what any fool can do for $2.00. This is not just a pithy idiom, but a guiding principal for how we operate at Kynetx.

3:12 PM | Comments () | Recommend This | Print This

Reading for January 7, 2011

• Beekeeper Who Leaked EPA Documents: "I Don't Think We Can Survive This Winter" - I sure hope my bees survive!
• The U.S.S. Prius - from Friedman in the NY Times
• In Pursuit of the Perfect Brainstorm - Corporate America wants help coming up with fresh ideas. Can a new breed of consultant teach companies how to think?
• The Unwelcome Return of Platform Dependencies - An app that depends on a dozen APIs has a dozen new ways to fail
• Live Web, Real Time . . . Call It What You Will, It's Gonna Take A While To Get It - An article by Narry Hodder from 2009
• What lies ahead: Data
• 10 Predictions for Web Development in 2011 - Check out 3 and 10
• Sideloading MOBI files to Kindle for iPad - This was a good find. I found that covers just seem to work, I didn't need to futz around at all.
• Fish Where the Developers Are from Steve Rubel
• Mac App Store Cracked Open for Piracy - Not released yet, but a new app will make "stealing" free Mac App store easy
• Why the Mac App Store Sucks - Can you say "walled garden?"
• Uncle Sam has $30M to bypass Chinese, Iranian 'Net filters - The State department wants you to help the crack firewalls
• The Gollum Effect - There's VRM ideas in here somewhere...
• Consumption, Innovation, and the Source of Wealth
• Blogging Forefather Seeks to Re-Invent Blogging, Again - "Unless we've all been Farmville-zombified too deeply to care anymore, My.ReallySimple is something we all ought to be cheering for."
• From Dave Winer...Upcoming: The minimal blogging tool - This looks like something I'd use.

2:35 PM | Comments () | Recommend This | Print This

January 3, 2011

Architectures and Patterns: Hitting the Sweet Spot

The Web, indeed the Internet, has a rich tradition of being made from "small pieces, loosely joined" to use David Weinberger's excellent phrase. The beauty of the Web is that it's all built from a handful of relatively simple standards:

• HTML, a standard for marking up documents with formatting instructions and creating rudimentary user interfaces called "forms,"
• HTTP, a simple request-response protocol for moving resource representations (which might be documents formatted in HTML) from place to place, and
• URIs, a standard for how resources will be universally reference on the Web

Using those simple pieces, the Web has evolved to rich collection of sites and services we now enjoy. Or, at least, that's the romantic view.

In fact, the modern Web is built on a large collection of conventions, patterns, architectures, libraries, and systems that have been built on top of the three pillars of the Web. In theory, you could build Facebook with nothing more than a CGI script that reads raw input on STDIN from the server, consults a database, and formats responses as a series of print statements on STDOUT--like we did in 1993--but that's not how it's done. The vast panoply of Web sites we use everyday exists because programmers make use of architectures and libraries built on top of the basics of the Web.

There's always some tension in this. For example, consider SOAP and the Web Services edifice that was constructed around it. A battle eventually ensued that pitted that elaborate structure of standards, protocols, and practices against a much simpler idea called "REST" that has, at this date, largely replaced it. REST accomplished many of the goals people had for SOAP in a way that was simpler. It didn't hurt that REST could invoke religious arguments that it was "truer" to the spirit of the Web. There's power in that, to be sure.

Now consider Rails, a programming framework built on top of Ruby. Rails asked much of its early adopters. First, it was built on a new and unfamiliar language called Ruby. Second, Rails introduced new and unfamiliar nomenclature to describe its core concepts. Third, it placed restrictions and boundaries on how programmers worked, requiring them to work in certain ways. And yet, developers who made the effort to overcome these hurdles were able achieve desirable results easier than with other alternatives.

Rails provided abstractions--conceptual models--that changed how programmers thought about the task of building interactive Web sites. Rails included conventions for certain, oft-repeated tasks to let programmers avoid choices that had little bearing on the final outcome of their project. Rails provided libraries and tools that made common tasks, like doing object to relational mapping, easy. But most importantly, Rails wrapped all of these into a system, with its own language and culture, so that they all worked together without the impedance mismatches developers face in "rolling their own."

We can ask "why did Rails succeed where Web Services failed?" While there might be a lot of technical arguments we could make, I think there are a few important ones:

1. Web Services was too heavy, requiring a huge investment in congnition and effort for even small jobs. Web Services tried to solve every problem with yet-another-standard. in the end, Web Services feels like lots of standards without a system to tie them together. Another way to put it: Web Services is a large cognitive model without a supporting linguistic framework. XML just doesn't do that--it's too flexible and too general.
2. More important, Web Services got dragged into the enterprise because that's where vendors pushing Web Services like IBM, Sun, and Microsoft could sell software using what I call the "two guys in suits" model. The enterprise is never as innovative as the non-enterprise, consumer-oriented developer space. Without the innovation, Web Services never got its groove on. It just wasn't cool.

I need to make a big disclaimer here: I'm not saying Web Services failed, just that it failed to get developer uptake in sufficient numbers to make it a player in the future Web. Indeed, Web Services has informed and left bread crumbs for a number of important discussions. For example, REST needs an API description language so that minor annoyances like generating libraries for various languages can be automated. Web Services has a lot of experience with that sort of thing that will inform thinking in this important area. Further, the problems Web Services was trying to solve are real. I think os Web Services as just the initial prototype of the services we'll have to build out to solve them.

I think a lot about what it takes to get developers over the hurdle of learning a new paradigm because the success or failure of Kynetx depends on that issue. I'm constantly balancing the height of that hurdle with expressiveness and cohesiveness. Over the course of the last several years, we've constantly adjusted the kinds of problems we're working on to find the ones where our ideas can have the bigget impact.

I believe strongly in the power of a programming language to shape the thoughts of developers along a particular path and in ways that make thinking about certain kinds of problems easier or harder. For example, Rails is domain specific language embedded in Ruby. The language of Rails, however, pushes developers to think about Web development problems in specific ways and, in so doing, shape and guide that thinking. Its success is dependent on being properly matched to the problem domain it's designed for a net effort savings by the developer.

I also believe that systems are more likely to be successful in that effort than mere patterns. While patterns are useful and form what we might think of as the first bastion of abstraction in the programmer's arsenal, they rarely have evengelical power sufficient to unite large crowds of developers around an idea. Systems, at the other end of the abstraction spectrum, unite a programing model with libraries and services to provide a platform (in the original sense) upon which community can be built.

With Kynetx, we're trying to introduce a new programming model for developing evented systems that understand entity (i.e. individual) context. That problem warrants a system with a new language to make it easy to solve. The problem is big, audaciuous, and hairy. The gap between a few patterns using the basic building blocks of the Web and the problem we're solving is too big for most developers to tackle and requires too much effort for all but the most profitable ventures. With a supporting system, however, smaller scale projects can be built with a level of effort comensurate to their value.

2:53 PM | Comments () | Recommend This | Print This