Fuse as an Experiment in Internet of Things Architectures

Yellow car

The Internet is decentralized, heterarchical, and interoperable. Unfortunately, today's Internet of Things is none of those. We might better call it the "CompuServe of Things." What architecture should the Internet of Things embrace?

This talk will discuss the problems with the current model for the Internet of Things and present the lessons we've learned from architecting an Internet of Things product that is more in keeping with the fundamental architectural principles of the Internet.

Fuse is a connected-car product built as an experiment with software architectures for the Internet of Things. Fuse is event-based, open-source, hostable, and has an extensible API enabled by a microservice architecture. Moreover, Fuse is designed to keep vehicle data and services private unless explicitly shared by the owner. At the same time, the architecture preserves the ability for the service and associated data to be sold with the car.

At the heart of the Fuse architecture are persistent compute objects (or picos). Picos are lightweight, general-purpose, online objects that have persistent state. Picos are used to represent anything with a unique identity including people, places, things, and even concepts. Picos are decentralized and networked. Fuse knits picos representing the owner, fleet, and the owner's vehicles together. The owner controls the picos, with whom and how they share data, and where the picos are hosted. Picos are individually extensible. Consequently, the set of services a given pico presents is controlled by what services the owner has installed.

Among the lessons learned in building fuse are the value of self-healing services, idempotent actions, asynchrony, and accretive functionality. These lessons have allowed us to build a connected-car service that is modular, loosely-coupled, and has unprecedented support for user control of data.

The Quantified Car

The quantified-self movement is all about measuring your personal activities and gaining insight from the data. Many of us dabble in it with a Fitbit or a Withing scale. Others measure everything and use the data to change their life. Data changes behavior. I walk more because I have a Fitbit.

One of the things I've noticed is that Fuse gives me data about a part of my life where I often make incorrect assumptions that cost me time and money: my driving. Here's a few things I've realized:

  • Travel costs both more and less than I'd have thought. I spend $4/day getting to and from BYU. That's less than I'd have thought. But, $80/month is still a significant spend. I spent $16 driving from work to REI in Sandy and back. That's more than I'd have thought and could change how I think about trips.
  • My doctor is in Alpine. My gut told me going from BYU to I-15 and then up to SR-92 and out to Alpine might not be the shortest rouute, but would be the fastest. I did an experiment and drove to the doctor's office for an appointment by going down State Street and then up Canyon Road from Plesant grove. Lots of lights and stop and go, but more direct. The trip took 30 minutes and was 19 miles. The trip back via the freeway was about 26 miles and took 40 minutes. So, neither faster nor shorter. IMG_7151
  • I was thinking about a new car with better mileage. How much will I really save? Fuse is able to show me my fuel costs for each vehicle and help me see what I really spend.

Having real data often reveals behaviors that seem logical but are, in fact, not optimal. Watching data from Fuse has changed how I drive. I think more about trips I take and the money I'm spending on gas. I'm looking forward to getting maintenance data in Fuse as well because I'm sure that will be eye opening.

Why Do CS Majors Study Calculus

NooNoo studying calculus

John Dougall asked me recently if I thought Calculus was really necessary for a CS degree. I do and here's why.

Calculus teaches and encourages abstract thinking, something that is necessary to be a good programmer. Those who are discouraged by Calculus may find that they can get along just fine in college programming exercises, but they won't do well in the modeling that is necessary for real-life software architecture—not because they're missing Calculus, but because they struggle with abstract thinking.

Engineers typically use calculus to optimize some part of a bridge, circuit, or something else. They do this by creating a model of the thing they're building. Computer scientists rarely optimize models in this way. Rather they create their own abstractions that model some real-world system. Thinking abstractly is vital to good system design. Seeing how others use mathematics to abstractly model things (i.e. learning calculus) is good preparation for that challenge.

In general, one of the chief reasons for going to college to study CS isn't to learn to program. You can learn that at Code Academy or whatever. You go to college to change the way you think and that's going to happen both inside and outside the courses in your major.

Fuse App Launches

Fuse Logo

I have two important announcements regarding Fuse. One is likely to be much more exciting than the other, but both are important, so please read the entire post.

First, we're excited to announce that apps are available (finally!) for Fuse for both iOS and Android. Thanks to our Kickstarter backers for helping to get this done and especially to Alex Olson for making it all happen. We're excited to have you download them and try them out.

A few notes:

  • The app provides fleet views for trips and fuel as well as details about trips, a feature for fuel management, and "Find My Car."
  • Log into the app with the Fuse account you created at Joinfuse.com in the Fuse Management Console
  • The app doesn't have functionality for managing your vehicles, etc. That's what the Fuse Management Console is for. You'll still need to use the Fuse Management Console.
  • We are still working on and plan to release maintenance features. The API for maintenance is complete, but it hasn't been surfaced in the app yet.
  • There is still one stability bug remaining that we're working on, but we decided it was usable as is (I've been using it for several months) and continue to work on the stability. The bug involves how the menu works and results in the app buttons being non-responsive. The workaround at this point is to kill the app.
  • The Android version is likely less polished than the iOS version because neither Alex nor I are Android users, so we're at a disadvantage there.

In addition to fixing the obvious bugs and getting maintenance done, we're working to open source the app code. It's in Cordova. The primary roadblock there is ensuring that we haven't left any keys lying around in the code. In the meantime, if you're interested in forking the project and having a look at the code (or better yet, submitting a pull request), send me your Github ID and I'm happy to add you to the project.

The second part of this announcement is that we have launched a forum for Fuse at forum.joinfuse.com. This will be our last post on Kickstarter. From now on we will treat the Fuse Forum as the primary channel for communicating with you about Fuse. Please look around and feel free to ask questions there. We're excited to have a real forum for interacting with everyone about Fuse. In addition to the forum, you'll often find interesting articles about the technical implementation of Fuse and its philosophical underpinnings here on my blog.

Building Docker on Centos

The Dockers Monument In Limerick City Ref-441

I couldn't find a single place with good instructions on getting the latest Docker installed on Centos. Sure I could use Ubuntu, but where's the fun in that. I found lots of places with partial solutions, but nothing that just walked me through it. Consequently, I am writing down what I did. Maybe this will help someone else.

The first step is to get some version of docker installed. Fortunately, docker is available via yum on late-model Centos installations, but it's old. There are slight differences in Centos 6.X and Centos 7. On Centos 7, just install docker via yum:

sudo yum -y update
sudo yum -y install docker
sudo service docker start

On 6.5, it's a little harder cause there's another package called docker in the standard RPMs and dockers in an EPEL.

sudo yum install epel-release
sudo yum -y update
sudo yum -y erase docker          # removes the conflicting package
sudo yum install docker-io
sudo service docker start

After you've got a version of docker installed, you can use these instructions on Setting Up a Dev Environment. Essentially, it uses the existing copy of docker to build an updated version of docker (in a container, of course). The build process takes a while, but I didn't have any problems. Note that the build process puts the binaries in a shared volume on the disk. You can copy them out or just link to them.

Fuse with Two Owners

If you've followed my posts on the Fuse Architecture, you'll likely recognize this picture:

fuse microservice overall

The architecture uses a pico to represent each vehicle, as well as one for the owner and one for the fleet. This organization provides essential communication pathways and establishes a structural representation of ownership and control.

I've also shown architectures where there was more than one owner, not to mention having relationships with everything from manufacturers, service vendors, and potholes.

With all that promise, however, I built Fuse with support for just one owner. That was a reasonable simplifying assumption when I was just getting started, but last week, with some prodding from Jim Pasquale, I decided I'd explore what it would take to support multiple owners, like so:

Fuse with multiple owners


The first step was to create the services inside the owner and fleet picos necessary to support introductions and subscriptions. Each pico subscription is unique. Each owner has a separate channel to the fleet and those channels have unique names.

I determined that the best way to support this was to re-use existing inter-pico subscription functionality as much as possible since that have been proven to work reliably over several years. The introduction process goes something like this:

  1. The prospective owner asks the current owner for an introduction to the fleet
  2. The current owner asks the fleet for a new channel and name to give to the prospective owner
  3. The prospective owner uses the new channel and name to subscribe to the fleet

The diagram below shows how this works. Green interactions depend on existing CloudOS services. Blue arrows are new interactions I wrote specifically to support introductions. I didn't implement the black interactions for this experiment, but they'd be needed to roll this out for Fuse generally.

introduction and subscription in picos

The fleet will only give up introductions to a pico with which it has an existing FleetOwner relationship. The fleet will only honor the subscription request if it is one that it has created and has the right name and channel identifier (shared key).

The overall experience for Fuse owners could mirror something like the Forever experiment we did several years ago. We'd likely build it into the Fuse Management Console (FMC).

The introduction pattern shows up in multiple place, albeit with different channel relationships (beside FleetOwner). For example, a similar process could be used in transferring ownership of a vehicle between two fleets. The primary difference would be that the fleet would drop the subscription to the old fleet once the transfer was complete.

Using Backchannels

I knew that giving a fleet two owners would cause some problems. After all, none of my testing—not even my mindset while programming—took two owners into account. Initial testing showed that both owners of the fleet could log into FMC and manipulate the fleet. So far so good.

The first place a problem showed up was weekly reports. Fuse sends each owner a weekly report. This can be disabled in the preferences. When enabled, the owner's pico has a scheduled event that fires once a week to send the report. The owner doesn't know how to generate a report, so it tells the fleet to generate a report. The fleet pico generates the report asynchronously.

The fleet, however, doesn't know how to email the owner. So, the fleet pico routes the completed report back to the owner for mailing. That's where things went wrong. Here the rule that routes events back to the owner when needed:n

rule route_to_owner {
  select when fuse email_for_owner ...
  pre {
    owner = CloudOS:subscriptionList(common:namespace(),"FleetOwner")
  event:send({"cid": owner}, "fuse", event:type())
     with attrs = event:attrs();

The problem is the head() operator. It simply returns the first member of the subscription list.

You can imagine what happened. When reports got sent out, one owner (which ever one happened to be first in the subscription list) received two reports and the other owner received none. This was easily remedied by having the rule find the owner channel based on who asked for the report in the first place (determined by the incoming channel as returned bymeta:eci()):

rule route_to_owner {
  select when fuse email_for_owner ...
  pre {
    owner_subs = CloudOS:subscriptionList(common:namespace(),"FleetOwner");
    matching_owner = owner_subs
                                 sub{"backChannel"} eq meta:eci() 
    owner = matching_owner.head().pick("$.eventChannel");
  event:send({"cid": owner}, "fuse", event:type())
    with attrs = event:attrs();

This solves the problem by using the backchannel to find who sent the initial event. Other times, it might be best to lookup the owner by pico name or other data. This is a good pattern for event routing in general: don't depend on stored or specific channels for communicating with other picos, look them up instead. In general this is a good idea since event channel identifiers can change.

I'm sure I'll find a few more problems as we play with this some more, but it was nice to know that the existing subscription process could form a large part of the introduction process and that finding the right owner involved only minor changes to look up, rather than rely on specific, channels. One of the things I like about modeling with picos is that having unique, persistent objects to represent real-world entities and concepts (like owners, fleets, and vehicles) makes handling concepts like multiple owners straightforward.

Self Sovereign Authorities and the Epic Struggle for IoT

I finished reading three things this week that all tied together in my mind and so I wanted to mention them. One is a full-length book, but the other two are short essays.

Jefferson and Hamilton: The Rivalry That Forged a Nation by John Ferling—I'm a fan of early American history and have read a lot about Jefferson (including Malone's six volume biography) and some about Hamilton. What was interesting about this volume was seeing the lives of the two foes play out in parallel and focusing on their interaction. The take away for me was they were both right. A contradiction, but one that is still played out daily in US politics. We need national strength (Hamilton) and democracy (Jefferson) even though those things are diametrically opposed and must be carefully balanced.

Hamilton is clearly the winner in the way history played out. He got his strong federal government and manufacturing-based society. Jefferson's dream for an agrarian society has faded with the 19th century. But his reason for wanting it—fear of the concentration of power and corruption (in the Lessig sense) that industry would bring—was spot on. Which brings me to the next two short essays.

The Epic Struggle of the Internet of Things by Bruce Sterling—This little essay will cost you $2.99 at Amazon, but it's worth it. Sterling lays bear the fallacy of the Internet of Things based on goods sold to us by the powers-that-be (namely the big five: Google, Apple, Facebook, Amazon, and Microsoft) and their lesser counterparts. The behemouths don't compete with each other, they seek to disrupt others in ways that are not connected with any free market that you're familiar with. We're up in arms about the NSA spying on us, but all too willing to sign up for surveillance from peddlers of connected things. Our freedom and independence as human beings is at stake, as I wrote about in The CompuServe of Things. How do we escape and regain our independence (the democracy that Jefferson fought for)? Read on.

Why Self-Sovereignty Matters by John Clippinger—This is Chapter 2 in a collection of essays edited by John H. Clippinger and David Bollier called From Bitcoin to Burning Man and Beyond: The Quest for Identity and Autonomy in a Digital Society (free PDF) John does a nice job of laying out in non-technical terms how we can be the source of our own identity (what is known as sovereign-source identity or self-sovereignity) rather than only being identified within the administrative domains of the Big Five, the Government, and other lesser administrations. This comes off as arcane, but it has a simple premise: why aren't you the source of your own identity? Why are all your identifiers given to you by others? This is the fundamental roadblock to true independence online and one that greatly interests me.

In my blog post on the CompuServe of Things, I wrote:

On the Net today we face a choice between freedom and captivity, independence and dependence.

I really believe that. I don't hate the Big Five. I love their products and use them regularly, but I want to do so as an independent entity with inalienable rights, not as a serf in their digital estates.

Events, Picos, and Microservices

I spoke at Apistrat 2014 today in Chicago on the Fuse architecture and API. The Fuse architecture is unique because it uses picos and event-query APIs to create a connected car platform. I found microservices to be a useful model for thinking about building features in the Fuse architecture. Here are my slides:

A University API

BYU Logo Big

BYU has been in the API and Web Services game for a long time. Kelly Flanagan, BYU's CIO, started promoting the idea to his team almost 10 years ago. The result? BYU has over 900 services in its Web Services registry. Some are small and some are big, but almost everything has a Web service of some kind.

Of course, this is both good news and bad news. Having services for everything is great. But a lot of them are quite tightly coupled to the underlying backend system that gives rise to them. On top of that, the same entity, say a student, will have different identifiers depending on which service you use. A developer writing an application that touches students will have to deal with multiple URL formats, identifiers, data formats, and error messages.

We're aiming to fix that by designing and implementing a University API. The idea is simple: identify the fundamental resources that make up the business of the university and design a single, consistent API around them. A facade layer will sit between the University API and the underlying systems to do the translation and deal with format, identifier, and other issues.

The name "API" reflects an important shift in how we view providing services. When you're providing a service, it's easy to fall into the trap of collecting an ad hoc mish-mash of service endpoints and thinking you're done. The "I" in API is for "interface." When you're providing an interface to the university, not just a collection of services, your mindset shifts. Specifically, in designing the University API we're aiming for something with the following properties:

  • Business-oriented—the API should be understandable to people who understand how a university works without having to understand anything about the underlying implementation. Many more people know how a university works than could ever know about the underlying implementation. An API based on resources familiar to anyone who understands a University makes the API useful even to non-programers.
  • Consistent—a developer should see a consistent pattern in URL formats, identifiers, data formats, and error messages. Consistency allows developers to anticipate how the API will work, even when they're working with a new resource.
  • Completeness—over time the University API ought to be an interface to every thing at the University that works via software (which is to say everything).
  • Obvious—using the API should be obvious to anyone who understands the general principles without needing to rely excessively on documentation.
  • Discoverable—a program should be able to discover allowed state transitions, query parameters, and so on to the extent possible.
  • Long-lived—An API is like a programming language in that it is a notation, not a technology. The goal is to create something that is not only intuitive, but stands the test of time. Designing for long-term use is more difficult than designing for short-term efficiency

The fundamental business of the university doesn't change rapidly. BYU has had students, classes, and instructors for 140 years. Likely, instructors will still be teaching classes to students in 20 years. The API to a university ought to reflect that stability. This doesn't mean it won't change, but ideally the University API will evolve over a period of decades in the same way a language does. Perl 5 is quite different and much more useful than Perl 2, for example, but it's still Perl. This gives the University API an importance that an ad hoc collection of services would be hard pressed to meet.

Building a University API has multiple advantages:

  1. First, and most obviously, making the API consistent and understandable will make it easier for developers to use it in building applications. This includes developers in the Office of Information Technology, BYU's central IT department, as well as developers in other units around campus. Further, there's no reason that students and others shouldn't be able to use the APIs, where authorized (see below), to create new services and GUIs on top of the standard university systems. The University API is the heart of a great university developer program.
  2. Beyond making it easy for developers, a consistent University API eases the pain of changing out underlying systems by introducing a layer of indirection. Once a University API is in place, underlying parts of the system can be changed out and the facade layer adjusted so that the API presented to developers doesn't change, or, more likely, only changes in response to new features.
  3. A third advantage of the University API is that it provides a single place to apply authorization policies. This is a huge advantage because it allows us to apply formal, specified policy parametrically to the API rather than doing it ad hoc. This results in more consistent and accurate data protection.
  4. Finally, a University API serves as a definition for the business of the University that befits the reality that more and more of the university's business is controlled and mediated via software systems. By designing a notation and semantics that matches what people believe the University to be, we document how the University's business is conducted.

How do you get started on such a monumental undertaking? We've created a University API team that is busy discussing, designing, and mocking-up APIs for a small set of interrelated, core resources. We hope to have mock-docs for review in the next few weeks. For now, we're focused on the following set:


Others that will eventually need to be considered include /colleges, /departments, /programs, and so on. There could easily be dozens of top-level resources in a university, but that's much more manageable than 900. And when they're logical and consistent that's especially true. For now we've identified five resources that form the core of the API and touch on activities that the university cares about most.

As you'd expect a GET on any of these resources returns a collection of all the members of that resources.

GET /students

Obviously some of these collections could be very large, so they will usually be filtered and paginated. Take /students for example. By rights, this resource should include not only all the current full-time students, but part-time students, independent study students, and so on—easily tens of thousands of records. Being able to filter this list so that it's the collection of students you want (e.g. all full-time students in the College of Engineering) will be critical.

Performing a GET on a resource with an identifier in the path returns the record for that identifier.

GET /students/:id

Again, the result could be very large. In theory, a student record contains everything the university knows about the student. In his white paper on University APIs, Kin Lane listed 11 types of data that might be in a student resource without even getting to things like transcripts, grades, applications, and so on. There are dozens of sub-resources inside a resource as complicated as /students. In practice, most programmers don't want (and aren't authorized to get) all the data about a particular student. We're attacking this problem by creating useful field sets for the most popular data for any given resource.

We're also dealing with issues such as the following:

  • What are the meta values for the API (e.g. what values are appropriate for a given field) and how should we represent them in the API?
  • How do we handle sub-resources? For example, a class has set of values for prerequisites that is a complex record in it's own right. But prerequisites doesn't deserve to be a top-level resource because it's only meaningful in the context of a course.
  • Many of the identifiers for a resource (like a class) are aggregate identifiers. In the case of a course the identifier is made from the term, department, course number, and section.
  • What is the right boundary between workflow and user interface. For example, when a student drops the class and that has cascading consequences, should the client or server be responsible for ensuring the student understands those consequences?
  • How deep do we go when returning a resource? For example, when we get a class enrollment, do we return links to the student records or the data about the students? If the latter, what does that communicate to developers about what can and can't be updated in the record?

There are new issues that come up all the time. We're still thinking, designing, and planning, so if you have suggestions, we'd love to hear them.

The effort has been fun and we're anxious to make at least part of this design exercise real. Watch for projects that tackle parts of this enterprise over the coming months.

Further reading:

Suggesting Changes to Google Places

Google Maps Icon Buttons

The Fuse app lets drivers record fuel purchases. One of the features it has is populating a pulldown of nearby gas stations so that the driver doesn't have to type all that in. I buy gas at the Costco in Lehi occasionally and I noticed it wasn't in the list while the Costco in Orem was. I got to wondering how the data for Google Places (the API Fuse uses) is collected and updated.

A quick search of the Costco in Lehi and the on in Orem on Google Maps didn't show any difference. There also wasn't an obvious difference when I drilled into the reviews. It was only when I clicked on the "Edit details" link that I noticed that the Costco in Orem had the following categories:

Optician, Discount Store, Gas Station, Department Store, Wholesale Club,...

whereas the one in Lehi only showed this:

Department Store

Fortunately, Google takes user comments on business listings, so I clicked "add category" and added "Gas Station." The change doesn't show up right away since Google reviews these types of changes.

So, the sequence for updating a business categorization:

  1. Find the business on Google Maps
  2. Click the "N reviews" link (or the "Write a review" link)
  3. Click "Edit details" under "Contact Information"
  4. Click the ">" to the left of "Category" to expose the category entry fields
  5. Edit as needed
  6. Submit

All in all, this is not easy to discover without some poking around. I'd tried looking from my phone a few times and never got far enough to make a change. But, once you know the secret, it's not too hard. I'm surprised that Costco doesn't do a better job of categorizing its stores in things like Google Places.