Phil Windley's Technometria

July 2, 2009

Automatically Building, Configuring, and Maintaining Complex Infrastructure

I’ve been heads down for the last few weeks getting a project out the door for a new customer. As I mentioned, this involves creating a virtual appliance. I decided, due to the circumstances of this deployment that the best option was the build an appliance factory that is capable of churning out new virtual machines at will. I’m going to describe how I did that in this post.

There are bascially three steps to creating a new image that runs the Kynetx Network Service (KNS):

Create a new virtual machine
Install packages and Perl libraries, create users, and otherwise configure the machine to run KNS
Deploy the KNS code and test it

I was exporing Kickstart files for automatically installing Fedora and CentOS when someone pointed me at Cobbler. Cobbler is a Linux installation server that is simply amazing. It includes templated kickstart files, DHCP and DNS servers, the ability to manage multiple distros and repositories, and a database for keeping it all straight.

You start by importing distros and images, then define profiles that combine those with kickstart files, and finally create system definitions for each machine refering to profiles. I pnly needed one distro, one repo, and one kickstart, so I ended up with multiple systems hanging off of one profile. Once that’s done, a command called koan (kickstart over a network) is used on the Dom0 machine to create virtual machines as defined by the system definitions cobbler.

I carefully edited the kickstart file to create just the machine I wanted with the right packages installed. At this point, I was building new VMs and taking them down 20-30 times a day as I tested this. That’s the beauty of automation—tacking up a machine is just dirt simple.

I was lucky that I’d already invested considerable effort in Puppet recipes for building the environment that KNS need to run, so the second step was almost done. In fact, with just a few edits, I had Puppet building the new VMs up.

The third step was also one that I’d spent some time on. I have a custom deploy script (in Perl) that deploys KNS code based on server role and takes care of all the little details like setting up the configuration files for the various servers.

Every system is slightly different, but I think there’s a definite distinction between machine setup, system configuration, and code deployment. The first creates a fairly standard environment, the second configures it to a specific purpose, and the third manages the code.

Some thoughts on all of this:

Some have asked “Why not put the code in Puppet (i.e. why use a deployment system)?” My answer is that code deployment is a dynamic process that I want more control of than puppet’s automatic configuration provides. You could probably press Puppet into this, but it didn’t seem to fit for me.
I had to create a simple YAML-based configuration file for KNS to pull everything together. YAML was the right answer for this. I chose to put that configuration file in Puppet, but I think I’ll pull it into the deployment process in the future.
One missing piece is a database that everything can read system configurations from. Cobbler provides a light-weight one that may serve our purposes for a while, but something like iClassify is more flexible. Right now there’s system information in Cobbler, Puppet, and the deploy script. There’s a way to put additional attributes in Cobbler that we could use in other places.
All of this—Cobbler, Puppet, and the deploy script—were installed and running on a virtual machine that we call the factory. That one image, once installed in Xen is capable of creating as many copies of each type of machine we run as needed.
This can all be done on physical boxes too, of course, but I prefer the flexibility of virtual machines—even when only one will be running on the physical hardware. They can be moved, replicated, and managed with a lot more ease that physical hardware. Plus I have the ability to fire up new ones for QA or whatever without buying and installing new physical hardware. When a 8 core, 32 Gb box costs $4K, you can amortize that investment a lot with virtual machines.

Startups need to be lean. Achieving that goal in a compute-intensive business requires automation. Fortunately with tools like Cobbler and Puppet, automating the build-side of your infrastructure is not only possible, but fairly easy. We manage several dozen machines with only a few hours a week of effort. What’s more, adding a new box for load or experimenting is as easy as typing a few commands and waiting 20-30 minutes.

June 25, 2009

Starting a High Tech Business: Selling the Third Deal

I’m starting a new business called Kynetx. As I go through some of the things I do, I’m planning to blog them. The whole series will be here. This is the nineteenth installment. You may find my efforts instructive. Or you may know a better way—if so, please let me know!

I have a theory that the third deal matters more than the first two. Here’s why.

The first time you sell your product—your first deal—is always exciting. But let’s be honest, it could be a fluke. If you beat the bushes long enough you’re likely to find someone who’ll buy almost anything.

The second deal feels good because you at least can convince yourself that the first deal wasn’t an accident.

But the third time you sell your product you have confidence around a few important things:

Proven repeatability - to get to the third deal you’ve proven that you understand what your selling and you’re able to explain it in a way that people connect with.
Turn the crank - at this point you ought to be able to “turn the crank” operationally and deliver. If you’re still doing one-offs by the third deal, you need to ask yourself what will change by the fourth, eighth or 100th deal? You can’t achieve scale without operational excellence.
Know your price - On the first deal you’re always a little unsure of the price you’ve set. Is it too high? Too low? Will you get laughed out of the room? By the third deal, you can go into pricing discussions with confidence. After all, two other customers have paid it—why won’t everyone?

I’ve found that to get any deal you usually have to put your ego aside. The sweetspot is when you’ve found (a) something you’re good at, (b) something you like to do, and (c) something someone will pay for. A deal implies (a) and (c). If you have to cave at all on (b), then your ego’s likely to get in the way of the deal.

Putting ego aside is not always easy for techies to do. After all, you’ve spent years working on this and generally have dreams and even fantasies about how people will use it. Take a deep breath and realize: someone’s willing to pay money for something you built. That’s a good feeling. Go with it and enjoy the ride.

Annotating Anything

Today I released build 299 of KNS. There are three important updates to KRL in this build.

First, KRL now support literal hashes. Hashes are creating by enclosing comma-delimted name-value pairs in curly braces like so:

{"foo" : "bar", "fizz" : 3, "flop" : [1, 2, 3]}

Second, the annotate_search_results action has been modified to support two new configuration parameters:

results_lister - (defaults to "li.g, div.g, li div.res, #results>ul>li") - jQuery selector for finding relevant results to annotate. The default finds search results for Google, Yahoo, and Bing.
element_to_modify - (defaults to "div.s,div.abstr,p") - the jQuery selector for finding the element to modify within the results returned by the result lister. The default finds the main body of a search result..

These parameters give you the ability to find and annotate other items on almost any page that displays results. For example, the following action will annotate the paid search results in Google given an appropriately defined my_selector function.

annotate_search_results(my_selector)
 with results_lister = "#mbEnd li" and
      element_to_modify = "a#an1";

Third, user-defined datasources are now more flexible. Previously, the arguments to the datasource function were assumed to be name-value pairs presented as strings. The function put them together to create a QUERY string. That wasn’t as flexible as needed for path-based APIs. So now the datasource takes a single parameter of either a string or a hash:

If the parameter is a string, then it is concatenated with the URL root given in the datasource declaration without modification. What you supply here, when appended to the datasource root URL must result in exactly the URL that you intend to call.
If the parameter is a hash, then the has is turned into a properly formatted HTTP QUERY string with names 7 values delimited by an equals sign (=) to create a name-value pair and each name-value pair delimited by an ampersand (&):
```
pre {
  tweets = datasource:twitter_search({"q": query});
}
```

This allows path-based API calls to be created as strings and QUERY string calls to be created by either creating the string or putting the name-value pairs in a hash and letting the function create the query string.

CTO Breakfast Tomorrow

Tomorrow is the CTO breakfast. It starts at 8am and goes to 9:30am. The location is, as usual, the Novell cafeteria. Sorry for the late notice; for some reason my calendar wasn’t showing the Google calendar event. Luckily an email prodded me from my stoopor.

The CTO breakfast isn’t just for CTOs, but also for those who aspire to be CTOs or are interested in building high-texh products. The discussion is open-format. We decide what to talk about when we get there. You’re welcome to bring your topic and bring it up.

Here are the scheduled dates for upcoming meetings:

No breakfast in July
August 28, 2009 (Friday)
September 24, 2009 (Thursday)

I’ve created a Utah CTO Breakfast group at LinkedIn. You’re invited to join.

I hope to see you there!

June 15, 2009

Kynetx Open House and Demo Day

Image via Wikipedia

This Friday at 11:30, Kynetx will host an open house and demo lunch. If you’d like to see what we’re up to, stop by, eat some pizza, see some demos, and chat. We’re especially interested in getting more people developing on our platform, so if you’d like to try it out, we’ll have sign up cards for anyone who wants them.

We’ll be at Kynetx World Headquarters in Thanksgiving Point. This map will give you directions; we’re in Suite 275 (metal door). Please RSVP so we know how much pizza to buy.

June 12, 2009

Building a Virtual Appliance: First Steps

Image by MrsWoman via Flickr

This week I’ve been working on creating a virtual appliance (VA) version of the Kynetx engine. This is a necessary step for customers who need a version of the Kynetx Rules Engine (KRE) running behind their firewall (think SAS70 compliance, for example).

This post documents some of what I discovered. (N.B. Since I’m working with Xen, much of what appears below could be Xen specific; I haven’t taken the time to generalize it.)

First, and most important, there’s no standard definition of what a virtual appliance is or how they are built. There are companies like rPath and others who have developed processes, but there are literally dozens of ways you might do it. They can be categorized into three camps:

You can, in the spirit of an appliance, hand over a disk image (or images) and an XML machine descriptor which can be used, over and over again if properly set up, to reconstitute a machine (using virt-image for example).
You can create a VA factory that when fired off creates a new machine from scratch, including the loading of the OS, configuration, and so on.
You can do a hybrid of these two that includes a mostly configured image that then bootstraps itself to a final configuration.

Let’s make a few assumptions:

You want to install multiple copies of the VA at the customer site.
The installation needs to be “appliance-like” implying that this isn’t just a custom software installtion but is relatively independent of customer environment and is configurable.

Given these requirements and some choices we’re already made—such as using a rather large puppet recipe collection to configure machines—I opted for for the VA factory method of deploying our appliances.

To make that all work, there are a number of pieces you need: DHCP, Kickstart files (if you are using RedHat, Centos, or Fedora), repositories, puppet recipes, and bootstrap scripts. Setting all these up individually can be done, but there’s a better way: Cobbler.

Cobbler is a Linux installation server that takes care of putting all these together for you into a nice package that you configure and then it takes care of the details of making things work. It took me most of a day to play with it, get it set up how I wanted, and then see success in creating a usable image.

You could put Cobbler on Dom0 (the host machine in Xen), but I chose to put Cobbler and it’s associated repositories on a virtual machine, which functions like a VA factory. That primordial system can be installed on a Xen host machine and then used to create as many copies and configurations of the VA as we need. The factory contains the Linux distro, the repositories we need, the kickstart file, and other configuration data. I haven’t decided whether to put the puppet recipes there or not, but I’m leaning that way.

For the record, here’s what I did in Cobbler. After installing and making sure the installation was good (cobbler check), I imported a distro, created a repository, created a profile for the particular kind of system I wanted to build, and added system definitions for the specific VA’s I was going to create. (N.B. the double \ indicates that the command ought to be on one line, but I’ve warpped it for presentation here.

cobbler import --mirror ~/fc8 --name fc8
cobbler repo add --mirror=http://archive.fedoraproject.org/...\\ 
  --name=fc8-newkey-repo --keep-updated=Y \\
  --rpm-list="puppet facter ruby-libs..."
cobbler profile add --name=base-fc8 --distro=fc8-xen-i386 \\
  --kickstart=/root/base-fc8.ks  --repos=fc8-newkey-repo
cobbler system add --name=kib0 --mac=00:16:3E:4B:4B:40 \\
  --ip=192.168.122.160 --profile=base-fc8 --hostname=kib0
cobbler system add --name=kib1 --mac=00:16:3E:4B:4B:41 \\
  --ip=192.168.122.161 --profile=base-fc8 --hostname=kib1
cobbler system add --name=kib2 --mac=00:16:3E:4B:4B:42 \\
  --ip=192.168.122.162 --profile=base-fc8 --hostname=kib2
cobbler system add --name=kib3 --mac=00:16:3E:4B:4B:43 \\
 --ip=192.168.122.163 --profile=base-fc8 --hostname=kib3

I customized things the most by editing the default Kickstart file that comes with Cobbler to customize the build to what I needed. After that was all set up, installing a virtual machine was as simple as typing the following command on Dom0:

koan --server=cobbler.kobj.net --virt --nogfx \\
   --system=kib0 --virt-bridge=virbr0

Koan stands for “kickstart over a network” and is the command that uses cobbler to install systems from a Cobbler server. I’ve still got some small kinks to work out, but I’m well on my way to having this all completely automated.

All this automation may seem like overkill, but it’s essential in a small shop. Without taking the time to create these systems, a few customers would quickly eat all the time we have and then we’ve got to hire more people. By creating automated system, we’re largely free to move onto creating new features and servicing new customers because existing customer maintenance has been automated to the greatest degree we can. Some might call that survivial—I just call it getting to sleep through the night.

June 5, 2009

A Tail of (Disk) Corruption and Eventual Salvation

Yesterday I accidentally overwrote the disk image of a running Xen server that represented a machine in production. I didn’t notice for two hours because the services on the machine kept going since they didn’t need the disk. In fact, the only reason I noticed was because I happened to need to do the same thing again and got the command from the shell command history and noticed the mistake—a one character typo.

Recovery from this event required rebuilding the machine. Fortunately, we’ve been working hard on infrastructure automation and have a set of Puppet recipes for completely building out a server and an automated deploy process that loads the required application code and tests it. In addition, since it’s all on Xen and the old server was still running, we just build up another machine while the old, corrupted server trundled along.

In about an hour a new machine was up and running. We have been meaning to put up redundent copies of this particualr service. In fact, that’s what I was working on when I caused the problem in the first place. We took this opportunity to put the new machine behind the load balancer and moved the DNS entry. Overnight the traffic has tailed off considerably to the damaged machine and the new machine is now handling most of the load. I’ll build up another copy on another host machine this morning so that we have the redundency we need.

Lessons learned:

Automate everything. More automation would have prevented the mistake I made. The automation we had saved us. Puppet and other tools are the only way to manage infrastructure.
Virtualization gives your incredible flexibility. I love the ability to put up new machines as needed wthout having to manage the hardware tasks at the same time.
Put things behind load balancers and have multiple copes of services running, even if you don’t need the capacity. If we’d done this earlier, my mistake would have been a non-event. We’d have just removed the damaged server from rotation and went on with life.

The end result is that Kynetx suffered no downtime, but we did have some tense moments. Changes we’re making to the infrastructure will improve our chances of achieving the same results with less stress and sweat.

June 4, 2009

GM, Its Shareholders, and Slower Rusting

Image via Wikipedia

As we all know, the US government is now GM’s largest shareholder. As someone who has served on the boards of several companies, this gives me great pause. It’s unlikely that GM is going to choose me to be on it’s board, but if they did, I’m not sure I’d understand what I was meant to do.

When you serve on the board of a for-profit company, your fiduciary duty is clear: increase shareholder value. Boards work for the shareholders and shareholders invest in companies to make money. Sure, the law imposes other duties on boards towards employees and customers, and most boards have legal counsel to help them sort those out; but the overall driving motivations are pretty simple.

Now take GM. (Please.) When your largest shareholder is the government, it’s not clear what their expectations of directors should be. Clearly they’re not interested in increasing their investment or they would have put their money in a different company. Of course, I’m being a little flip—taxpayers would like their money back.

Still, that isn’t why the US government put the money in. Should directors focus on saving jobs? Increasing fuel economy and going green? Saving the supply chain? The areas for potential conflicts of interest are huge.

The reason the government provided what amounts to DIP funding the GM is that no one else would. Smart people obviously thought the risks were too great. But not for the US taxpayer. Sure, saving GM would be nice, but only if we really save GM. I think the chances of GM being a major player on the world stage in the future to be slim. Perhaps we’ve provided a soft landing for someone. Like most government programs the ROI is going to be very hard to find.

I think we’d have been better off in the long term if we’d taken the $50 billion and put it in a fund of funds for investing in new companies. We could put limits on it so that it promotes certain technologies and certain geographies. In the end we’d have jobs and a better way forward. As it is, the most we’re going to get a longer, slower decline of the rust belt.

Phil Windley's Technometria

Organizations Get the IT They Deserve

Technometria

About Phil

Essays

Software

Topic Guides

Categories

Archives

Utah Tech Organizations

Friends

Copyright

Twitter

Free Newsletter!

Search

What I'm Doing

Find Me Elsewhere

Digital Identity

Upcoming Events

Recent Entries