Phil Windley's Technometria

« December 2008 | Main | February 2009 »

January 29, 2009

Kynetx Demo Day

I've had a handful of people ask if they could stop by Kynetx and see what we do. Steve has had similar requests. In an effort to not miss anyone who would like to visit Kynetx and get a demo of our fledgling product, we're hosting a Kynetx Demo Lunch on Friday, Feb 6th at 11:30 at Kynetx World Headquarters in Thanksgiving Point. This map will give you directions and we're in Suite 275 (metal doors). We'll supply the pizza, you bring your curiosity. Please RSVP so we know how much pizza to buy.

7:18 PM | Comments () | Recommend This | Print This

January 27, 2009

CTO Breakfast this Friday

We'll be holding January's CTO Breakfast this Friday in the Novell cafeteria in Provo (Building G). Come prepared for an awesome discussion of technology and companies--especially startups. Feel free to bring topics for discussion. Anyone interested in building high-tech products and services is welcome to attend--not just CTOs.

Please put future CTO breakfasts on your calendar so you can be sure and be there. Here are the scheduled dates so far:

• Jan 30, 2009 (Friday)
• Feb 26, 2009 (Thursday)
• Mar 27, 2009 (Friday)
• Apr 24, 2009 (Friday)

I have created a Google Calendar with dates for the CTO breakfast that you can subscribe to.

Or if you'd rather subscribe from iCal or Outlook, here's the iCalendar link.

9:22 PM | Comments () | Recommend This | Print This

Interactive Map of Utah Legislators

Back in 2003, I lamented the fact that there was no interactive map to finding your legislator in Utah. Indeed, the process involved a lot of steps that introduced considerable friction.

Now, thanks to the power of mash-ups and open data, Scott Riding has created an interactive map of Utah legislative districts and the legislators representing them. I typed in my address and was presented with pictures and contact information of my legislators along with a pin in the map showing my house so I could verify everything was right. Thanks Scott!

8:54 AM | Comments () | Recommend This | Print This

January 26, 2009

Real-Time Regular Expressions

Grant Skinner has a nifty little regular expression tool, written in Flex, that let's you put text in one window and type regular expressions in the other and show the match. There's a "replace" tab as well for seeing how a regular expression replace would modify the text. The right side shows regexp components and describes what they do. I love it.

8:58 PM | Comments () | Recommend This | Print This

I'm Going to Gluecon

I'm going to be speaking at GlueCon in Denver on May 12-13. The overall theme of the conference is that there is a lot of interesting stuff happening in what we might have thought of as "glue" before--all the code that holds things together. Turns out that there's plenty of value you can add in the glue that makes the resulting mash-up better.

Glue is a new conference (the best kind) and is being organized by Eric Norlin, Seth Levine, and Phil Becker. These guys do good conferences. Eric and Phil were the founders of DIDW. More recently Eric's been doing Defrag. Follow the Gluecon blog for more info.

My session is titled "Building Context-aware Applications using Identity as a Foundation." The gist of the talk is about how Web-site independent identity allows us to create context that spans Web-sites. Of course the most obvious context is an authentication context that we refer to as "single sign on" but it goes well beyond that. This is exactly the problem we're tackling at Kynetx. More on these ideas in this blog between now and then.

2:11 PM | Comments () | Recommend This | Print This

Mounting Remote Filesystems Using SSH and Fuse

Paul Figgiani, the Senior Audio Engineer at IT Conversations, sent me a link to a program called ExpanDrive, that allows you to mount any remote directory to which you have SSH access on your Mac. The cost: $39.

ExpanDrive is based on MacFUSE, an extension which extends OS X's native file handling capabilities to programs in user space (that is, outside of the kernel). I first heard about this when Scott and I interviewed Amit Singh on IT Conversations. Amit is probably the world's leading expert on OS X internals and the creator of MacFUSE.

Because MacFUSE allows filesystem extensions to live in user space, you can write a regular program that looks like a file system. That's what ExtenDrive is. Another example is the Cryptomfs which creates an encrypted directory that can be mounted and then read in plain text. I've used the ntfs-3g filesystem to read and write NTFS files to a USB drive. It also works great for accessing Bootcamp partitions.

If you're geeky, there's a free Fuse FS called sshfs that does essentially the same thing but without all the bells and whistles. You can use the OS X automounting capability to automount sshfs volumes.

Having something like this--where a remote directory just shows up in your file system is nice for backups. Mount a directory on a remote machine and then use rsync and a cronjob to perform automated backup over the 'Net. All for free.

Update: CIO has a feature on dynamic languages where ExpanDrive (written in Python) was the major factor in turning someone from C to Python for a project involving ZFS.

1:46 PM | Comments () | Recommend This | Print This

January 22, 2009

Seeing the Dynamic in the Static

Jeff Atwood opines on why many programmers are also musicians of some kind--or at least appreciate music. I've heard this said of mathematicians as well. I have long held that all of these disciplines hold in common the ability to relate the dynamic and the static. Good programmers can see how the program will operate by looking at a lexically scoped program listings. Musicians do the same thing with music (I'm not just talking about music notation, but that's part of it).

7:41 PM | Comments () | Recommend This | Print This

January 21, 2009

A Retweeting Twitterbot in Perl

I'm trying an experiment with this year's Utah legislative session, I've created a Twitter account (@utahpolitics) and set up an autofollower on it (hat tip to @jesse). I wanted to also set up a retweeting twitterbot so that people following the account would see what anyone else following the account said when it contained certain keywords.

The world probably doesn't need yet another retweeter, but I couldn't find exactly what I was looking for and decided to build one for a few reasons:

1. I like to program
2. I want to understand the Twitter API more deeply
3. I didn't want to modify someone's PHP or Python code
4. Oh, and I like to program

Armed with those few requirements and Chris Thompson's Net::Twitter library, I wrote the following program in an hour or so:

#!/usr/bin/perl -w
use strict;

use Net::Twitter;
use DateTime;
use DateTime::Format::HTTP;
use Fcntl;
use SDBM_File;
use File::Copy;

my $dt = DateTime->now;
$dt->subtract( minutes => 60); 

my $class = 'DateTime::Format::HTTP';
my $since = $class->format_datetime($dt);

my $today_string = $dt->strftime("%M");
my $seen_dir = "./seen";
my $seen_file = "$seen_dir/latest.db";
my $backup_file = "$seen_dir/$today_string.db";

# make a backup of the hash and open it
unless(-e "$backup_file.pag") {
    copy("$seen_file.pag","$backup_file.pag");
    copy("$seen_file.dir","$backup_file.dir");
}

my %seen;
tie %seen, "SDBM_File", $seen_file, O_CREAT|O_RDWR, 0644 || 
  die "Can't link to $seen_file, $!\n";

# set your own username and password here
my $user = 'put_your_twitter_screenname_here';
my $password = 'put_your_password_here';

my $twit = Net::Twitter->new(
   username=>$user, 
   password=>$password, 
   source => "Utah Politics Retweeter",
   clientname=>"UtahPolitics ReTweeter");

# find replies
my $retweets = [];
my $twit_replies = 
     $twit->friends_timeline({since => $since, count=>100}) ;

foreach my $reply (@{ $twit_replies }) {
  my $text = $reply->{'text'};
  my $id = $reply->{'id'};
  my $name = $reply->{'user'}->{'screen_name'};
  print ".";
  if($text =~ m/utahpolitics|#utpolitics/  &&
     ! $seen{$id} 
    ){
        unshift @{ $retweets}, 
	        {'name' => $name,
	         'id' => $id,
	         'text' => $text
	 } unless $name eq $user;
  }
}

print "\n";

foreach my $retweet (@{ $retweets }) {
  print ".";
  my $status = "(@".$retweet->{'name'}.") ".
               $retweet->{'text'};
  my $code = $twit->update($status);
  $seen{$retweet->{'id'}} = 1 if $code;
}

print "\n";

1;

I made heavy use of Data::Dumper to Dump data structures I got back from the library during development. This could be generalized in lots of ways. For example, passing the username and password along with the keywords to look for as arguments would allow it to be used for more than one ID. I run this as a cronjob every five minutes and so far it seems to be working fine.

2:16 PM | Comments () | Recommend This | Print This

Technometria Podcast: A New Year and New Projects

In this week's Technometria podcast, we talk about the new year and some new projects. With the beginning of a new year, it's always a good time to look ahead to upcoming activities and products.

In this podcast Dion, Ben, Scott, and I talk about what we're are expecting in 2009. We also discuss the problems with having to raise funds for a business startup, a necessary but often difficult process. We also talk about some of the new products announced at CES and Macworld. Scott also talks about his download of the Windows 7 Beta. The discussion ends with the upcoming transition to digital TV.

2:01 PM | Comments () | Recommend This | Print This

January 19, 2009

jQuery, Monads, and Functional Programming

I tweeted this over the weekend, but it deserves a bigger mention than that. Patrick Thomson has written a wonderful description of why jQuery is a monad including a discussion of cautious computation and state transformations. No need to know monads or jQuery (although an interest in one will help you appreciate the other).

As Patrick explains nicely, jQuery is monadic in that is meets all three requirements of a monad. A monad is a concept from category theory. A type is a monad if it meets three requirements

1. Monads wrap themselves around other data types
2. Monads have an operator that performs this wrapping
3. Monads can feed the wrapped value to a function as long as that function also returns a monad

Anyone familiar with jQuery will immediately recognize that this is a good, abstract description of jQuery's operation. The beauty of this, from jQuery's perspective is a nice programming style called "chaining" where wrapped chunks of the DOM are passed from operator to operator in pipeline fashion. Using this style effectively results in compact, yet readable code with little need for intermediate variables.

You may find it ironic to think of DOM manipulation happening in a functional style, but that's just what jQuery allows. So, if you're a jQuery programmer you may be moving more and more toward a functional style of programming without even knowing it.

10:31 AM | Comments () | Recommend This | Print This

January 16, 2009

The Netherlands Song

Scott Dastrup and his brother Jordan put this video together for their sister's school project on the Netherlands. Scott was one of my scouts many years ago. I still remember a song he taught me about peanut butter. The melody in this song is catchy and Scott and Jordan are good performers.

9:19 PM | Comments () | Recommend This | Print This

January 14, 2009

Consolidating State Data Centers

Word going around is that the State of Utah is looking at possibly consolidating some data centers. In government, over the years, state agencies built data centers and ran them independently. A lot of these were really just machine rooms with not much in the way of power and air conditioning. Some were full-on Class A data centers.

As State CIOs have looked for ways to save money, data center consolidation was a favorite example of how more interagency cooperation could result in tax payer saving. After all, the DNS server for Tax doesn't really care if it's running next to the Web server from Commerce. It's all bits. Oregon, Michigan, Texas, Wisconsin and Georgia are all among states that have taken on consolidation projects--with varying degrees of success.

Over the last ten years there have been a lot of changes in data centers--many driven by changes in the density of the machines that run in them. No longer is space an issue. Rather the issue becomes one of how much power you can get in and how much heat you can get out. Another trend changing data centers is virtualization.

Consolidation can save money if it's done correctly and with a clear understanding of costs, benefits, potential pitfalls, and likely future needs. Costs come from a number of factors:

• Air handling - old data centers may have inefficient air handling units and poor design leading to excessive costs.
• Power conditioning and distribution - older data centers may be inefficient at providing power or may not do a good job of conditioning and providing backup power.
• Facilities management - managing a building costs money. Reusing space for other purposes saves the money needed to buy or rent new space.
• Personnel - older data centers and the applications running in them may not be instrumented for remote monitoring and operation and thus require personnel onsite.
• Security - data centers need physical security to protect the machines inside of them.
• Ping - running multiple lines to multiple data centers may cost more than running a few larger lines to bigger data centers. The need for redundancy multiplies this.

There are also plenty of potential pitfalls. Here are some areas where people run into trouble:

• Undersizing is common - Many data center plans don't make provision for enough space, power, and cooling.
• Relocation is hard - moving servers can mean downed services. More problematic is that services that have grown acretively over the years have lots of interdependencies.
• Baselines are ignored - If the reason for consolidating is to save money, you'd better know how much you're spending before and after so you can understand what happened.
• Too many changes happening at once - consolidations look like a good time to drive other changes, like virtualization, as well. Consolidation is complex enough without adding more moving parts to the mix.
• Experienced hands are unavailable - Not very many people have experience in consolidating data centers and yet it's vital to success.

Here's a few things--by no means exhaustive--that you can do to mitigate some of these problems.

• Consolidate in phases - consolidate servers using virtualization and blade technology before you consolidate data centers so that you have a better idea of size requirement and interdependencies.
• Don't upgrade and move simultaneously - Don't try to upgrade or change systems as part of the consolidation. Freeze systems to be moved in the months beforehand and move the systems as they are.
• Pick low hanging fruit first - concentrate initial efforts on machine rooms and racks in closets. Consider leaving good data centers that can be remotely managed until the end.
• Understand security and privacy requirements of all players - Many agency programs have particular, legally mandate, requirements. Health and education are two that come to mind.
• Don't promise that you'll eliminate servers--at least not at first. That will increase pressure to change systems at the same time you're consolidating.
• Don't underestimate the emotion attached to servers--people like to be able to see their servers. I used to be this way, so I understand it. They're cool. Blinking lights give the impression that important, cool things are happening. Get them over this by moving some things to the cloud or another data center before you take it all away.
• Don't try to get down to one massive data center--multiple, smaller data centers may be cheaper to operate than one large one as long as they can be remotely managed. The model is a small number of smaller, high watt per square foot data centers and one NOC.

One question that seems to come up is what should be outsourced and what should be kept in house. I don't think there's one answer to this question. But there are some ground rules that can help make the decision:

• You should definitely get outside help to plan the consolidation. Experience can make a world of difference.
• Backup data centers and redundancy is a great thing to outsource and often can be placed somewhere that yields additional benefits (like being in a different earthquake zone or where power is cheaper).
• While some system might need to reside on State premises for security or legal reasons, many do not and could be somewhere else--even in the cloud.
• Using consolidation as an excuse to build a new shiny data center can add additional scrutiny. Outsourcing takes away much of the spotlight of a new construction project.
• Make sure that you understand the true costs of running things yourself so that you can make reasoned decisions instead of emotional ones.

Surely, I've forgotten some things that ought to be said. Add your comments below if you have other ideas about how to go about data center consolidation.

9:33 AM | Comments () | Recommend This | Print This

January 12, 2009

The Kynetx Move to CloudFront

One of the components of Kynetx Network Service (KNS) is a 30K (compressed) static Javascript library. This is mostly a slightly modified jQuery along with some other components. We set the Expires header so that it is cached in the browser for 24 hours. Even still, it's a significant load on our network bandwidth and, consequently, our budget.

When Amazon's CloudFront (CF) was announced, we realized that we could move these kinds of static files to CF as a way to reduce our bandwidth and maybe get a performance improvement. If you aren't familiar with it, CF is an Amazon service that fronts their S3 storage system to provide worldwide access & distribution over the 'Net. It's not quite a content delivery network (CDN) but is a substitute for one for many companies.

CloudFront works by fronting S3. Thus, to get something on CloudFront, you upload it to an S3 bucket. You configure CloudFront to point to an S3 bucket and then anything you put in that bucket is available through CloudFront.

Because CloudFront is a simple pointer, if you will, to an S3 bucket, there's no way to tell CF when that item has changed and naturally Amazon doesn't want to check it each time. CF just happily serves up whatever is in it's cache until the object expires. The shortest expiration time you can set on a CF object is 24 hours.

Many items you might serve from CloudFront never, or rarely, change--pictures, for example. In Kynetx' case, the files do change from time to time as we update the library. While you could decide to just live with that, it's not acceptable if you accidentally push a faulty library out. Can you imagine having this conversation? "Gee we're sorry Mr. Customer. We know it's broken. If you'll wait 24 hours, we'll try again."

The common solution seems to be timestamping the filenames so that you just create a new file to serve from CF each time. When you also control the reference to that file, that's a great solution. We don't. The reference is controlled by our customer and telling them all to update their systems each time we want to push a new file is untenable.

You know the saying "there's no Computer Science problem that can't be solved with a layer of indirection." In homage to that, our solution is to redirect from a static URL to the timestamped filename. That's not ideal, but it works. The downside is the other half of that homily: "there's no performance problem that can't be solved by eliminating a layer of indirection." Still, our measurements show that the price we're paying isn't too high.

We put this change in place and tested it for many weeks and then rolled it out to all our customers. So what do we get? Look at this graph showing the dramatic drop in our outbound traffic when we put the change in place:

I did a workup comparing what we'll pay through Amazon and what we were paying for our network traffic and I'm convinced the savings will be nearly as dramatic as the preceding graph.

7:06 AM | Comments () | Recommend This | Print This

January 8, 2009

Delay Digital TV Transition? NOT!

John Podesta, the Obama transition team co-chairman, has sent a letter to lawmakers urging Congress to postpone the Feb. 17 switch from analog to digital television (DTV) broadcasting. Really.

This kind of things was inevitable, of course. We've been warned about this for years and now that it's finally here, people are crying because they're not ready. The real problem, of course, is that the coupon program needs a bailout. I'm not sure why we would bail it out. There was a budget ($1.3 billion) allocated to ease the pain and it's done that. I don't think anyone believed it would take care of the full burden. Full disclosure: I applied for my two coupons early and spent them to help boost the economy. :-) I wanted DTV everywhere I could get it as soon as I could get it.

Consumer Union (publishers of Consumer Reports) is behind this push. In a letter to President Bush, President-elect Barack Obama, and the heads of the House and Senate commerce committees that said, in part:

"Millions of consumers could now be forced to spend their own money to navigate this federally mandated transition," says the letter from Consumers Union. "This economic climate is not the right time to ask consumers to dig deeper into their own pockets to pay for the miscalculation by the federal government."

From DTV troubles: Consumer group pushes to delay digital transition | csmonitor.com
Referenced Thu Jan 08 2009 13:13:22 GMT-0700 (MST)

This isn't what I'd call a federally mandated transition so much as an effort to reallocate impacted bandwidth (a commonly held resource) to places where it has more societal benefit. And DTV is better anyway: more channels, clearer signal--what's not to like?

The pandering politicians and consumer groups trying to make a name for themselves need to get out of the way to let's free up some bandwidth now so we can reap the benefit later.

2:05 PM | Comments () | Recommend This | Print This

January 7, 2009

Good Advice for Switchers

Todd Ogasawara has some good advice for Mac Switchers that might keep you from lamenting your move. I switched in 2002 but had never really been a Windows user (Sun mostly) and I knew Unix cold, so switching wasn't such a big deal for me. But if you've been a long time Windows user and think a Mac might be fun, read Todd's advice first. I love number two:

2. If you do go cold turkey, don't drag your wife, girlfriend, significant other, parents, child, best friend along for the ride until you begin get comfortable with Mac OS X yourself.

From A Brief Guide for Mac Switchers/Try-ers
Referenced Wed Jan 07 2009 07:32:21 GMT-0700 (MST)

Wait until they ask. That's the best way. And they will ask if they see you having fun on your new Mac.

7:34 AM | Comments () | Recommend This | Print This

January 5, 2009

IT Conversations Needs Web Editors

As you're probably aware, IT Conversations, and other Conversations Network Channels, are made possible through the efforts of a small army of Web site editors and audio engineers.

We have a terrific team of people who help out and Doug's put together a great system for managing the workfow of producing shows Now, with a bit of attrition in the ranks of both our website editors and series producers and a new channel on the way, it's time to add to the team once again.

If you'd like to help us write descriptions for our programs, track down and crop photos and sync the occasional slideshow, here's your chance. The word "volunteer" isn't quite right since in fact Web editors are paid (thanks to donations from our paid members), but you're not going to get rich. You'll get a whopping US$15 for each description you write - US$25 if you also sync a presentation's slides to the audio.

You'll find details on TeamITC (as we call it for historical reasons) and our Apprenticeship Program on The Conversations Network web site.

8:56 AM | Comments () | Recommend This | Print This

Pricing Bulk Cold Storage and Real Engineering

James Hamilton has put together an analysis of the cost of bulk cold storage. That is, the cost of storing data, including the fully burdened cost of power in a data center, without the associated transport fees. The answer: $0.80/GB/year.

Wow--that's cheap. And of course it's getting cheaper. When James did a similar analysis using numbers from two years ago, the cost was $2.50/GB/year.

One thought I had as I looked at James' analysis is that we don't teach enough people to do these kinds of calculations. Not that there's anything particularly difficult about the math, but there are things to know and techniques to use.

I often say, quoting Pat Taylor, one of my professors in my undergraduates days in metallurgical engineering, that an engineer is some one who can do for a dollar what any fool could do for two. Of course, building performant, efficient code is part of this, but so is understanding the cost of bulk storage and other resources and using that in the trade-off.

8:26 AM | Comments () | Recommend This | Print This

January 2, 2009

Installing IE6 and IE7 Side by Side

If you do Web design, and who doesn't these days, then you might be interested in seeing how your site operates in multiple browsers. You could, of course, create a virtual machine for each version of Internet Explorer that you want to test against. But there's a better way.

There is a repository of standalone IE versions all the way back to 3.0. Tredsoft has an installer that will install any or all of these in one convenient package. A few minutes after starting the download, I had IE6 up and running in all it's, ahem, glory. IE7 runs right along side without a hiccup.

11:27 AM | Comments () | Recommend This | Print This